# Partner DLQ Design Issues & Fixes (RESOLVED)

## Critical Design Flaws Identified (Historical - Fixed in Step 8)

> **Status**: ✅ All issues below have been resolved with the queue-native DLQ implementation (Step 8).
> This document is preserved for historical context and architecture decision documentation.

### Issue #1: DLQ Only Handles Log File Processing ❌ → ✅ RESOLVED

**Old Implementation (Deprecated):**
- DLQ retry/archive endpoints expected `PartnerLogTracker` ID
- Routes: `POST /api/dlq/:queueName/retryByPosition (queue-native)`, `POST /api/partners/dlq/archive/:id`
- Only works for `PROCESS_PARTNER_LOG` tasks
- **Fails for other task types:**
  - `UPLOAD_PARTNER_JOB` - Uploading jobs to partner aircraft
  - Future task types (sync, health check, etc.)

**Problem:**
```javascript
// Current implementation in partner_dlq.js
exports.retryFailedTask_post = async (req, res, next) => {
  const { id } = req.params;
  
  // ❌ Assumes task is always a log file!
  const tracker = await PartnerLogTracker.findById(id);
  
  const taskInfo = {
    logFileName: tracker.logFileName,  // ❌ What if it's a job upload?
    partnerId: tracker.partnerId,
    customerId: tracker.customerId
  };
};
```

### Issue #2: Message Content Discarded ❌ → ✅ RESOLVED

**Old Flow (Deprecated):**
```mermaid
flowchart LR
    A[Task Fails] --> B[Sent to DLQ]
    B --> C[getMessage from DLQ]
    C --> D[❌ Extract only logFileName]
    D --> E[Lookup in PartnerLogTracker]
    E --> F[Recreate task from DB]
```

**Problems:**
1. ❌ Original task data lost
2. ❌ Can only retry log processing tasks
3. ❌ No way to retry job uploads or other operations
4. ❌ Task type information discarded

## Proposed Solution

### Design #1: Generic Task Queue with Type Discrimination ✅

```mermaid
flowchart TD
    A[Task Created] --> B{Task Type?}
    B -->|PROCESS_PARTNER_LOG| C[Queue with Log Metadata]
    B -->|UPLOAD_PARTNER_JOB| D[Queue with Job Metadata]
    B -->|Other| E[Queue with Generic Metadata]
    
    C --> F[Task Fails]
    D --> F
    E --> F
    
    F --> G[DLQ Message]
    G -->|Stored as| H[Complete Task Info JSON]
    H --> I{Retry Request}
    I --> J{Check Task Type}
    J -->|PROCESS_PARTNER_LOG| K[Query PartnerLogTracker]
    J -->|UPLOAD_PARTNER_JOB| L[Query JobAssign]
    J -->|Other| M[Use Stored Data]
```

### Implementation Strategy

#### 1. Standardize DLQ Message Format

**Current:**
```javascript
// Inconsistent task data
{
  logFileName: "150-12-06-2025.log",  // ❌ Only for log tasks
  partnerId: "...",
  customerId: "..."
}
```

**Proposed:**
```javascript
{
  taskType: "PROCESS_PARTNER_LOG" | "UPLOAD_PARTNER_JOB" | "SYNC_PARTNER_DATA",
  taskData: {
    // Type-specific data
  },
  metadata: {
    attemptNumber: 3,
    firstFailedAt: "2025-12-17T...",
    lastError: "Network timeout",
    originalQueuedAt: "2025-12-17T..."
  },
  trackerId: "ObjectId(...)"  // Reference to tracker if exists
}
```

#### 2. Update DLQ Retry Logic

```javascript
exports.retryFailedTask_post = async (req, res, next) => {
  const { id } = req.params;
  
  // id can be either:
  // 1. PartnerLogTracker._id (for log processing)
  // 2. JobAssign._id (for job uploads)
  // 3. Generic task ID from DLQ message itself
  
  // Step 1: Try to find in PartnerLogTracker
  let tracker = await PartnerLogTracker.findById(id);
  
  if (tracker) {
    // It's a log processing task
    const taskInfo = {
      type: PartnerTasks.PROCESS_PARTNER_LOG,
      data: {
        logFileName: tracker.logFileName,
        partnerId: tracker.partnerId,
        customerId: tracker.customerId
      }
    };
    return await requeueTask(taskInfo);
  }
  
  // Step 2: Try to find in JobAssign
  let assignment = await JobAssign.findById(id);
  
  if (assignment) {
    // It's a job upload task
    const taskInfo = {
      type: PartnerTasks.UPLOAD_PARTNER_JOB,
      data: {
        assignId: assignment._id,
        jobId: assignment.jobId,
        partnerCode: assignment.partnerCode,
        aircraftId: assignment.aircraftId
      }
    };
    return await requeueTask(taskInfo);
  }
  
  // Step 3: Check if it's a raw DLQ message ID
  // ... lookup in RabbitMQ or a DLQ tracking collection
  
  throw new AppParamError('Task not found in any registry');
};
```

#### 3. Create Task Registry Collection

```javascript
// New model: PartnerTaskRegistry
{
  _id: ObjectId,
  taskType: String,  // PROCESS_PARTNER_LOG, UPLOAD_PARTNER_JOB, etc.
  taskData: Mixed,   // Original task data
  status: String,    // 'queued', 'processing', 'failed', 'completed', 'archived'
  
  // Tracking fields
  queuedAt: Date,
  processingStartedAt: Date,
  completedAt: Date,
  failedAt: Date,
  
  // Retry tracking
  attemptCount: Number,
  maxAttempts: Number,
  lastError: String,
  errorHistory: [{ error: String, occurredAt: Date }],
  
  // References
  relatedId: ObjectId,  // PartnerLogTracker._id, JobAssign._id, etc.
  relatedModel: String, // 'PartnerLogTracker', 'JobAssign', etc.
  
  customerId: ObjectId,
  partnerId: ObjectId,
  
  // Audit
  createdAt: Date,
  updatedAt: Date
}
```

### Design #2: Separate DLQ Per Task Type ✅

**Alternative Approach:**
```
partner_log_tasks          → partner_log_tasks_failed
partner_job_upload_tasks   → partner_job_upload_tasks_failed  
partner_sync_tasks         → partner_sync_tasks_failed
```

**Pros:**
- ✅ Clear separation of concerns
- ✅ Each DLQ tailored to specific task type
- ✅ Easier to implement type-specific retry logic

**Cons:**
- ❌ More queues to monitor
- ❌ More complex worker setup
- ❌ Duplicated DLQ management code

## Recommended Approach

### Hybrid Solution: Single Queue + Task Registry ✅

```mermaid
flowchart TD
    A[Partner Tasks] --> B[Single Partner Queue]
    B --> C{Worker Processes}
    C -->|Success| D[Mark Complete in Registry]
    C -->|Fail| E[DLQ + Update Registry]
    
    E --> F[PartnerTaskRegistry Collection]
    F --> G{Task Type}
    G -->|PROCESS_PARTNER_LOG| H[PartnerLogTracker Reference]
    G -->|UPLOAD_PARTNER_JOB| I[JobAssign Reference]
    G -->|Other| J[Standalone Task Data]
    
    F --> K[DLQ API Endpoints]
    K -->|GET /stats| L[Aggregate by Type]
    K -->|POST /:queueName/retryAll| M[Retry All Messages]
    K -->|POST /:queueName/retryByPosition| N[Retry By Position]
    K -->|POST /:queueName/retryByHeader| O[Retry By Header]
```

### Implementation Steps

1. **Create PartnerTaskRegistry Model** ✅
   - Tracks all partner tasks regardless of type
   - References related entities (PartnerLogTracker, JobAssign, etc.)
   - Maintains complete task history

2. **Update Worker to Use Registry** ✅
   - Create registry entry when processing task
   - Update on success/failure
   - Store complete error history

3. **Refactor DLQ Endpoints** ✅
   - Accept registry ID instead of tracker ID
   - Support all task types
   - Provide type-specific handling

4. **Update DLQ HTML Monitor** ✅
   - Display task type
   - Show appropriate details per type
   - Enable retry/archive for any task type

5. **Backward Compatibility** ✅
   - Keep PartnerLogTracker for log-specific tracking
   - Registry provides unified view
   - Gradually migrate to registry-first approach

## Updated API Endpoints

### Queue-Native Retry Operations

#### POST /api/dlq/:queueName/retryAll

Retries all messages in the DLQ back to the main queue.

#### POST /api/dlq/:queueName/retryByPosition

Retries messages by position range (e.g., messages 1-10).

#### POST /api/dlq/:queueName/retryByHeader

Retries messages matching specific header values (e.g., partner code).
```javascript
// :id is now PartnerTaskRegistry._id (not PartnerLogTracker._id)

// Request
POST /api/partners/dlq/retry/674abc123...

// Response
{
  success: true,
  taskType: "UPLOAD_PARTNER_JOB",
  message: "Job upload task requeued for retry",
  task: {
    id: "674abc123...",
    type: "UPLOAD_PARTNER_JOB",
    attemptNumber: 4,
    status: "queued"
  }
}
```

### GET /api/dlq/partner_tasks/stats
```javascript
// Response includes breakdown by task type
{
  dlq: {
    messageCount: 12,
    byType: {
      PROCESS_PARTNER_LOG: 8,
      UPLOAD_PARTNER_JOB: 3,
      SYNC_PARTNER_DATA: 1
    }
  },
  registry: {
    failed: 15,
    processing: 2,
    queued: 5,
    archived: 10,
    byType: { ... }
  },
  recentFailures: [
    {
      id: "...",
      taskType: "UPLOAD_PARTNER_JOB",
      errorMessage: "Partner API timeout",
      failedAt: "2025-12-17T...",
      attemptCount: 3
    },
    // ...
  ]
}
```

## Migration Path

### Phase 1: Add Registry (Non-Breaking) ✅
- Create PartnerTaskRegistry model
- Update worker to create registry entries
- Keep existing PartnerLogTracker functionality

### Phase 2: Update DLQ Endpoints ✅
- Add new endpoints using registry ID
- Keep old endpoints for backward compatibility
- Add deprecation warnings

### Phase 3: Update Clients ✅
- Update HTML monitor to use new endpoints
- Update any scripts/tools using DLQ API
- Update documentation

### Phase 4: Remove Old Endpoints ✅
- Deprecate tracker-based endpoints
- Remove after migration period
- Keep PartnerLogTracker for log-specific needs

## Benefits

1. **Unified Task Management** ✅
   - Single source of truth for all partner tasks
   - Consistent retry/archive logic
   - Comprehensive task history

2. **Type Safety** ✅
   - Explicit task type discrimination
   - Type-specific handling logic
   - Prevents type confusion errors

3. **Better Monitoring** ✅
   - See all failed tasks in one place
   - Filter/sort by task type
   - Track success rates per type

4. **Scalability** ✅
   - Easy to add new task types
   - Registry pattern supports any task
   - DLQ management code reusable

5. **Debugging** ✅
   - Complete task lifecycle visible
   - Error history preserved
   - Easier root cause analysis

## Conclusion

The current DLQ implementation has a critical flaw - it only handles log processing tasks. The proposed PartnerTaskRegistry solution provides a unified, type-safe approach that supports all partner task types while maintaining backward compatibility.

**Next Steps:**
1. Implement PartnerTaskRegistry model
2. Update partner_sync_worker to use registry
3. Refactor DLQ endpoints to support all task types
4. Update documentation and monitoring tools

---

**Status**: 📝 Design Proposal  
**Priority**: 🔴 High - Affects production reliability  
**Estimated Effort**: 2-3 days implementation + testing