agmission/server/docs/archived/PARTNER_DLQ_DESIGN_ISSUES_AND_FIXES.md

Partner DLQ Design Issues & Fixes (RESOLVED)

Critical Design Flaws Identified (Historical - Fixed in Step 8)

Status: ✅ All issues below have been resolved with the queue-native DLQ implementation (Step 8). This document is preserved for historical context and architecture decision documentation.

Issue #1: DLQ Only Handles Log File Processing ❌ → ✅ RESOLVED

Old Implementation (Deprecated):

• DLQ retry/archive endpoints expected PartnerLogTracker ID
• Routes: POST /api/dlq/:queueName/retryByPosition (queue-native), POST /api/partners/dlq/archive/:id
• Only works for PROCESS_PARTNER_LOG tasks
• Fails for other task types:
  • UPLOAD_PARTNER_JOB - Uploading jobs to partner aircraft
  • Future task types (sync, health check, etc.)

Problem:

// Current implementation in partner_dlq.js
exports.retryFailedTask_post = async (req, res, next) => {
  const { id } = req.params;
  
  // ❌ Assumes task is always a log file!
  const tracker = await PartnerLogTracker.findById(id);
  
  const taskInfo = {
    logFileName: tracker.logFileName,  // ❌ What if it's a job upload?
    partnerId: tracker.partnerId,
    customerId: tracker.customerId
  };
};

Issue #2: Message Content Discarded ❌ → ✅ RESOLVED

Old Flow (Deprecated):

flowchart LR
    A[Task Fails] --> B[Sent to DLQ]
    B --> C[getMessage from DLQ]
    C --> D[❌ Extract only logFileName]
    D --> E[Lookup in PartnerLogTracker]
    E --> F[Recreate task from DB]

Problems:

1. ❌ Original task data lost
2. ❌ Can only retry log processing tasks
3. ❌ No way to retry job uploads or other operations
4. ❌ Task type information discarded

Proposed Solution

Design #1: Generic Task Queue with Type Discrimination ✅

flowchart TD
    A[Task Created] --> B{Task Type?}
    B -->|PROCESS_PARTNER_LOG| C[Queue with Log Metadata]
    B -->|UPLOAD_PARTNER_JOB| D[Queue with Job Metadata]
    B -->|Other| E[Queue with Generic Metadata]
    
    C --> F[Task Fails]
    D --> F
    E --> F
    
    F --> G[DLQ Message]
    G -->|Stored as| H[Complete Task Info JSON]
    H --> I{Retry Request}
    I --> J{Check Task Type}
    J -->|PROCESS_PARTNER_LOG| K[Query PartnerLogTracker]
    J -->|UPLOAD_PARTNER_JOB| L[Query JobAssign]
    J -->|Other| M[Use Stored Data]

Implementation Strategy

1. Standardize DLQ Message Format

Current:

// Inconsistent task data
{
  logFileName: "150-12-06-2025.log",  // ❌ Only for log tasks
  partnerId: "...",
  customerId: "..."
}

Proposed:

{
  taskType: "PROCESS_PARTNER_LOG" | "UPLOAD_PARTNER_JOB" | "SYNC_PARTNER_DATA",
  taskData: {
    // Type-specific data
  },
  metadata: {
    attemptNumber: 3,
    firstFailedAt: "2025-12-17T...",
    lastError: "Network timeout",
    originalQueuedAt: "2025-12-17T..."
  },
  trackerId: "ObjectId(...)"  // Reference to tracker if exists
}

2. Update DLQ Retry Logic

exports.retryFailedTask_post = async (req, res, next) => {
  const { id } = req.params;
  
  // id can be either:
  // 1. PartnerLogTracker._id (for log processing)
  // 2. JobAssign._id (for job uploads)
  // 3. Generic task ID from DLQ message itself
  
  // Step 1: Try to find in PartnerLogTracker
  let tracker = await PartnerLogTracker.findById(id);
  
  if (tracker) {
    // It's a log processing task
    const taskInfo = {
      type: PartnerTasks.PROCESS_PARTNER_LOG,
      data: {
        logFileName: tracker.logFileName,
        partnerId: tracker.partnerId,
        customerId: tracker.customerId
      }
    };
    return await requeueTask(taskInfo);
  }
  
  // Step 2: Try to find in JobAssign
  let assignment = await JobAssign.findById(id);
  
  if (assignment) {
    // It's a job upload task
    const taskInfo = {
      type: PartnerTasks.UPLOAD_PARTNER_JOB,
      data: {
        assignId: assignment._id,
        jobId: assignment.jobId,
        partnerCode: assignment.partnerCode,
        aircraftId: assignment.aircraftId
      }
    };
    return await requeueTask(taskInfo);
  }
  
  // Step 3: Check if it's a raw DLQ message ID
  // ... lookup in RabbitMQ or a DLQ tracking collection
  
  throw new AppParamError('Task not found in any registry');
};

3. Create Task Registry Collection

// New model: PartnerTaskRegistry
{
  _id: ObjectId,
  taskType: String,  // PROCESS_PARTNER_LOG, UPLOAD_PARTNER_JOB, etc.
  taskData: Mixed,   // Original task data
  status: String,    // 'queued', 'processing', 'failed', 'completed', 'archived'
  
  // Tracking fields
  queuedAt: Date,
  processingStartedAt: Date,
  completedAt: Date,
  failedAt: Date,
  
  // Retry tracking
  attemptCount: Number,
  maxAttempts: Number,
  lastError: String,
  errorHistory: [{ error: String, occurredAt: Date }],
  
  // References
  relatedId: ObjectId,  // PartnerLogTracker._id, JobAssign._id, etc.
  relatedModel: String, // 'PartnerLogTracker', 'JobAssign', etc.
  
  customerId: ObjectId,
  partnerId: ObjectId,
  
  // Audit
  createdAt: Date,
  updatedAt: Date
}

Design #2: Separate DLQ Per Task Type ✅

Alternative Approach:

partner_log_tasks          → partner_log_tasks_failed
partner_job_upload_tasks   → partner_job_upload_tasks_failed  
partner_sync_tasks         → partner_sync_tasks_failed

Pros:

• ✅ Clear separation of concerns
• ✅ Each DLQ tailored to specific task type
• ✅ Easier to implement type-specific retry logic

Cons:

• ❌ More queues to monitor
• ❌ More complex worker setup
• ❌ Duplicated DLQ management code

Recommended Approach

Hybrid Solution: Single Queue + Task Registry ✅

flowchart TD
    A[Partner Tasks] --> B[Single Partner Queue]
    B --> C{Worker Processes}
    C -->|Success| D[Mark Complete in Registry]
    C -->|Fail| E[DLQ + Update Registry]
    
    E --> F[PartnerTaskRegistry Collection]
    F --> G{Task Type}
    G -->|PROCESS_PARTNER_LOG| H[PartnerLogTracker Reference]
    G -->|UPLOAD_PARTNER_JOB| I[JobAssign Reference]
    G -->|Other| J[Standalone Task Data]
    
    F --> K[DLQ API Endpoints]
    K -->|GET /stats| L[Aggregate by Type]
    K -->|POST /:queueName/retryAll| M[Retry All Messages]
    K -->|POST /:queueName/retryByPosition| N[Retry By Position]
    K -->|POST /:queueName/retryByHeader| O[Retry By Header]

Implementation Steps

1. Create PartnerTaskRegistry Model ✅

   • Tracks all partner tasks regardless of type
   • References related entities (PartnerLogTracker, JobAssign, etc.)
   • Maintains complete task history

2. Update Worker to Use Registry ✅

   • Create registry entry when processing task
   • Update on success/failure
   • Store complete error history

3. Refactor DLQ Endpoints ✅

   • Accept registry ID instead of tracker ID
   • Support all task types
   • Provide type-specific handling

4. Update DLQ HTML Monitor ✅

   • Display task type
   • Show appropriate details per type
   • Enable retry/archive for any task type

5. Backward Compatibility ✅

   • Keep PartnerLogTracker for log-specific tracking
   • Registry provides unified view
   • Gradually migrate to registry-first approach

Updated API Endpoints

Queue-Native Retry Operations

POST /api/dlq/:queueName/retryAll

Retries all messages in the DLQ back to the main queue.

POST /api/dlq/:queueName/retryByPosition

Retries messages by position range (e.g., messages 1-10).

POST /api/dlq/:queueName/retryByHeader

Retries messages matching specific header values (e.g., partner code).

// :id is now PartnerTaskRegistry._id (not PartnerLogTracker._id)

// Request
POST /api/partners/dlq/retry/674abc123...

// Response
{
  success: true,
  taskType: "UPLOAD_PARTNER_JOB",
  message: "Job upload task requeued for retry",
  task: {
    id: "674abc123...",
    type: "UPLOAD_PARTNER_JOB",
    attemptNumber: 4,
    status: "queued"
  }
}

GET /api/dlq/partner_tasks/stats

// Response includes breakdown by task type
{
  dlq: {
    messageCount: 12,
    byType: {
      PROCESS_PARTNER_LOG: 8,
      UPLOAD_PARTNER_JOB: 3,
      SYNC_PARTNER_DATA: 1
    }
  },
  registry: {
    failed: 15,
    processing: 2,
    queued: 5,
    archived: 10,
    byType: { ... }
  },
  recentFailures: [
    {
      id: "...",
      taskType: "UPLOAD_PARTNER_JOB",
      errorMessage: "Partner API timeout",
      failedAt: "2025-12-17T...",
      attemptCount: 3
    },
    // ...
  ]
}

Migration Path

Phase 1: Add Registry (Non-Breaking) ✅

• Create PartnerTaskRegistry model
• Update worker to create registry entries
• Keep existing PartnerLogTracker functionality

Phase 2: Update DLQ Endpoints ✅

• Add new endpoints using registry ID
• Keep old endpoints for backward compatibility
• Add deprecation warnings

Phase 3: Update Clients ✅

• Update HTML monitor to use new endpoints
• Update any scripts/tools using DLQ API
• Update documentation

Phase 4: Remove Old Endpoints ✅

• Deprecate tracker-based endpoints
• Remove after migration period
• Keep PartnerLogTracker for log-specific needs

Benefits

1. Unified Task Management ✅

   • Single source of truth for all partner tasks
   • Consistent retry/archive logic
   • Comprehensive task history

2. Type Safety ✅

   • Explicit task type discrimination
   • Type-specific handling logic
   • Prevents type confusion errors

3. Better Monitoring ✅

   • See all failed tasks in one place
   • Filter/sort by task type
   • Track success rates per type

4. Scalability ✅

   • Easy to add new task types
   • Registry pattern supports any task
   • DLQ management code reusable

5. Debugging ✅

   • Complete task lifecycle visible
   • Error history preserved
   • Easier root cause analysis

Conclusion

The current DLQ implementation has a critical flaw - it only handles log processing tasks. The proposed PartnerTaskRegistry solution provides a unified, type-safe approach that supports all partner task types while maintaining backward compatibility.

Next Steps:

1. Implement PartnerTaskRegistry model
2. Update partner_sync_worker to use registry
3. Refactor DLQ endpoints to support all task types
4. Update documentation and monitoring tools

---

Status: 📝 Design Proposal
Priority: 🔴 High - Affects production reliability
Estimated Effort: 2-3 days implementation + testing