agmission/Development/server/docs/DLQ_SYSTEM_GUIDE.md

# Dead Letter Queue (DLQ) System Guide

**Navigation:** [📖 Index](DLQ_INDEX.md) | [🚀 Quick Start](DLQ_QUICKSTART.md) | [📚 API Reference](DLQ_API_REFERENCE.md) | [🔧 Operations](DLQ_OPERATIONS.md) | [🏗️ System Guide](DLQ_SYSTEM_GUIDE.md)

---

## Overview

The AgMission DLQ system provides robust error handling, monitoring, and archival for failed partner integration tasks. Built on RabbitMQ's Dead Letter Exchange (DLX) mechanism, it automatically captures failed messages, enriches them with diagnostic metadata, and provides configurable retention with smart alerting.

**Key Features:**
- Automatic message enrichment with error categorization
- Configurable retention (default: 365 days) with TTL-based auto-archival
- Smart admin email alerts based on thresholds (warning @ 20, critical @ 50 messages)
- Real-time dashboard monitoring at `/dlq-monitor.html`
- Health check integration for infrastructure monitoring
- Complete audit trail preservation in organized filesystem archives

---

## Architecture

### Message Flow

```mermaid
flowchart LR
    Task[Partner Task] --> Main[Main Queue<br/>partner_tasks]
    Main -->|Processing| Success[✓ Success]
    Main -->|On Failure| DLQ[Dead Letter Queue<br/>partner_tasks_failed]
    DLQ -->|After DLQ_RETENTION_DAYS| Archive[Archive Queue<br/>partner_tasks_archive]
    Archive --> Filesystem[Filesystem Archive<br/>./dlq_archives/YYYY/MM/DD/]
```

### Components

1. **Main Worker** (`workers/partner_sync_worker.js`)
   - Processes partner tasks from `partner_tasks` queue
   - Enriches messages with error metadata before DLQ routing
   - Implements retry logic with max attempts (default: 5)

2. **DLQ API** (`routes/dlq.js`, `controllers/dlq.js`)
   - Global queue-native endpoints at `/api/dlq/:queueName/*`
   - Direct RabbitMQ operations (no MongoDB coupling)
   - Supports all queue types (partner_tasks, jobs, etc.)
   - Provides retry operations: retryAll, retryByPosition, retryByHeader

3. **Archival Worker** (`workers/dlq_archival_worker.js`)
   - Consumes TTL-expired messages from archive queue
   - Writes JSON files organized by date: `YYYY/MM/DD/timestamp_tasktype_filename.json`
   - Preserves full message content + headers + properties for compliance

4. **Alert Worker** (`workers/dlq_alert_worker.js`)
   - Monitors DLQ message counts for all queues
   - Sends email alerts when thresholds exceeded (warning @ 20, critical @ 50)
   - Smart throttling prevents alert spam (1 hour minimum between similar alerts)
   - Configurable via DLQ_ALERT_* environment variables

5. **Health Check** (`controllers/health.js`)
   - GET `/api/health` includes DLQ component status
   - Status levels: healthy (<20 msgs), degraded (20-49), unhealthy (≥50)
   - Exposes: messageCount, threshold, critical, retentionDays, consumerEnabled

5. **Dashboard** (`public/dlq-monitor.html`)
   - Real-time monitoring UI with auto-refresh
   - Visual alert indicators (green/yellow/red) based on thresholds
   - Manual actions: retry, archive, process, purge
   - Authentication via Bearer token

---

## Configuration

### Environment Variables

Add to your `.env` file or environment:

```bash
# DLQ Retention & Archival
DLQ_RETENTION_DAYS=365              # Days to keep messages in DLQ before auto-archive (default: 365)
DLQ_ARCHIVE_PATH=./dlq_archives     # Where to store archived messages (default: ./dlq_archives)

# Alert Configuration
DLQ_ALERT_ENABLED=true              # Enable admin email alerts (default: true)
DLQ_ALERT_THRESHOLD=20              # Warning threshold for message count (default: 20)
DLQ_ALERT_CRITICAL=50               # Critical threshold for escalated alerts (default: 50)
DLQ_ALERT_INTERVAL_MS=300000        # Check interval in milliseconds (default: 300000 = 5 min)

# DLQ Consumer Control
DLQ_CONSUMER_ENABLED=false          # Enable DLQ message reprocessing (default: false, manual control)
```

### RabbitMQ Queue Configuration

Automatically configured by `partner_sync_worker.js` on startup:

1. **Main Queue** (`partner_tasks` or `dev_partner_tasks`)
   - Durable: yes
   - DLX: default exchange
   - DLX routing key: `{queue_name}_failed`

2. **DLQ** (`partner_tasks_failed`)
   - Durable: yes
   - TTL: `DLQ_RETENTION_DAYS * 86400000` ms
   - DLX: `dlq_archive` exchange
   - DLX routing key: `archive`

3. **Archive Queue** (`partner_tasks_archive`)
   - Durable: yes
   - Consumed by `dlq_archival_worker.js`

### Message Headers

Messages sent to DLQ include headers for filtering and diagnostics. **All other data is in the message body** for reprocessing.

#### Diagnostic Headers (All Queue Types)

| Header | Purpose | Example Values |
|--------|---------|----------------|
| `x-error-category` | Error classification | transient, validation, processing, infrastructure, partner_api, unknown |
| `x-error-reason` | Error message | "Connection timeout", "Invalid file format" |
| `x-task-type` | Task being processed | PROCESS_PARTNER_LOG, UPLOAD_JOB, SEND_NOTIFICATION |
| `x-severity` | Alert severity | low, medium, high, critical |
| `x-first-death-time` | Timestamp of failure | 1734567890123 |

#### Context Headers (partner_tasks queue only)

| Header | Purpose | Example Values |
|--------|---------|----------------|
| `x-partner-code` | Partner identifier (for filtering) | SATLOC, AGIDRONEX |
| `x-customer-id` | Customer ObjectId (for filtering) | 507f1f77bcf86cd799439011 |

**Note**: Headers are for filtering/identification only. Job IDs, user IDs, filenames, etc. are in the message body where they belong.

---

## Operations

### Starting Workers

```bash
# Start main partner sync worker (includes DLQ routing)
node workers/partner_sync_worker.js

# Start archival worker (consumes expired DLQ messages)
node workers/dlq_archival_worker.js

# Start alert worker (monitors DLQs and sends email alerts)
node workers/dlq_alert_worker.js

# Or use start_workers.js to manage all workers
node start_workers.js
```

### Manual DLQ Operations

#### Web Dashboard (Recommended)
Navigate to: `https://localhost:4100/dlq-monitor.html`

1. Enter admin Bearer token (from login)
2. Select queue type (partner_tasks, jobs, etc.)
3. View statistics and messages
4. One-click retry operations

#### API Operations
```bash
# Check DLQ statistics
curl http://localhost:4100/api/dlq/partner_tasks/stats \
  -H "Authorization: Bearer $TOKEN"

# Retry all messages
curl -X POST http://localhost:4100/api/dlq/partner_tasks/retryAll \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"maxMessages": 100}'
```

#### View Dashboard Details
1. Enter admin Bearer token (from login)
2. View real-time statistics
3. Actions available:
   - **Refresh**: Update stats immediately
   - **Process DLQ**: Retry eligible messages
   - **Dry Run**: Analyze without processing
   - **Purge DLQ**: Delete all messages (requires confirmation)

---

## Monitoring

### Health Check Integration

```bash
curl http://localhost:4100/api/health
```

Response includes DLQ component:
```json
{
  "timestamp": "2024-12-18T10:30:00.000Z",
  "overall": "healthy",
  "components": {
    "dlq": {
      "status": "healthy",
      "message": "DLQ operating normally",
      "messageCount": 5,
      "threshold": 20,
      "critical": 50,
      "consumerEnabled": false,
      "retentionDays": 365,
      "queueName": "partner_tasks_failed"
    }
  }
}
```

### Alert Email Format

When DLQ message count exceeds thresholds, admin receives:

**Subject:** `[AgMission][WARNING] DLQ Alert: 25 failed partner tasks`

**Body:**
```
Dead Letter Queue Alert - Partner Tasks
========================================

Status: WARNING
Current DLQ Message Count: 25
Alert Threshold: 20
Critical Threshold: 50
Time: 2024-12-18T10:30:00.000Z

Error Breakdown:
  - transient: 12 (48.0%)
  - partner_api: 8 (32.0%)
  - validation: 5 (20.0%)

Recommended Actions:
1. Monitor DLQ dashboard at /dlq-monitor.html
2. Review error patterns and plan fixes
3. Messages will auto-archive after 365 days

Configuration:
- DLQ Retention: 365 days
- Archive Path: ./dlq_archives
- Max Retries: 5

To disable these alerts, set DLQ_ALERT_ENABLED=false in environment config.
```

**Throttling:** 1 email per hour per severity level to prevent floods.

---

## Error Categorization

Based on [DLQ Best Practices](https://rashadansari.medium.com/strategies-for-successful-dead-letter-queue-event-handling-e354f7dfbb3e):

### Transient Errors
**Auto-retry within 2 hours**  
Patterns: `timeout`, `connection`, `network`, `ECONNREFUSED`, `DNS`  
Action: May resolve automatically, suitable for retry

### Validation Errors
**Archive immediately**  
Patterns: `validation`, `invalid`, `malformed`, `missing required`, `400`  
Action: Requires code/data fix, not retryable

### Processing Errors
**Manual review**  
Patterns: `processing`, `calculation`, `parse`, `transform`  
Action: Business logic issue, investigate root cause

### Infrastructure Errors
**Escalate**  
Patterns: `database`, `mongo`, `transaction`, `filesystem`, `disk`  
Action: System-level problem, may affect multiple tasks

### Partner API Errors
**Extended backoff**  
Patterns: `partner api`, `authentication`, `401`, `403`, `5xx`  
Action: External service issue, retry with longer delays

### Unknown Errors
**Admin notification**  
Patterns: Anything not matching above  
Action: New error type, update categorization logic

---

## Archive Structure

Messages are archived to filesystem with this structure:

```
dlq_archives/
├── 2024/
│   ├── 12/
│   │   ├── 18/
│   │   │   ├── 1734567890123_PROCESS_PARTNER_LOG_satloc_flight_456.json
│   │   │   ├── 1734567891234_UPLOAD_PARTNER_JOB_job_789.json
│   │   │   └── ...
│   │   ├── 19/
│   │   └── ...
│   └── ...
└── ...
```

### Archive File Format

```json
{
  "archived_at": "2024-12-18T10:30:00.000Z",
  "timestamp": 1734567890123,
  "queue_name": "partner_tasks",
  "dlq_name": "partner_tasks_failed",
  "task_type": "PROCESS_PARTNER_LOG",
  "error_category": "transient",
  "severity": "medium",
  "headers": {
    "x-error-category": "transient",
    "x-error-reason": "Connection timeout after 30s",
    "x-task-type": "PROCESS_PARTNER_LOG",
    "x-severity": "medium",
    "x-first-death-time": 1734567890000,
    "x-partner-code": "SATLOC",
    "x-customer-id": "507f1f77bcf86cd799439011",
    "x-log-filename": "2024-12-18_aircraft_456.log"
  },
  "properties": {
    "contentType": "application/json",
    "deliveryMode": 2,
    "timestamp": 1734567890000,
    "expiration": "31536000000"
  },
  "message": {
    "type": "PROCESS_PARTNER_LOG",
    "customerId": "507f1f77bcf86cd799439011",
    "partnerCode": "SATLOC",
    "aircraftId": "aircraft_456",
    "logFileName": "2024-12-18_aircraft_456.log",
    "logId": "log_123",
    "retryCount": 5,
    "lastError": "Connection timeout after 30s",
    "failedAt": "2024-12-18T10:25:00.000Z"
  }
}
```

---

## PartnerLogTracker vs DLQ

### Separation of Concerns

**PartnerLogTracker** (MongoDB) - Business Intelligence Layer:
- Duplicate prevention via unique compound index (`logId`, `partnerCode`)
- Job matching audit trail with confidence scores
- Processing analytics (processTime, success rate)
- Customer reporting and dashboard queries
- File lifecycle tracking (download → process → complete)
- Timeout detection for stuck tasks

**DLQ System** (RabbitMQ + Filesystem) - Error Handling Layer:
- Failed message capture and retry orchestration
- Error categorization and severity assessment
- Configurable retention with TTL-based archival
- Admin alerting for operational issues
- Audit compliance with immutable archive logs

### Why Keep Both?

1. **Different Query Patterns**: PartnerLogTracker optimized for customer/job lookups, DLQ for error analysis
2. **Data Lifecycle**: PartnerLogTracker persists forever (business records), DLQ archives after 365 days
3. **Performance**: DLQ operations don't impact business query performance
4. **Compliance**: Archive files provide immutable audit trail separate from operational database

### Integration Points

- DLQ messages include `x-partner-code` and `x-customer-id` headers for filtering
- Original message body contains full task data (logFileName, jobId, etc.) for reprocessing
- PartnerLogTracker.status field reflects processing state independent of DLQ
- Health check queries both systems for comprehensive status

---

## Troubleshooting

### DLQ Buildup

**Symptom:** Message count > 20  
**Causes:**
1. Partner API downtime → Check partner service status
2. Network connectivity issues → Review infrastructure health
3. Database connection problems → Check MongoDB connection pool
4. Code bugs in processing logic → Review recent deployments

**Resolution:**
1. Identify root cause via error breakdown in dashboard
2. Fix underlying issue
3. Enable DLQ consumer: `DLQ_CONSUMER_ENABLED=true`
4. Monitor processing until queue drains
5. Disable consumer: `DLQ_CONSUMER_ENABLED=false`

### Archive Worker Not Running

**Symptom:** DLQ messages not archiving after TTL expiry  
**Check:**
```bash
ps aux | grep dlq_archival_worker
```

**Fix:**
```bash
node workers/dlq_archival_worker.js &
```

### No Alert Emails

**Symptom:** DLQ > threshold but no email received  
**Checks:**
1. `DLQ_ALERT_ENABLED=true` in env?
2. `NO_EMAIL_MODE=false` in env?
3. SMTP credentials configured?
4. Check last alert time (throttled to 1/hour)?

**Test Email:**
```javascript
const mailer = require('./helpers/mailer');
await mailer.sendAdminNotification('Test', 'Testing DLQ alerts');
```

### High Memory Usage

**Symptom:** Archival worker consuming excessive memory  
**Cause:** Large message backlog in archive queue  
**Fix:**
1. Reduce `prefetch` in archival worker (currently 1)
2. Process archive queue in batches
3. Add memory monitoring alerts

---

## Performance Tuning

### Adjust Check Interval

Reduce alert latency:
```bash
DLQ_ALERT_INTERVAL_MS=60000  # Check every minute instead of 5
```

### Optimize Archive Storage

Compress archived files:
```javascript
// In dlq_archival_worker.js, add gzip compression:
const zlib = require('zlib');
const compressed = zlib.gzipSync(JSON.stringify(archiveRecord));
await fs.writeFile(filepath + '.gz', compressed);
```

### Adjust Retention

For high-volume systems:
```bash
DLQ_RETENTION_DAYS=30  # Reduce to 30 days
```

---

## API Reference

### GET /api/health
Returns overall system health including DLQ component.

**Response:**
```json
{
  "components": {
    "dlq": {
      "status": "healthy|degraded|unhealthy",
      "messageCount": 5,
      "threshold": 20,
      "critical": 50
    }
  }
}
```

### GET /api/dlq/partner_tasks/stats
Get comprehensive DLQ statistics.

**Response:**
```json
{
  "dlq": {
    "messageCount": 25,
    "consumerCount": 0,
    "queueName": "partner_tasks_failed"
  },
  "trackers": {
    "failed": 25,
    "processing": 3,
    "downloaded": 10,
    "processed": 1523,
    "archived": 45
  },
  "recentFailures": [...]
}
```

### GET /api/dlq/partner_tasks/messages?limit=50
Peek at DLQ messages without consuming.

### POST /api/dlq/:queueName/retryAll
Retry all messages currently in the DLQ.

### POST /api/dlq/:queueName/retryByPosition
Retry messages by position range.

### POST /api/dlq/:queueName/retryByHeader
Retry messages matching specific header values.

### POST /api/dlq/:queueName/process
Process all eligible DLQ messages with intelligent categorization.

### DELETE /api/dlq/:queueName/purge
Purge entire DLQ (requires confirmation).

---

## Best Practices

1. **Never Enable DLQ Consumer Permanently**
   - Set `DLQ_CONSUMER_ENABLED=true` only during active recovery
   - Return to `false` after queue drains to prevent blind retries

2. **Monitor Error Categories**
   - Review dashboard regularly for patterns
   - High validation errors → Data quality issues
   - High transient errors → Infrastructure problems

3. **Adjust Thresholds Per Environment**
   - Production: `DLQ_ALERT_THRESHOLD=20`, `DLQ_ALERT_CRITICAL=50`
   - Staging: Higher thresholds acceptable
   - Development: Consider disabling alerts (`DLQ_ALERT_ENABLED=false`)

4. **Archive Retention Planning**
   - Default 365 days suitable for most compliance needs
   - Consider lifecycle policy for `dlq_archives/` directory
   - Old archives can be compressed or moved to cold storage

5. **Correlate with PartnerLogTracker**
   - Query PartnerLogTracker by customerId and partnerCode
   - Check message body for logFileName and other task details
   - Match against tracker records using filter criteria
   - Cross-reference for complete failure analysis
   - Update PartnerLogTracker.errorMessage for business reporting

6. **Regular Archive Cleanup**
   - Setup cron job to delete/compress old archives
   - Example: Keep 2 years, then delete
   ```bash
   find ./dlq_archives -type f -mtime +730 -delete
   ```

7. **Test Alert System**
   - Manually trigger test alerts during setup
   - Verify email delivery and formatting
   - Confirm throttling behavior

8. **Health Check Integration**
   - Add DLQ health to monitoring dashboards (Grafana, Datadog)
   - Alert on `status: "unhealthy"` in CI/CD pipeline
   - Include in uptime checks

---

## Migration Guide

### From Old DLQ System

If migrating from PartnerLogTracker-based retry to queue-native DLQ:

1. **Deploy New System:**
   ```bash
   # Update env with DLQ settings
   DLQ_RETENTION_DAYS=365
   DLQ_ALERT_ENABLED=true
   DLQ_CONSUMER_ENABLED=false
   
   # Restart workers
   pm2 restart partner_sync_worker
   pm2 start workers/dlq_archival_worker.js --name dlq-archival
   ```

2. **Monitor Both Systems:**
   - Old DLQ code still handles in-flight messages
   - New system captures new failures
   - Gradually phase out old retry logic

3. **Cleanup:**
   - Remove old DLQ retry code from controllers
   - Keep PartnerLogTracker for BI purposes
   - Update documentation

---

## Support

For issues or questions:
- Check logs: `workers/*.rlog` files
- Review RabbitMQ management console: `http://localhost:15672`
- Check archive directory: `ls -lah dlq_archives/`
- Contact: trungh@agnav.com

---

## See Also

- **[📖 DLQ Index](DLQ_INDEX.md)** - Documentation overview
- **[🚀 Quick Start Guide](DLQ_QUICKSTART.md)** - Get started quickly
- **[📚 API Reference](DLQ_API_REFERENCE.md)** - Complete API documentation
- **[🔧 Operations Guide](DLQ_OPERATIONS.md)** - Advanced operations

### 🔗 Related Resources

- [Web Dashboard](../public/dlq-monitor.html) - Monitoring interface

---

**Last Updated:** December 18, 2025  
**System Version:** 1.0.0  
**Author:** AgMission Platform Team