# Partner DLQ API - Deployment Checklist

## Pre-Deployment Verification

### ✅ Code Review
- [ ] All controller functions implemented (`controllers/partner_dlq.js`)
- [ ] Routes properly configured (`routes/partner.js`)
- [ ] Authentication middleware applied to all endpoints
- [ ] Error handling implemented for all operations
- [ ] Logging configured for critical operations
- [ ] Input validation added (ObjectId, parameters)
- [ ] No hardcoded credentials or sensitive data

### ✅ Testing
- [ ] Run automated test suite: `./scripts/test_dlq_api.sh`
- [ ] Test all API endpoints with Postman collection
- [ ] Verify web dashboard functionality
- [ ] Test CLI monitoring tool
- [ ] Test background worker operation
- [ ] Verify error categorization logic
- [ ] Test with actual failed messages
- [ ] Load test critical endpoints

### ✅ Documentation
- [ ] API documentation complete (`docs/PARTNER_DLQ_API.md`)
- [ ] Quick start guide available (`docs/PARTNER_DLQ_QUICKSTART.md`)
- [ ] Architecture diagrams created (`docs/PARTNER_DLQ_ARCHITECTURE_DIAGRAMS.md`)
- [ ] Implementation summary documented (`docs/PARTNER_DLQ_IMPLEMENTATION.md`)
- [ ] README.md updated with DLQ section
- [ ] Troubleshooting guide available

### ✅ Security
- [ ] Admin authentication required on all endpoints
- [ ] JWT token validation working
- [ ] Input sanitization implemented
- [ ] Dangerous operations require confirmation
- [ ] Audit logging configured
- [ ] No sensitive data in error messages
- [ ] CORS configured appropriately
- [ ] Rate limiting considered

## Deployment Steps

### 1. Environment Configuration

```bash
# Add to .env or environment_prod.env

# Queue Configuration
QUEUE_HOST=<rabbitmq-host>
QUEUE_PORT=5672
QUEUE_USR=<username>
QUEUE_PWD=<password>
QUEUE_NAME_PARTNER=partner_tasks  # Auto-prefixes with 'dev_' when PRODUCTION=false

# DLQ Configuration
PARTNER_MAX_RETRIES=5
DLQ_CHECK_INTERVAL=300000          # 5 minutes
MAX_DLQ_AGE_MS=86400000           # 24 hours
AUTO_RETRY_WINDOW_MS=7200000      # 2 hours

# MongoDB
MONGO_URI=<mongodb-connection-string>
```

**Checklist:**
- [ ] Environment variables configured
- [ ] RabbitMQ connection details verified
- [ ] MongoDB connection string updated
- [ ] Queue names match environment (dev vs prod)
- [ ] Timeout values appropriate for environment

### 2. Database Preparation

```bash
# Verify PartnerLogTracker indexes
mongo <connection-string> --eval '
  db.partnerlogtrackers.getIndexes()
'
```

**Checklist:**
- [ ] Database connection verified
- [ ] PartnerLogTracker collection exists
- [ ] Indexes created for performance
- [ ] Partner and Customer collections accessible

### 3. RabbitMQ Setup

```bash
# Verify queue configuration
rabbitmqadmin list queues name messages consumers

# Create DLQ if not exists (should auto-create)
rabbitmqadmin declare queue name=partner_tasks_failed durable=true
```

**Checklist:**
- [ ] RabbitMQ service running
- [ ] Main queue exists (`partner_tasks`)
- [ ] DLQ exists (`partner_tasks_failed`)
- [ ] Dead letter exchange configured
- [ ] Queue permissions verified

### 4. Deploy Code

```bash
# Pull latest code
git pull origin <branch>

# Install dependencies (if any new ones)
npm install

# Restart server
pm2 restart agm-server

# Or systemctl restart
sudo systemctl restart agm-server
```

**Checklist:**
- [ ] Code deployed to server
- [ ] Dependencies installed
- [ ] Server restarted successfully
- [ ] No startup errors in logs
- [ ] API endpoints accessible

### 5. Deploy Web Dashboard

```bash
# Verify public directory
ls -la public/dlq-monitor.html

# Check file permissions
chmod 644 public/dlq-monitor.html

# Test access
curl http://localhost:3000/dlq-monitor.html
```

**Checklist:**
- [ ] HTML file in public directory
- [ ] File permissions correct
- [ ] Static file serving configured
- [ ] Dashboard loads in browser
- [ ] API calls working from dashboard

### 6. Deploy Background Services

#### Option A: PM2 (Recommended)

```bash
# Start DLQ handler
pm2 start workers/partner_dlq_handler.js \
  --name partner-dlq-handler \
  -- monitor

# Save PM2 configuration
pm2 save

# Enable PM2 startup
pm2 startup
```

#### Option B: Systemd

```bash
# Create systemd service
sudo nano /etc/systemd/system/partner-dlq-handler.service

# Enable and start
sudo systemctl enable partner-dlq-handler
sudo systemctl start partner-dlq-handler
```

#### Option C: Cron Job

```bash
# Edit crontab
crontab -e

# Add line (process every 4 hours)
0 */4 * * * cd /path/to/server && node workers/partner_dlq_handler.js process >> /var/log/dlq-processing.log 2>&1
```

**Checklist:**
- [ ] Background service method chosen
- [ ] Service configured and started
- [ ] Service running without errors
- [ ] Auto-restart on failure configured
- [ ] Logs being written correctly

### 7. Verify Deployment

```bash
# Run deployment verification tests
./scripts/test_dlq_api.sh

# Check endpoint health
curl -X GET http://localhost:3000/api/dlq/partner_tasks/stats \
  -H "Authorization: Bearer $TOKEN"

# Check web dashboard
open http://localhost:3000/dlq-monitor.html

# Check logs
tail -f /var/log/agm-server.log | grep -i dlq
```

**Checklist:**
- [ ] All API endpoints responding
- [ ] Status codes correct
- [ ] Response format valid
- [ ] Web dashboard loading
- [ ] Background service running
- [ ] No errors in logs

## Post-Deployment Configuration

### 1. Monitoring Setup

```bash
# Add monitoring alerts (example with Prometheus)
# Add to prometheus.yml

- job_name: 'partner-dlq'
  static_configs:
    - targets: ['localhost:3000']
  metrics_path: '/api/partners/dlq/stats'
```

**Checklist:**
- [ ] Monitoring system configured
- [ ] DLQ metrics being collected
- [ ] Alert rules defined
- [ ] Alert notification channels configured
- [ ] Dashboard created (Grafana/similar)

### 2. Alert Thresholds

```yaml
# Example alert rules

- alert: DLQHighMessageCount
  expr: dlq_messages > 20
  for: 5m
  annotations:
    summary: "DLQ has {{ $value }} messages"
    
- alert: DLQCriticalMessageCount
  expr: dlq_messages > 50
  for: 2m
  annotations:
    summary: "DLQ is critically full: {{ $value }} messages"
    
- alert: DLQStaleMessages
  expr: dlq_oldest_message_age > 21600  # 6 hours
  annotations:
    summary: "DLQ has stale messages"
```

**Checklist:**
- [ ] Warning threshold alerts (> 20 messages)
- [ ] Critical threshold alerts (> 50 messages)
- [ ] Message age alerts (> 6 hours)
- [ ] Failed task rate alerts
- [ ] Alert destinations configured (email/Slack)

### 3. Access Control

```bash
# Configure admin access
# Add admins to admin role in database

mongo <connection-string> --eval '
  db.users.updateOne(
    { email: "admin@example.com" },
    { $set: { role: "admin" } }
  )
'
```

**Checklist:**
- [ ] Admin users identified
- [ ] Admin role assigned in database
- [ ] Access permissions verified
- [ ] Non-admin access blocked
- [ ] Authentication tokens distributed

### 4. Backup Procedures

```bash
# Backup PartnerLogTracker collection
mongodump \
  --uri="<connection-string>" \
  --collection=partnerlogtrackers \
  --out=/backup/$(date +%Y%m%d)

# Backup cron job
0 2 * * * /usr/bin/mongodump --uri="..." --collection=partnerlogtrackers --out=/backup/$(date +\%Y\%m\%d)
```

**Checklist:**
- [ ] Backup strategy defined
- [ ] Automated backups configured
- [ ] Backup retention policy set
- [ ] Restore procedure documented
- [ ] Backup tested successfully

## Training & Documentation

### 1. Administrator Training

**Topics to Cover:**
- [ ] Web dashboard overview and features
- [ ] How to interpret statistics
- [ ] Error categories and their meanings
- [ ] When to retry vs archive tasks
- [ ] Using the CLI monitoring tool
- [ ] Emergency procedures
- [ ] Escalation procedures

**Training Materials:**
- [ ] Quick start guide provided
- [ ] Video walkthrough created (optional)
- [ ] FAQ document available
- [ ] Troubleshooting guide accessible

### 2. Operations Documentation

**Required Documents:**
- [ ] Standard Operating Procedures (SOP)
- [ ] Incident Response Plan
- [ ] Escalation Matrix
- [ ] On-call Runbook
- [ ] Change Management Procedures

### 3. Developer Documentation

**Required Resources:**
- [ ] API documentation accessible
- [ ] Code documentation (JSDoc)
- [ ] Architecture diagrams available
- [ ] Integration examples provided
- [ ] Testing procedures documented

## Operational Procedures

### 1. Daily Operations

**Daily Checklist:**
- [ ] Check DLQ message count
- [ ] Review recent failures
- [ ] Verify background service running
- [ ] Check error category distribution
- [ ] Review logs for anomalies

**Automated:**
```bash
# Daily health check script
#!/bin/bash
STATS=$(curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/dlq/partner_tasks/stats)

DLQ_COUNT=$(echo $STATS | jq -r '.dlq.messageCount')
FAILED_COUNT=$(echo $STATS | jq -r '.trackers.failed')

echo "DLQ Messages: $DLQ_COUNT"
echo "Failed Tasks: $FAILED_COUNT"

if [ "$DLQ_COUNT" -gt 20 ]; then
  echo "WARNING: High DLQ count!"
fi
```

### 2. Weekly Operations

**Weekly Checklist:**
- [ ] Review archived tasks
- [ ] Analyze error trends
- [ ] Update documentation if needed
- [ ] Clean up old archived records (> 30 days)
- [ ] Review and optimize alert thresholds

### 3. Monthly Operations

**Monthly Checklist:**
- [ ] Performance review
- [ ] Capacity planning
- [ ] Update monitoring dashboards
- [ ] Review and update procedures
- [ ] Training refresher for new team members

## Incident Response

### High DLQ Count (> 50 messages)

1. [ ] Check DLQ statistics via dashboard
2. [ ] Identify error category distribution
3. [ ] Check for systemic issues
4. [ ] Fix root cause if identified
5. [ ] Process DLQ with dry run first
6. [ ] Process DLQ for real
7. [ ] Monitor recovery
8. [ ] Document incident

### Service Down

1. [ ] Check service status: `pm2 status` or `systemctl status`
2. [ ] Review recent logs
3. [ ] Restart service if needed
4. [ ] Verify recovery
5. [ ] Identify root cause
6. [ ] Implement preventive measures

### Database Issues

1. [ ] Check MongoDB connection
2. [ ] Verify indexes
3. [ ] Check disk space
4. [ ] Review slow queries
5. [ ] Optimize if needed
6. [ ] Document resolution

## Rollback Procedure

If issues arise after deployment:

### 1. Quick Rollback

```bash
# Stop new services
pm2 stop partner-dlq-handler

# Revert code
git checkout <previous-commit>

# Restart server
pm2 restart agm-server

# Verify old code working
curl http://localhost:3000/api/health
```

**Checklist:**
- [ ] Services stopped
- [ ] Code reverted
- [ ] Server restarted
- [ ] Functionality verified
- [ ] Issue documented

### 2. Database Rollback (if needed)

```bash
# Restore from backup
mongorestore \
  --uri="<connection-string>" \
  --collection=partnerlogtrackers \
  /backup/YYYYMMDD/agmission/partnerlogtrackers.bson
```

**Checklist:**
- [ ] Backup identified
- [ ] Database restored
- [ ] Data integrity verified
- [ ] Services restarted

## Sign-Off

### Deployment Team

- [ ] **Developer**: Code reviewed and tested
  - Signature: _______________ Date: _______________

- [ ] **QA Engineer**: Testing completed successfully
  - Signature: _______________ Date: _______________

- [ ] **DevOps**: Infrastructure configured and verified
  - Signature: _______________ Date: _______________

- [ ] **Security**: Security review completed
  - Signature: _______________ Date: _______________

- [ ] **Operations Manager**: Ready for production
  - Signature: _______________ Date: _______________

### Post-Deployment Verification

- [ ] All tests passing
- [ ] No critical issues in first 24 hours
- [ ] Performance metrics acceptable
- [ ] Monitoring alerts configured
- [ ] Team trained and ready

**Final Approval Date**: _______________

---

## Support Contacts

**Technical Issues:**
- Developer Team: <dev-team-email>
- On-call Engineer: <on-call-number>

**Business Issues:**
- Product Owner: <product-owner-email>
- Operations Manager: <ops-manager-email>

**Emergency Escalation:**
- Level 1: Team Lead
- Level 2: Engineering Manager
- Level 3: CTO

---

**Deployment Completed**: [ ] Yes [ ] No  
**Deployment Date**: _______________  
**Deployed By**: _______________  
**Version**: 1.0.0