agmission/server/docs/archived/PARTNER_DLQ_DEPLOYMENT_CHECKLIST.md

Partner DLQ API - Deployment Checklist

Pre-Deployment Verification

✅ Code Review

• All controller functions implemented (controllers/partner_dlq.js)
• Routes properly configured (routes/partner.js)
• Authentication middleware applied to all endpoints
• Error handling implemented for all operations
• Logging configured for critical operations
• Input validation added (ObjectId, parameters)
• No hardcoded credentials or sensitive data

✅ Testing

• Run automated test suite: ./scripts/test_dlq_api.sh
• Test all API endpoints with Postman collection
• Verify web dashboard functionality
• Test CLI monitoring tool
• Test background worker operation
• Verify error categorization logic
• Test with actual failed messages
• Load test critical endpoints

✅ Documentation

• API documentation complete (docs/PARTNER_DLQ_API.md)
• Quick start guide available (docs/PARTNER_DLQ_QUICKSTART.md)
• Architecture diagrams created (docs/PARTNER_DLQ_ARCHITECTURE_DIAGRAMS.md)
• Implementation summary documented (docs/PARTNER_DLQ_IMPLEMENTATION.md)
• README.md updated with DLQ section
• Troubleshooting guide available

✅ Security

• Admin authentication required on all endpoints
• JWT token validation working
• Input sanitization implemented
• Dangerous operations require confirmation
• Audit logging configured
• No sensitive data in error messages
• CORS configured appropriately
• Rate limiting considered

Deployment Steps

1. Environment Configuration

# Add to .env or environment_prod.env

# Queue Configuration
QUEUE_HOST=<rabbitmq-host>
QUEUE_PORT=5672
QUEUE_USR=<username>
QUEUE_PWD=<password>
QUEUE_NAME_PARTNER=partner_tasks  # Auto-prefixes with 'dev_' when PRODUCTION=false

# DLQ Configuration
PARTNER_MAX_RETRIES=5
DLQ_CHECK_INTERVAL=300000          # 5 minutes
MAX_DLQ_AGE_MS=86400000           # 24 hours
AUTO_RETRY_WINDOW_MS=7200000      # 2 hours

# MongoDB
MONGO_URI=<mongodb-connection-string>

Checklist:

• Environment variables configured
• RabbitMQ connection details verified
• MongoDB connection string updated
• Queue names match environment (dev vs prod)
• Timeout values appropriate for environment

2. Database Preparation

# Verify PartnerLogTracker indexes
mongo <connection-string> --eval '
  db.partnerlogtrackers.getIndexes()
'

Checklist:

• Database connection verified
• PartnerLogTracker collection exists
• Indexes created for performance
• Partner and Customer collections accessible

3. RabbitMQ Setup

# Verify queue configuration
rabbitmqadmin list queues name messages consumers

# Create DLQ if not exists (should auto-create)
rabbitmqadmin declare queue name=partner_tasks_failed durable=true

Checklist:

• RabbitMQ service running
• Main queue exists (partner_tasks)
• DLQ exists (partner_tasks_failed)
• Dead letter exchange configured
• Queue permissions verified

4. Deploy Code

# Pull latest code
git pull origin <branch>

# Install dependencies (if any new ones)
npm install

# Restart server
pm2 restart agm-server

# Or systemctl restart
sudo systemctl restart agm-server

Checklist:

• Code deployed to server
• Dependencies installed
• Server restarted successfully
• No startup errors in logs
• API endpoints accessible

5. Deploy Web Dashboard

# Verify public directory
ls -la public/dlq-monitor.html

# Check file permissions
chmod 644 public/dlq-monitor.html

# Test access
curl http://localhost:3000/dlq-monitor.html

Checklist:

• HTML file in public directory
• File permissions correct
• Static file serving configured
• Dashboard loads in browser
• API calls working from dashboard

6. Deploy Background Services

Option A: PM2 (Recommended)

# Start DLQ handler
pm2 start workers/partner_dlq_handler.js \
  --name partner-dlq-handler \
  -- monitor

# Save PM2 configuration
pm2 save

# Enable PM2 startup
pm2 startup

Option B: Systemd

# Create systemd service
sudo nano /etc/systemd/system/partner-dlq-handler.service

# Enable and start
sudo systemctl enable partner-dlq-handler
sudo systemctl start partner-dlq-handler

Option C: Cron Job

# Edit crontab
crontab -e

# Add line (process every 4 hours)
0 */4 * * * cd /path/to/server && node workers/partner_dlq_handler.js process >> /var/log/dlq-processing.log 2>&1

Checklist:

• Background service method chosen
• Service configured and started
• Service running without errors
• Auto-restart on failure configured
• Logs being written correctly

7. Verify Deployment

# Run deployment verification tests
./scripts/test_dlq_api.sh

# Check endpoint health
curl -X GET http://localhost:3000/api/dlq/partner_tasks/stats \
  -H "Authorization: Bearer $TOKEN"

# Check web dashboard
open http://localhost:3000/dlq-monitor.html

# Check logs
tail -f /var/log/agm-server.log | grep -i dlq

Checklist:

• All API endpoints responding
• Status codes correct
• Response format valid
• Web dashboard loading
• Background service running
• No errors in logs

Post-Deployment Configuration

1. Monitoring Setup

# Add monitoring alerts (example with Prometheus)
# Add to prometheus.yml

- job_name: 'partner-dlq'
  static_configs:
    - targets: ['localhost:3000']
  metrics_path: '/api/partners/dlq/stats'

Checklist:

• Monitoring system configured
• DLQ metrics being collected
• Alert rules defined
• Alert notification channels configured
• Dashboard created (Grafana/similar)

2. Alert Thresholds

# Example alert rules

- alert: DLQHighMessageCount
  expr: dlq_messages > 20
  for: 5m
  annotations:
    summary: "DLQ has {{ $value }} messages"
    
- alert: DLQCriticalMessageCount
  expr: dlq_messages > 50
  for: 2m
  annotations:
    summary: "DLQ is critically full: {{ $value }} messages"
    
- alert: DLQStaleMessages
  expr: dlq_oldest_message_age > 21600  # 6 hours
  annotations:
    summary: "DLQ has stale messages"

Checklist:

• Warning threshold alerts (> 20 messages)
• Critical threshold alerts (> 50 messages)
• Message age alerts (> 6 hours)
• Failed task rate alerts
• Alert destinations configured (email/Slack)

3. Access Control

# Configure admin access
# Add admins to admin role in database

mongo <connection-string> --eval '
  db.users.updateOne(
    { email: "admin@example.com" },
    { $set: { role: "admin" } }
  )
'

Checklist:

• Admin users identified
• Admin role assigned in database
• Access permissions verified
• Non-admin access blocked
• Authentication tokens distributed

4. Backup Procedures

# Backup PartnerLogTracker collection
mongodump \
  --uri="<connection-string>" \
  --collection=partnerlogtrackers \
  --out=/backup/$(date +%Y%m%d)

# Backup cron job
0 2 * * * /usr/bin/mongodump --uri="..." --collection=partnerlogtrackers --out=/backup/$(date +\%Y\%m\%d)

Checklist:

• Backup strategy defined
• Automated backups configured
• Backup retention policy set
• Restore procedure documented
• Backup tested successfully

Training & Documentation

1. Administrator Training

Topics to Cover:

• Web dashboard overview and features
• How to interpret statistics
• Error categories and their meanings
• When to retry vs archive tasks
• Using the CLI monitoring tool
• Emergency procedures
• Escalation procedures

Training Materials:

• Quick start guide provided
• Video walkthrough created (optional)
• FAQ document available
• Troubleshooting guide accessible

2. Operations Documentation

Required Documents:

• Standard Operating Procedures (SOP)
• Incident Response Plan
• Escalation Matrix
• On-call Runbook
• Change Management Procedures

3. Developer Documentation

Required Resources:

• API documentation accessible
• Code documentation (JSDoc)
• Architecture diagrams available
• Integration examples provided
• Testing procedures documented

Operational Procedures

1. Daily Operations

Daily Checklist:

• Check DLQ message count
• Review recent failures
• Verify background service running
• Check error category distribution
• Review logs for anomalies

Automated:

# Daily health check script
#!/bin/bash
STATS=$(curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/dlq/partner_tasks/stats)

DLQ_COUNT=$(echo $STATS | jq -r '.dlq.messageCount')
FAILED_COUNT=$(echo $STATS | jq -r '.trackers.failed')

echo "DLQ Messages: $DLQ_COUNT"
echo "Failed Tasks: $FAILED_COUNT"

if [ "$DLQ_COUNT" -gt 20 ]; then
  echo "WARNING: High DLQ count!"
fi

2. Weekly Operations

Weekly Checklist:

• Review archived tasks
• Analyze error trends
• Update documentation if needed
• Clean up old archived records (> 30 days)
• Review and optimize alert thresholds

3. Monthly Operations

Monthly Checklist:

• Performance review
• Capacity planning
• Update monitoring dashboards
• Review and update procedures
• Training refresher for new team members

Incident Response

High DLQ Count (> 50 messages)

1. Check DLQ statistics via dashboard
2. Identify error category distribution
3. Check for systemic issues
4. Fix root cause if identified
5. Process DLQ with dry run first
6. Process DLQ for real
7. Monitor recovery
8. Document incident

Service Down

1. Check service status: pm2 status or systemctl status
2. Review recent logs
3. Restart service if needed
4. Verify recovery
5. Identify root cause
6. Implement preventive measures

Database Issues

1. Check MongoDB connection
2. Verify indexes
3. Check disk space
4. Review slow queries
5. Optimize if needed
6. Document resolution

Rollback Procedure

If issues arise after deployment:

1. Quick Rollback

# Stop new services
pm2 stop partner-dlq-handler

# Revert code
git checkout <previous-commit>

# Restart server
pm2 restart agm-server

# Verify old code working
curl http://localhost:3000/api/health

Checklist:

• Services stopped
• Code reverted
• Server restarted
• Functionality verified
• Issue documented

2. Database Rollback (if needed)

# Restore from backup
mongorestore \
  --uri="<connection-string>" \
  --collection=partnerlogtrackers \
  /backup/YYYYMMDD/agmission/partnerlogtrackers.bson

Checklist:

• Backup identified
• Database restored
• Data integrity verified
• Services restarted

Sign-Off

Deployment Team

• Developer: Code reviewed and tested

  • Signature: _______________ Date: _______________

• QA Engineer: Testing completed successfully

  • Signature: _______________ Date: _______________

• DevOps: Infrastructure configured and verified

  • Signature: _______________ Date: _______________

• Security: Security review completed

  • Signature: _______________ Date: _______________

• Operations Manager: Ready for production

  • Signature: _______________ Date: _______________

Post-Deployment Verification

• All tests passing
• No critical issues in first 24 hours
• Performance metrics acceptable
• Monitoring alerts configured
• Team trained and ready

Final Approval Date: _______________

---

Support Contacts

Technical Issues:

• Developer Team:
• On-call Engineer:

Business Issues:

• Product Owner:
• Operations Manager:

Emergency Escalation:

• Level 1: Team Lead
• Level 2: Engineering Manager
• Level 3: CTO

---

Deployment Completed: [ ] Yes [ ] No
Deployment Date: _______________
Deployed By: _______________
Version: 1.0.0