agnav/agmission
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
01881947a7
agmission/Development/server/docs/WORKER_RESPONSIBILITIES_UPDATE.md

3.7 KiB
Raw Blame History

Worker Responsibilities Update Summary

Key Changes Made

1. Dedicated Partner Queue Architecture

  • New Queue: partner_jobs (production) / dev_partner_jobs (development)
  • Purpose: Separates partner tasks from internal job processing
  • Benefit: Better scalability and error isolation

2. Worker Responsibility Separation

Partner Data Polling Worker (workers/partner_data_polling_worker.js)

  • ✅ ENHANCED: Downloads log files from partner systems using partnerService.downloadLogFile()
  • ✅ ENHANCED: Stores files locally in partner-specific directories
  • ✅ ENHANCED: Updates PartnerLogTracker with download status and local file paths
  • ✅ ENHANCED: Enqueues PROCESS_PARTNER_DATA_FILE tasks with local file paths
  • ✅ RELIABLE: Separates file acquisition from file processing for better error handling

Partner Sync Worker (workers/partner_sync_worker.js)

  • ✅ PRIMARY: Handle job upload/assignment to partner aircraft
  • ✅ PRIMARY: Handle partner job data synchronization from partners' systems
  • ✅ ENHANCED: Process local binary log files using SatLocBinaryProcessor
  • ✅ ENHANCED: Comprehensive statistics calculation and application metrics
  • ✅ ENHANCED: Queue-based task processing via dedicated partner queue
  • ✅ ENHANCED: Scheduled periodic sync as backup

Job Worker (workers/job_worker.js)

  • ✅ FOCUSED: Handle only internal data submitted by internal systems/clients
  • ✅ REMOVED: Partner task processing (moved to dedicated worker)
  • ✅ CLEAN: Simplified codebase focusing on core job processing

3. Credential Management Update

  • ✅ REMOVED: Global environment variables (SATLOC_EMAIL, SATLOC_PASSWORD)
  • ✅ ENHANCED: Individual partner system user credentials per customer/applicator
  • ✅ SECURE: Database-stored encrypted credentials with proper access control

4. Automatic Data Sync Triggers

  • ✅ SMART: Data sync automatically triggered after successful job upload
  • ✅ DELAYED: 30-second delay to allow partner system processing
  • ✅ EFFICIENT: Reduces unnecessary polling and improves responsiveness

Flow Diagram

Job Assignment Request
         ↓
Partner Sync Worker (via partner queue)
         ↓
Upload Job to Partner API
         ↓
Success? → Auto-queue data sync task (30s delay)
         ↓
Partner Data Polling Worker
         ↓
Download & Store Log Files Locally
         ↓
Enqueue PROCESS_PARTNER_DATA_FILE task
         ↓
Partner Sync Worker processes local file
         ↓
SatLocBinaryProcessor parses binary data
         ↓
Enhanced Statistics & Application Details
         ↓
Save to Database (100% success rate)

Benefits Achieved

  1. Better Separation of Concerns: Partner operations isolated from internal job processing
  2. Improved Scalability: Dedicated queue allows independent scaling of partner workers
  3. Enhanced Security: Individual credentials instead of shared environment variables
  4. Automatic Sync: Intelligent triggering reduces manual intervention
  5. Better Error Handling: Partner failures don't affect internal job processing
  6. Cleaner Codebase: Simplified job worker focusing on core functionality

Migration Notes

  • No breaking changes to existing APIs
  • Partner system users need proper credentials configured
  • New environment variable: QUEUE_NAME_PARTNER (optional, defaults to 'partner_tasks', auto-prefixes 'dev_' in development)
  • Existing job assignments continue to work
  • Enhanced with automatic sync capabilities

This update establishes a robust, scalable foundation for partner integrations while maintaining clean separation between internal and external operations.