87 lines
3.7 KiB
Markdown
87 lines
3.7 KiB
Markdown
# Worker Responsibilities Update Summary
|
|
|
|
## Key Changes Made
|
|
|
|
### 1. **Dedicated Partner Queue Architecture**
|
|
- **New Queue**: `partner_jobs` (production) / `dev_partner_jobs` (development)
|
|
- **Purpose**: Separates partner tasks from internal job processing
|
|
- **Benefit**: Better scalability and error isolation
|
|
|
|
### 2. **Worker Responsibility Separation**
|
|
|
|
#### **Partner Data Polling Worker** (`workers/partner_data_polling_worker.js`)
|
|
- ✅ **ENHANCED**: Downloads log files from partner systems using `partnerService.downloadLogFile()`
|
|
- ✅ **ENHANCED**: Stores files locally in partner-specific directories
|
|
- ✅ **ENHANCED**: Updates `PartnerLogTracker` with download status and local file paths
|
|
- ✅ **ENHANCED**: Enqueues `PROCESS_PARTNER_DATA_FILE` tasks with local file paths
|
|
- ✅ **RELIABLE**: Separates file acquisition from file processing for better error handling
|
|
|
|
#### **Partner Sync Worker** (`workers/partner_sync_worker.js`)
|
|
- ✅ **PRIMARY**: Handle job upload/assignment to partner aircraft
|
|
- ✅ **PRIMARY**: Handle partner job data synchronization from partners' systems
|
|
- ✅ **ENHANCED**: Process local binary log files using `SatLocBinaryProcessor`
|
|
- ✅ **ENHANCED**: Comprehensive statistics calculation and application metrics
|
|
- ✅ **ENHANCED**: Queue-based task processing via dedicated partner queue
|
|
- ✅ **ENHANCED**: Scheduled periodic sync as backup
|
|
|
|
|
|
#### **Job Worker** (`workers/job_worker.js`)
|
|
- ✅ **FOCUSED**: Handle only internal data submitted by internal systems/clients
|
|
- ✅ **REMOVED**: Partner task processing (moved to dedicated worker)
|
|
- ✅ **CLEAN**: Simplified codebase focusing on core job processing
|
|
|
|
### 3. **Credential Management Update**
|
|
- ✅ **REMOVED**: Global environment variables (`SATLOC_EMAIL`, `SATLOC_PASSWORD`)
|
|
- ✅ **ENHANCED**: Individual partner system user credentials per customer/applicator
|
|
- ✅ **SECURE**: Database-stored encrypted credentials with proper access control
|
|
|
|
### 4. **Automatic Data Sync Triggers**
|
|
- ✅ **SMART**: Data sync automatically triggered after successful job upload
|
|
- ✅ **DELAYED**: 30-second delay to allow partner system processing
|
|
- ✅ **EFFICIENT**: Reduces unnecessary polling and improves responsiveness
|
|
|
|
## Flow Diagram
|
|
|
|
```
|
|
Job Assignment Request
|
|
↓
|
|
Partner Sync Worker (via partner queue)
|
|
↓
|
|
Upload Job to Partner API
|
|
↓
|
|
Success? → Auto-queue data sync task (30s delay)
|
|
↓
|
|
Partner Data Polling Worker
|
|
↓
|
|
Download & Store Log Files Locally
|
|
↓
|
|
Enqueue PROCESS_PARTNER_DATA_FILE task
|
|
↓
|
|
Partner Sync Worker processes local file
|
|
↓
|
|
SatLocBinaryProcessor parses binary data
|
|
↓
|
|
Enhanced Statistics & Application Details
|
|
↓
|
|
Save to Database (100% success rate)
|
|
```
|
|
|
|
## Benefits Achieved
|
|
|
|
1. **Better Separation of Concerns**: Partner operations isolated from internal job processing
|
|
2. **Improved Scalability**: Dedicated queue allows independent scaling of partner workers
|
|
3. **Enhanced Security**: Individual credentials instead of shared environment variables
|
|
4. **Automatic Sync**: Intelligent triggering reduces manual intervention
|
|
5. **Better Error Handling**: Partner failures don't affect internal job processing
|
|
6. **Cleaner Codebase**: Simplified job worker focusing on core functionality
|
|
|
|
## Migration Notes
|
|
|
|
- No breaking changes to existing APIs
|
|
- Partner system users need proper credentials configured
|
|
- New environment variable: `QUEUE_NAME_PARTNER` (optional, defaults to 'partner_tasks', auto-prefixes 'dev_' in development)
|
|
- Existing job assignments continue to work
|
|
- Enhanced with automatic sync capabilities
|
|
|
|
This update establishes a robust, scalable foundation for partner integrations while maintaining clean separation between internal and external operations.
|