Devin Major 0836fc0fbc first commit (copy of Trunk as of April 22 2026)

2026-04-22 15:00:02 -04:00

8.2 KiB

Raw Blame History

Partner Log File Download and Storage Implementation

This document describes the implementation of partner log file download and local storage before processing.

Overview

Modified the partner data polling system to download and store partner log files locally before enqueueing them for processing. This approach separates file acquisition from file processing, improving reliability and performance.

Changes Made

1. Partner Data Polling Worker (`workers/partner_data_polling_worker.js`)

Enhanced pollAircraftData() Function:

File Download: Downloads log files from partner systems using partnerService.downloadLogFile()
Local Storage: Stores files in configured partner storage directory with timestamped filenames
Storage Management: Creates storage directories if they don't exist
Error Handling: Comprehensive error handling with cleanup of partial files
Task Enhancement: Includes localFilePath in enqueued tasks

Key Features Added:

Partner storage configuration integration
Atomic file download and storage operations
Filename sanitization to prevent conflicts
Cleanup of partial downloads on failure
Enhanced PartnerLogTracker updates with local file path

Storage Path Format:

{basePath}/{aircraftId}_{timestamp}_{sanitized_filename}
Example: /data/partners/satloc/aircraft123_2025-08-27T10-30-00-000Z_logfile.log

2. PartnerLogTracker Model (`model/partner_log_tracker.js`)

Added Fields:

localFilePath: { type: String, required: false, trim: true }, // Path to downloaded log file
downloadedAt: { type: Date, required: false }, // When log was downloaded from partner system

Enhanced Tracking:

Tracks local file storage location
Records download timestamp
Maintains processing state throughout download → process → complete lifecycle

3. Partner Sync Worker (`workers/partner_sync_worker.js`)

Enhanced processPartnerLog() Function:

Local File Processing: Checks for localFilePath in task data first
File Verification: Verifies local file exists before processing
Fallback Support: Falls back to partner API if no local file (backward compatibility)
Performance: Processes local files directly without API calls

Added processLocalLogFile() Function:

Direct File Access: Reads log files from local storage
SatLoc Parser Integration: Uses binary parser on local files
Assignment Matching: Same matching logic as API-based processing
Application Data Saving: Identical data processing and storage

Benefits:

No API calls during processing phase
Faster processing due to local file access
Better error recovery - files remain available for retry
Separation of concerns between download and processing

Process Flow

Before Enhancement

Poll → API Call to Get Log List → Enqueue Task → Process Task (API Call to Download) → Parse → Save

After Enhancement

Poll → API Call to Get Log List → Download & Store Files → Enqueue Task → Process Task (Local File) → Parse → Save

Configuration Requirements

Environment Variables

# Partner storage paths
SATLOC_STORAGE_PATH=/data/partners/satloc
SATLOC_TEMP_PATH=/tmp/satloc
SATLOC_MAX_FILE_AGE=7776000000

AGIDRONEX_STORAGE_PATH=/data/partners/agidronex
AGIDRONEX_TEMP_PATH=/tmp/agidronex
AGIDRONEX_MAX_FILE_AGE=7776000000

Directory Structure

/data/partners/
├── satloc/
│   ├── aircraft123_2025-08-27T10-30-00-000Z_flight001.log
│   ├── aircraft123_2025-08-27T10-35-00-000Z_flight002.log
│   └── aircraft456_2025-08-27T11-00-00-000Z_flight003.log
└── agidronex/
    ├── drone001_2025-08-27T09-00-00-000Z_mission_a.log
    └── drone001_2025-08-27T09-30-00-000Z_mission_b.log

Task Message Format

Enhanced Task Message

{
  type: 'process_partner_log',
  data: {
    customerId: 'customer_id',
    partnerCode: 'SATLOC',
    aircraftId: 'aircraft123',
    logId: 'partner_log_id',
    logFileName: 'original_filename.log',
    uploadedDate: '2025-08-27T10:00:00Z',
    localFilePath: '/data/partners/satloc/aircraft123_2025-08-27T10-30-00-000Z_original_filename.log', // NEW
    assignments: [/* assignment objects */]
  }
}

Error Handling

Download Phase (Polling Worker)

Storage Directory Creation Failure: Logs error and aborts processing for that partner
File Download Failure: Cleans up partial files, updates tracker with error message
File Write Failure: Cleans up partial files, allows retry on next poll
Task Enqueue Failure: Removes local file path from tracker to allow retry

Processing Phase (Sync Worker)

Local File Missing: Falls back to partner API download
File Read Failure: Reports error and fails task
Parser Failure: Reports error with file details
Database Save Failure: Reports error but keeps local file for retry

Performance Benefits

Reduced API Calls

Before: 2 API calls per log (list + download during processing)
After: 1 API call per log (list + download during polling)

Improved Processing Speed

Local file access vs. API download during processing
No network latency during processing phase
Better batch processing capabilities

Enhanced Reliability

Files remain available even if partner API goes down during processing
Easier retry logic - files are already local
Better debugging - files can be examined directly

Monitoring and Maintenance

Storage Space Monitoring

# Monitor partner storage usage
du -sh /data/partners/*

# Clean old files based on maxFileAge configuration
find /data/partners -name "*.log" -mtime +90 -delete

Log File Analysis

# Count downloaded files by partner
ls /data/partners/satloc/*.log | wc -l
ls /data/partners/agidronex/*.log | wc -l

# Check file sizes
ls -lh /data/partners/satloc/*.log | tail -10

Database Queries

// Check download status
db.partner_log_trackers.find({
  "localFilePath": { $exists: true },
  "downloadedAt": { $gte: new Date(Date.now() - 24*60*60*1000) }
}).count()

// Check processing status
db.partner_log_trackers.find({
  "localFilePath": { $exists: true },
  "processed": false,
  "downloadedAt": { $lt: new Date(Date.now() - 60*60*1000) }
})

Backward Compatibility

Legacy Support

Partner sync worker falls back to API download if localFilePath not provided
Existing tasks in queue will continue to work
No database migration required - new fields are optional

Migration Strategy

Deploy updated code
Let polling worker start downloading new files
Existing queued tasks process normally via fallback
Monitor for successful local file processing

Testing

Manual Testing Steps

Verify Storage Directory Creation:

# Check if directories are created
ls -la /data/partners/

Monitor File Downloads:

# Watch for new files
watch 'ls -la /data/partners/satloc/'

Check Database Updates:

// Verify tracker updates
db.partner_log_trackers.find({
  "localFilePath": { $exists: true }
}).limit(5)

Verify Processing:

# Check logs for local file processing messages
grep "Using local file for processing" logs/partner_sync_worker.log

Performance Testing

Compare processing times before/after implementation
Monitor API call reduction
Test error recovery scenarios
Validate disk space usage patterns

Future Enhancements

Possible Improvements

File Compression: Compress stored files to save disk space
Checksums: Verify file integrity with checksums
Parallel Downloads: Download multiple files concurrently
Smart Cleanup: Automated cleanup of old files based on processing status
File Deduplication: Avoid storing identical files multiple times
Progress Tracking: Track download progress for large files

Storage Optimization

Implement file rotation based on age and processing status
Add file compression for long-term storage
Consider cloud storage integration for archive purposes

This implementation provides a robust foundation for partner log file processing with improved reliability, performance, and maintainability.

8.2 KiB Raw Blame History