# Partner Log File Download and Storage Implementation

This document describes the implementation of partner log file download and local storage before processing.

## Overview

Modified the partner data polling system to download and store partner log files locally before enqueueing them for processing. This approach separates file acquisition from file processing, improving reliability and performance.

## Changes Made

### 1. Partner Data Polling Worker (`workers/partner_data_polling_worker.js`)

**Enhanced `pollAircraftData()` Function:**
- **File Download**: Downloads log files from partner systems using `partnerService.downloadLogFile()`
- **Local Storage**: Stores files in configured partner storage directory with timestamped filenames
- **Storage Management**: Creates storage directories if they don't exist
- **Error Handling**: Comprehensive error handling with cleanup of partial files
- **Task Enhancement**: Includes `localFilePath` in enqueued tasks

**Key Features Added:**
- Partner storage configuration integration
- Atomic file download and storage operations
- Filename sanitization to prevent conflicts
- Cleanup of partial downloads on failure
- Enhanced PartnerLogTracker updates with local file path

**Storage Path Format:**
```
{basePath}/{aircraftId}_{timestamp}_{sanitized_filename}
Example: /data/partners/satloc/aircraft123_2025-08-27T10-30-00-000Z_logfile.log
```

### 2. PartnerLogTracker Model (`model/partner_log_tracker.js`)

**Added Fields:**
```javascript
localFilePath: { type: String, required: false, trim: true }, // Path to downloaded log file
downloadedAt: { type: Date, required: false }, // When log was downloaded from partner system
```

**Enhanced Tracking:**
- Tracks local file storage location
- Records download timestamp
- Maintains processing state throughout download → process → complete lifecycle

### 3. Partner Sync Worker (`workers/partner_sync_worker.js`)

**Enhanced `processPartnerLog()` Function:**
- **Local File Processing**: Checks for `localFilePath` in task data first
- **File Verification**: Verifies local file exists before processing
- **Fallback Support**: Falls back to partner API if no local file (backward compatibility)
- **Performance**: Processes local files directly without API calls

**Added `processLocalLogFile()` Function:**
- **Direct File Access**: Reads log files from local storage
- **SatLoc Parser Integration**: Uses binary parser on local files
- **Assignment Matching**: Same matching logic as API-based processing
- **Application Data Saving**: Identical data processing and storage

**Benefits:**
- No API calls during processing phase
- Faster processing due to local file access
- Better error recovery - files remain available for retry
- Separation of concerns between download and processing

## Process Flow

### Before Enhancement
```
Poll → API Call to Get Log List → Enqueue Task → Process Task (API Call to Download) → Parse → Save
```

### After Enhancement
```
Poll → API Call to Get Log List → Download & Store Files → Enqueue Task → Process Task (Local File) → Parse → Save
```

## Configuration Requirements

### Environment Variables
```bash
# Partner storage paths
SATLOC_STORAGE_PATH=/data/partners/satloc
SATLOC_TEMP_PATH=/tmp/satloc
SATLOC_MAX_FILE_AGE=7776000000

AGIDRONEX_STORAGE_PATH=/data/partners/agidronex
AGIDRONEX_TEMP_PATH=/tmp/agidronex
AGIDRONEX_MAX_FILE_AGE=7776000000
```

### Directory Structure
```
/data/partners/
├── satloc/
│   ├── aircraft123_2025-08-27T10-30-00-000Z_flight001.log
│   ├── aircraft123_2025-08-27T10-35-00-000Z_flight002.log
│   └── aircraft456_2025-08-27T11-00-00-000Z_flight003.log
└── agidronex/
    ├── drone001_2025-08-27T09-00-00-000Z_mission_a.log
    └── drone001_2025-08-27T09-30-00-000Z_mission_b.log
```

## Task Message Format

### Enhanced Task Message
```javascript
{
  type: 'process_partner_log',
  data: {
    customerId: 'customer_id',
    partnerCode: 'SATLOC',
    aircraftId: 'aircraft123',
    logId: 'partner_log_id',
    logFileName: 'original_filename.log',
    uploadedDate: '2025-08-27T10:00:00Z',
    localFilePath: '/data/partners/satloc/aircraft123_2025-08-27T10-30-00-000Z_original_filename.log', // NEW
    assignments: [/* assignment objects */]
  }
}
```

## Error Handling

### Download Phase (Polling Worker)
1. **Storage Directory Creation Failure**: Logs error and aborts processing for that partner
2. **File Download Failure**: Cleans up partial files, updates tracker with error message
3. **File Write Failure**: Cleans up partial files, allows retry on next poll
4. **Task Enqueue Failure**: Removes local file path from tracker to allow retry

### Processing Phase (Sync Worker)
1. **Local File Missing**: Falls back to partner API download
2. **File Read Failure**: Reports error and fails task
3. **Parser Failure**: Reports error with file details
4. **Database Save Failure**: Reports error but keeps local file for retry

## Performance Benefits

### Reduced API Calls
- **Before**: 2 API calls per log (list + download during processing)
- **After**: 1 API call per log (list + download during polling)

### Improved Processing Speed
- Local file access vs. API download during processing
- No network latency during processing phase
- Better batch processing capabilities

### Enhanced Reliability
- Files remain available even if partner API goes down during processing
- Easier retry logic - files are already local
- Better debugging - files can be examined directly

## Monitoring and Maintenance

### Storage Space Monitoring
```bash
# Monitor partner storage usage
du -sh /data/partners/*

# Clean old files based on maxFileAge configuration
find /data/partners -name "*.log" -mtime +90 -delete
```

### Log File Analysis
```bash
# Count downloaded files by partner
ls /data/partners/satloc/*.log | wc -l
ls /data/partners/agidronex/*.log | wc -l

# Check file sizes
ls -lh /data/partners/satloc/*.log | tail -10
```

### Database Queries
```javascript
// Check download status
db.partner_log_trackers.find({
  "localFilePath": { $exists: true },
  "downloadedAt": { $gte: new Date(Date.now() - 24*60*60*1000) }
}).count()

// Check processing status
db.partner_log_trackers.find({
  "localFilePath": { $exists: true },
  "processed": false,
  "downloadedAt": { $lt: new Date(Date.now() - 60*60*1000) }
})
```

## Backward Compatibility

### Legacy Support
- Partner sync worker falls back to API download if `localFilePath` not provided
- Existing tasks in queue will continue to work
- No database migration required - new fields are optional

### Migration Strategy
1. Deploy updated code
2. Let polling worker start downloading new files
3. Existing queued tasks process normally via fallback
4. Monitor for successful local file processing

## Testing

### Manual Testing Steps
1. **Verify Storage Directory Creation**:
   ```bash
   # Check if directories are created
   ls -la /data/partners/
   ```

2. **Monitor File Downloads**:
   ```bash
   # Watch for new files
   watch 'ls -la /data/partners/satloc/'
   ```

3. **Check Database Updates**:
   ```javascript
   // Verify tracker updates
   db.partner_log_trackers.find({
     "localFilePath": { $exists: true }
   }).limit(5)
   ```

4. **Verify Processing**:
   ```bash
   # Check logs for local file processing messages
   grep "Using local file for processing" logs/partner_sync_worker.log
   ```

### Performance Testing
- Compare processing times before/after implementation
- Monitor API call reduction
- Test error recovery scenarios
- Validate disk space usage patterns

## Future Enhancements

### Possible Improvements
1. **File Compression**: Compress stored files to save disk space
2. **Checksums**: Verify file integrity with checksums
3. **Parallel Downloads**: Download multiple files concurrently
4. **Smart Cleanup**: Automated cleanup of old files based on processing status
5. **File Deduplication**: Avoid storing identical files multiple times
6. **Progress Tracking**: Track download progress for large files

### Storage Optimization
- Implement file rotation based on age and processing status
- Add file compression for long-term storage
- Consider cloud storage integration for archive purposes

This implementation provides a robust foundation for partner log file processing with improved reliability, performance, and maintainability.