361 lines
17 KiB
Markdown
361 lines
17 KiB
Markdown
# Partner Integration System
|
|
|
|
**Last Updated**: February 2026 | **Active partners**: SatLoc (`SATLOC`) | **Planned**: AGIDRONEX
|
|
|
|
A multi-partner integration framework built on a dual-user model with a generic service layer. Originally implemented for SatLoc; designed to extend to any partner providing job-upload and log-download APIs.
|
|
|
|
For full architecture diagrams → [docs/PARTNER_INTEGRATION_ARCHITECTURE.md](docs/PARTNER_INTEGRATION_ARCHITECTURE.md)
|
|
For queue/DLQ data-flow → [docs/PARTNER_TASK_DATA_FLOW_ANALYSIS.md](docs/PARTNER_TASK_DATA_FLOW_ANALYSIS.md)
|
|
|
|
## Architecture Overview
|
|
|
|
### Dual User System
|
|
|
|
1. **Partner Organizations**: Companies providing integration services (e.g., SatLoc)
|
|
- Stored as User entities with `kind: "PARTNER"`
|
|
- Contains partner-specific configuration and metadata
|
|
|
|
2. **Partner System Users**: Customer credential records for each partner system
|
|
- Stored as User entities with `kind: "PARTNER_SYSTEM_USER"`
|
|
- Links AgMission customers to their partner system accounts
|
|
- Contains partner-specific credentials and identifiers
|
|
- **Used by workers and services to communicate with partner APIs**
|
|
|
|
### Assignment Design
|
|
|
|
**Job assignments are created with normal internal Aircraft user IDs**:
|
|
- Assignments use standard internal Aircraft user IDs (`kind: "9"`) for compatibility with existing job assignment logic
|
|
- Partner integration is determined by checking if the assigned user has associated PartnerSystemUser records
|
|
- Partner-specific metadata (like `partnerAircraftId`) is stored as assignment fields
|
|
- Workers and services use the Partner System User records to obtain credentials for external API calls
|
|
|
|
## Service Layer
|
|
|
|
```
|
|
helpers/partner_service_factory.js — ★ singleton factory used by all workers
|
|
services/base_partner_service.js — abstract base; defines uploadJobDataToAircraft,
|
|
healthCheck, getAircraftList, getStoragePath, etc.
|
|
services/satloc_service.js — SatLoc implementation; Redis-backed auth cache
|
|
services/partner_sync_service.js — orchestrates upload/sync; used by sync worker
|
|
services/task_id_generator.js — generates unique task/execution IDs for TaskTracker
|
|
```
|
|
|
|
### Auth / Credential Flow
|
|
1. Worker receives `customerId`
|
|
2. `satloc_service.getCachedAuth(customerId)` checks **Redis** for valid JWT
|
|
3. Cache miss → queries `PartnerSystemUser` for credentials → calls SatLoc API
|
|
4. JWT cached in Redis with TTL; auto-retries once on expiry
|
|
5. No global credentials — every API call uses the customer's own SatLoc account
|
|
|
|
### Tracking Models
|
|
- **`PartnerLogTracker`** — per-log-file: `PENDING → DOWNLOADING → DOWNLOADED → PROCESSING → PROCESSED | FAILED | SKIPPED`
|
|
- **`TaskTracker`** — per-queue-task (Jan 2026): `QUEUED → PROCESSING → COMPLETED | FAILED`; retry-chain and stuck-task detection
|
|
|
|
## Quick Start
|
|
|
|
### 1. Environment Configuration (`environment.env`)
|
|
|
|
```bash
|
|
# SatLoc API — customer credentials are per-customer in PartnerSystemUser records
|
|
SATLOC_API_ENDPOINT=https://www.satloccloudfc.com/api/Satloc
|
|
SATLOC_API_TIMEOUT=30000
|
|
SATLOC_RETRY_ATTEMPTS=3
|
|
SATLOC_RETRY_DELAY=10000
|
|
SATLOC_RATE_LIMIT=60
|
|
SATLOC_BURST_LIMIT=10
|
|
SATLOC_STORAGE_PATH=/path/to/uploads/satloc
|
|
SATLOC_REALTIME_ENABLED=true
|
|
SATLOC_FILE_UPLOAD_ENABLED=true
|
|
SATLOC_1ST_ASSIGNMENT_ALWAYS_MATCH=true
|
|
|
|
# Worker control
|
|
PARTNER_POLLING_ENABLED=true
|
|
PARTNER_SYNC_ENABLED=true
|
|
PARTNER_MAX_RETRIES=5
|
|
STUCK_TIMEOUT_MS=18*60*1000
|
|
```
|
|
|
|
### 2. Run Setup Script
|
|
|
|
```bash
|
|
node setup_partners.js
|
|
```
|
|
|
|
Creates: SatLoc partner organization, sample customer, and a PartnerSystemUser linking them.
|
|
|
|
### 3. Start Workers
|
|
|
|
```bash
|
|
node start_workers.js
|
|
# or individually:
|
|
node workers/partner_data_polling_worker.js
|
|
node workers/partner_sync_worker.js
|
|
```
|
|
|
|
## API Reference
|
|
|
|
All endpoints at `/api/partners` (authentication required).
|
|
|
|
### Partner Organization CRUD
|
|
|
|
| Method | Path | Description |
|
|
|--------|------|-------------|
|
|
| `GET` | `/api/partners` | List all partners |
|
|
| `POST` | `/api/partners` | Create partner |
|
|
| `GET` | `/api/partners/:id` | Get partner details |
|
|
| `PUT` | `/api/partners/:id` | Update partner |
|
|
| `DELETE` | `/api/partners/:id` | Soft-delete (`active: false`) |
|
|
|
|
### Partner System User CRUD
|
|
|
|
| Method | Path | Description |
|
|
|--------|------|-------------|
|
|
| `GET` | `/api/partners/systemUsers` | List all system users |
|
|
| `POST` | `/api/partners/systemUsers` | Create system user |
|
|
| `POST` | `/api/partners/systemUsers/testAuth` | Test credentials against partner API |
|
|
| `GET` | `/api/partners/systemUsers/:id` | Get system user |
|
|
| `PUT` | `/api/partners/systemUsers/:id` | Update (partial updates supported) |
|
|
| `DELETE` | `/api/partners/systemUsers/:id` | Soft-delete |
|
|
|
|
**Create body:**
|
|
```json
|
|
{
|
|
"partnerId": "<partner_org_id>",
|
|
"customerId": "<agmission_customer_id>",
|
|
"username": "customer_satloc_username",
|
|
"password": "customer_satloc_password",
|
|
"name": "Customer SatLoc Account",
|
|
"active": true
|
|
}
|
|
```
|
|
|
|
### Other Partner Routes
|
|
|
|
| Method | Path | Description |
|
|
|--------|------|-------------|
|
|
| `GET` | `/api/partners/customers` | List customers for a partner (with subscription info). Query: `?partnerId=<id>` |
|
|
| `GET` | `/api/partners/aircraft` | Get aircraft list from partner API |
|
|
| `POST` | `/api/partners/uploadJob` | Manually trigger job upload to partner |
|
|
| `POST` | `/api/partners/syncData` | Trigger partner data sync |
|
|
|
|
### Global DLQ API (for `partner_tasks`)
|
|
|
|
```bash
|
|
GET /api/dlq/partner_tasks/list
|
|
POST /api/dlq/partner_tasks/retryAll
|
|
POST /api/dlq/partner_tasks/retryByPosition # body: {"position":0}
|
|
POST /api/dlq/partner_tasks/retryByHeader # body: {"headerName":"x-partner-code","headerValue":"SATLOC"}
|
|
POST /api/dlq/partner_tasks/purge
|
|
```
|
|
|
|
### Assign Jobs with Partner Integration
|
|
|
|
Use **internal** user IDs — partner detection is automatic:
|
|
|
|
```json
|
|
POST /api/jobs/assign
|
|
{
|
|
"jobId": "job_id",
|
|
"dlOp": { "type": 1 },
|
|
"asUsers": [
|
|
{
|
|
"uid": "<internal_pilot_user_id>",
|
|
"notes": "High priority mission"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Flow:
|
|
1. Assignment created in `controllers/job.js`
|
|
2. Partner integration detected from assigned user's context
|
|
3. **Immediate upload attempted** — if partner API is live, calls `partnerSyncService.uploadJobToPartner()` directly
|
|
4. **If immediate upload fails or API is offline** → queues `UPLOAD_PARTNER_JOB` task to `partner_tasks` queue
|
|
5. `partner_sync_worker` processes task → calls `uploadJobToPartner()` → stores `extJobId` on `JobAssign`, status set to `UPLOADED`
|
|
6. Polling worker (cron) finds `JobAssign` records with `status = UPLOADED` → polls partner for aircraft logs
|
|
7. New logs downloaded → `PROCESS_PARTNER_LOG` task queued → `partner_sync_worker` processes → `ApplicationDetail` records created
|
|
|
|
|
|
|
|
## Partner Data Flow
|
|
|
|
```
|
|
Polling Worker (cron — every 15 min prod / 1 min dev)
|
|
├─ Query JobAssign WHERE status = UPLOADED (jobs successfully sent to partner)
|
|
├─ Group by partnerCode + customerId
|
|
├─ Per aircraft: call partnerService.getAircraftLogs(customerId, aircraftId)
|
|
├─ Filter out already-processed logs (PartnerLogTracker where processed = true)
|
|
├─ For each new log:
|
|
│ ├─ Atomic upsert PartnerLogTracker (PENDING)
|
|
│ ├─ Claim for download → status DOWNLOADING
|
|
│ ├─ partnerService.getAircraftLogData() → save to SATLOC_STORAGE_PATH
|
|
│ ├─ Update PartnerLogTracker → DOWNLOADED
|
|
│ └─ Enqueue PROCESS_PARTNER_LOG → partner_tasks queue
|
|
└─ Periodic cleanup of stuck DOWNLOADING / DOWNLOADED / PROCESSING trackers
|
|
|
|
Sync Worker (queue consumer — partner_tasks)
|
|
├─ UPLOAD_PARTNER_JOB
|
|
│ ├─ Check idempotency (redelivered check via extJobId on JobAssign)
|
|
│ ├─ partnerSyncService.uploadJobToPartner(assignId)
|
|
│ ├─ POST to SatLoc UploadJobData
|
|
│ ├─ Store extJobId on JobAssign
|
|
│ └─ Set JobAssign.status = UPLOADED (AssignStatus.UPLOADED = 2)
|
|
└─ PROCESS_PARTNER_LOG
|
|
├─ TaskTracker idempotency check (claim or skip if already processing)
|
|
├─ Circuit breaker: skip repeatedly-failing files
|
|
├─ Atomic claim PartnerLogTracker → PROCESSING
|
|
├─ SatLocLogParser → SatLocApplicationProcessor
|
|
├─ Creates ApplicationFile + ApplicationDetail records
|
|
├─ PartnerLogTracker → PROCESSED
|
|
└─ On failure → nack → DLQ partner_tasks_failed
|
|
```
|
|
|
|
See [docs/PARTNER_TASK_DATA_FLOW_ANALYSIS.md](docs/PARTNER_TASK_DATA_FLOW_ANALYSIS.md) for full Mermaid diagrams.
|
|
|
|
## File Structure
|
|
|
|
### Core Partner Files
|
|
|
|
```
|
|
server/
|
|
├── model/
|
|
│ ├── partner.js # Partner + PartnerSystemUser discriminators
|
|
│ ├── partner_log_tracker.js # Per-log-file lifecycle tracking
|
|
│ └── task_tracker.js # Per-queue-task tracking (Phase 2, Jan 2026)
|
|
├── controllers/partner.js # RESTful CRUD + action endpoints
|
|
├── routes/partner.js # Mounted at /api/partners
|
|
├── helpers/
|
|
│ ├── partner_config.js # Partner configuration (SATLOC, AGIDRONEX stub)
|
|
│ ├── partner_service_factory.js # ★ Singleton factory (used by all workers)
|
|
│ ├── satloc_log_parser.js # Binary .LOG file parser
|
|
│ ├── satloc_application_processor.js # Parsed records → ApplicationDetail
|
|
│ ├── file_satlog.js # Low-level binary read utilities
|
|
│ └── satloc_util.js # SatLoc utilities
|
|
├── services/
|
|
│ ├── base_partner_service.js # Abstract base class
|
|
│ ├── satloc_service.js # SatLoc API client + Redis auth cache
|
|
│ ├── partner_sync_service.js # Orchestrates upload/sync flows
|
|
│ └── task_id_generator.js # Task/execution ID generation
|
|
├── workers/
|
|
│ ├── partner_data_polling_worker.js # Cron: polls APIs, downloads, enqueues
|
|
│ ├── partner_sync_worker.js # Consumer: processes partner_tasks queue
|
|
│ ├── dlq_archival_worker.js # Archives expired DLQ messages to disk
|
|
│ └── dlq_alert_worker.js # Threshold-based email alerts
|
|
├── controllers/dlq.js # Global DLQ controller (all queues)
|
|
├── routes/dlq.js # Mounted at /api/dlq/:queueName/*
|
|
├── public/dlq-monitor.html # ★ Web DLQ monitoring dashboard
|
|
└── setup_partners.js # Dev setup script
|
|
```
|
|
|
|
## Key Milestones
|
|
|
|
| Date | Change |
|
|
|------|--------|
|
|
| Jul 2025 | RESTful API standardization — consistent `:id` params, soft deletes, controller reuse |
|
|
| Aug 2025 | Binary processing architecture — `SatLocBinaryProcessor`, file download + local storage |
|
|
| Oct 2025 | Auth refactoring — `authenticate()` vs `authenticateAndCache()` separation; Redis cache |
|
|
| Dec 2025 | DLQ Step 8 — queue-native global `/api/dlq/:queueName/*` endpoints; web dashboard |
|
|
| Jan 2026 | TaskTracker (Phase 2) — per-task lifecycle tracking with retry chain and stuck-task detection |
|
|
|
|
## Environment Variables
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `SATLOC_API_ENDPOINT` | `https://www.satloccloudfc.com/api/Satloc` | SatLoc Cloud base URL |
|
|
| `SATLOC_API_TIMEOUT` | `30000` | Request timeout (ms) |
|
|
| `SATLOC_RETRY_ATTEMPTS` | `3` | Max retry attempts |
|
|
| `SATLOC_RETRY_DELAY` | `10000` | Delay between retries (ms) |
|
|
| `SATLOC_RATE_LIMIT` | `60` | Requests per minute |
|
|
| `SATLOC_BURST_LIMIT` | `10` | Burst request limit |
|
|
| `SATLOC_STORAGE_PATH` | `/uploads/satloc` | Local log file storage |
|
|
| `SATLOC_MAX_FILE_SIZE` | `10485760` | Max log file size (bytes) |
|
|
| `SATLOC_REALTIME_ENABLED` | `true` | Enable real-time polling |
|
|
| `SATLOC_FILE_UPLOAD_ENABLED` | `true` | Enable file download/storage |
|
|
| `SATLOC_1ST_ASSIGNMENT_ALWAYS_MATCH` | `true` | Match first assignment when no extJobId |
|
|
| `PARTNER_POLLING_ENABLED` | `true` | Enable polling worker |
|
|
| `PARTNER_SYNC_ENABLED` | `true` | Enable sync worker |
|
|
| `PARTNER_MAX_RETRIES` | `5` | Max retries for stuck tasks |
|
|
| `STUCK_TIMEOUT_MS` | `18*60*1000` | Stuck-task detection threshold |
|
|
|
|
## Monitoring
|
|
|
|
### DLQ Web Dashboard
|
|
```
|
|
http://localhost:4100/public/dlq-monitor.html
|
|
```
|
|
|
|
### DLQ CLI
|
|
```bash
|
|
node scripts/publish_to_dlq.js --env ./environment.env --queue dev_partner_tasks --count 3
|
|
node scripts/test_dlq_e2e.js --env ./environment.env --queue dev_partner_tasks
|
|
```
|
|
|
|
### Debug Logging
|
|
```bash
|
|
# All partner + satloc modules:
|
|
DEBUG=agm:partner*,agm:satloc* node server.js
|
|
# Workers:
|
|
DEBUG=agm:* node workers/partner_data_polling_worker.js
|
|
```
|
|
|
|
### Log Level Filtering (Pino)
|
|
See `PINO_MODULE_FILTERING_GUIDE.md` for filtering by module (e.g., `LOG_MODULES=partner*,satloc*`).
|
|
|
|
## Benefits
|
|
|
|
### For Development
|
|
- **Simplified Architecture**: Reuses existing User model infrastructure
|
|
- **Environment Configuration**: No complex partner management UI needed
|
|
- **Easy Testing**: Standard User entity patterns
|
|
- **Maintainable**: Fewer moving parts than separate partner models
|
|
|
|
### For Operations
|
|
- **Customer Isolation**: Each customer has their own partner system account
|
|
- **Secure Credentials**: Environment-based credential management
|
|
- **Simple Monitoring**: Basic health checks without complex infrastructure
|
|
- **Scalable**: Easy to add new customers and partners
|
|
|
|
### For Business
|
|
- **Partner Flexibility**: Easy to add new partners with environment configuration
|
|
- **Cost Effective**: No complex monitoring infrastructure required
|
|
- **Mixed Assignments**: Support for both internal and partner aircraft in same job
|
|
|
|
## Troubleshooting
|
|
|
|
| Problem | Likely Cause | Solution |
|
|
|---------|-------------|----------|
|
|
| `partner_service_not_implemented` | Unregistered `partnerCode` | Check `helpers/partner_service_factory.js` `serviceMapping` |
|
|
| `No SatLoc system user found` | Missing `PartnerSystemUser` record | Run `setup_partners.js` or `POST /api/partners/systemUsers` |
|
|
| Auth fails after token expiry | Stale Redis cache | Service auto-retries once; check `REDIS_PWD` and Redis connectivity |
|
|
| Log files not processed | `PartnerLogTracker` stuck in `DOWNLOADING` | Startup cleanup resets stuck tasks automatically; tune `STUCK_TIMEOUT_MS` |
|
|
| Tasks accumulating in DLQ | Worker processing failures | `POST /api/dlq/partner_tasks/retryAll` or use web dashboard |
|
|
| Queue connection lost | RabbitMQ restart | Workers reconnect automatically; in-flight tasks cached in memory offline queue |
|
|
|
|
## Documentation Index
|
|
|
|
| Document | Purpose | Status |
|
|
|----------|---------|--------|
|
|
| `docs/PARTNER_INTEGRATION_ARCHITECTURE.md` | Full architecture, diagrams, current state | ✅ Current |
|
|
| `docs/PARTNER_TASK_DATA_FLOW_ANALYSIS.md` | Queue/DLQ message lifecycle (Mermaid) | ✅ Current (Jan 2026) |
|
|
| `docs/PARTNER_RESPONSIBILITIES_ANALYSIS.md` | Worker responsibility boundaries | ✅ Current |
|
|
| `docs/PARTNER_AUTH_REFACTORING.md` | SatLoc auth separation + Redis cache design | ✅ Current |
|
|
| `docs/PARTNER_MODEL_SCHEMA_UPDATES.md` | PartnerSystemUser + Customer schema relationships | ✅ Current |
|
|
| `docs/PARTNER_LOG_FILE_PROCESSING.md` | Log processing pipeline | ✅ Current |
|
|
| `docs/PARTNER_LOG_DOWNLOAD_IMPLEMENTATION.md` | Polling worker download implementation | ✅ Current |
|
|
| `docs/PARTNER_INTEGRATION_IMPLEMENTATION.md` | Implementation patterns and examples | ✅ Current |
|
|
| `docs/PARTNER_SYSTEM_REFACTORING_SUMMARY.md` | Historical: Jul 2025 RESTful refactor | 📁 Historical |
|
|
| `docs/PARTNER_SYNC_INTEGRATION_SUMMARY.md` | Historical: Aug 2025 sync improvements | 📁 Historical |
|
|
| `docs/PARTNER_LOG_MIGRATION_SUMMARY.md` | Historical: log tracker schema migration | 📁 Historical |
|
|
| `docs/PARTNER_SYNC_WORKER_REFACTORING.md` | Historical: worker refactoring milestone | 📁 Historical |
|
|
| `docs/SATLOC_API_SPECIFICATION.md` | SatLoc external API endpoint reference | ✅ Reference |
|
|
| `docs/SATLOC_API_ACTUAL_BEHAVIOR.md` | Documented deviations from spec (important!) | ✅ Reference |
|
|
| `docs/SATLOC_BINARY_PROCESSING_ARCHITECTURE.md` | Binary log parsing architecture | ✅ Reference |
|
|
| `docs/SATLOC_APPLICATION_PROCESSOR_README.md` | `satloc_application_processor.js` usage | ✅ Reference |
|
|
| `docs/SATLOC_ERROR_PATTERNS.md` | Known error patterns and handling | ✅ Reference |
|
|
| `docs/SATLOC_LOG_NOTES.md` | Raw log format notes | ✅ Reference |
|
|
| `docs/Transland_SATLOC_Log_File_Formats_v3_76.md` | Official SatLoc log format spec | ✅ Reference |
|
|
| `docs/SATLOC_COMPLETE_IMPLEMENTATION.md` | Historical: implementation milestone | 📁 Historical |
|
|
| `docs/SATLOC_IMPLEMENTATION_SUMMARY.md` | Historical: implementation summary | 📁 Historical |
|
|
| `docs/SATLOC_INTEGRATION_SUMMARY.md` | Historical: integration summary | 📁 Historical |
|
|
| `docs/SATLOC_TESTING_SUMMARY.md` | Historical: testing milestone | 📁 Historical |
|