Devin Major 0836fc0fbc first commit (copy of Trunk as of April 22 2026)

2026-04-22 15:00:02 -04:00

17 KiB

Raw Blame History

Partner Integration System

Last Updated: February 2026 | Active partners: SatLoc (SATLOC) | Planned: AGIDRONEX

A multi-partner integration framework built on a dual-user model with a generic service layer. Originally implemented for SatLoc; designed to extend to any partner providing job-upload and log-download APIs.

For full architecture diagrams → docs/PARTNER_INTEGRATION_ARCHITECTURE.md
For queue/DLQ data-flow → docs/PARTNER_TASK_DATA_FLOW_ANALYSIS.md

Architecture Overview

Dual User System

Partner Organizations: Companies providing integration services (e.g., SatLoc)
- Stored as User entities with kind: "PARTNER"
- Contains partner-specific configuration and metadata
Partner System Users: Customer credential records for each partner system
- Stored as User entities with kind: "PARTNER_SYSTEM_USER"
- Links AgMission customers to their partner system accounts
- Contains partner-specific credentials and identifiers
- Used by workers and services to communicate with partner APIs

Assignment Design

Job assignments are created with normal internal Aircraft user IDs:

Assignments use standard internal Aircraft user IDs (kind: "9") for compatibility with existing job assignment logic
Partner integration is determined by checking if the assigned user has associated PartnerSystemUser records
Partner-specific metadata (like partnerAircraftId) is stored as assignment fields
Workers and services use the Partner System User records to obtain credentials for external API calls

Service Layer

helpers/partner_service_factory.js   — ★ singleton factory used by all workers
services/base_partner_service.js     — abstract base; defines uploadJobDataToAircraft,
                                       healthCheck, getAircraftList, getStoragePath, etc.
services/satloc_service.js           — SatLoc implementation; Redis-backed auth cache
services/partner_sync_service.js     — orchestrates upload/sync; used by sync worker
services/task_id_generator.js        — generates unique task/execution IDs for TaskTracker

Auth / Credential Flow

Worker receives customerId
satloc_service.getCachedAuth(customerId) checks Redis for valid JWT
Cache miss → queries PartnerSystemUser for credentials → calls SatLoc API
JWT cached in Redis with TTL; auto-retries once on expiry
No global credentials — every API call uses the customer's own SatLoc account

Tracking Models

PartnerLogTracker — per-log-file: PENDING → DOWNLOADING → DOWNLOADED → PROCESSING → PROCESSED | FAILED | SKIPPED
TaskTracker — per-queue-task (Jan 2026): QUEUED → PROCESSING → COMPLETED | FAILED; retry-chain and stuck-task detection

Quick Start

1. Environment Configuration (`environment.env`)

# SatLoc API — customer credentials are per-customer in PartnerSystemUser records
SATLOC_API_ENDPOINT=https://www.satloccloudfc.com/api/Satloc
SATLOC_API_TIMEOUT=30000
SATLOC_RETRY_ATTEMPTS=3
SATLOC_RETRY_DELAY=10000
SATLOC_RATE_LIMIT=60
SATLOC_BURST_LIMIT=10
SATLOC_STORAGE_PATH=/path/to/uploads/satloc
SATLOC_REALTIME_ENABLED=true
SATLOC_FILE_UPLOAD_ENABLED=true
SATLOC_1ST_ASSIGNMENT_ALWAYS_MATCH=true

# Worker control
PARTNER_POLLING_ENABLED=true
PARTNER_SYNC_ENABLED=true
PARTNER_MAX_RETRIES=5
STUCK_TIMEOUT_MS=18*60*1000

2. Run Setup Script

node setup_partners.js

Creates: SatLoc partner organization, sample customer, and a PartnerSystemUser linking them.

3. Start Workers

node start_workers.js
# or individually:
node workers/partner_data_polling_worker.js
node workers/partner_sync_worker.js

API Reference

All endpoints at /api/partners (authentication required).

Partner Organization CRUD

Method	Path	Description
`GET`	`/api/partners`	List all partners
`POST`	`/api/partners`	Create partner
`GET`	`/api/partners/:id`	Get partner details
`PUT`	`/api/partners/:id`	Update partner
`DELETE`	`/api/partners/:id`	Soft-delete (`active: false`)

Partner System User CRUD

Method	Path	Description
`GET`	`/api/partners/systemUsers`	List all system users
`POST`	`/api/partners/systemUsers`	Create system user
`POST`	`/api/partners/systemUsers/testAuth`	Test credentials against partner API
`GET`	`/api/partners/systemUsers/:id`	Get system user
`PUT`	`/api/partners/systemUsers/:id`	Update (partial updates supported)
`DELETE`	`/api/partners/systemUsers/:id`	Soft-delete

Create body:

{
  "partnerId": "<partner_org_id>",
  "customerId": "<agmission_customer_id>",
  "username": "customer_satloc_username",
  "password": "customer_satloc_password",
  "name": "Customer SatLoc Account",
  "active": true
}

Other Partner Routes

Method	Path	Description
`GET`	`/api/partners/customers`	List customers for a partner (with subscription info). Query: `?partnerId=<id>`
`GET`	`/api/partners/aircraft`	Get aircraft list from partner API
`POST`	`/api/partners/uploadJob`	Manually trigger job upload to partner
`POST`	`/api/partners/syncData`	Trigger partner data sync

Global DLQ API (for `partner_tasks`)

GET  /api/dlq/partner_tasks/list
POST /api/dlq/partner_tasks/retryAll
POST /api/dlq/partner_tasks/retryByPosition  # body: {"position":0}
POST /api/dlq/partner_tasks/retryByHeader    # body: {"headerName":"x-partner-code","headerValue":"SATLOC"}
POST /api/dlq/partner_tasks/purge

Assign Jobs with Partner Integration

Use internal user IDs — partner detection is automatic:

POST /api/jobs/assign
{
  "jobId": "job_id",
  "dlOp": { "type": 1 },
  "asUsers": [
    {
      "uid": "<internal_pilot_user_id>",
      "notes": "High priority mission"
    }
  ]
}

Flow:

Assignment created in controllers/job.js
Partner integration detected from assigned user's context
Immediate upload attempted — if partner API is live, calls partnerSyncService.uploadJobToPartner() directly
If immediate upload fails or API is offline → queues UPLOAD_PARTNER_JOB task to partner_tasks queue
partner_sync_worker processes task → calls uploadJobToPartner() → stores extJobId on JobAssign, status set to UPLOADED
Polling worker (cron) finds JobAssign records with status = UPLOADED → polls partner for aircraft logs
New logs downloaded → PROCESS_PARTNER_LOG task queued → partner_sync_worker processes → ApplicationDetail records created

Partner Data Flow

Polling Worker (cron — every 15 min prod / 1 min dev)
  ├─ Query JobAssign WHERE status = UPLOADED (jobs successfully sent to partner)
  ├─ Group by partnerCode + customerId
  ├─ Per aircraft: call partnerService.getAircraftLogs(customerId, aircraftId)
  ├─ Filter out already-processed logs (PartnerLogTracker where processed = true)
  ├─ For each new log:
  │     ├─ Atomic upsert PartnerLogTracker (PENDING)
  │     ├─ Claim for download → status DOWNLOADING
  │     ├─ partnerService.getAircraftLogData() → save to SATLOC_STORAGE_PATH
  │     ├─ Update PartnerLogTracker → DOWNLOADED
  │     └─ Enqueue PROCESS_PARTNER_LOG → partner_tasks queue
  └─ Periodic cleanup of stuck DOWNLOADING / DOWNLOADED / PROCESSING trackers

Sync Worker (queue consumer — partner_tasks)
  ├─ UPLOAD_PARTNER_JOB
  │     ├─ Check idempotency (redelivered check via extJobId on JobAssign)
  │     ├─ partnerSyncService.uploadJobToPartner(assignId)
  │     ├─ POST to SatLoc UploadJobData
  │     ├─ Store extJobId on JobAssign
  │     └─ Set JobAssign.status = UPLOADED (AssignStatus.UPLOADED = 2)
  └─ PROCESS_PARTNER_LOG
        ├─ TaskTracker idempotency check (claim or skip if already processing)
        ├─ Circuit breaker: skip repeatedly-failing files
        ├─ Atomic claim PartnerLogTracker → PROCESSING
        ├─ SatLocLogParser → SatLocApplicationProcessor
        ├─ Creates ApplicationFile + ApplicationDetail records
        ├─ PartnerLogTracker → PROCESSED
        └─ On failure → nack → DLQ partner_tasks_failed

See docs/PARTNER_TASK_DATA_FLOW_ANALYSIS.md for full Mermaid diagrams.

File Structure

Core Partner Files

server/
├── model/
│   ├── partner.js                           # Partner + PartnerSystemUser discriminators
│   ├── partner_log_tracker.js               # Per-log-file lifecycle tracking
│   └── task_tracker.js                      # Per-queue-task tracking (Phase 2, Jan 2026)
├── controllers/partner.js                   # RESTful CRUD + action endpoints
├── routes/partner.js                        # Mounted at /api/partners
├── helpers/
│   ├── partner_config.js                    # Partner configuration (SATLOC, AGIDRONEX stub)
│   ├── partner_service_factory.js           # ★ Singleton factory (used by all workers)
│   ├── satloc_log_parser.js                 # Binary .LOG file parser
│   ├── satloc_application_processor.js      # Parsed records → ApplicationDetail
│   ├── file_satlog.js                       # Low-level binary read utilities
│   └── satloc_util.js                       # SatLoc utilities
├── services/
│   ├── base_partner_service.js              # Abstract base class
│   ├── satloc_service.js                    # SatLoc API client + Redis auth cache
│   ├── partner_sync_service.js              # Orchestrates upload/sync flows
│   └── task_id_generator.js                # Task/execution ID generation
├── workers/
│   ├── partner_data_polling_worker.js       # Cron: polls APIs, downloads, enqueues
│   ├── partner_sync_worker.js               # Consumer: processes partner_tasks queue
│   ├── dlq_archival_worker.js               # Archives expired DLQ messages to disk
│   └── dlq_alert_worker.js                  # Threshold-based email alerts
├── controllers/dlq.js                       # Global DLQ controller (all queues)
├── routes/dlq.js                            # Mounted at /api/dlq/:queueName/*
├── public/dlq-monitor.html                  # ★ Web DLQ monitoring dashboard
└── setup_partners.js                        # Dev setup script

Key Milestones

Date	Change
Jul 2025	RESTful API standardization — consistent `:id` params, soft deletes, controller reuse
Aug 2025	Binary processing architecture — `SatLocBinaryProcessor`, file download + local storage
Oct 2025	Auth refactoring — `authenticate()` vs `authenticateAndCache()` separation; Redis cache
Dec 2025	DLQ Step 8 — queue-native global `/api/dlq/:queueName/*` endpoints; web dashboard
Jan 2026	TaskTracker (Phase 2) — per-task lifecycle tracking with retry chain and stuck-task detection

Environment Variables

Variable	Default	Description
`SATLOC_API_ENDPOINT`	`https://www.satloccloudfc.com/api/Satloc`	SatLoc Cloud base URL
`SATLOC_API_TIMEOUT`	`30000`	Request timeout (ms)
`SATLOC_RETRY_ATTEMPTS`	`3`	Max retry attempts
`SATLOC_RETRY_DELAY`	`10000`	Delay between retries (ms)
`SATLOC_RATE_LIMIT`	`60`	Requests per minute
`SATLOC_BURST_LIMIT`	`10`	Burst request limit
`SATLOC_STORAGE_PATH`	`/uploads/satloc`	Local log file storage
`SATLOC_MAX_FILE_SIZE`	`10485760`	Max log file size (bytes)
`SATLOC_REALTIME_ENABLED`	`true`	Enable real-time polling
`SATLOC_FILE_UPLOAD_ENABLED`	`true`	Enable file download/storage
`SATLOC_1ST_ASSIGNMENT_ALWAYS_MATCH`	`true`	Match first assignment when no extJobId
`PARTNER_POLLING_ENABLED`	`true`	Enable polling worker
`PARTNER_SYNC_ENABLED`	`true`	Enable sync worker
`PARTNER_MAX_RETRIES`	`5`	Max retries for stuck tasks
`STUCK_TIMEOUT_MS`	`18601000`	Stuck-task detection threshold

Monitoring

DLQ Web Dashboard

http://localhost:4100/public/dlq-monitor.html

DLQ CLI

node scripts/publish_to_dlq.js --env ./environment.env --queue dev_partner_tasks --count 3
node scripts/test_dlq_e2e.js   --env ./environment.env --queue dev_partner_tasks

Debug Logging

# All partner + satloc modules:
DEBUG=agm:partner*,agm:satloc* node server.js
# Workers:
DEBUG=agm:* node workers/partner_data_polling_worker.js

Log Level Filtering (Pino)

See PINO_MODULE_FILTERING_GUIDE.md for filtering by module (e.g., LOG_MODULES=partner*,satloc*).

Benefits

For Development

Simplified Architecture: Reuses existing User model infrastructure
Environment Configuration: No complex partner management UI needed
Easy Testing: Standard User entity patterns
Maintainable: Fewer moving parts than separate partner models

For Operations

Customer Isolation: Each customer has their own partner system account
Secure Credentials: Environment-based credential management
Simple Monitoring: Basic health checks without complex infrastructure
Scalable: Easy to add new customers and partners

For Business

Partner Flexibility: Easy to add new partners with environment configuration
Cost Effective: No complex monitoring infrastructure required
Mixed Assignments: Support for both internal and partner aircraft in same job

Troubleshooting

Problem	Likely Cause	Solution
`partner_service_not_implemented`	Unregistered `partnerCode`	Check `helpers/partner_service_factory.js` `serviceMapping`
`No SatLoc system user found`	Missing `PartnerSystemUser` record	Run `setup_partners.js` or `POST /api/partners/systemUsers`
Auth fails after token expiry	Stale Redis cache	Service auto-retries once; check `REDIS_PWD` and Redis connectivity
Log files not processed	`PartnerLogTracker` stuck in `DOWNLOADING`	Startup cleanup resets stuck tasks automatically; tune `STUCK_TIMEOUT_MS`
Tasks accumulating in DLQ	Worker processing failures	`POST /api/dlq/partner_tasks/retryAll` or use web dashboard
Queue connection lost	RabbitMQ restart	Workers reconnect automatically; in-flight tasks cached in memory offline queue

Documentation Index

Document	Purpose	Status
`docs/PARTNER_INTEGRATION_ARCHITECTURE.md`	Full architecture, diagrams, current state	✅ Current
`docs/PARTNER_TASK_DATA_FLOW_ANALYSIS.md`	Queue/DLQ message lifecycle (Mermaid)	✅ Current (Jan 2026)
`docs/PARTNER_RESPONSIBILITIES_ANALYSIS.md`	Worker responsibility boundaries	✅ Current
`docs/PARTNER_AUTH_REFACTORING.md`	SatLoc auth separation + Redis cache design	✅ Current
`docs/PARTNER_MODEL_SCHEMA_UPDATES.md`	PartnerSystemUser + Customer schema relationships	✅ Current
`docs/PARTNER_LOG_FILE_PROCESSING.md`	Log processing pipeline	✅ Current
`docs/PARTNER_LOG_DOWNLOAD_IMPLEMENTATION.md`	Polling worker download implementation	✅ Current
`docs/PARTNER_INTEGRATION_IMPLEMENTATION.md`	Implementation patterns and examples	✅ Current
`docs/PARTNER_SYSTEM_REFACTORING_SUMMARY.md`	Historical: Jul 2025 RESTful refactor	📁 Historical
`docs/PARTNER_SYNC_INTEGRATION_SUMMARY.md`	Historical: Aug 2025 sync improvements	📁 Historical
`docs/PARTNER_LOG_MIGRATION_SUMMARY.md`	Historical: log tracker schema migration	📁 Historical
`docs/PARTNER_SYNC_WORKER_REFACTORING.md`	Historical: worker refactoring milestone	📁 Historical
`docs/SATLOC_API_SPECIFICATION.md`	SatLoc external API endpoint reference	✅ Reference
`docs/SATLOC_API_ACTUAL_BEHAVIOR.md`	Documented deviations from spec (important!)	✅ Reference
`docs/SATLOC_BINARY_PROCESSING_ARCHITECTURE.md`	Binary log parsing architecture	✅ Reference
`docs/SATLOC_APPLICATION_PROCESSOR_README.md`	`satloc_application_processor.js` usage	✅ Reference
`docs/SATLOC_ERROR_PATTERNS.md`	Known error patterns and handling	✅ Reference
`docs/SATLOC_LOG_NOTES.md`	Raw log format notes	✅ Reference
`docs/Transland_SATLOC_Log_File_Formats_v3_76.md`	Official SatLoc log format spec	✅ Reference
`docs/SATLOC_COMPLETE_IMPLEMENTATION.md`	Historical: implementation milestone	📁 Historical
`docs/SATLOC_IMPLEMENTATION_SUMMARY.md`	Historical: implementation summary	📁 Historical
`docs/SATLOC_INTEGRATION_SUMMARY.md`	Historical: integration summary	📁 Historical
`docs/SATLOC_TESTING_SUMMARY.md`	Historical: testing milestone	📁 Historical

17 KiB Raw Blame History