agmission/server/docs/DATA_EXPORT_API_DESIGN.md

Data Export API — Design & Implementation Guide

Single Source of Truth: This is the canonical document for Data Export API design, implementation status, and next steps in this branch.

Branch: data-export-api
Date: April 10, 2026
Status: Phase A complete — Phase B in progress

Change Log

Date	Update
2026-04-14	Added ApiKeyServices and ExportUnits frozen constants to helpers/constants.js; wired throughout models and controllers. Added service field to ApiKey, units field to ExportJob, US unit conversion support to async export.
2026-04-10	Marked this file as the single source of truth for Data Export API design and implementation tracking.
2026-04-10	Consolidated documentation into this file and removed duplicate summary document.

---

1. Overview

The Data Export API allows authorised external systems (data warehouses, Power BI, ArcGIS) to pull mission data from AgMission on demand or on a scheduled basis. It exposes the same data already shown in the web application's Data Playback screen, served through a versioned REST API authenticated with API keys.

Two functional areas:

1. REST API (/api/v1/) — session summaries, per-point GPS trace, spray-area polygons, async bulk export
2. UI enhancement — improved Job List filter controls (order number, date range) and an API Key management screen in the web app settings

---

2. Architecture

2.1 Request Flow & Authentication Architecture

sequenceDiagram
    participant External as External System
    participant API as Express Server
    participant Auth as checkApiKey Middleware
    participant DB as ApiKey DB
    participant Handler as Route Handler
    
    External->>API: GET /api/v1/jobs/:id/sessions
    External->>API: Header X-API-Key
    API->>Auth: req.headers x-api-key
    Auth->>Auth: Extract prefix first 8 chars
    Auth->>DB: Find by prefix active=true
    DB-->>Auth: ApiKey candidates
    Auth->>Auth: bcrypt.compare plainKey vs keyHash
    Auth->>Auth: On match set req.uid
    Auth->>Handler: next with req.uid set
    Handler->>Handler: ownerJob verify req.uid
    Handler-->>External: JSON response

Web UI caller (JWT-authenticated, unchanged):

graph LR
    A[Web App] -->|Bearer token| B[checkUser Middleware]
    B -->|Verify JWT| C[req.uid set]
    C -->|api/keys routes| D[Key Management]
    D -->|CRUD ops| E[ApiKey Model]

2.2 Data Model Hierarchy

graph TD
    Job[Job model]
    App[App - Session data]
    AppFile[AppFile - Metadata]
    AppDetail[AppDetail - GPS points]
    ExportJob[ExportJob - Export tracker]
    ApiKey[ApiKey - Authentication]
    
    Job -->|has many| App
    App -->|has many| AppFile
    AppFile -->|has many| AppDetail
    Job -.->|triggers| ExportJob
    Job -.->|auth via| ApiKey
    App -.->|derived from| AppDetail
    style Job fill:#e1f5ff
    style App fill:#f3e5f5
    style AppFile fill:#fff3e0
    style AppDetail fill:#fce4ec
    style ExportJob fill:#e8f5e9
    style ApiKey fill:#f1f8e9

Fields summary:

• Job: jobId, byPuid, rptOp, weatherInfo, sprayAreas
• App: avgSpraySpeed, totalSprayed, totalSprayTime, totalFlightTime
• AppFile: meta (operator, appRate, fcName, sprOnLag), totalSprayed, totalSprayTime
• AppDetail: gpsTime, lat, lon, grSpeed, lminApp, swath, sprayStat, windSpd, temp, humid
• ExportJob: owner, jobId, format, status, filePath, expiresAt
• ApiKey: owner, keyHash, prefix, active, lastUsedAt

2.3 Route Prefix Strategy

Prefix	Auth	Purpose
/api/v1/	X-API-Key header (new checkApiKey)	Public data export endpoints
/api/keys	Authorization: Bearer (existing checkUser)	Key management for web UI
All other /api/...	Authorization: Bearer (existing checkUser)	Existing application routes — unchanged

The /api/v1/ path is added to the checkUser bypass whitelist (in isSecuredRoute()) so the existing JWT middleware skips these routes.

2.4 Async Export Lifecycle

stateDiagram-v2
    [*] --> pending: POST /export
    pending --> processing: async generate
    processing --> ready: success
    processing --> error: fail I/O error
    ready --> pending: download cleanup
    error --> [*]: TTL expiry
    ready --> [*]: 24h TTL
    pending --> [*]: TTL index

Lifecycle details:

• pending: ExportJob created and returned to caller; caller polls GET /exports/:id
• processing: Streams AppDetail cursor to CSV or JSON format (memory-efficient)
• ready: File written to disk at filePath, TTL (expiresAt) set and ready for download
• error: Error message recorded, awaits manual retry via queue or TTL cleanup
• Cleanup: After file download completes, filePath cleared and status reset to pending for potential re-download

---

3. New Files

Backend

File	Status	Purpose
model/api_key.js	✅ Done	ApiKey Mongoose model
model/export_job.js	✅ Done	ExportJob tracking model
middlewares/app_validator.js	✅ Done	Added checkApiKey function + whitelist entry
routes/api_pub.js	✅ Done	/api/v1/ route definitions
routes/api_keys.js	✅ Done	/api/keys route definitions
routes/index.js	✅ Done	Registers api_pub and api_keys
controllers/api_key.js	✅ Done	createKey, listKeys, revokeKey
controllers/api_pub.js	✅ Done	getSessions, getSessionRecords, getAreas
controllers/api_export.js	✅ Done	triggerExport, getExportStatus, downloadExport
scripts/migrate_avg_spray_speed.js	✅ Done	One-time back-fill for existing jobs

Modified Files (existing)

File	Change
model/application.js	Added avgSpraySpeed: Number field
workers/job_worker.js	Accumulates avgSpraySpeed during file import at lines ~528, ~944–1090, ~1309–1381

Frontend (pending)

File	Status	Purpose
job-list.component.ts/.html	⬜ Pending	Add orderNumber filter input
src/app/settings/api-keys/	⬜ Pending	API Key management feature module

---

4. Model Designs

4.1 ApiKey (model/api_key.js)

Field	Type	Notes
owner	ObjectId → User	The applicator this key authorises
label	String	Human-readable name (max 100 chars)
prefix	String	First 8 chars of plain key — stored clear-text for O(1) candidate lookup
keyHash	String	bcryptjs hash of the full plain key — plain key never stored
service	ApiKeyServices	Which service the key grants access to: 'data_export' (default) or 'partner_api'
active	Boolean	Revoke by setting false
managedBy	'owner' | 'admin'	Who created the key
createdAt	Date
lastUsedAt	Date	Updated async (fire-and-forget) — no added request latency

Key lookup flow: prefix → find candidates → bcrypt.compare(incomingKey, candidate.keyHash) → match → set req.uid = key.owner.

Limit: 10 active keys per owner (enforced in createKey).

4.2 ExportJob (model/export_job.js)

Field	Type	Notes
owner	ObjectId → User	Scoped to requesting applicator
jobId	Number	AgMission job ID
format	'csv' | 'json'	Requested output format
interval	Number | null	GPS point thinning in seconds; null = all points
units	ExportUnits	Output measurement system: 'metric' (default) or 'us'
status	'pending' | 'processing' | 'ready' | 'error'	Lifecycle state
filePath	String	Absolute path on disk (set when ready)
errorMsg	String	Populated on error
createdAt	Date
expiresAt	Date	MongoDB TTL index — document auto-deleted after expiry

Files are written to env.TEMP_DIR. TTL defaults to 24 hours (EXPORT_TTL_HOURS env var).

---

5. API Endpoint Reference

5.1 Authentication

All /api/v1/ requests require:

X-API-Key: <full 64-char hex key>

No Authorization header needed. On failure the middleware returns 401.

---

5.2 GET /api/v1/jobs/:jobId/sessions

Returns one summary record per uploaded application file ("session") for the job.

Response shape:

{
  "jobId": 12345,
  "clientId": "664f1a...",
  "clientName": "Fazenda São Paulo Ltda",
  "assignedPilotId": "664f1b...",
  "assignedPilotName": "Carlos Mendes",
  "assignedAircraftId": "664f1a...",
  "assignedAircraftName": "Agrinova 01",
  "assignedAircraftTailNumber": "PR-XYZ",
  "planAircraftName": "Agrinova 01",
  "planAircraftTailNumber": "PR-XYZ",
  "assignedDate": "2025-07-13T18:00:00Z",
  "mappedArea_ha": 50.0,
  "reportConfirmed": true,
  "areaSize_ha": 50.0,
  "coverage_ha": 48.3,
  "overSprayed_pct": -3.40,
  "appRate": 2.5,
  "appRateUnit": "lit/ha",
  "appRateConfirmed": 2.5,
  "sprayVolume": 120.75,
  "volumeUnit": "lit",
  "useConfirmedVolume": false,
  "actualSprayVolume": 118.42,
  "confirmedActualVolume": 120.75,
  "effectiveVolume": 120.75,
  "useCustomWeather": false,
  "weather": null,
  "data": [
    {
      "sessionId": "...",
      "fileName": "2507140724SatlocG4.log",
      "startDateTime": "2025-07-14T10:24:00Z",
      "endDateTime": "2025-07-14T11:05:42Z",
      "totalFlightTime_s": 2462,
      "totalSprayTime_s": 1840,
      "totalTurnTime_s": 622,
      "totalSprayed_ha": 48.3,
      "totalSprayMat": 120.5,
      "totalSprayMatUnit": "lit",
      "avgSpraySpeed_ms": 14.2,
      "sprayZoneName": "Field A North",
      "sprayZoneArea_ha": 25.0,
      "appRate": 2.5,
      "appRateUnit": "lit/ha",
      "flowController": "SatLoc G4",
      "sprayOnLag_s": 0.2,
      "sprayOffLag_s": 0.15,
      "pulsesPerLiter": 1800,
      "files": [{ "fileId": "...", "name": "2507140724SatlocG4.log" }],
      "sessionPilotName": "João Silva"
    }
  ]
}

Output field definitions (sessions endpoint)

Response envelope fields:

Field	Type	Required	Description
jobId	number	✓	Numeric job identifier from the URL path.
clientId	string | null	—	Client account ObjectId (the applicator's customer this job was performed for).
clientName	string | null	—	Client account name.
assignedPilotId	string | null	—	Assigned pilot ObjectId from the job operator relation.
assignedPilotName	string | null	—	Assigned pilot name from the job operator relation.
assignedAircraftId	string | null	—	Assigned aircraft ObjectId from latest live JobAssign when assignment user is DEVICE; otherwise null.
assignedAircraftName	string | null	—	Assigned aircraft display name from latest live JobAssign when assignment user is DEVICE; otherwise null.
assignedAircraftTailNumber	string | null	—	Assigned aircraft tail number from latest live JobAssign when assignment user is DEVICE; otherwise null.
planAircraftName	string | null	—	Planned aircraft name from Job.vehicle.name. Always from the job plan, regardless of live assignment.
planAircraftTailNumber	string | null	—	Planned aircraft tail number from Job.vehicle.tailNumber. Always from the job plan.
assignedDate	string | null	—	Latest job assignment timestamp (ISO 8601 UTC).
mappedArea_ha	number | null	—	Job mapped area in hectares from Job.rptOp.areaSize, falling back to Job.ttSprArea. 2 dp.
reportConfirmed	boolean	✓	True when report settings are confirmed (rptOp.coverage != null).
areaSize_ha	number | null	—	Confirmed area size, or fallback mapped area when not confirmed. 2 dp.
coverage_ha	number | null	—	Confirmed coverage, or fallback total sprayed area across sessions. 2 dp.
overSprayed_pct	number | null	—	(coverage_ha − areaSize_ha) / areaSize_ha × 100. 2 dp.
appRate	number | null	—	Confirmed app rate, or first-session fallback app rate.
appRateUnit	string | null	—	App rate unit label from job setting.
appRateConfirmed	number | null	—	Confirmed app rate only; null when not confirmed.
sprayVolume	number | null	—	Planned/estimated spray volume: coverage_ha × appRate, converted by Job.measureUnit. 3 dp.
volumeUnit	string | null	—	Volume unit derived from Job.measureUnit and material type: "lit" / "gal" for liquid, "kg" / "lb" for solid (dry).
useConfirmedVolume	boolean	✓	True when applicator selected confirmed actual volume override in Report Settings.
actualSprayVolume	number | null	—	Actual spray volume calculated from applications: SUM(App.totalSprayMat) normalized to metric base, then converted by Job.measureUnit. 3 dp.
confirmedActualVolume	number | null	—	Confirmed actual spray volume from rptOp.actualVol (stored in metric base: L/Kg), converted by Job.measureUnit. 3 dp.
effectiveVolume	number | null	—	Authoritative volume: confirmedActualVolume when useConfirmedVolume=true; otherwise actualSprayVolume. 3 dp.
useCustomWeather	boolean	✓	True when custom weather was manually entered.
weather	object | null	—	Weather block when custom weather exists; otherwise null.
data	array	✓	Array of per-session summary records.

Per-session fields in data[]:

| Field | Type | Required | Description | |---|---|---| | sessionId | string | ✓ | Session identifier (App._id). | | fileName | string | null | — | Session file name from App.fileName. | | startDateTime | string | null | — | Session start datetime (ISO 8601 UTC). | | endDateTime | string | null | — | Session end datetime (ISO 8601 UTC). | | totalFlightTime_s | number | null | — | Total flight time in seconds. 3 dp. | | totalSprayTime_s | number | null | — | Total spray time in seconds. 3 dp. | | totalTurnTime_s | number | null | — | Total turn time in seconds. 3 dp. | | totalSprayed_ha | number | null | — | Total sprayed area in hectares. 2 dp. | | totalSprayMat | number | null | — | Total sprayed material amount. 3 dp. | | totalSprayMatUnit | string | null | — | Spray material unit label (e.g. "lit", "kg") — decoded from raw code via rateUnitString(). | | avgSpraySpeed_ms | number | null | — | Average spray speed in m/s. 2 dp. | | sprayZoneName | string | null | — | Zone/area name from AppFile.meta.areaOrZone. | | sprayZoneArea_ha | number | null | — | Zone area in hectares from AppFile.meta.sprCoverage[1]. 2 dp. | | appRate | number | null | — | Session target app rate from file metadata. | | appRateUnit | string | null | — | App rate unit label from job setting (canonical, matches top-level). | | flowController | string | — | Flow controller name from file metadata. 'No FC' when absent or when the value is 'none' (case-insensitive), matching the playback display. | | sprayOnLag_s | number | null | — | Spray-on lag in seconds. | | sprayOffLag_s | number | null | — | Spray-off lag in seconds. | | pulsesPerLiter | number | null | — | Pulses-per-liter. | | files | array | ✓ | Session file list: [{ fileId, name }]. | | sessionPilotName | string | null | — | Pilot name recorded inside the imported data file. May differ from the job-assigned pilot. |

reportConfirmed Fallback Logic Diagram:

flowchart TD
    A{Is rptOp.coverage<br/>defined?}
    A -->|Yes| B["reportConfirmed=true"]
    A -->|No| C["reportConfirmed=false"]
    
    B --> D["Use Report Settings<br/>values"]
    C --> E["Compute from raw<br/>data"]
    
    D --> F{useActualVol?}
    E --> G{useActualVol?}
    
    F -->|Yes| H["effective=actual"]
    F -->|No| I["effective=coverage*rate"]
    
    G -->|Yes| J["effective=computed"]
    G -->|No| K["effective=computed"]
    
    H --> L["Confirmed block"]
    I --> L
    J --> M["Fallback block"]
    K --> M

Field	reportConfirmed: true	reportConfirmed: false
areaSize_ha	Job.rptOp.areaSize	Job.ttSprArea
coverage_ha	Job.rptOp.coverage	Sum of App.totalSprayed
appRate	Job.rptOp.appRate	AppFile.meta.appRate (first session)
sprayVolume	planned coverage × appRate	same formula using fallback coverage/appRate values
effectiveVolume	actualVol if useActualVol, else calculated from applications	calculated from applications
weather fields	Job.weatherInfo.* when useCustWI=true	omitted

When reportConfirmed: false, re-fetch this record after the applicator confirms in Report Settings.

---

5.3 GET /api/v1/jobs/:jobId/sessions/:fileId/records

Per-point GPS trace records, cursor-paginated. Uses the same paginateWithCursor helper as the existing filesdata_post.

Query parameters:

Param	Default	Description
after	—	Cursor (_id of last record received) — preferred by customer requirements
startingAfter	—	Cursor (_id of last record received)
limit	500	Max records per page (hard cap: 2000)
interval	—	Return one record per N seconds of GPS time (e.g. 1, 5, 10). Records where sprayStat changes are always kept.
interval	null or 0	Set interval=0 (or omit it) to disable interval thinning for full-fidelity results.
fm	false	Set fm=true to include Flight Master/AgDisp FM fields (see below). Off by default — only for customers with FM-enabled equipment.

Field groups per record:

GPS Data: timeUtc, lat, lon, utmX, utmY, alt, grSpeed, heading, xTrack, lockedLine, hdop, satsIn, tslu, calcodeFreq, sprayStat

Application Info: flowRateApplied, flowRateRequired, appRateRequired, appRateApplied*, swathWidth, boomPressure_psi, sprayOnLag_s†, sprayOffLag_s†, pulsesPerLiter†, rpm[]

MET: windSpeed_kt, windDir_deg, temp_c, humidity_pct

Compatibility aliases returned by implementation for existing consumers:

• None — aliases were removed; this is a new API with no existing consumers.

Output field definitions (records endpoint)

Response envelope fields:

Field	Type	Required	Description
data	array	✓	Array of per-point records after pagination and optional interval thinning.
hasMore	boolean	✓	True when additional pages exist.
startingAfter	string | undefined	—	Last record _id — pass as ?startingAfter= to fetch the next page. Present whenever data is non-empty.
endingBefore	string | undefined	—	First record _id — pass as ?endingBefore= to fetch the previous page. Present whenever data is non-empty.

Per-record fields in data[]:

Field	Type	Required	Description
timeUtc	string | null	—	GPS timestamp formatted as ISO 8601 UTC.
gpsTime	number | null	—	Raw GPS epoch seconds.
lat	number | null	—	Latitude (WGS84 decimal degrees).
lon	number | null	—	Longitude (WGS84 decimal degrees).
utmX	number | null	—	UTM X coordinate in meters.
utmY	number | null	—	UTM Y coordinate in meters.
alt	number | null	—	Altitude in meters.
grSpeed	number | null	—	Ground speed in m/s.
heading	number | null	—	Aircraft heading in degrees.
xTrack	number | null	—	Cross-track error in meters.
lockedLine	number | null	—	Locked line index from guidance data.
hdop	number | null	—	Horizontal dilution of precision.
satsIn	number | null	—	Raw satellite/inside-area composite from AgNav native NT binary. Encoding is satellite count with inside offset: 0..99 = outside area (satellites = value), 100..199 = inside area (satellites = value - 100).
tslu	number | null	—	Raw "time since last update" in seconds for GPS differential correction.
calcodeFreq	number | null	—	Raw calibration/frequency field. 30000-60000 indicates frequency/RPM (true RPM = value - 30000). Also used for spray offset in decimeter: <20000 positive offset; >60000 negative offset where stored value is 65536 - abs(offset).
sprayStat	number | null	—	Spray state from source data (returned as-is). 0 = OFF. 1 = ON, inside area. 3 = ON, first point of new spray line (start-of-line marker; boom IS open). 10 = ON, outside area. Any non-zero value = boom open.
flowRateApplied	number | null	—	Applied flow rate (L/min).
flowRateRequired	number | null	—	Required flow rate (L/min).
appRateRequired	number | null	—	Required app rate from source data.
appRateApplied	number | null	—	Playback-aligned app rate applied; null when spray is off.
swathWidth	number | null	—	Swath width in meters.
boomPressure_psi	number | null	—	Boom pressure in PSI.
sprayOnLag_s	number | null	—	Session constant, repeated per record.
sprayOffLag_s	number | null	—	Session constant, repeated per record.
pulsesPerLiter	number | null	—	Session constant.
rpm	array | null	—	RPM array from raw data.
windSpeed_kt	number | null	—	Wind speed in knots (converted from m/s on output to match playback display).
windDir_deg	number | null	—	Wind direction in degrees.
temp_c	number | null	—	Temperature in Celsius.
humidity_pct	number | null	—	Relative humidity percentage.

Interval thinning rule: keep first record in window, then keep records at least interval seconds after last kept record; always keep records where sprayStat changes. Use interval=0 (or omit interval) to bypass thinning.
† Session constants from AppFile.meta — same value repeated on every record for flat-file consumers.

FM fields (included only when ?fm=true is set):

Field	Type	DB source	Description
sprayHeight_m	number | null	sprayHeight	Target spray height in metres (AgDisp).
driftX_m	number | null	driftX	Lateral drift offset X in metres (AgDisp).
driftY_m	number | null	driftY	Lateral drift offset Y in metres (AgDisp).
depositX_m	number | null	depositX	Deposit offset X in metres (AgDisp).
depositY_m	number | null	depositY	Deposit offset Y in metres (AgDisp).
radarAlt_m	number | null	radarAlt	Radar altimeter reading in metres.
laserAlt_m	number | null	raserAlt ¹	Laser altimeter reading in metres.

¹ The source DB field is named raserAlt (schema typo). The API exposes it as laserAlt_m with the correct name.

Record Decoding Transformation Pipeline:

graph LR
    A[Raw AppDetail] --> B[Interval Thinning]
    B --> C[Decode GPS Fields]
    C --> D[Compute appRateApplied]
    D --> E[Inject Session Meta]
    E --> F[Format ISO 8601 UTC]
    F --> G[Return API Record]
    
    style A fill:#fce4ec
    style B fill:#f3e5f5
    style C fill:#e8eaf6
    style D fill:#f3e5f5
    style E fill:#e0f2f1
    style F fill:#fff9c4
    style G fill:#c8e6c9

Raw quality-field semantics:

• satsIn: AgNav native NT binary encoding uses inside offset: inside = (value >= 100), satellites = inside ? value - 100 : value
• tslu: time since last update in seconds for GPS differential correction
• calcodeFreq: 30000-60000 indicates frequency/RPM (true RPM = calcodeFreq - 30000); also used for spray offset in decimeter (<20000 positive, >60000 negative with stored value 65536 - abs(offset))
• sprayStat values are returned as stored in source data (no filtering): 0=OFF, 1=ON inside, 3=ON first-point-of-line, 10=ON outside
• appRateApplied = lminApp / (grSpeed × swath) × 10000; null when grSpeed or swath = 0

---

5.4 Public API Endpoint Architecture

graph LR
    subgraph External[External Callers]
        PBI[Power BI]
        ARCGIS[ArcGIS]
        DW[Data Warehouse]
    end
    
    subgraph PublicAPI[Public API /api/v1]
        SESSIONS[GET /sessions]
        RECORDS[GET /records]
        AREAS[GET /areas]
        TRIGEXP[POST /export]
        POLLEXP[GET /export-status]
        DOWNLOAD[GET /download]
    end
    
    subgraph Internal[Backend Models]
        APP[(App)]
        APPFILE[(AppFile)]
        APPDETAIL[(AppDetail)]
        EXPORTJOB[(ExportJob)]
    end
    
    PBI --> SESSIONS
    ARCGIS --> AREAS
    DW --> DOWNLOAD
    
    SESSIONS --> APP
    RECORDS --> APPDETAIL
    AREAS --> APP
    TRIGEXP --> EXPORTJOB
    POLLEXP --> EXPORTJOB
    DOWNLOAD --> EXPORTJOB
    
    SESSIONS --> APPFILE
    RECORDS --> APPFILE
    
    style External fill:#e3f2fd
    style PublicAPI fill:#f3e5f5
    style Internal fill:#e8f5e9

5.5 GET /api/v1/jobs/:jobId/areas

Returns the planned spray-area polygons as a GeoJSON FeatureCollection.

Output field definitions (areas endpoint)

Field	Type	Required	Description
type	string	✓	Always FeatureCollection.
jobId	number	✓	Numeric job identifier from path.
features	array	✓	Array of polygon features from planned spray areas.

Per-feature fields in features[]:

Field	Type	Required	Description
type	string	✓	Always Feature.
properties.name	string | null	—	Spray area name.
properties.appRate	number | null	—	Planned app rate for the area.
properties.area_ha	number | null	—	Planned area size in hectares.
properties.type	string | null	—	Area type metadata when present.
geometry	object | null	—	GeoJSON polygon geometry copied from job.sprayAreas.

Only implement / expose once customer confirms this is needed for ArcGIS layer import (pending).

---

5.6 Async Export

Trigger:

POST /api/v1/jobs/:jobId/export
Body: { "format": "csv", "interval": 1, "units": "us" }
→ 202 { "exportId": "...", "status": "pending", "units": "us" }

Body parameters:

Parameter	Required	Values	Default
format	Yes	'csv', 'json'	—
interval	No	seconds (e.g. 1, 5)	null (all points)
units	No	'metric' (ExportUnits.METRIC), 'us' (ExportUnits.US)	'metric'
fm	No	true / false	false — include Flight Master/AgDisp FM fields

Bulk export interval behavior:

• Records are read in stable _id ascending order per file.
• With interval set, records are thinned by GPS time window.
• Records where sprayStat changes are always included (not thinned out).
• Thinning is applied per file stream (not across a global merged timeline).
• For bulk export, omit interval (or set interval=0) to export all points.

Poll:

GET /api/v1/exports/:exportId
→ { "status": "processing" }   (repeat)
→ { "status": "ready", "downloadUrl": "/api/v1/exports/:id/download" }
→ { "status": "error", "errorMsg": "..." }

Download:

GET /api/v1/exports/:exportId/download
→ streams file with Content-Disposition: attachment

Output field definitions (export endpoints)

POST /api/v1/jobs/:jobId/export response fields (HTTP 202):

Field	Type	Required	Description
exportId	string	✓	Export tracker identifier.
status	string	✓	Initial export status (pending).
format	string	✓	Selected format (csv or json).
units	string	✓	Selected units (metric or us).
createdAt	string	✓	Export tracker creation timestamp (ISO 8601 UTC).

GET /api/v1/exports/:exportId response fields:

Field	Type	Required	Description
exportId	string	✓	Export tracker identifier.
status	string	✓	pending, processing, ready, or error.
format	string	✓	Export format.
units	string	✓	Export units mode.
createdAt	string	✓	Creation timestamp.
expiresAt	string | null	—	Expiry timestamp for downloaded file cleanup.
error	string | null	—	Error message when generation fails.
downloadUrl	string | undefined	—	Present only when status is ready.

GET /api/v1/exports/:exportId/download response:

Item	Value
Body	Streamed file content (CSV or JSON).
Content-Type	text/csv or application/geo+json.
Content-Disposition	Attachment filename with format extension.

CSV structure: one row per AppDetail record. All raw trace fields plus job/session header columns (jobId, orderNumber, jobName, clientId, clientName, sessionId, fileName, pilotName) repeated on every row — no joins required for Power BI or data warehouse import. Column headers include unit suffix when units='us' (e.g. groundSpeed_mph vs groundSpeed_ms, temp_f vs temp_c).

US unit conversions (units='us'):

Metric field	US field	Factor
alt_m	alt_ft	× 3.28084
groundSpeed_ms	groundSpeed_mph	× 2.23694
crossTrackError_m	crossTrackError_ft	× 3.28084
swathWidth_m	swathWidth_ft	× 3.28084
flowRateApplied_Lmin	flowRateApplied_galMin	× 0.264172
flowRateRequired_Lmin	flowRateRequired_galMin	× 0.264172
appRateRequired_Lha	appRateRequired_galAc	× 0.10694
appRateApplied_Lha	appRateApplied_galAc	× 0.10694
windSpeed_kt	windSpeed_mph	× 1.15078 (kt → mph)
temp_c	temp_f	× 9/5 + 32
boomPressure_psi	boomPressure_psi	already PSI — no conversion

Implementation: Node.js Transform stream over AppDetail cursor (sorted by _id: 1) → writes to env.TEMP_DIR. Keeps memory flat regardless of file size. interval thinning preserves spray-state transition points.

---

5.7 Key Management Endpoints (Web UI, JWT-authenticated)

Method	Path	Description
GET	/api/keys	List active keys for the signed-in applicator
POST	/api/keys	Create a key — returns full plain key once in the response
DELETE	/api/keys/:keyId	Revoke a key (sets active: false)

Key management body (POST /api/keys):

{ "label": "Power BI Prod", "service": "data_export" }

service is optional and defaults to 'data_export'. Valid values are defined in ApiKeyServices in helpers/constants.js.

Admin users may append ?ownerId=<ObjectId> or include ownerId in the POST body to manage keys for another account.

---

6. avgSpraySpeed — Storage Strategy

Rather than computing average spray speed on demand (which would require scanning all AppDetail records for every session summary request), it is computed once at import time and stored in App.avgSpraySpeed.

Accumulation logic in job_worker.js:

// Per GPS point during file parsing (in importDataFiles, per-file pass):
// sprayStat=3 is the start-of-line spray marker (boom IS open, but it records
// the position anchor for area calculation). Excluded from speed averaging
// because it may capture the low-speed moment of spray transition, which would
// skew the average spray speed downward.
if (record.sprayStat !== 3 && record.sprayStat > 0 && utils.isNumber(record.grSpeed)) {
  totalSpeedAcc += record.grSpeed;
  spraySpeedCount++;
}
// At end of file:
importInfo.avgSpraySpeed = spraySpeedCount > 0 ? totalSpeedAcc / spraySpeedCount : null; // m/s

One-time back-fill: scripts/migrate_avg_spray_speed.js — iterates existing App docs via cursor, re-scans their AppDetail records (sprayStat > 0, grSpeed !== 0) and bulk-writes the value. It targets apps with missing/null/zero avgSpraySpeed. Safe to run on production (cursor-based, low memory, progress logging every 100 docs).

---

7. Frontend Design

7.1 Job List Filter Enhancement (Step 3 — pending)

File: src/app/job/job-list/job-list.component.ts

Add an orderNumber text filter control to the existing filter bar alongside client, status, and date pickers. Wire into the existing Job.Fetch() NgRx action that calls jobService.loadJobs(). Minor backend check: ensure searchJobs_post / getJobs_get accepts orderNumber as a partial-match filter.

7.2 API Key Management UI (Step 8 — pending)

New lazy-loaded feature module following the same NgRx pattern as PartnerListComponent / ClientListComponent.

Structure:

src/app/settings/api-keys/
  api-keys.module.ts
  api-keys-routing.module.ts
  api-keys-list/
    api-keys-list.component.ts
    api-keys-list.component.html
  store/
    api-key.actions.ts
    api-key.reducer.ts
    api-key.effects.ts
  services/
    api-key.service.ts

UX flow:

1. PrimeNG p-table listing keys — columns: Label, Prefix, Created, Last Used, Status
2. "Generate Key" button → calls POST /api/keys → shows full key in a p-dialog with copy-to-clipboard — key masked after dialog is closed, never retrievable again
3. "Revoke" button per row → p-confirmDialog → calls DELETE /api/keys/:id
4. Admin view: additional applicator selector (p-dropdown) to manage keys on behalf of any account

---

8. Implementation Status

Step	Feature	Status	Notes
1	App.avgSpraySpeed — model field + import worker + migration script	✅ Done	model/application.js, workers/job_worker.js, scripts/migrate_avg_spray_speed.js
2	ApiKey model + checkApiKey middleware + /api/keys CRUD	✅ Done	model/api_key.js, middlewares/app_validator.js, routes/api_keys.js, controllers/api_key.js. ApiKeyServices frozen constant controls valid service values.
3	Job List UI filter enhancements	⬜ Pending	Frontend only — job-list.component
4	GET /api/v1/jobs/:id/sessions — session summary	✅ Done	controllers/api_pub.js getSessions
5	GET /api/v1/jobs/:id/sessions/:fid/records — raw trace	✅ Done	controllers/api_pub.js getSessionRecords
6	GET /api/v1/jobs/:id/areas — spray-area GeoJSON	✅ Done	controllers/api_pub.js getAreas — awaiting customer confirmation to expose
7	Async export (POST /export, GET /exports/:id, download)	✅ Done	model/export_job.js, controllers/api_export.js. ExportUnits frozen constant controls valid units values; US unit conversions applied at output time.
8	API Key management UI (Angular)	⬜ Pending	New settings/api-keys feature module
9	Sandbox seeding script	⬜ Pending	scripts/seed_sandbox.js
—	Tests	⬜ Pending	checkApiKey unit tests, session summary integration tests

---

9. Key Design Decisions

Decision	Rationale
Separate checkApiKey middleware (not extending checkUser)	Zero risk to existing JWT-protected routes; req.uid set identically so all ownership filters work unchanged
prefix stored clear-text in ApiKey	O(1) candidate row lookup before expensive bcrypt.compare; prefix alone is not usable as a key
ApiKeyServices frozen constant for service enum	Single source of truth in helpers/constants.js; adding a new service type requires editing the constant only — model and controller stay in sync via Object.values()
ExportUnits frozen constant for units enum	Same principle — consistent with project convention for all enumeric text constants
avgSpraySpeed stored at import, not computed on demand	Session summary endpoint must never touch AppDetail (billion-scale collection); O(1) read from App model
Cursor pagination on AppDetail._id	Consistent with existing filesdata_post pattern; no skip-based offset that degrades on large collections
interval thinning on both records endpoint and export	Consistent behaviour; reduces Power BI payload for overview queries; daily batch export at 17:00 can use interval=1 to shrink CSV size significantly
reportConfirmed boolean + always-populated fallback	Consumer's data warehouse always has a usable record; can upsert when field flips to true
Async export with TTL (ExportJob.expiresAt + MongoDB TTL index)	Files self-clean after 24 hours; no manual housekeeping job needed
CSV columns include job/session header repeated per row	Direct Power BI / warehouse import without requiring a separate join step
Unit conversion at output time, not at storage	Raw data stored in metric throughout; conversion applied in recordToRow() with unit-labelled column headers so output is self-documenting

---

10. Constraints & Notes

• All API responses use metric units by default (ha, m/s, L/min, L/ha, Kg/ha, °C, metres). Callers may request US customary output via units: 'us' on the export endpoint — see Section 5.6.
• All dates/times are ISO 8601 UTC strings.
• Coordinates are WGS84 decimal degrees (EPSG:4326) — numerically equivalent to SIRGAS 2000 (EPSG:4674) for Brazil.
• AppDetail.sprayStat === 3 is the first sprayed point of a new spray line — the boom IS open at this record. It is written when the spray transitions from OFF to ON (or at the start of a new save target). It also serves as an area anchor (records UTM X/Y, swath, line number) that the import worker uses to compute spray segment coverage. Do NOT exclude sprayStat = 3 from coverage calculations.
• AppDetail.raserAlt (typo in source schema) is exposed as laserAlt_m in the API.
• rpm[] array semantics differ between liquid and dry material types.
• Pending: customer confirmation on whether GET /api/v1/jobs/:id/areas (spray-area GeoJSON) is required for their ArcGIS workflow — endpoint is implemented but not yet scheduled for release.

---

11. Canonical Field Reference — DB Source Map

Every public /api/v1 output field mapped to its exact MongoDB source. Use this table as the authoritative reference when debugging a field returning wrong or null values.

11.1 /sessions Envelope Fields

API field	DB model	DB field path	Notes
jobId	—	URL param Job.id (integer)
clientId	Job	client._id (populated)
clientName	Job	client.name (populated)
mappedArea_ha	Job	rptOp.areaSize → fallback ttSprArea	getJobMappedAreaHa() helper; 2 dp
reportConfirmed	Job	rptOp.coverage != null
areaSize_ha	Job	rptOp.areaSize (confirmed) / ttSprArea (fallback)	2 dp
coverage_ha	Job / App[]	rptOp.coverage (confirmed) / SUM(App.totalSprayed) (fallback)	2 dp
overSprayed_pct	derived	(coverage_ha - areaSize_ha) / areaSize_ha × 100	null when either area is 0 or null; 2 dp
appRate	Job / AppFile	rptOp.appRate (confirmed) / AppFile.meta.appRate first session (fallback)
appRateUnit	Job	appRateUnit code → rateUnitString(code, true)
appRateConfirmed	Job	rptOp.appRate	null when not confirmed
sprayVolume	derived	coverage_ha × appRate converted by Job.measureUnit	planned estimate; 3 dp
volumeUnit	derived	material type + Job.measureUnit → "lit"/"gal" (liquid) or "kg"/"lb" (solid)	liquid default when appRateUnit unset
useConfirmedVolume	Job	rptOp.useActualVol	false when not confirmed
actualSprayVolume	App[]	SUM(App.totalSprayMat) normalized to metric base, then converted by Job.measureUnit	null when no application totals; 3 dp
confirmedActualVolume	Job	rptOp.actualVol converted by Job.measureUnit	null when not confirmed or not set; 3 dp
effectiveVolume	derived	confirmedActualVolume (if useConfirmedVolume) else actualSprayVolume	3 dp
useCustomWeather	Job	useCustWI
weather.windSpeed_kt	Job	weatherInfo.windSpd	only when useCustWI=true
weather.windDir	Job	weatherInfo.windDir	only when useCustWI=true
weather.temp_c	Job	weatherInfo.temp	only when useCustWI=true
weather.humidity_pct	Job	weatherInfo.humid	only when useCustWI=true
assignedPilotId	Job	operator._id (populated)
assignedPilotName	Job	operator.name (populated)
assignedAircraftId	JobAssign	user._id (latest, when user.kind=DEVICE)	live workflow assignment traceability
assignedAircraftName	JobAssign / Job	user.name when live-assigned; fallback vehicle.name
assignedAircraftTailNumber	JobAssign / Job	user.tailNumber when live-assigned; fallback vehicle.tailNumber
planAircraftName	Job	vehicle.name (populated)	always the job-plan aircraft regardless of live assignment
planAircraftTailNumber	Job	vehicle.tailNumber (populated)	always the job-plan aircraft regardless of live assignment
assignedDate	JobAssign	date	latest assignment; sorted by date desc

11.2 /sessions Per-Session data[] Fields

API field	DB model	DB field path	Notes
sessionId	App	_id
fileName	App	fileName
startDateTime	App	startDateTime	ISO 8601 UTC
endDateTime	App	endDateTime	ISO 8601 UTC
totalFlightTime_s	App	totalFlightTime	3 dp
totalSprayTime_s	App	totalSprayTime	3 dp
totalTurnTime_s	App	totalTurnTime	3 dp
totalSprayed_ha	App	totalSprayed	2 dp
totalSprayMat	App	totalSprayMat	3 dp
totalSprayMatUnit	App	totalSprayMatUnit code → rateUnitString(code, true, 1)	decoded to string e.g. "lit"
avgSpraySpeed_ms	App	avgSpraySpeed	stored at import time; m/s; 2 dp
sprayZoneName	AppFile	meta.areaOrZone	first file
sprayZoneArea_ha	AppFile	meta.sprCoverage[1]	first file; 2 dp
appRate	AppFile	meta.appRate	first file
appRateUnit	Job	see envelope appRateUnit
flowController	AppFile	meta.fcName	'No FC' when absent or "none"
sprayOnLag_s	AppFile	meta.sprOnLag	first file
sprayOffLag_s	AppFile	meta.sprOffLag	first file
pulsesPerLiter	AppFile	meta.pulsesPerLit	first file
files	AppFile[]	[{ fileId: _id, name }]	all files for this session
sessionPilotName	AppFile	meta.operator	name as written in the data file; may differ from job-assigned pilot

11.3 /records Per-Record Fields

API field	DB model	DB field path	Transform
timeUtc	AppDetail	gpsTime	epoch-s → ISO 8601 UTC (toRecordTimeUtc)
gpsTime	AppDetail	gpsTime	raw
lat	AppDetail	lat	7 dp
lon	AppDetail	lon	7 dp
utmX	AppDetail	utmX	1 dp
utmY	AppDetail	utmY	1 dp
alt	AppDetail	alt	m; US: × 3.28084 → ft; 2 dp
grSpeed	AppDetail	grSpeed	m/s; US: × 2.23694 → mph; 2 dp
heading	AppDetail	head	degrees; 2 dp
xTrack	AppDetail	xTrack	m; US: × 3.28084 → ft; 2 dp
lockedLine	AppDetail	llnum
hdop	AppDetail	stdHdop	2 dp
satsIn	AppDetail	satsIn	raw NT value: 0..99 outside area, 100..199 inside area; satellites = value (outside) or value-100 (inside)
tslu	AppDetail	tslu	raw time since last GPS differential correction update (seconds)
calcodeFreq	AppDetail	calcodeFreq	raw frequency/calibration field; see semantics in section 5.3
sprayStat	AppDetail	sprayStat	raw (0/1/3/10)
flowRateApplied	AppDetail	lminApp	L/min; US: × 0.264172 → gal/min; 3 dp
flowRateRequired	AppDetail	lminReq	L/min; US: × 0.264172 → gal/min; 3 dp
appRateRequired	AppDetail	lhaReq	L/ha; US: × 0.10694 → gal/ac; 2 dp
appRateApplied	derived	lminApp / (grSpeed × swath) × 10000	null on zero-division; US: × 0.10694; 2 dp
swathWidth	AppDetail	swath	m; US: × 3.28084 → ft
boomPressure_psi	AppDetail	psi	already PSI; 2 dp
flowController	AppFile	meta.fcName	session constant; 'No FC' when absent
sprayOnLag_s	AppFile	meta.sprOnLag	session constant
sprayOffLag_s	AppFile	meta.sprOffLag	session constant
pulsesPerLiter	AppFile	meta.pulsesPerLit	session constant
rpm	AppDetail	rpm	raw array
windSpeed_kt	AppDetail	windSpd	m/s × 1.94384 → kt; US: m/s × 2.23694 → mph; 2 dp
windDir_deg	AppDetail	windDir	1 dp
temp_c	AppDetail	temp	°C; US: × 9/5 + 32 → °F; 1 dp
humidity_pct	AppDetail	humid	1 dp
sprayHeight_m (fm)	AppDetail	sprayHeight	?fm=true only
driftX_m (fm)	AppDetail	driftX	?fm=true only
driftY_m (fm)	AppDetail	driftY	?fm=true only
depositX_m (fm)	AppDetail	depositX	?fm=true only
depositY_m (fm)	AppDetail	depositY	?fm=true only
radarAlt_m (fm)	AppDetail	radarAlt	?fm=true only
laserAlt_m (fm)	AppDetail	laserAlt → raserAlt (typo fallback)	?fm=true only

11.4 /areas Feature Properties

API field	DB model	DB field path	Notes
name	Job	sprayAreas[i].properties.name
appRate	Job	sprayAreas[i].properties.appRate
area_ha	Job	sprayAreas[i].properties.area	polygon-level metadata only; NOT used for session area totals
type	Job	sprayAreas[i].properties.type	e.g. "area", "xcl"
appRateUnit	Job	appRateUnit code → rateUnitString
fallbackAreaHa	Job	rptOp.areaSize → ttSprArea	envelope-level fallback, not per-polygon
geometry	Job	sprayAreas[i].geometry	copied verbatim

11.5 CSV Export Column → DB Source

CSV column (metric)	CSV column (US)	DB model	DB field path	Transform
jobId	same	ExportJob	jobId
orderNumber	same	Job	orderNumber
jobName	same	Job	name
clientId	same	Job	client._id
clientName	same	Job	client.name
sessionId	same	App	_id
fileName	same	App	fileName
pilotName	same	AppFile	meta.operator
timeUtc	same	AppDetail	gpsTime	epoch-s → ISO 8601 UTC
gpsTime	same	AppDetail	gpsTime	raw
lat	same	AppDetail	lat
lon	same	AppDetail	lon
utmX	same	AppDetail	utmX
utmY	same	AppDetail	utmY
alt_m	alt_ft	AppDetail	alt	US: × 3.28084
groundSpeed_ms	groundSpeed_mph	AppDetail	grSpeed	US: × 2.23694
heading	same	AppDetail	head
crossTrackError_m	crossTrackError_ft	AppDetail	xTrack	US: × 3.28084
lockedLine	same	AppDetail	llnum
hdop	same	AppDetail	stdHdop
satsIn	same	AppDetail	satsIn	raw
tslu	same	AppDetail	tslu	raw
calcodeFreq	same	AppDetail	calcodeFreq	raw
sprayStat	same	AppDetail	sprayStat	raw
flowRateApplied_Lmin	flowRateApplied_galMin	AppDetail	lminApp	US: × 0.264172
flowRateRequired_Lmin	flowRateRequired_galMin	AppDetail	lminReq	US: × 0.264172
appRateRequired_Lha	appRateRequired_galAc	AppDetail	lhaReq	US: × 0.10694
appRateApplied_Lha	appRateApplied_galAc	derived	lminApp / (grSpeed × swath) × 10000	US: × 0.10694
swathWidth_m	swathWidth_ft	AppDetail	swath	US: × 3.28084
boomPressure_psi	same	AppDetail	psi
flowController	same	AppFile	meta.fcName	'No FC' fallback
sprayOnLag_s	same	AppFile	meta.sprOnLag
sprayOffLag_s	same	AppFile	meta.sprOffLag
pulsesPerLiter	same	AppFile	meta.pulsesPerLit
rpm	same	AppDetail	rpm	JSON-serialised array
windSpeed_kt	windSpeed_mph	AppDetail	windSpd	m/s × 1.94384 (kt); US: m/s × 2.23694 (mph)
windDir_deg	same	AppDetail	windDir
temp_c	temp_f	AppDetail	temp	US: × 9/5 + 32
humidity_pct	same	AppDetail	humid
sprayHeight_m (fm)	same	AppDetail	sprayHeight
driftX_m (fm)	same	AppDetail	driftX
driftY_m (fm)	same	AppDetail	driftY
depositX_m (fm)	same	AppDetail	depositX
depositY_m (fm)	same	AppDetail	depositY
radarAlt_m (fm)	same	AppDetail	radarAlt
laserAlt_m (fm)	same	AppDetail	laserAlt → raserAlt	schema typo fallback