85 KiB
Pilot Analytics Dashboard — API Design Reference
Version: 2.5 — (2026-06-15) Scope: Backend API contract for the Pilot Analytics Dashboard. This document is the single source of truth for both backend and frontend/client development.
Table of Contents
- 1 Overview
- 2 Authentication
- 3 Status Constants
- 4 Timezone Handling
- 5 Endpoints
- 6 Error Responses
- 7 Data Model Notes
- 8 Frontend Integration Guide
- 9 Backend Architecture Notes
- 10 Open Decisions
- 11 Changelog
1 Overview
The Pilot Dashboard is a read-only analytics surface (plus one write action) scoped to the currently authenticated Pilot user. All endpoints derive the pilot identity from the JWT token — no client-side user ID parameter is accepted or trusted.
Base path: /api/dashboard/pilot
Implemented in:
controllers/dashboard.jsroutes/dashboard.jsmodel/setting.js(fordashboardthreshold storage)routes/job.js(for the complete action)
Testing resources:
- Postman collection:
docs/Pilot_Dashboard_API.postman_collection.json - Node.js test script:
tests/test_pilot_dashboard_api.js - Test command:
npm run test:dashboard
Dashboard Test Suite Requirements
npm run test:dashboard runs live integration tests (not mocked/unit-only). To execute all non-optional checks reliably:
- Server must be running and reachable at
https://localhost:4100(or setPILOT_DASHBOARD_BASE_URL). DASHBOARD_TEST_TOKENmust be set to a valid Pilot JWT.- For complete-job tests, set:
RUN_COMPLETE_TEST=1DASHBOARD_TEST_JOB_ID=<jobId in SPRAYED(3) status>
Optional toggles:
RUN_SNAPSHOT_TESTS=0skips/snapshotendpoint tests.RUN_COMPLETE_TEST=0(default) skips complete-job mutation tests.
If required env vars are missing, Mocha reports pending (skipped) tests by design.
Migration Script: scripts/migrate_applications.js
Purpose:
- All-in-one replacement for the former
backfill_application_datetimes.jsandmigrate_app_aggregates.jsscripts (both kept as deprecated stubs). - Pass 1A — Aggregate metrics:
Application.avgSpraySpeed,totalSprLength,totalFlightLength,avgXtError,avgHdop; andAppFile.totalSprLength,AppFile.totalFlightLength. - Pass 1B — Datetime fields (for apps that have legacy
startDateTime):Application.utcOffset,startDateTimeUTC,endDateTimeUTC.- First valid lat/lon coordinate captured during the existing Pass 1A AppDetail stream — no extra DB query needed.
- Pass 2 —
Application.flowAccuracyPct(Application-level only, no AppDetail streaming required).
Selection criteria (non-force mode):
- Union of both former scripts'
$orconditions:- Datetime:
utcOffsetmissing or 0,startDateTimeUTC/endDateTimeUTCmissing, orstartDateTimeUTC > endDateTimeUTC. - Aggregates: any of
avgSpraySpeed,totalSprLength,totalFlightLength,avgXtError,avgHdopmissing/null/0.
- Datetime:
- Use
--skip-datetimeor--skip-aggregatesto restrict to one set of conditions.
Why Apps to process: 0 can happen:
- All eligible apps are already backfilled, OR
- Remaining records missing UTC companions do not have legacy
startDateTime(script skips those for datetime, but still processes aggregates).
Usage:
node scripts/migrate_applications.js
node scripts/migrate_applications.js --dry-run
node scripts/migrate_applications.js --env ./environment.env --dry-run
node scripts/migrate_applications.js --missing-limit 200
# After a formula fix: reprocess ALL apps to correct previously stored values
node scripts/migrate_applications.js --force
# Tier 1 — most recent 90 days only
node scripts/migrate_applications.js --tier-days=90
# Datetime fields only (fast, uses lightweight findOne)
node scripts/migrate_applications.js --skip-aggregates
# Aggregate metrics only (skip datetime computation)
node scripts/migrate_applications.js --skip-datetime
Operational diagnostics:
- When
Apps to process: 0and legacy datetime is missing/null on records, prints each affected app and related job context:appId,App-jobId,jobId,jobStatus,appStatusstartDateTime,endDateTimeappFiles,appDetails,likelyNoDataFiles
- This makes "zip has no data files" cases directly visible from script output.
All dashboard endpoints share the same scoping logic:
pilot → Job.operator = req.uid → jobIds → Application.jobId IN jobIds
2 Authentication
All routes require a valid JWT bearer token. The checkUser middleware is applied globally
in server.js before any route is mounted.
Authorization: Bearer <jwt>
The token payload sets req.uid (user ID string) and req.userInfo on every request.
Dashboard endpoints check req.uid and throw 401 Not Authorized if absent.
The dashboard route group (/api/dashboard) does not require a subscription package
(checkRqPkgSubscription is not applied there). The complete-job endpoint lives under
/api/jobs which does apply that middleware.
3 Status Constants
Job Status (helpers/job_constants.js)
| Constant | Value | Meaning |
|---|---|---|
NEW |
0 |
Job created, not yet prepared |
READY |
1 |
Prepared and available for aircraft download |
DOWNLOADED |
2 |
Downloaded to aircraft |
SPRAYED |
3 |
Application data uploaded, not yet reviewed |
COMPLETED |
4 |
Reviewed and marked done by Applicator |
INVOICED |
5 |
Invoice issued |
ARCHIVED |
9 |
Archived |
Display Status (frontend label mapping)
The activeJobs endpoint returns a displayStatus string derived from the raw numeric status:
status value(s) |
displayStatus |
Suggested UI treatment |
|---|---|---|
0 (NEW) |
"NEW" |
Grey badge, no progress bar |
1, 2, 3 |
"IN_PROGRESS" |
Blue/active badge, show progress |
4 (COMPLETED) |
"COMPLETED" |
Green badge, full bar |
INVOICED (
5) and ARCHIVED (9) are excluded from the active jobs panel entirely.
Application Status
Only status: 3 (processed) applications are counted in all aggregations. Uploading (1),
in-progress (2), and error (0) applications are excluded automatically.
4 Timezone Handling
All time-windowed endpoints accept an optional ?tz=<IANA> query parameter.
- Default:
UTC - Example:
?tz=America/Sao_Paulo,?tz=America/Chicago,?tz=Australia/Brisbane - Invalid values silently fall back to
UTC— the frontend should always send a valid tz.
What timezone affects:
- "Today" and "yesterday" day boundaries
- Current week (Mon–Sun) boundaries
- Current month and year boundaries
- Date labels in the trend chart (
YYYY-MM-DD)
What timezone does NOT affect:
- Raw UTC timestamps stored in MongoDB
- Job
createdAt(always UTC)
Recommended frontend behaviour: read the browser's timezone once (Intl.DateTimeFormat().resolvedOptions().timeZone) and pass it on every dashboard request. The API still needs tz to derive calendar boundaries and day labels, even though the Application filters now use the stored UTC fields.
5 Endpoints
5.1 KPI Cards
URL: GET /api/dashboard/pilot/kpi
Returns top-level KPI cards and historical breakdowns for the authenticated pilot.
Query Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
tz |
String | UTC |
IANA timezone for period windows |
Response 200 OK
{
"operations": {
"missionsFlown": 1,
"distanceTravelledKm": 256.22,
"distanceSprayedKm": 50.82,
"sprayEfficiencyPct": 68.40,
"ferryTimePct": 31.60,
"flowAccuracyPct": 97.50,
"avgHdop": 1.20
},
"periods": {
"day": {
"assignedJobs": 1,
"assignedHectares": 939.02,
"sprayedHectares": 148.49,
"flightHours": 1.53,
"sprayEfficiencyPct": 68.40,
"ferryTimePct": 31.60,
"flowAccuracyPct": 97.50,
"avgHdop": 1.20,
"jobCounts": { "new": 0, "inProgress": 1, "completed": 0 }
},
"week": {
"assignedJobs": 4,
"assignedHectares": 9807.32,
"sprayedHectares": 15762.43,
"flightHours": 18.37,
"sprayEfficiencyPct": 71.00,
"ferryTimePct": 29.00,
"flowAccuracyPct": 96.25,
"avgHdop": 1.15,
"jobCounts": { "new": 1, "inProgress": 3, "completed": 1 }
},
"month": {
"assignedJobs": 7,
"assignedHectares": 27908.71,
"sprayedHectares": 22838.75,
"flightHours": 21.73,
"sprayEfficiencyPct": 70.50,
"ferryTimePct": 29.50,
"flowAccuracyPct": 98.10,
"avgHdop": 1.08,
"jobCounts": { "new": 1, "inProgress": 6, "completed": 4 }
},
"year": {
"assignedJobs": 7,
"assignedHectares": 27908.71,
"sprayedHectares": 22838.75,
"flightHours": 21.73,
"sprayEfficiencyPct": 69.80,
"ferryTimePct": 30.20,
"flowAccuracyPct": 97.50,
"avgHdop": 1.18,
"jobCounts": { "new": 2, "inProgress": 3, "completed": 2 }
},
"all": {
"assignedJobs": 7,
"assignedHectares": 27908.71,
"sprayedHectares": 22838.75,
"flightHours": 21.71,
"sprayEfficiencyPct": 69.80,
"ferryTimePct": 30.20,
"flowAccuracyPct": 97.50,
"avgHdop": 1.18,
"jobCounts": { "new": 2, "inProgress": 3, "completed": 2 }
}
}
}
New in v2.4: All efficiency and GPS quality metrics now available in each period, not just today's operations. Pilots can track historical trends in spray efficiency, ferry percentage, flow control accuracy, and GPS health across day/week/month/year/all-time windows.
jobCountsis present on all periods:day,week,month,year, andall. New metrics (sprayEfficiencyPct,ferryTimePct,flowAccuracyPct,avgHdop) are also present on all periods.
Field Notes
| Field | Unit | Notes |
|---|---|---|
operations.missionsFlown |
count | Number of Application records uploaded today (scoped to current day window) |
operations.distanceTravelledKm |
km | Sum of Application.totalFlightLength / 1000 for today — all GPS segments including turns (segments with time gap > 120 s or distance > 1000 m excluded) |
operations.distanceSprayedKm |
km | Sum of Application.totalSprLength / 1000 for today — spray-on segments only (same time/distance gates as above) |
operations.sprayEfficiencyPct |
% | SUM(totalSprayTime) / SUM(totalFlightTime) × 100 for today. null when no flight time. Higher = more time actively spraying vs turning/ferrying |
operations.ferryTimePct |
% | (SUM(totalFlightTime) − SUM(totalSprayTime)) / SUM(totalFlightTime) × 100 for today. Complement of sprayEfficiencyPct; both sum to 100 when non-null. null when no flight time |
operations.flowAccuracyPct |
% | Average of Application.flowAccuracyPct for today — (actual rate / prescribed appRate) × 100 per session. null when no sessions with prescribed rate |
operations.avgHdop |
— | Average of Application.avgHdop for today — mean HDOP across spray-on records. null when no data. < 1 excellent; 1–2 good; 2–5 moderate; > 5 poor |
periods.<p>.assignedJobs |
count | Open jobs (NEW / READY / DOWNLOADED / SPRAYED / COMPLETED) created within period <p> |
periods.<p>.assignedHectares |
ha | Sum of Job.ttSprArea for jobs created within period <p> |
periods.<p>.sprayedHectares |
ha | Sum of Application.totalSprayed for applications uploaded within period <p> |
periods.<p>.flightHours |
hours | Sum of Application.totalFlightTime / 3600 for applications uploaded within period <p> (all flight records, same validity rules as totalFlightTime) |
periods.<p>.sprayEfficiencyPct |
% | SUM(totalSprayTime) / SUM(totalFlightTime) × 100 for period <p>. null when no flight time. Higher = more time actively spraying vs turning/ferrying |
periods.<p>.ferryTimePct |
% | (SUM(totalFlightTime) − SUM(totalSprayTime)) / SUM(totalFlightTime) × 100 for period <p>. Complement of spray efficiency; both sum to 100 when non-null. null when no flight time |
periods.<p>.flowAccuracyPct |
% | Average of Application.flowAccuracyPct for applications in period <p>. null when no sessions with prescribed rate |
periods.<p>.avgHdop |
— | Average of Application.avgHdop for period <p>. null when no data. < 1 excellent; 1–2 good; 2–5 moderate; > 5 poor |
periods.<p>.jobCounts.new |
count | Jobs with status NEW (0) created within that period |
periods.<p>.jobCounts.inProgress |
count | Jobs with status READY (1), DOWNLOADED (2), or SPRAYED (3) created within that period |
periods.<p>.jobCounts.completed |
count | Jobs with status COMPLETED (4) created within that period |
<p> = day | week | month | year | all. The all period has no time boundary — it covers all records for this pilot.
All numeric values are rounded to 2 decimal places.
5.2 Daily Summary
URL: GET /api/dashboard/pilot/summary
Returns today vs. yesterday operational metrics with percentage change deltas.
Query Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
tz |
String | UTC |
IANA timezone for day boundaries |
Response 200 OK
{
"today": {
"hectares": 120.50,
"flightHours": 3.25,
"haPerHour": 37.08,
"avgSpeedKmh": 28.50,
"sprayVolumeLiters": 2400.00
},
"yesterday": {
"hectares": 95.00,
"flightHours": 2.80,
"haPerHour": 33.93,
"avgSpeedKmh": 26.10,
"sprayVolumeLiters": 1900.00
},
"todayHasData": true,
"deltas": {
"hectaresPct": 27,
"flightHoursPct": 16,
"haPerHourPct": 9,
"avgSpeedPct": 9,
"sprayVolumePct": 26
}
}
Field Notes
| Field | Unit | Notes |
|---|---|---|
today.haPerHour |
ha/hr | Derived: hectares / flightHours (0 if no flight time) |
today.avgSpeedKmh |
km/h | Application.avgSpraySpeed (m/s) × 3.6, averaged |
todayHasData |
bool | false when no Application records exist for today. Use this to distinguish "no upload yet" from a real operational drop |
deltas.*Pct |
% | round((today − yesterday) / yesterday × 100). All fields are null when todayHasData is false |
deltas.*Pct = null |
— | Either no data uploaded today (todayHasData: false), or yesterday value was 0 (division by zero avoided) |
Positive delta = improvement today vs. yesterday. Negative = drop.
Frontend guidance: only render colored delta arrows (red/green) when
todayHasDataistrueand the delta value is non-null. WhentodayHasDataisfalse, show a neutral "—" or "Awaiting data" state — a-100%would be misleading when the cause is a missing upload, not a real performance drop.
5.3 Trend Charts
URL: GET /api/dashboard/pilot/trend
Returns daily hours flown and hectares sprayed for a date range. Default range is the current calendar week (Monday–Sunday) in the given timezone.
Query Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
tz |
String | UTC |
IANA timezone for day grouping and boundaries |
startDate |
String | Monday of this week | Start date inclusive. Format: YYYY-MM-DD |
endDate |
String | Sunday of this week | End date inclusive. Format: YYYY-MM-DD |
- Maximum range: 90 days. Returns
409if exceeded. - If either
startDateorendDateis omitted, both are silently defaulted to the current week (Mon–Sun). No409is returned for a one-sided pair — the partial value is discarded.
Response 200 OK
{
"labels": ["2026-04-27", "2026-04-28", "2026-04-29", "2026-04-30", "2026-05-01", "2026-05-02", "2026-05-03"],
"hoursFlown": [2.50, 3.25, 0, 1.80, 0, 0, 0],
"hectaresPerDay":[80.0, 120.5, 0, 65.0, 0, 0, 0]
}
Field Notes
| Field | Type | Notes |
|---|---|---|
labels |
String[] |
One entry per calendar day in YYYY-MM-DD, tz-adjusted |
hoursFlown |
Number[] |
Parallel array. Days with no activity = 0 |
hectaresPerDay |
Number[] |
Parallel array. Days with no activity = 0 |
Array contract: all three arrays are always the same length and in the same order.
Frontend can zip them: labels[i] ↔ hoursFlown[i] ↔ hectaresPerDay[i].
5.4 Active Jobs Panel
URL: GET /api/dashboard/pilot/activeJobs
Returns the pilot's active-status jobs with per-job progress and applied totals.
- Statuses included:
NEW (0),READY (1),DOWNLOADED (2),SPRAYED (3),COMPLETED (4) - Statuses excluded:
INVOICED (5),ARCHIVED (9) - Maximum 50 most recent jobs returned (sorted by
createdAtdescending)
Query Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
tz |
String | UTC |
IANA timezone used to compute all period boundaries |
period |
String | — | Time window for both job list and Application totals: day | week | month | year. Filters jobs by createdAt and Application totals by startDateTimeUTC. The tz parameter still controls how those calendar windows are derived. |
Parameter priority: period > (none — all-time)
When a
periodis provided, only jobs created within that window are returned and thehaSprayed/volumeAppliedLiterstotals are scoped to applications whosestartDateTimeUTCfalls in the same calendar window.haTotalandprogressPctalways reflect the job's planned area regardless of the filter. The legacy string fields remain display-only;tzis still needed to translate the requested calendar period into UTC boundaries.
Response 200 OK
{
"jobs": [
{
"jobId": 1042,
"name": "North Block — Canola",
"clientName": "Sunrise Farms Ltd.",
"aircraftReg": "C-FABM",
"status": 3,
"displayStatus": "IN_PROGRESS",
"createdDate": "2026-06-01T14:22:00.000Z",
"haTotal": 250.00,
"haSprayed": 187.50,
"progressPct": 75.00,
"volumeAppliedLiters": 3750.00
}
]
}
Field Notes
| Field | Type | Notes |
|---|---|---|
jobId |
Number | Numeric job ID (Job._id, not a MongoDB ObjectId) |
name |
String | Job name. Empty string if not set |
clientName |
String | From linked Client user record. Empty if not linked |
aircraftReg |
String | Vehicle.tailNumber or Vehicle.unitId. Empty if not linked |
status |
Number | Raw numeric status (see constants table) |
displayStatus |
String | "NEW" | "IN_PROGRESS" | "COMPLETED" |
createdDate |
Date | Job.createdAt — UTC ISO 8601 timestamp of job creation. null if not set |
haTotal |
Number | Job.ttSprArea — planned area in ha. 0 if not set |
haSprayed |
Number | Sum of Application.totalSprayed for this job within the active window (processed apps only) |
progressPct |
Number | 0–100 (2 decimal places). 0 if haTotal is 0 or no applications |
volumeAppliedLiters |
Number | Sum of Application.totalSprayMat for this job within the active window |
progressPct formula: parseFloat(min(100, max(0, haSprayed / haTotal × 100)).toFixed(2))
Returns a float with up to 2 decimal places (e.g. 0.47 for a job under 1% complete). The backend caps at 100 if haSprayed > haTotal.
5.5 Performance Gauges
URL: GET /api/dashboard/pilot/performance
Returns average XT cross-track error and spray altitude gauges for all processed application files within the requested date range.
All values are calculated using spray-on records only (sprayStat === 1),
giving meaningful agronomic metrics free from transit/ferry-flight pollution.
The average is record-weighted across all individual GPS data points (one row
per second of flight) in the period — days with more spray-on time have more
influence than days with fewer records.
Note
: ApplicationDetail is a billion-row collection. Queries are strictly bounded to a set of
fileIdvalues to use the collection's primary index. Never query this collection without afileIdfilter.
Query Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
tz |
String | UTC |
IANA timezone string for date boundary calculations |
startDate |
String | Monday of current week | Range start date YYYY-MM-DD (inclusive) |
endDate |
String | Sunday of current week | Range end date YYYY-MM-DD (inclusive). Max range: 90 days |
Date resolution: startDate+endDate → current week default
Response 200 OK — data available
{
"startDate": "2026-05-19",
"endDate": "2026-05-25",
"avgXtError": 2.82,
"xtThreshold": {
"good": 1.0,
"monitor": 3.0
},
"hasXtData": true,
"avgSprayAltitudeMeters": 3.62,
"altitudeSource": "sprayHeight",
"altThreshold": {
"target": 3.7,
"goodBand": 0.15,
"monitorBand": 0.46
},
"hasAltitudeData": true,
"sampleSize": 4
}
Response 200 OK — no data (new pilot, no apps uploaded)
{
"startDate": "2026-05-19",
"endDate": "2026-05-25",
"avgXtError": null,
"xtThreshold": { "good": 1.0, "monitor": 3.0 },
"hasXtData": false,
"avgSprayAltitudeMeters": null,
"altitudeSource": null,
"altThreshold": { "target": 3.7, "goodBand": 0.15, "monitorBand": 0.46 },
"hasAltitudeData": false,
"sampleSize": 0
}
Field Notes
| Field | Unit | Notes |
|---|---|---|
startDate |
String | Effective start of the analysis window (YYYY-MM-DD) |
endDate |
String | Effective end of the analysis window (YYYY-MM-DD) |
avgXtError |
metres | Mean abs(xTrack) across all spray-on GPS records. null if no data |
xtThreshold.good |
metres | Below this → green zone |
xtThreshold.monitor |
metres | Above this → red zone; between good and monitor → yellow |
hasXtData |
bool | false = no valid xTrack readings in sample (show "No data") |
avgSprayAltitudeMeters |
metres | Mean spray height from best available sensor, spray-on records only. null if none |
altitudeSource |
String | "sprayHeight" (FM dedicated sensor) or "radarAlt" (AGL fallback). null if no data |
altThreshold.target |
metres | Ideal spray height (~3.7 m) |
altThreshold.goodBand |
metres | ±0.15 m of target → green |
altThreshold.monitorBand |
metres | ±0.46 m of target → yellow; outside → red |
hasAltitudeData |
bool | false = no altitude sensor data in sample (show "No data") |
sampleSize |
count | Number of AppFile records included in the analysis window |
Threshold Gauge Logic (frontend)
XT Cross-track error:
value < good → green (acceptable precision)
good ≤ value < monitor → yellow (monitor)
value ≥ monitor → red (needs attention)
Spray Altitude:
|value - target| < goodBand → green
|value - target| < monitorBand → yellow
|value - target| ≥ monitorBand → red
Altitude Source Priority
The backend selects the best available altitude sensor in this order:
sprayHeight— dedicated FM spray height sensor (most accurate)radarAlt— radar altimeter (AGL, good fallback)- No data —
hasAltitudeData: false,avgSprayAltitudeMeters: null
gpsAlt(AMSL) is not used for spray height — it measures altitude above sea level, not above the crop canopy.
xTrack = 0 values are excluded from the XT average because 0 means "no reading",
not "perfectly on track".
5.6 Save Performance Thresholds
Persists custom XT error and altitude gauge thresholds for the authenticated pilot.
Values are stored per-user in Setting.dashboard and are automatically applied
the next time GET /pilot/performance is called.
All fields are optional — send only the fields you want to change. Pass null for a
field to reset it to the system default.
URL: PUT /api/dashboard/pilot/performance/thresholds
Request Body (JSON)
| Field | Type | Default (system) | Description |
|---|---|---|---|
xtGood |
Number | null | 1.0 |
XT ideal threshold in metres (top of green zone) |
xtMonitor |
Number | null | 3.0 |
XT caution threshold in metres (top of yellow zone) |
altTarget |
Number | null | 3.7 |
Altitude target in metres (centre of altitude gauge) |
altGoodBand |
Number | null | 0.15 |
±band from target for green zone |
altMonitorBand |
Number | null | 0.46 |
±band from target for yellow zone |
Constraints:
- All supplied values must be positive finite numbers (
> 0). xtMonitormust be greater thanxtGood.altMonitorBandmust be greater thanaltGoodBand.- Constraints are checked against the effective post-save value for each field: the value being saved in this request → if being reset (
null), the system default → the currently stored custom value → the system default. This means partial updates (sending only some fields) are correctly validated against the real in-DB state.
Example Request Body
{
"xtGood": 7.0,
"xtMonitor": 13.0
}
Only the two XT fields are sent — altitude thresholds remain unchanged.
Response 200 OK
Returns the effective thresholds after saving (saved value, or system default if not customised):
{
"xtThreshold": {
"good": 7.0,
"monitor": 13.0
},
"altThreshold": {
"target": 3.7,
"goodBand": 0.15,
"monitorBand": 0.46
}
}
The frontend can use this response to immediately update the gauge without a separate
GET /performancecall.
Reset to Defaults
Pass null to clear a custom value and revert to the system default:
{ "xtGood": null, "xtMonitor": null }
Error Cases
| Condition | Status | Error tag |
|---|---|---|
| Missing or invalid JWT | 401 |
not_authorized |
| Non-positive or non-finite value | 409 |
invalid_param |
xtMonitor ≤ xtGood (effective values) |
409 |
invalid_param |
altMonitorBand ≤ altGoodBand (effective) |
409 |
invalid_param |
5.7 Mark Job as Completed
URL: PATCH /api/jobs/:job_id/complete
Transitions a job from SPRAYED (3) to COMPLETED (4).
This endpoint lives under /api/jobs (not /api/dashboard) and requires an active
subscription package (enforced by checkRqPkgSubscription middleware).
URL Parameters
| Parameter | Type | Description |
|---|---|---|
job_id |
Number | Numeric job ID |
Request Body
None.
Response 200 OK
Returns the full updated job document (with client, operator, and vehicle populated).
{
"_id": 1042,
"name": "North Block — Canola",
"status": 4,
"client": { "_id": "...", "name": "Sunrise Farms Ltd." },
"operator": { "_id": "...", "name": "Jane Pilot" },
"vehicle": { "_id": "...", "name": "Agri-One", "tailNumber": "C-FABM" },
"..."
}
Authorization
The Applicator who owns the job (Job.byPuid) may complete it, as may any sub-user
(Pilot) operating under that Applicator. The check compares Job.byPuid against
req.userInfo.puid (the caller's root Applicator ID), which resolves correctly for both
the Applicator themselves and their sub-users.
| Caller type | req.userInfo.puid |
Allowed? |
|---|---|---|
| Applicator | same as req.uid |
✅ Yes |
| Pilot sub-user under the owning Applicator | parent's _id |
✅ Yes |
| Any other user | different from Job.byPuid |
❌ 401 |
Error Cases
| Condition | Status | Error tag |
|---|---|---|
job_id is not a valid positive number |
409 |
invalid_param |
| Job not found | 409 |
job_not_found |
Caller is not the job owner (byPuid) |
401 |
not_authorized |
Job status is not SPRAYED (3) |
409 |
status_job_invalid |
5.8 Snapshot (Composite Dashboard)
Returns multiple dashboard modules in a single request. Eliminates N+1 API calls on initial page load and during periodic polling.
URL: GET /api/dashboard/pilot/snapshot
All sub-modules share the same pilot-scoped job/app data fetch internally — no redundant
database queries. Each module uses its own query parameter validation (date-range modules
independently validate startDate/endDate; kpi, summary, and activeJobs use tz;
activeJobs additionally accepts period for time-scoped sub-totals).
Query Parameters
| Parameter | Type | Default | Applies to module(s) | Description |
|---|---|---|---|---|
include |
String | kpi,summary,activeJobs,performance,trend |
— | Comma-separated list of modules to return. Unknown names are silently ignored. |
tz |
String | UTC |
all | IANA timezone string |
period |
String | (omit = all-time) | activeJobs |
Time window for both job list and Application totals: day | week | month | year. Filters jobs by createdAt and Application totals by startDateTimeUTC. The tz parameter still controls the period boundaries. |
startDate |
String | Monday of current week | performance, trend |
YYYY-MM-DD |
endDate |
String | Sunday of current week | performance, trend |
YYYY-MM-DD. Max range: 90 days |
Valid include values: kpi · summary · activeJobs · performance · trend
Note on
invalid_paramfrom snapshot with trend: The 90-day cap applies inside the snapshot just as it does on the standalone/trendendpoint. IfstartDate→endDatespans more than 90 days andtrend(orperformance) is in theincludelist, the entire request returns409 invalid_param. To avoid this, keep the date range ≤ 90 days.
Response 200 OK
Only requested modules are present in the response object. Example with all modules:
{
"kpi": {
"operations": {
"missionsFlown": 2,
"distanceTravelledKm": 45.2,
"distanceSprayedKm": 32.1,
"sprayEfficiencyPct": 71.00,
"ferryTimePct": 29.00,
"flowAccuracyPct": 98.50,
"avgHdop": 1.10
},
"periods": {
"day": { "assignedJobs": 3, "assignedHectares": 45.5, "sprayedHectares": 32.1, "flightHours": 1.53, "jobCounts": { "new": 1, "inProgress": 1, "completed": 1 } },
"week": { "...": "same shape" },
"month": { "...": "same shape" },
"year": { "...": "same shape" },
"all": { "...": "same shape" }
}
},
"summary": {
"today": { "hectares": 32.1, "flightHours": 1.53, "haPerHour": 21.0, "avgSpeedKmh": 45.2, "sprayVolumeLiters": 256.0 },
"yesterday": { "...": "same shape" },
"todayHasData": true,
"deltas": { "hectaresPct": 15, "flightHoursPct": 10, "haPerHourPct": null, "avgSpeedPct": null, "sprayVolumePct": null }
},
"activeJobs": {
"jobs": [
{ "jobId": 42, "name": "North Block", "status": 3, "displayStatus": "IN_PROGRESS", "createdDate": "2026-06-01T14:22:00.000Z", "progressPct": 75.00, "haTotal": 250.0, "haSprayed": 187.5, "volumeAppliedLiters": 3750.0 }
]
},
"performance": {
"startDate": "2026-05-19", "endDate": "2026-05-25",
"avgXtError": 2.82, "xtThreshold": { "good": 1.0, "monitor": 3.0 }, "hasXtData": true,
"avgSprayAltitudeMeters": 3.62, "altitudeSource": "sprayHeight",
"altThreshold": { "target": 3.7, "goodBand": 0.15, "monitorBand": 0.46 },
"hasAltitudeData": true, "sampleSize": 4
},
"trend": {
"labels": ["2026-05-19", "2026-05-20", "2026-05-21"],
"hoursFlown": [1.53, 2.1, 0],
"hectaresPerDay":[32.1, 45.5, 0]
}
}
Selective fetch examples:
GET /api/dashboard/pilot/snapshot?include=kpi → only kpi
GET /api/dashboard/pilot/snapshot?include=kpi,summary,activeJobs → 3 modules (recommended for periodic polling)
GET /api/dashboard/pilot/snapshot?include=kpi,summary,activeJobs&period=week → activeJobs haSprayed/volume scoped to current week
GET /api/dashboard/pilot/snapshot?include=performance,trend&startDate=2026-05-01&endDate=2026-05-14
Error Cases
| Condition | Status | Error tag |
|---|---|---|
| Missing/invalid JWT | 401 |
not_authorized |
All include values are unrecognised |
409 |
invalid_param |
Date range > 90 days (when trend/performance included) |
409 |
invalid_param |
| Bad date format | 409 |
invalid_param |
Invalid period value (not day/week/month/year) |
409 |
invalid_param |
6 Error Responses
All errors follow the standard AgMission error format:
{
"error": {
".tag": "error_constant_value",
"message": "Human-readable detail (development mode only)"
}
}
| HTTP Status | .tag value |
When it occurs |
|---|---|---|
401 |
not_authorized |
Missing/invalid JWT, or caller not the job owner |
409 |
invalid_param |
Bad date format, range > 90 days, invalid job_id, all include modules unrecognised |
409 |
job_not_found |
Job does not exist |
409 |
status_job_invalid |
Job is not in required status for transition |
7 Data Model Notes
Pilot Scoping
All Application metrics are scoped via the Job's operator field, not byUser on Application:
Job.operator = pilotId → collect jobIds → Application.jobId IN jobIds
Application.byUser is the master Applicator account — it is not the pilot. Never use it
to scope pilot metrics.
Application UTC fields
Time-window filtering uses Application.startDateTimeUTC (the canonical spray-time timestamp),
with Application.endDateTimeUTC available for end-of-flight queries and utcOffset available
for display. The legacy startDateTime / endDateTime strings remain display-only and are not
used for dashboard date arithmetic.
Job._id is a Number
Job._id is an auto-incrementing Number (via mongoose-sequence), not a MongoDB ObjectId.
Frontend must treat jobId values as integers, not hex strings.
Vehicle and Client are User discriminators
Both Vehicle (kind=DEVICE) and Client (kind=CLIENT) are stored in the users MongoDB
collection. The $lookup in activeJobs joins against users for both.
Models and Collections
| Model name in code | Collection | Key dashboard fields |
|---|---|---|
Job |
jobs |
operator, byPuid, ttSprArea, status, client, vehicle |
App (Application) |
applications |
jobId, status, startDateTimeUTC, endDateTimeUTC, utcOffset, totalSprayed, totalFlightTime, totalSprayTime, totalSprLength, totalFlightLength, totalSprayMat, avgSpraySpeed, appRate, avgHdop, flowAccuracyPct |
AppFile |
appfiles |
appId |
AppDetail |
application_details |
fileId, xTrack, sprayHeight, radarAlt |
User |
users |
operator, byPuid (scoping only — no dashboard-specific fields) |
Setting |
settings |
dashboard (custom gauge thresholds per pilot) |
Setting.dashboard
Each pilot's custom gauge thresholds are stored as an optional nested object in their Setting
document (the settings collection, linked by userId). All fields default to undefined
(not set) — the system constants are used when the field is absent.
Setting.dashboard: {
xtGood: Number | undefined // XT ideal threshold (m)
xtMonitor: Number | undefined // XT caution threshold (m)
altTarget: Number | undefined // Altitude target (m)
altGoodBand: Number | undefined // ±band for green zone (m)
altMonitorBand: Number | undefined // ±band for yellow zone (m)
}
System defaults (used when the field is absent):
| Field | Default |
|---|---|
xtGood |
1.0 |
xtMonitor |
3.0 |
altTarget |
3.7 |
altGoodBand |
0.15 |
altMonitorBand |
0.46 |
8 Frontend Integration Guide
Suggested Fetch Strategy
Load the dashboard in two tiers to keep the initial paint fast:
Tier 1 — above the fold (load in parallel on mount)
GET /api/dashboard/pilot/kpi?tz=...
GET /api/dashboard/pilot/summary?tz=...
GET /api/dashboard/pilot/activeJobs?tz=...&period=week
Tier 2 — charts and gauges (load after Tier 1 resolves or in parallel)
GET /api/dashboard/pilot/trend?tz=...&startDate=...&endDate=...
GET /api/dashboard/pilot/performance?tz=...
When the user saves custom thresholds
PUT /api/dashboard/pilot/performance/thresholds { xtGood: 7, xtMonitor: 13 }
→ use the response directly to update the gauge (no extra GET needed)
Period filter — when the user switches Day / Week / Month / Year tab
GET /api/dashboard/pilot/activeJobs?tz=...&period=day
GET /api/dashboard/pilot/activeJobs?tz=...&period=week
GET /api/dashboard/pilot/activeJobs?tz=...&period=month
GET /api/dashboard/pilot/activeJobs?tz=...&period=year
Timezone Snippet
// Read once and reuse
const tz = Intl.DateTimeFormat().resolvedOptions().timeZone; // e.g. "America/Sao_Paulo"
const params = new URLSearchParams({ tz });
fetch(`/api/dashboard/pilot/kpi?${params}`, { headers: { Authorization: `Bearer ${token}` } });
Empty State Handling
Every endpoint returns meaningful zero-filled data when the pilot has no jobs:
- Number fields return
0 - Null fields (
avgXtError,altitudeSource) returnnull - Boolean guard fields (
hasXtData,hasAltitudeData) returnfalse - Array fields (
labels,hoursFlown,hectaresPerDay) return empty[]
The frontend should check hasXtData and hasAltitudeData before rendering gauges, and
render a "No data yet" placeholder instead of a gauge at null.
Progress Bar Colour Logic
Use displayStatus from activeJobs for badge colour, and progressPct for bar width:
displayStatus = "NEW" → grey badge, hide progress bar
displayStatus = "IN_PROGRESS" → blue badge, show progress bar at progressPct%
displayStatus = "COMPLETED" → green badge, show bar at 100%
Mark as Completed Button Visibility
The Complete action is restricted to the Applicator (Job.byPuid), not the Pilot.
If the frontend is used by an Applicator who is reviewing a pilot's job, show the button only
when the raw status === 3 (SPRAYED). After a successful PATCH, update the job in local
state to status = 4, displayStatus = "COMPLETED".
Trend Chart Date Pickers
- Pass
startDate/endDateasYYYY-MM-DDstrings in the user's local timezone. - Maximum selectable range: 90 days.
- Default range on first render: current week (omit both parameters and let the API default).
Filter Parameters Quick Reference
| Scenario | Parameters to use |
|---|---|
| User switches Day/Week/Month/Year tab | period=day|week|month|year on /activeJobs |
| User picks a date range in the trend | startDate + endDate on /trend and /performance |
| Dashboard reset to today | Omit period — all endpoints revert to live defaults |
Polling Strategy
Dashboard widgets show live data that changes as jobs progress and applications are uploaded. The recommended approach is interval-based pull polling — no WebSockets needed for Phase 1.
| Data group | Changes when | Recommended interval |
|---|---|---|
| KPI · Summary · ActiveJobs | Job status changes, apps uploaded | Every 60 s |
| Trend | Application uploaded on the day | Every 5 min |
| Performance | Heavy aggregation, historical | On demand / 5 min |
Guards against excessive requests:
-
Page Visibility API — pause polling when the tab is hidden, restart on focus:
Plain JS (framework-agnostic):
document.addEventListener('visibilitychange', () => { document.hidden ? clearInterval(pollTimer) : (pollTimer = startPolling()); });Angular 9.x with NgRx + RxJS (
@ngrx/effects):import { Injectable, NgZone } from '@angular/core'; import { Actions, createEffect, ofType } from '@ngrx/effects'; import { Store } from '@ngrx/store'; import { fromEvent, merge, of, timer, EMPTY } from 'rxjs'; import { switchMap, map, exhaustMap, takeUntil, filter } from 'rxjs/operators'; import { HttpClient } from '@angular/common/http'; import * as DashboardActions from './dashboard.actions'; @Injectable() export class DashboardPollEffects { /** Emits true when the tab is visible, false when hidden. */ private readonly visibility$ = merge( of(!document.hidden), // initial state fromEvent(document, 'visibilitychange').pipe( map(() => !document.hidden) ) ); /** Core poll stream: 60 s interval, paused while tab is hidden. */ readonly pollSnapshot$ = createEffect(() => this.actions$.pipe( ofType(DashboardActions.startPolling), switchMap(() => this.visibility$.pipe( switchMap(visible => visible ? timer(0, 60_000).pipe( // tick immediately, then every 60 s exhaustMap(() => this.http.get<SnapshotResponse>('/api/dashboard/pilot/snapshot') .pipe(map(data => DashboardActions.snapshotLoaded({ data }))) ) ) : EMPTY // tab hidden — no requests ), takeUntil(this.actions$.pipe(ofType(DashboardActions.stopPolling))) ) ) ) ); /** Stop polling on component destroy (dispatched from ngOnDestroy). */ readonly stopPolling$ = createEffect(() => this.actions$.pipe( ofType(DashboardActions.stopPolling), map(() => DashboardActions.pollingStopped()) ) ); constructor( private actions$: Actions, private http: HttpClient, private ngZone: NgZone ) {} }Corresponding actions (
dashboard.actions.ts):import { createAction, props } from '@ngrx/store'; export const startPolling = createAction('[Dashboard] Start Polling'); export const stopPolling = createAction('[Dashboard] Stop Polling'); export const pollingStopped = createAction('[Dashboard] Polling Stopped'); export const snapshotLoaded = createAction( '[Dashboard] Snapshot Loaded', props<{ data: SnapshotResponse }>() );Component wiring:
// dashboard.component.ts ngOnInit() { this.store.dispatch(DashboardActions.startPolling()); } ngOnDestroy() { this.store.dispatch(DashboardActions.stopPolling()); }Angular 9 note:
createEffectrequires@ngrx/effects≥ 9.x (ships with Angular 9 LTS). TheexhaustMapinside the timer prevents a slow response from queuing duplicate requests — equivalent to theclearIntervalguard in the plain-JS version. -
Batch the live-update group — see proposed
GET /snapshotin §9.4. Three concurrent requests per tick become one, reducing server load 3× at scale. -
Skip Tier 2 on polls —
/trendand/performancerequire heavy aggregation. Only re-fetch them when the user changes the date range or when the tab regains focus after more than 5 min of inactivity.
Gauge Rendering Reference
XT Error gauge (lower is better):
0 m ──────── 1.0 m ──────────── 3.0 m ──────────→
green yellow red
↑ good ↑ monitor
Altitude gauge (target 3.7 m):
← red ── 3.24 ── yellow ── 3.55 ── [3.7] ── 3.85 ── yellow ── 4.16 ── red →
↑ -0.46 ↑ -0.15 target ↑ +0.15 ↑ +0.46
9 Backend Architecture Notes
9.1 Pilot Scope Data Flow Diagram
flowchart LR
A[Authenticated User req uid] --> B[Job Query operator equals uid]
B --> C[Collect Job IDs]
C --> D[Application Query jobId in Job IDs and status 3]
D --> E[KPI Summary Trend Active Jobs Aggregates]
9.2 Endpoint Interaction Diagram
flowchart TD
FE[Frontend Pilot Dashboard] --> K[GET pilot kpi]
FE --> S[GET pilot summary]
FE --> T[GET pilot trend]
FE --> A[GET pilot activeJobs]
FE --> P[GET pilot performance]
FE --> TH[PUT pilot performance thresholds]
FE --> C[PATCH jobs job_id complete]
K --> J[(jobs)]
K --> AP[(applications)]
S --> AP
T --> AP
A --> J
A --> AP
P --> AP
P --> AF[(appfiles)]
P --> AD[(application_details)]
TH --> ST[(settings)]
C --> J
9.3 Performance Query Safety Diagram
flowchart LR
A[Scoped Job IDs] --> B[Processed Applications within date range]
B --> C[AppFile Lookup by appId]
C --> D[ApplicationDetail Match by fileId IN list]
D --> E[Aggregate spray-on records only sprayStat eq 1]
E --> F[Return XT error and altitude averages]
9.4 Periodic Polling Sequence
The sequence below covers the full client lifecycle: two-tier initial load, periodic
polling, and user-driven drilldown. The proposed /snapshot endpoint replaces three
parallel Tier-1 poll requests with one, cutting polling overhead by 3×.
sequenceDiagram
participant FE as Frontend
participant API as Dashboard API
participant DB as MongoDB
Note over FE,DB: 1 — Initial load
par Tier 1 (parallel)
FE->>API: GET /kpi?tz=...
and
FE->>API: GET /summary?tz=...
and
FE->>API: GET /activeJobs?tz=...&period=week
end
API->>DB: Aggregate jobs + applications
DB-->>API: Results
API-->>FE: KPI · Summary · ActiveJobs
par Tier 2 (parallel)
FE->>API: GET /trend?tz=...
and
FE->>API: GET /performance?tz=...
end
API->>DB: Aggregate application + detail records
DB-->>API: Results
API-->>FE: Trend · Performance gauge
Note over FE,DB: 2 — Periodic refresh (tab visible, every 60 s)
loop Poll interval
FE->>API: GET /snapshot?include=kpi,summary,activeJobs&tz=...
API->>DB: Aggregate jobs + applications (lightweight)
DB-->>API: Batch results
API-->>FE: { kpi, summary, activeJobs }
end
Note over FE,DB: trend + performance refreshed on date filter change or interval change or refresh button click only
GET /api/dashboard/pilot/snapshot
A single endpoint that returns any combination of KPI, Summary, ActiveJobs, Performance, and Trend in one round-trip, eliminating the N+1 overhead during periodic polling.
Query parameters: include (module list) + tz + startDate/endDate (for trend/performance modules). See §5.8 for full documentation.
Response shape (all modules):
{
"kpi": { },
"summary": { },
"activeJobs": { },
"performance":{ },
"trend": { }
}
Each nested object has the same shape as the corresponding individual endpoint response.
Recommended polling payload:
?include=kpi,summary,activeJobs— see § Why trend and performance are excluded from periodic polling for the rationale.
Why trend and performance are excluded from periodic polling
The 60 s poll uses ?include=kpi,summary,activeJobs deliberately. trend and performance
are omitted for two independent reasons:
1. Query cost
kpi,summary, andactiveJobsaggregate overapplicationsandjobsonly — collections that are indexed onoperator/jobIdand contain O(thousands) of documents per pilot.performanceadditionally fans out intoapplication_details(potentially billions of rows) via{ fileId: 1 }index lookups — one query per AppFile in the date range.trendruns a per-day aggregation across the fullapplicationscollection for the chosen date window. Wider windows compound the cost.- Running these every 60 s for every open browser tab creates avoidable DB pressure.
2. Data-change frequency
- KPI counters, today's operational totals, and active-job progress change continuously throughout the working day — sub-minute freshness is meaningful.
- Trend charts (daily hectares/hours) and performance gauges (cross-track error, spray height) are computed from uploaded flight records. They only change when a new Application is uploaded, which happens at most a few times per day. There is no value in re-fetching them every 60 s.
When to refresh trend and performance
- On initial page load (already in Tier 2 of the sequence diagram above).
- When the user changes the date filter.
- On tab re-focus after a long absence (
visibilitychangeevent + staleness check). - After the user manually triggers a refresh (for example, by clicking a refresh button or changing the refresh interval from a dropdown list (5/10/30/60 minutes)).
Custom startDate/endDate parameters are a secondary consideration: the frontend would
need to remember the current filter state to include them in a poll, which adds complexity;
but the cost and staleness arguments above are the primary reason for the separation.
Why Job.operator and not Application.byUser
Application.byUser is the master Applicator account that uploaded the file — not the
pilot assigned to fly the job. Using it would mix data from all jobs the Applicator manages,
not just those assigned to this pilot. The correct field is Job.operator (set when a pilot
is assigned to a job).
Why Application.startDateTimeUTC and not the legacy string fields
Application.startDateTime and endDateTime are legacy display fields. They are still computed
during file import by two different code paths, and the two paths produce incompatible values
in both format and timezone semantics. The dashboard now queries the UTC companion fields
instead: startDateTimeUTC, endDateTimeUTC, and utcOffset.
AgNav binary (.nt) files — workers/job_worker.js: computeStartEndDate()
AgNav filename → YYYYMMDD (LOCAL mission date assigned by the device)
GPS seconds-of-day → HHmmss (GPS UTC time-of-day — already UTC, not local)
combined as-is → "20250522T000632"
Result: legacy hybrid string: datePart = local mission date, timePart = GPS UTC
time-of-day. Both parts carry independent semantics — the date is local, the time is UTC.
Converting to a proper UTC Date (startDateTimeUTC) requires a UTC offset derived from the
flight location; see helpers/application_datetime.js: toUtcDateFromAppDateTime() for the
date-shift logic that handles timezone crossings correctly.
SatLoc (.log) files — helpers/satloc_application_processor.js
`
record.gpsTime ← Unix epoch seconds (UTC)
new Date(gpsTime * 1000).toISOString() → "2020-07-29T00:15:38.030Z"
Result: UTC time as ISO 8601 string with Z suffix. The new startDateTimeUTC and
endDateTimeUTC fields store the canonical UTC Date values directly.
Why this makes range queries impossible
A pilot spraying at 10:06 local time (UTC+10) produces two entirely different stored values depending on which file type was uploaded:
| Source | Stored value | Meaning |
|---|---|---|
| AgNav | "20250729T100634" |
legacy hybrid string (local date + UTC time-of-day) |
| SatLoc | "2025-07-29T00:06:34.000Z" |
UTC 00:06 (same moment) |
| Both | startDateTimeUTC / endDateTimeUTC |
canonical UTC Date values for query/filter use |
| Both | utcOffset |
minutes east of UTC, derived from flight location/timezone |
MongoDB string-range queries ($gte/$lt) compare lexicographically. Across these two formats,
the comparison is meaningless — the strings are ordered differently and represent different
timezones. Additionally, String fields cannot use a date index.
createdDate remains the stable server-side ingest timestamp for upload/ingest operational
reports. For dashboard filtering, charting, and per-job application totals, use
startDateTimeUTC / endDateTimeUTC with utcOffset rather than the legacy string fields.
For AgNav records the local timezone is never stored in the legacy string fields, so there is no
way to convert "20250522T000632" to UTC without external context (pilot's location/timezone at
time of flight). The new utcOffset and UTC companion fields address that gap for new and
backfilled applications.
Frontend display guidance
Recommended client-side rule set:
- Use
startDateTimeUTC/endDateTimeUTCas the canonical values for all filtering, sorting, chart bucketing, and API query parameters. - Display local pilot time by taking the UTC field and shifting it by
utcOffsetminutes. - If the UI needs to show both, render the UTC value as the primary canonical timestamp and a
secondary localized label such as
22 May 2025, 10:06using the offset. - If
utcOffsetis missing, fall back to the browser timezone only for display. Do not use the browser timezone for server-side filtering or export logic. - For day-based summaries, derive the day boundary from
startDateTimeUTC+utcOffsetso the displayed day matches the pilot's working day instead of the browser's locale.
Correct field to use
createdDate is still useful when you want to group by upload time instead of spray time. It is a
Date field set by the server at upload time — consistent, UTC, and independent of file type.
Known tradeoff: createdDate is the upload date, not the spray date. A pilot who
completes fieldwork on Monday but uploads on Friday will have those records attributed to
Friday if you filter by upload time. The UTC companion fields are the correct choice for spray
window queries.
ApplicationDetail Query Safety
application_details is a very large collection (potentially billions of rows). The only
index available for dashboard use is { fileId: 1 }. All performance queries follow a
three-step pattern to avoid collection scans:
1. App.find({ jobId: { $in: jobIds }, startDateTimeUTC: { $gte: start, $lt: end } }) → appIds in range
2. AppFile.find({ appId: { $in: appIds } }) → fileIds
3. AppDetail.aggregate([{ $match: { fileId: { $in: fileIds } } }, ...]) → spray-on metrics
Never add additional ApplicationDetail queries without scoping by fileId first.
Why Spray-On Records Only (sprayStat === 3 || sprayStat === 1)
ApplicationDetail records cover the entire flight including taxi, transit, and ferry legs.
During these phases xTrack can be millions of metres from the spray line and altitude
can be hundreds of metres AGL — both meaningless for agronomic gauges. Filtering to
sprayStat === 3 || sprayStat === 1 (pump active or spray active) isolates only the records where the aircraft was
actually spraying, producing accurate XT error and altitude metrics.
Unit Conversions (done server-side)
| Raw storage | Conversion | API field |
|---|---|---|
totalFlightTime |
seconds → hours ÷3600 | flightHours, hoursFlown |
totalFlightLength |
metres → km ÷1000 | operations.distanceTravelledKm |
totalSprLength |
metres → km ÷1000 | operations.distanceSprayedKm |
avgSpraySpeed |
m/s → km/h ×3.6 | avgSpeedKmh |
All other fields (totalSprayed, totalSprayMat, xTrack, sprayHeight, radarAlt)
are stored and returned in their natural units (ha, L, m).
10 Open Decisions
These items affect backend behaviour but have not been finalised by the Product Owner. They are Phase 2 scope and the current implementation uses the defaults noted.
| ID | Question | Current default |
|---|---|---|
| Resolved: open jobs only (NEW/READY/DOWNLOADED/SPRAYED) | ||
| Resolved: Excluded. Only NEW/READY/DOWNLOADED/SPRAYED/COMPLETED shown. | ||
| ACT-Q1 | Should the period filter on Active Jobs also filter the job list (by createdAt), or only the Application totals per job? |
Resolved: both. period filters the job list by createdAt AND scopes Application totals by startDateTimeUTC. tz still controls the calendar boundary calculation. |
Resolved: Any user under the same Applicator account may complete the job, except users with role inspector or client. The check uses req.userInfo.puid (the caller's root Applicator ID), which resolves correctly for the Applicator themselves and all their sub-users (Pilot, co-pilot, etc). |
||
| PERF-1 | When no altitude sensor exists, should the gauge show N/A or be hidden? | Returns hasAltitudeData: false, value null |
| JOBS-1 | Are jobs with operator = null (unassigned) relevant to any view? |
Not included in any pilot-scoped query |
11 Changelog
All revisions target
#3054 — Operational Analytics - Pilot Dashboardunless noted.
Quick-Reference Table
| Version | Date | SVN Rev | Task# | Summary |
|---|---|---|---|---|
| 2.5 | 2026-06-15 | — | #3054 | Endpoint URLs added to §5.1–§5.7; parseDateRange 90-day cap now DST-immune with invalid-date guard |
| 2.4 | 2026-06-08 | — | #3054 | New operations metrics: sprayEfficiencyPct, ferryTimePct, flowAccuracyPct, avgHdop; new Application fields avgHdop/flowAccuracyPct; migration script updated |
| 2.3 | 2026-06-05 | r1227 | #3054 | todayHasData flag in summary; progressPct float precision; createdDate in activeJobs; COMPLETED in KPI assignedJobs; fix avgXtErrorMeters→avgXtError in §5.5 docs |
| 2.2 | 2026-06-03 | — | #3054 | Datetime root-cause fix: AgNav hybrid UTC conversion corrected; backfill --force; docs sync |
| 2.1 | 2026-06-02 | — | #3054 | Distance aggregates: inline streaming calc + 120s/1000m gates; docs/tests alignment |
| 2.0 | 2026-05-28 | — | #3054 | Add GET /snapshot; rename avgXtErrorMeters→avgXtError; resolve Q11; update KPI tests |
| 1.9 | 2026-05-26 | — | #3054 | Remove selectedDate drill-down from all endpoints; remove parseDateWindow helper |
| 1.8 | 2026-05-25 | — | #3054 | Move dashboardSettings → Setting.dashboard; fix completeJob auth (allow sub-users); move completeJob to job controller |
| 1.7 | 2026-05-22 | — | #3054 | Bug fixes: displayStatus underscore (code not applied in v1.3); threshold null reset + cross-field validation |
| 1.6 | 2026-05-22 | r1181 | #3054 | KPI: rename sprayed → sprayedHectares |
| 1.5 | 2026-05-21 | r1176 | #3054 | New PUT /performance/thresholds endpoint; User.dashboardSettings per-pilot threshold store |
| 1.4 | 2026-05-20 | r1173 | #3054 | Performance: date-range mode, spray-on records only, startDate/endDate in response |
| 1.3 | 2026-05-15 | r1155 | #3054 | Breaking: KPI response restructure; period filter on activeJobs; displayStatus typo fix |
| 1.2 | 2026-05-14 | — | #3054 | External baseline (frontend-distributed copy; equivalent to server v1.1) |
| 1.1 | 2026-05-14 | r1144 | #3054 | selectedDate drilldown param on all endpoints; frontend drilldown integration guide |
| 1.0 | 2026-04-29 | r1081 | #3064 | Initial release |
v2.5 — 2026-06-15 (—, #3054)
Endpoint URLs added to §5.1–§5.7
- Every endpoint section now opens with a
**URL**:line (GET/PUT/PATCH+ full path). - Previously only §5.6 and §5.8 had an explicit URL line; §5.1–§5.5 and §5.7 were missing it.
parseDateRange — 90-day cap made DST-immune
- Replaced the millisecond-arithmetic
diffDays(Math.round((endExcl − startUTC) / 86400000)) with a direct calendar-day count from the date strings:(endDay − startDay) / 86400000 + 1using UTC midnight (T00:00:00Z). - This eliminates the theoretical off-by-one risk in DST-observing timezones where the UTC offset of
startDateandendDatecan differ by up to 1 hour, causingMath.roundto produce the wrong integer. - Added an explicit invalid-date guard (
isNaNcheck onstartDay/endDay): date strings that pass theYYYY-MM-DDregex but represent non-existent dates (e.g.2025-02-30) now return409 invalid_paraminstead of producingNaN-based DB queries that bypass the cap silently. - Applies to both
GET /pilot/trendandGET /pilot/performance(both callparseDateRange).
§8 formatting fix
- Tier 2 fetch example: the two endpoint URLs were incorrectly merged onto a single line; restored as two separate lines.
v2.4 — 2026-06-08 (—, #3054)
KPI Cards (/kpi) — three new operations metrics
Added to the operations block (today-scoped) in the KPI response:
| Field | Formula | Notes |
|---|---|---|
sprayEfficiencyPct |
SUM(totalSprayTime) / SUM(totalFlightTime) × 100 |
null when no flight time |
ferryTimePct |
(SUM(totalFlightTime) − SUM(totalSprayTime)) / SUM(totalFlightTime) × 100 |
Complement of sprayEfficiencyPct; both sum to 100 |
flowAccuracyPct |
AVG(Application.flowAccuracyPct) |
null when no sessions with a prescribed rate |
avgHdop |
AVG(Application.avgHdop) |
null when no HDOP data uploaded today |
New Application schema fields
avgHdop— average HDOP across spray-on (sprayStat > 0) records, computed at import time injob_worker.js importDataFiles(). Lower is better (< 1 excellent, 1–2 good, > 5 poor).flowAccuracyPct—(totalSprayMat / totalSprayed / appRate) × 100, computed injob_worker.js work()after all three source fields are available.nullwhen any is zero/absent.
Migration script updates (now in scripts/migrate_applications.js)
processFile()selectsstdHdopand accumulateshdopSum/hdopCountduring spray-on records within valid segments.processApplication()aggregates per-file HDOP intoavgHdopand writes it toApplication.backfillFlowAccuracy()runs as a second pass: a server-side aggregation pipeline update that computesflowAccuracyPctfor all Application documents already havingtotalSprayMat,totalSprayed, andappRate.--forceand--dry-runflags both apply to the new pass.
emptyOps fallback — sprayEfficiencyPct, ferryTimePct, flowAccuracyPct, and avgHdop default to null (not 0) when there are no Application records for today, distinguishing "no data" from a genuine zero.
v2.3 — 2026-06-05 (r1227, #3054)
Daily Summary (/summary) — todayHasData flag
- Added
todayHasData: booleanto the summary response. - When
false(no Application records uploaded for today), alldeltas.*Pctfields are returned asnullinstead of computing a misleading-100%. - When
true, delta calculation is unchanged:round((today − yesterday) / yesterday × 100). - Why: today values of
0because no data was uploaded are indistinguishable from a genuine zero-activity day at the aggregation level. A-100%delta in that state is alarming and misleading. Frontend should checktodayHasDatabefore rendering colored arrows.
Active Jobs (/activeJobs) — createdDate field
- Added
createdDate(ISO 8601 UTC string,nullif absent) to each job object in the response. - Sourced from
Job.createdAt.
Active Jobs (/activeJobs) — progressPct precision fix
progressPctis now a float with up to 2 decimal places (e.g.0.47) instead of an integer rounded value.- Previously
round()truncated any value below0.5%to0, hiding real progress on large jobs with small completions. - Cap and floor behaviour unchanged: clamped to
[0, 100].
KPI Cards (/kpi) — assignedJobs now includes COMPLETED
periods.<p>.assignedJobsnow counts jobs with status NEW / READY / DOWNLOADED / SPRAYED / COMPLETED.- Previously COMPLETED jobs were excluded, causing the count to drop when a job was marked done within the period.
Performance (/performance) — docs align avgXtError field name
- §5.5 response examples and field notes updated from
avgXtErrorMeters→avgXtErrorto match the actual API response (renamed in code at v2.0 / r1227).
v2.2 — 2026-06-03 (—, #3054)
Application Datetime Conversion — Root-Cause Fix
- Fixed
helpers/application_datetime.js: toUtcDateFromAppDateTime()for AgNav hybrid strings where:datePartis local mission date (YYYYMMDD), andtimePartis GPS UTC time-of-day (HHmmss).
- Replaced prior
dayShift = floor((utcSecondsOfDay + offsetSeconds) / 86400)logic with explicit local-day rollover handling:localSecondsOfDay < 0→ shift UTC date-1daylocalSecondsOfDay >= 86400→ shift UTC date+1day- otherwise no date shift
- This removes the systematic +1-day start-time drift seen on western timezones with early UTC start times.
Safety Guard Behavior
- Kept the
startDateTimeUTC > endDateTimeUTCguard inbuildApplicationDateFields()as a fallback safety net for genuinely bad/corrupt data, not as a primary correction path.
Backfill Script Improvements
- Updated datetime backfill (now in
scripts/migrate_applications.js --skip-aggregates):- Added
--forcemode to recompute datetime fields for all apps with legacystartDateTime. - Retained targeted mode for missing/zero/inverted UTC fields.
- Updated script usage/help comments accordingly.
- Added
- Re-ran backfill with
--forceso existing records are recomputed with the corrected formula.
Documentation Updates
- Updated §Overview Backfill Script usage/selection criteria in this document.
- Updated AgNav datetime semantics section to explicitly document hybrid format and date-shift conversion rationale.
v2.1 — 2026-06-02 (—, #3054)
Distance Aggregate Calculation — Efficiency + Correctness Alignment
totalSprLengthandtotalFlightLengthare now computed inline during file-read loops injob_worker(readNTFile,readShapeDataFile,readSatLogAsc) instead of relying on a post-read full-array rescan.- Cross-file boundary segments are now included when shape spray-on/spray-off data are merged, so join-point distance is not dropped.
- Distance validity gates are aligned and explicitly documented:
0 < dt <= 120s(with midnight rollover handling)dist <= 1000m- spray-distance additionally requires spray-on segment (
prev.sprayStat > 0 || curr.sprayStat > 0)
_computeSprLength/_computeFlightLengthremain as fallback helpers and now apply the same gate logic for consistency.
Migration Script Alignment
- Distance calculation in
scripts/migrate_applications.jsuses the same time+distance gates as runtime worker logic.
Documentation + Test Alignment
- Updated metric definitions in this document and
AGGREGATED_FIELDS_CALCULATION.mdto reflect gate rules and inline streaming aggregation. - Dashboard test suite re-run after updates:
66 passing. (with RUN_COMPLETE_TEST=1, DASHBOARD_TEST_JOB_ID set to a job with a long spray leg to confirm the new logic is working as intended)
v2.0 — 2026-05-28 (—, #3054, #3064, #3063)
Add GET /snapshot composite endpoint
- New endpoint
GET /api/dashboard/pilot/snapshotreturns any combination ofkpi,summary,activeJobs,performance,trendin one request ?include=param selects modules (comma-separated); unrecognised module names silently ignored; default returns all modules?startDate/?endDateapply toperformanceandtrendmodules; 90-day cap enforced per module- Shared job/app data fetched once internally — no N+1 database calls
- §5.8 added; §9.4 updated from "Proposed" to "Implemented"
Rename avgXtErrorMeters → avgXtError (§5.5 Performance)
- Field name no longer carries unit suffix for consistency with other fields (
avgSpraySpeed,avgSprayAltitudeMetersare unaffected) - Schema field
Application.avgXtErrorupdated; migration script updated - Performance tests confirm new field name working
Resolve Q11 — complete-job access
- Confirmed: any user under the same Applicator account may complete a job, except
inspectorandclientrole users - §5.7 Authorization table already reflects this; Q11 row struck through in §10
v1.9 — 2026-05-26 (—, #3054)
Remove selectedDate drill-down (all affected endpoints)
- Removed
selectedDatequery parameter from/kpi,/summary,/activeJobs, and/performance - Removed internal
parseDateWindow()helper — it existed solely to serveselectedDate /kpi: always returns live period windows (day / week / month / year / all); periods no longer collapse to a single day/summary: always compares today vs. yesterday; "today" is no longer overridable via query param/activeJobs:periodparam retained;selectedDatepriority chain removed — new priority:period> all-time/performance:startDate/endDateand the current-week default retained;selectedDateoverride removed- §8 Frontend Integration Guide: removed "Drilldown" fetch pattern, removed
Drilldown Snippetcode example, renamed "Drilldown vs. Date Range" table to "Filter Parameters Quick Reference", removedselectedDaterow - §9.4 Sequence diagram: removed "3 — User drilldown" sequence block; removed
selectedDatefrom proposed/snapshotquery params
v1.8 — 2026-05-25 (—, #3054)
Architecture: dashboardSettings renamed to Setting.dashboard; moved from User (no API change)
User.dashboardSettingssubdocument removed frommodel/user.js. The five threshold fields (xtGood,xtMonitor,altTarget,altGoodBand,altMonitorBand) are now stored asSetting.dashboardon the pilot'sSettingdocument in thesettingscollection.- Field renamed from
dashboardSettingstodashboard— theSettingssuffix is redundant given the document already lives in theSettingcollection. - All per-user preferences (measurement units, spray-path options, map colours, etc.) are now
consolidated in the
Settingcollection. Reading theUserdocument for dashboard purposes is no longer required. controllers/dashboard.jsnow usesSetting.findOne/findOneAndUpdate(withupsert: true) instead ofUser.findById/findByIdAndUpdatefor threshold reads and writes.- The API contract, endpoint paths, request/response shapes, and error codes are unchanged.
completeJob auth fix — sub-users allowed
- The ownership check was
job.byPuid.toString() !== req.uid, which rejected Pilot sub-users because their own_id≠byPuid. Changed tojob.byPuid.toString() !== req.userInfo.puidso both the Applicator and any sub-user under that Applicator can complete the job.
completeJob moved to controllers/job.js (no API change)
- Handler relocated from
controllers/dashboard.jstocontrollers/job.jsfor cohesion. Route unchanged:PATCH /api/jobs/:job_id/complete.
v1.7 — 2026-05-22 (—, #3054)
Active Jobs (/activeJobs) — Bug Fix
displayStatusfor READY/DOWNLOADED/SPRAYED jobs was being returned as"IN PROGRESS"(with a space) despite the v1.3 changelog documenting the fix to"IN_PROGRESS"(underscore). The code was not updated in v1.3. Fixed now — this is a breaking change for any consumer that string-compared against"IN PROGRESS".
Save Performance Thresholds (PUT /performance/thresholds) — Bug Fixes
- Passing
nullto reset a threshold field to the system default was silently ignored:$setwith a JavaScriptundefinedvalue is stripped by the MongoDB driver, leaving the stored value unchanged. The endpoint now uses$unsetfornullfields so the stored custom value is correctly removed. - Cross-field validation (
xtMonitor > xtGood,altMonitorBand > altGoodBand) was resolved against system defaults when a field was absent from the request, not the user's currently stored values. A partial update such as{xtMonitor: 3}could pass validation while leaving a stored combination ofxtGood=5,xtMonitor=3(invalid). The endpoint now fetches the storeddashboardSettingsbefore validation and uses the stored values as the fallback.
Documentation
- §5.3 Trend: corrected the note about partial
startDate/endDatepairs — both are silently defaulted (no409) - §5.6 Thresholds: updated constraint description to reflect the improved cross-field validation fallback chain
v1.6 — 2026-05-22 (r1181, #3054)
KPI Cards (/kpi)
- Renamed
periods.<p>.sprayed→periods.<p>.sprayedHectaresacross all period objects (day,week,month,year,all) for clarity and consistency with the field's unit
v1.5 — 2026-05-21 (r1176, #3054)
New Endpoint
- Added
PUT /api/dashboard/pilot/performance/thresholds(§5.6): persists custom XT error and altitude gauge thresholds per pilot; returns effective thresholds after save; supportsnullto reset individual fields to system defaults - Mark Job as Completed moved from §5.6 → §5.7 to accommodate new thresholds endpoint
Data Model
- Added
User.dashboardSettingsoptional subdocument (xtGood,xtMonitor,altTarget,altGoodBand,altMonitorBand) for per-pilot threshold overrides; system defaults used when fields absent - Added
model/user.jsto the Implemented In list (§1)
Frontend Guide
- Added "When the user saves custom thresholds" fetch pattern — response from
PUT /thresholdscan be used directly to update the gauge without a follow-upGET /performance
Diagrams
- Updated endpoint interaction diagram: added
TH[PUT pilot performance thresholds]node withTH → U[(users)]edge
v1.4 — 2026-05-20 (r1173, #3054)
Performance Gauges (/performance) — Breaking Change
- Replaced last-10-files static sample with a date-range query
- Added
startDateandendDatequery params (default: current calendar week Mon–Sun); max range 90 days - Date resolution priority:
selectedDate→startDate+endDate→ current week default - Response now includes
startDateandendDatefields reflecting the effective analysis window - Metrics now computed from spray-on records only (
sprayStat === 1) — transit, taxi, and ferry-flight records are excluded; this eliminates spurious XT error spikes (millions of metres off-line during turns) and AMSL altitude readings from non-spray legs sampleSizenow reflects the number ofAppFilerecords in the date window (was: "up to ~20" files cap)- Updated
hasAltitudeDatafield note: now means "no altitude sensor data in sample" (was: "aircraft has no height sensor")
Backend Notes (§9)
- Added "Why Spray-On Records Only (
sprayStat === 1)" explanatory section - Updated ApplicationDetail query pattern: step 1 now uses
createdDaterange filter instead of.limit(10) - Updated Performance Query Safety diagram to reflect date-range scoping and spray-on filter step
v1.3 — 2026-05-15 (r1155, #3054)
KPI Cards (/kpi) — Breaking Response Shape Change
Old flat top-level fields removed: assignedJobs, assignedHectares, totalSprayed, totalFlightHours, jobCounts, and the historical block.
New structure:
operationsblock (today-scoped):missionsFlown,distanceTravelledKm,distanceSprayedKmdistanceTravelledKmreplacesdistanceKmand now usesApplication.totalFlightLength(all GPS segments including turns) instead oftotalSprLengthdistanceSprayedKmis new — spray-on segments only (Application.totalSprLength / 1000)sprayVolumeLitersremoved fromoperations
periodsblock:day,week,month,year,allsub-objects each containingassignedJobs,assignedHectares,sprayed,flightHoursjobCounts(new,inProgress,completed) present onday,week,monthonly — not onyearorallallperiod added (no time boundary — covers all records for this pilot)
- Added
totalFlightLengthfield tracking toApplicationmodel,AppFile, andjob_worker
Active Jobs (/activeJobs)
- Added
periodquery param (day|week|month|year): filters both the job list (byJob.createdAt) and Application sub-totals (byApplication.startDateTimeUTC) to the selected window;tzcontinues to control the boundary math - Priority chain:
selectedDate>period> all-time (no filter) - Updated field notes for
haSprayedandvolumeAppliedLitersto clarify they are scoped to the active window
Status / Display
- Fixed
displayStatusvalue:"IN PROGRESS"(with space) →"IN_PROGRESS"(underscore) — breaking change for frontend consumers
Unit Conversions (§9)
- Added
totalFlightLength(metres → km ÷ 1000) →operations.distanceTravelledKmto the server-side conversion table
Frontend Guide (§8)
- Added "Period filter" fetch pattern for Day / Week / Month / Year tab switching on
/activeJobs - Expanded Drilldown vs. Date Range table with "User switches Day/Week/Month/Year tab" scenario
- Updated Tier 1 fetch example:
/activeJobs?tz=...&period=week
Open Decisions
ACT-Q1resolved:periodfilter applies to both the job list and Application totals
v1.1 — 2026-05-14 (r1144, #3054)
- Added
selectedDatequery param (YYYY-MM-DD) to/kpi,/summary,/activeJobs,/performancefor single-day drilldown filtering from chart interactions - Added
tzparam documentation to/activeJobsand/performance(previously these endpoints acceptedtzbut it was undocumented) - Added frontend Drilldown Snippet JavaScript code example
- Added Drilldown vs. Date Range reference table
- Fixed DST-safe yesterday derivation in
/summary(no longer a simple 24-hour subtraction)
v1.0 — 2026-04-29 (r1081, #3064)
Initial release implementing all Pilot Dashboard endpoints:
GET /api/dashboard/pilot/kpi— KPI cards with historical breakdownsGET /api/dashboard/pilot/summary— today vs. yesterday metrics with deltasGET /api/dashboard/pilot/trend— daily hours flown and hectares for date rangeGET /api/dashboard/pilot/activeJobs— active job list with progressGET /api/dashboard/pilot/performance— XT error and spray altitude gaugesPATCH /api/jobs/:job_id/complete— transition job SPRAYED → COMPLETED- Postman collection:
docs/Pilot_Dashboard_API.postman_collection.json - Mocha/Chai integration test script:
tests/test_pilot_dashboard_api.js