agmission/server/docs/PILOT_DASHBOARD_API.md

85 KiB
Raw Blame History

Pilot Analytics Dashboard — API Design Reference

Version: 2.5 — (2026-06-15) Scope: Backend API contract for the Pilot Analytics Dashboard. This document is the single source of truth for both backend and frontend/client development.


Table of Contents


1 Overview

The Pilot Dashboard is a read-only analytics surface (plus one write action) scoped to the currently authenticated Pilot user. All endpoints derive the pilot identity from the JWT token — no client-side user ID parameter is accepted or trusted.

Base path: /api/dashboard/pilot

Implemented in:

  • controllers/dashboard.js
  • routes/dashboard.js
  • model/setting.js (for dashboard threshold storage)
  • routes/job.js (for the complete action)

Testing resources:

  • Postman collection: docs/Pilot_Dashboard_API.postman_collection.json
  • Node.js test script: tests/test_pilot_dashboard_api.js
  • Test command: npm run test:dashboard

Dashboard Test Suite Requirements

npm run test:dashboard runs live integration tests (not mocked/unit-only). To execute all non-optional checks reliably:

  1. Server must be running and reachable at https://localhost:4100 (or set PILOT_DASHBOARD_BASE_URL).
  2. DASHBOARD_TEST_TOKEN must be set to a valid Pilot JWT.
  3. For complete-job tests, set:
  • RUN_COMPLETE_TEST=1
  • DASHBOARD_TEST_JOB_ID=<jobId in SPRAYED(3) status>

Optional toggles:

  • RUN_SNAPSHOT_TESTS=0 skips /snapshot endpoint tests.
  • RUN_COMPLETE_TEST=0 (default) skips complete-job mutation tests.

If required env vars are missing, Mocha reports pending (skipped) tests by design.

Migration Script: scripts/migrate_applications.js

Purpose:

  • All-in-one replacement for the former backfill_application_datetimes.js and migrate_app_aggregates.js scripts (both kept as deprecated stubs).
  • Pass 1A — Aggregate metrics: Application.avgSpraySpeed, totalSprLength, totalFlightLength, avgXtError, avgHdop; and AppFile.totalSprLength, AppFile.totalFlightLength.
  • Pass 1B — Datetime fields (for apps that have legacy startDateTime): Application.utcOffset, startDateTimeUTC, endDateTimeUTC.
    • First valid lat/lon coordinate captured during the existing Pass 1A AppDetail stream — no extra DB query needed.
  • Pass 2 — Application.flowAccuracyPct (Application-level only, no AppDetail streaming required).

Selection criteria (non-force mode):

  • Union of both former scripts' $or conditions:
    • Datetime: utcOffset missing or 0, startDateTimeUTC / endDateTimeUTC missing, or startDateTimeUTC > endDateTimeUTC.
    • Aggregates: any of avgSpraySpeed, totalSprLength, totalFlightLength, avgXtError, avgHdop missing/null/0.
  • Use --skip-datetime or --skip-aggregates to restrict to one set of conditions.

Why Apps to process: 0 can happen:

  • All eligible apps are already backfilled, OR
  • Remaining records missing UTC companions do not have legacy startDateTime (script skips those for datetime, but still processes aggregates).

Usage:

node scripts/migrate_applications.js
node scripts/migrate_applications.js --dry-run
node scripts/migrate_applications.js --env ./environment.env --dry-run
node scripts/migrate_applications.js --missing-limit 200
# After a formula fix: reprocess ALL apps to correct previously stored values
node scripts/migrate_applications.js --force
# Tier 1 — most recent 90 days only
node scripts/migrate_applications.js --tier-days=90
# Datetime fields only (fast, uses lightweight findOne)
node scripts/migrate_applications.js --skip-aggregates
# Aggregate metrics only (skip datetime computation)
node scripts/migrate_applications.js --skip-datetime

Operational diagnostics:

  • When Apps to process: 0 and legacy datetime is missing/null on records, prints each affected app and related job context:
    • appId, App-jobId, jobId, jobStatus, appStatus
    • startDateTime, endDateTime
    • appFiles, appDetails, likelyNoDataFiles
  • This makes "zip has no data files" cases directly visible from script output.

All dashboard endpoints share the same scoping logic:

pilot → Job.operator = req.uid → jobIds → Application.jobId IN jobIds

2 Authentication

All routes require a valid JWT bearer token. The checkUser middleware is applied globally in server.js before any route is mounted.

Authorization: Bearer <jwt>

The token payload sets req.uid (user ID string) and req.userInfo on every request. Dashboard endpoints check req.uid and throw 401 Not Authorized if absent.

The dashboard route group (/api/dashboard) does not require a subscription package (checkRqPkgSubscription is not applied there). The complete-job endpoint lives under /api/jobs which does apply that middleware.


3 Status Constants

Job Status (helpers/job_constants.js)

Constant Value Meaning
NEW 0 Job created, not yet prepared
READY 1 Prepared and available for aircraft download
DOWNLOADED 2 Downloaded to aircraft
SPRAYED 3 Application data uploaded, not yet reviewed
COMPLETED 4 Reviewed and marked done by Applicator
INVOICED 5 Invoice issued
ARCHIVED 9 Archived

Display Status (frontend label mapping)

The activeJobs endpoint returns a displayStatus string derived from the raw numeric status:

status value(s) displayStatus Suggested UI treatment
0 (NEW) "NEW" Grey badge, no progress bar
1, 2, 3 "IN_PROGRESS" Blue/active badge, show progress
4 (COMPLETED) "COMPLETED" Green badge, full bar

INVOICED (5) and ARCHIVED (9) are excluded from the active jobs panel entirely.

Application Status

Only status: 3 (processed) applications are counted in all aggregations. Uploading (1), in-progress (2), and error (0) applications are excluded automatically.


4 Timezone Handling

All time-windowed endpoints accept an optional ?tz=<IANA> query parameter.

  • Default: UTC
  • Example: ?tz=America/Sao_Paulo, ?tz=America/Chicago, ?tz=Australia/Brisbane
  • Invalid values silently fall back to UTC — the frontend should always send a valid tz.

What timezone affects:

  • "Today" and "yesterday" day boundaries
  • Current week (MonSun) boundaries
  • Current month and year boundaries
  • Date labels in the trend chart (YYYY-MM-DD)

What timezone does NOT affect:

  • Raw UTC timestamps stored in MongoDB
  • Job createdAt (always UTC)

Recommended frontend behaviour: read the browser's timezone once (Intl.DateTimeFormat().resolvedOptions().timeZone) and pass it on every dashboard request. The API still needs tz to derive calendar boundaries and day labels, even though the Application filters now use the stored UTC fields.


5 Endpoints

5.1 KPI Cards

URL: GET /api/dashboard/pilot/kpi

Returns top-level KPI cards and historical breakdowns for the authenticated pilot.

Query Parameters

Parameter Type Default Description
tz String UTC IANA timezone for period windows

Response 200 OK

{
  "operations": {
    "missionsFlown":       1,
    "distanceTravelledKm": 256.22,
    "distanceSprayedKm":   50.82,
    "sprayEfficiencyPct": 68.40,
    "ferryTimePct":       31.60,
    "flowAccuracyPct":    97.50,
    "avgHdop":             1.20
  },
  "periods": {
    "day": {
      "assignedJobs":      1,
      "assignedHectares":  939.02,
      "sprayedHectares":   148.49,
      "flightHours":       1.53,
      "sprayEfficiencyPct": 68.40,
      "ferryTimePct":      31.60,
      "flowAccuracyPct":   97.50,
      "avgHdop":           1.20,
      "jobCounts":         { "new": 0, "inProgress": 1, "completed": 0 }
    },
    "week": {
      "assignedJobs":      4,
      "assignedHectares":  9807.32,
      "sprayedHectares":   15762.43,
      "flightHours":       18.37,
      "sprayEfficiencyPct": 71.00,
      "ferryTimePct":      29.00,
      "flowAccuracyPct":   96.25,
      "avgHdop":           1.15,
      "jobCounts":         { "new": 1, "inProgress": 3, "completed": 1 }
    },
    "month": {
      "assignedJobs":      7,
      "assignedHectares":  27908.71,
      "sprayedHectares":   22838.75,
      "flightHours":       21.73,
      "sprayEfficiencyPct": 70.50,
      "ferryTimePct":      29.50,
      "flowAccuracyPct":   98.10,
      "avgHdop":           1.08,
      "jobCounts":         { "new": 1, "inProgress": 6, "completed": 4 }
    },
    "year": {
      "assignedJobs":      7,
      "assignedHectares":  27908.71,
      "sprayedHectares":   22838.75,
      "flightHours":       21.73,
      "sprayEfficiencyPct": 69.80,
      "ferryTimePct":      30.20,
      "flowAccuracyPct":   97.50,
      "avgHdop":           1.18,
      "jobCounts":         { "new": 2, "inProgress": 3, "completed": 2 }
    },
    "all": {
      "assignedJobs":      7,
      "assignedHectares":  27908.71,
      "sprayedHectares":   22838.75,
      "flightHours":       21.71,
      "sprayEfficiencyPct": 69.80,
      "ferryTimePct":      30.20,
      "flowAccuracyPct":   97.50,
      "avgHdop":           1.18,
      "jobCounts":         { "new": 2, "inProgress": 3, "completed": 2 }
    }
  }
}

New in v2.4: All efficiency and GPS quality metrics now available in each period, not just today's operations. Pilots can track historical trends in spray efficiency, ferry percentage, flow control accuracy, and GPS health across day/week/month/year/all-time windows.

jobCounts is present on all periods: day, week, month, year, and all. New metrics (sprayEfficiencyPct, ferryTimePct, flowAccuracyPct, avgHdop) are also present on all periods.

Field Notes

Field Unit Notes
operations.missionsFlown count Number of Application records uploaded today (scoped to current day window)
operations.distanceTravelledKm km Sum of Application.totalFlightLength / 1000 for today — all GPS segments including turns (segments with time gap > 120 s or distance > 1000 m excluded)
operations.distanceSprayedKm km Sum of Application.totalSprLength / 1000 for today — spray-on segments only (same time/distance gates as above)
operations.sprayEfficiencyPct % SUM(totalSprayTime) / SUM(totalFlightTime) × 100 for today. null when no flight time. Higher = more time actively spraying vs turning/ferrying
operations.ferryTimePct % (SUM(totalFlightTime) SUM(totalSprayTime)) / SUM(totalFlightTime) × 100 for today. Complement of sprayEfficiencyPct; both sum to 100 when non-null. null when no flight time
operations.flowAccuracyPct % Average of Application.flowAccuracyPct for today — (actual rate / prescribed appRate) × 100 per session. null when no sessions with prescribed rate
operations.avgHdop Average of Application.avgHdop for today — mean HDOP across spray-on records. null when no data. < 1 excellent; 12 good; 25 moderate; > 5 poor
periods.<p>.assignedJobs count Open jobs (NEW / READY / DOWNLOADED / SPRAYED / COMPLETED) created within period <p>
periods.<p>.assignedHectares ha Sum of Job.ttSprArea for jobs created within period <p>
periods.<p>.sprayedHectares ha Sum of Application.totalSprayed for applications uploaded within period <p>
periods.<p>.flightHours hours Sum of Application.totalFlightTime / 3600 for applications uploaded within period <p> (all flight records, same validity rules as totalFlightTime)
periods.<p>.sprayEfficiencyPct % SUM(totalSprayTime) / SUM(totalFlightTime) × 100 for period <p>. null when no flight time. Higher = more time actively spraying vs turning/ferrying
periods.<p>.ferryTimePct % (SUM(totalFlightTime) SUM(totalSprayTime)) / SUM(totalFlightTime) × 100 for period <p>. Complement of spray efficiency; both sum to 100 when non-null. null when no flight time
periods.<p>.flowAccuracyPct % Average of Application.flowAccuracyPct for applications in period <p>. null when no sessions with prescribed rate
periods.<p>.avgHdop Average of Application.avgHdop for period <p>. null when no data. < 1 excellent; 12 good; 25 moderate; > 5 poor
periods.<p>.jobCounts.new count Jobs with status NEW (0) created within that period
periods.<p>.jobCounts.inProgress count Jobs with status READY (1), DOWNLOADED (2), or SPRAYED (3) created within that period
periods.<p>.jobCounts.completed count Jobs with status COMPLETED (4) created within that period

<p> = day | week | month | year | all. The all period has no time boundary — it covers all records for this pilot.

All numeric values are rounded to 2 decimal places.


5.2 Daily Summary

URL: GET /api/dashboard/pilot/summary

Returns today vs. yesterday operational metrics with percentage change deltas.

Query Parameters

Parameter Type Default Description
tz String UTC IANA timezone for day boundaries

Response 200 OK

{
  "today": {
    "hectares":          120.50,
    "flightHours":       3.25,
    "haPerHour":         37.08,
    "avgSpeedKmh":       28.50,
    "sprayVolumeLiters": 2400.00
  },
  "yesterday": {
    "hectares":          95.00,
    "flightHours":       2.80,
    "haPerHour":         33.93,
    "avgSpeedKmh":       26.10,
    "sprayVolumeLiters": 1900.00
  },
  "todayHasData": true,
  "deltas": {
    "hectaresPct":    27,
    "flightHoursPct": 16,
    "haPerHourPct":   9,
    "avgSpeedPct":    9,
    "sprayVolumePct": 26
  }
}

Field Notes

Field Unit Notes
today.haPerHour ha/hr Derived: hectares / flightHours (0 if no flight time)
today.avgSpeedKmh km/h Application.avgSpraySpeed (m/s) × 3.6, averaged
todayHasData bool false when no Application records exist for today. Use this to distinguish "no upload yet" from a real operational drop
deltas.*Pct % round((today yesterday) / yesterday × 100). All fields are null when todayHasData is false
deltas.*Pct = null Either no data uploaded today (todayHasData: false), or yesterday value was 0 (division by zero avoided)

Positive delta = improvement today vs. yesterday. Negative = drop.

Frontend guidance: only render colored delta arrows (red/green) when todayHasData is true and the delta value is non-null. When todayHasData is false, show a neutral "—" or "Awaiting data" state — a -100% would be misleading when the cause is a missing upload, not a real performance drop.


5.3 Trend Charts

URL: GET /api/dashboard/pilot/trend

Returns daily hours flown and hectares sprayed for a date range. Default range is the current calendar week (MondaySunday) in the given timezone.

Query Parameters

Parameter Type Default Description
tz String UTC IANA timezone for day grouping and boundaries
startDate String Monday of this week Start date inclusive. Format: YYYY-MM-DD
endDate String Sunday of this week End date inclusive. Format: YYYY-MM-DD
  • Maximum range: 90 days. Returns 409 if exceeded.
  • If either startDate or endDate is omitted, both are silently defaulted to the current week (MonSun). No 409 is returned for a one-sided pair — the partial value is discarded.

Response 200 OK

{
  "labels":        ["2026-04-27", "2026-04-28", "2026-04-29", "2026-04-30", "2026-05-01", "2026-05-02", "2026-05-03"],
  "hoursFlown":    [2.50, 3.25, 0,    1.80, 0,    0,    0],
  "hectaresPerDay":[80.0, 120.5, 0,   65.0, 0,    0,    0]
}

Field Notes

Field Type Notes
labels String[] One entry per calendar day in YYYY-MM-DD, tz-adjusted
hoursFlown Number[] Parallel array. Days with no activity = 0
hectaresPerDay Number[] Parallel array. Days with no activity = 0

Array contract: all three arrays are always the same length and in the same order. Frontend can zip them: labels[i]hoursFlown[i]hectaresPerDay[i].


5.4 Active Jobs Panel

URL: GET /api/dashboard/pilot/activeJobs

Returns the pilot's active-status jobs with per-job progress and applied totals.

  • Statuses included: NEW (0), READY (1), DOWNLOADED (2), SPRAYED (3), COMPLETED (4)
  • Statuses excluded: INVOICED (5), ARCHIVED (9)
  • Maximum 50 most recent jobs returned (sorted by createdAt descending)

Query Parameters

Parameter Type Default Description
tz String UTC IANA timezone used to compute all period boundaries
period String Time window for both job list and Application totals: day | week | month | year. Filters jobs by createdAt and Application totals by startDateTimeUTC. The tz parameter still controls how those calendar windows are derived.

Parameter priority: period > (none — all-time)

When a period is provided, only jobs created within that window are returned and the haSprayed/volumeAppliedLiters totals are scoped to applications whose startDateTimeUTC falls in the same calendar window. haTotal and progressPct always reflect the job's planned area regardless of the filter. The legacy string fields remain display-only; tz is still needed to translate the requested calendar period into UTC boundaries.

Response 200 OK

{
  "jobs": [
    {
      "jobId":               1042,
      "name":                "North Block — Canola",
      "clientName":          "Sunrise Farms Ltd.",
      "aircraftReg":         "C-FABM",
      "status":              3,
      "displayStatus":       "IN_PROGRESS",
      "createdDate":         "2026-06-01T14:22:00.000Z",
      "haTotal":             250.00,
      "haSprayed":           187.50,
      "progressPct":         75.00,
      "volumeAppliedLiters": 3750.00
    }
  ]
}

Field Notes

Field Type Notes
jobId Number Numeric job ID (Job._id, not a MongoDB ObjectId)
name String Job name. Empty string if not set
clientName String From linked Client user record. Empty if not linked
aircraftReg String Vehicle.tailNumber or Vehicle.unitId. Empty if not linked
status Number Raw numeric status (see constants table)
displayStatus String "NEW" | "IN_PROGRESS" | "COMPLETED"
createdDate Date Job.createdAt — UTC ISO 8601 timestamp of job creation. null if not set
haTotal Number Job.ttSprArea — planned area in ha. 0 if not set
haSprayed Number Sum of Application.totalSprayed for this job within the active window (processed apps only)
progressPct Number 0100 (2 decimal places). 0 if haTotal is 0 or no applications
volumeAppliedLiters Number Sum of Application.totalSprayMat for this job within the active window

progressPct formula: parseFloat(min(100, max(0, haSprayed / haTotal × 100)).toFixed(2)) Returns a float with up to 2 decimal places (e.g. 0.47 for a job under 1% complete). The backend caps at 100 if haSprayed > haTotal.


5.5 Performance Gauges

URL: GET /api/dashboard/pilot/performance

Returns average XT cross-track error and spray altitude gauges for all processed application files within the requested date range.

All values are calculated using spray-on records only (sprayStat === 1), giving meaningful agronomic metrics free from transit/ferry-flight pollution. The average is record-weighted across all individual GPS data points (one row per second of flight) in the period — days with more spray-on time have more influence than days with fewer records.

Note

: ApplicationDetail is a billion-row collection. Queries are strictly bounded to a set of fileId values to use the collection's primary index. Never query this collection without a fileId filter.

Query Parameters

Parameter Type Default Description
tz String UTC IANA timezone string for date boundary calculations
startDate String Monday of current week Range start date YYYY-MM-DD (inclusive)
endDate String Sunday of current week Range end date YYYY-MM-DD (inclusive). Max range: 90 days

Date resolution: startDate+endDate → current week default

Response 200 OK — data available

{
  "startDate": "2026-05-19",
  "endDate":   "2026-05-25",

  "avgXtError": 2.82,
  "xtThreshold": {
    "good":    1.0,
    "monitor": 3.0
  },
  "hasXtData": true,

  "avgSprayAltitudeMeters": 3.62,
  "altitudeSource":         "sprayHeight",
  "altThreshold": {
    "target":      3.7,
    "goodBand":    0.15,
    "monitorBand": 0.46
  },
  "hasAltitudeData": true,
  "sampleSize": 4
}

Response 200 OK — no data (new pilot, no apps uploaded)

{
  "startDate": "2026-05-19",
  "endDate":   "2026-05-25",

  "avgXtError":             null,
  "xtThreshold": { "good": 1.0, "monitor": 3.0 },
  "hasXtData":              false,

  "avgSprayAltitudeMeters": null,
  "altitudeSource":         null,
  "altThreshold": { "target": 3.7, "goodBand": 0.15, "monitorBand": 0.46 },
  "hasAltitudeData":        false,
  "sampleSize":             0
}

Field Notes

Field Unit Notes
startDate String Effective start of the analysis window (YYYY-MM-DD)
endDate String Effective end of the analysis window (YYYY-MM-DD)
avgXtError metres Mean abs(xTrack) across all spray-on GPS records. null if no data
xtThreshold.good metres Below this → green zone
xtThreshold.monitor metres Above this → red zone; between good and monitor → yellow
hasXtData bool false = no valid xTrack readings in sample (show "No data")
avgSprayAltitudeMeters metres Mean spray height from best available sensor, spray-on records only. null if none
altitudeSource String "sprayHeight" (FM dedicated sensor) or "radarAlt" (AGL fallback). null if no data
altThreshold.target metres Ideal spray height (~3.7 m)
altThreshold.goodBand metres ±0.15 m of target → green
altThreshold.monitorBand metres ±0.46 m of target → yellow; outside → red
hasAltitudeData bool false = no altitude sensor data in sample (show "No data")
sampleSize count Number of AppFile records included in the analysis window

Threshold Gauge Logic (frontend)

XT Cross-track error:
  value < good          → green  (acceptable precision)
  good ≤ value < monitor → yellow (monitor)
  value ≥ monitor       → red   (needs attention)

Spray Altitude:
  |value - target| < goodBand    → green
  |value - target| < monitorBand → yellow
  |value - target| ≥ monitorBand → red

Altitude Source Priority

The backend selects the best available altitude sensor in this order:

  1. sprayHeight — dedicated FM spray height sensor (most accurate)
  2. radarAlt — radar altimeter (AGL, good fallback)
  3. No datahasAltitudeData: false, avgSprayAltitudeMeters: null

gpsAlt (AMSL) is not used for spray height — it measures altitude above sea level, not above the crop canopy.

xTrack = 0 values are excluded from the XT average because 0 means "no reading", not "perfectly on track".


5.6 Save Performance Thresholds

Persists custom XT error and altitude gauge thresholds for the authenticated pilot. Values are stored per-user in Setting.dashboard and are automatically applied the next time GET /pilot/performance is called.

All fields are optional — send only the fields you want to change. Pass null for a field to reset it to the system default.

URL: PUT /api/dashboard/pilot/performance/thresholds

Request Body (JSON)

Field Type Default (system) Description
xtGood Number | null 1.0 XT ideal threshold in metres (top of green zone)
xtMonitor Number | null 3.0 XT caution threshold in metres (top of yellow zone)
altTarget Number | null 3.7 Altitude target in metres (centre of altitude gauge)
altGoodBand Number | null 0.15 ±band from target for green zone
altMonitorBand Number | null 0.46 ±band from target for yellow zone

Constraints:

  • All supplied values must be positive finite numbers (> 0).
  • xtMonitor must be greater than xtGood.
  • altMonitorBand must be greater than altGoodBand.
  • Constraints are checked against the effective post-save value for each field: the value being saved in this request → if being reset (null), the system default → the currently stored custom value → the system default. This means partial updates (sending only some fields) are correctly validated against the real in-DB state.

Example Request Body

{
  "xtGood": 7.0,
  "xtMonitor": 13.0
}

Only the two XT fields are sent — altitude thresholds remain unchanged.

Response 200 OK

Returns the effective thresholds after saving (saved value, or system default if not customised):

{
  "xtThreshold": {
    "good":    7.0,
    "monitor": 13.0
  },
  "altThreshold": {
    "target":      3.7,
    "goodBand":    0.15,
    "monitorBand": 0.46
  }
}

The frontend can use this response to immediately update the gauge without a separate GET /performance call.

Reset to Defaults

Pass null to clear a custom value and revert to the system default:

{ "xtGood": null, "xtMonitor": null }

Error Cases

Condition Status Error tag
Missing or invalid JWT 401 not_authorized
Non-positive or non-finite value 409 invalid_param
xtMonitorxtGood (effective values) 409 invalid_param
altMonitorBandaltGoodBand (effective) 409 invalid_param

5.7 Mark Job as Completed

URL: PATCH /api/jobs/:job_id/complete

Transitions a job from SPRAYED (3) to COMPLETED (4).

This endpoint lives under /api/jobs (not /api/dashboard) and requires an active subscription package (enforced by checkRqPkgSubscription middleware).

URL Parameters

Parameter Type Description
job_id Number Numeric job ID

Request Body

None.

Response 200 OK

Returns the full updated job document (with client, operator, and vehicle populated).

{
  "_id":    1042,
  "name":   "North Block — Canola",
  "status": 4,
  "client": { "_id": "...", "name": "Sunrise Farms Ltd." },
  "operator": { "_id": "...", "name": "Jane Pilot" },
  "vehicle":  { "_id": "...", "name": "Agri-One", "tailNumber": "C-FABM" },
  "..."
}

Authorization

The Applicator who owns the job (Job.byPuid) may complete it, as may any sub-user (Pilot) operating under that Applicator. The check compares Job.byPuid against req.userInfo.puid (the caller's root Applicator ID), which resolves correctly for both the Applicator themselves and their sub-users.

Caller type req.userInfo.puid Allowed?
Applicator same as req.uid Yes
Pilot sub-user under the owning Applicator parent's _id Yes
Any other user different from Job.byPuid 401

Error Cases

Condition Status Error tag
job_id is not a valid positive number 409 invalid_param
Job not found 409 job_not_found
Caller is not the job owner (byPuid) 401 not_authorized
Job status is not SPRAYED (3) 409 status_job_invalid

5.8 Snapshot (Composite Dashboard)

Returns multiple dashboard modules in a single request. Eliminates N+1 API calls on initial page load and during periodic polling.

URL: GET /api/dashboard/pilot/snapshot

All sub-modules share the same pilot-scoped job/app data fetch internally — no redundant database queries. Each module uses its own query parameter validation (date-range modules independently validate startDate/endDate; kpi, summary, and activeJobs use tz; activeJobs additionally accepts period for time-scoped sub-totals).

Query Parameters

Parameter Type Default Applies to module(s) Description
include String kpi,summary,activeJobs,performance,trend Comma-separated list of modules to return. Unknown names are silently ignored.
tz String UTC all IANA timezone string
period String (omit = all-time) activeJobs Time window for both job list and Application totals: day | week | month | year. Filters jobs by createdAt and Application totals by startDateTimeUTC. The tz parameter still controls the period boundaries.
startDate String Monday of current week performance, trend YYYY-MM-DD
endDate String Sunday of current week performance, trend YYYY-MM-DD. Max range: 90 days

Valid include values: kpi · summary · activeJobs · performance · trend

Note on invalid_param from snapshot with trend: The 90-day cap applies inside the snapshot just as it does on the standalone /trend endpoint. If startDateendDate spans more than 90 days and trend (or performance) is in the include list, the entire request returns 409 invalid_param. To avoid this, keep the date range ≤ 90 days.

Response 200 OK

Only requested modules are present in the response object. Example with all modules:

{
  "kpi": {
    "operations": {
      "missionsFlown":       2,
      "distanceTravelledKm": 45.2,
      "distanceSprayedKm":   32.1,
      "sprayEfficiencyPct": 71.00,
      "ferryTimePct":       29.00,
      "flowAccuracyPct":    98.50,
      "avgHdop":             1.10
    },
    "periods": {
      "day":   { "assignedJobs": 3, "assignedHectares": 45.5, "sprayedHectares": 32.1, "flightHours": 1.53, "jobCounts": { "new": 1, "inProgress": 1, "completed": 1 } },
      "week":  { "...": "same shape" },
      "month": { "...": "same shape" },
      "year":  { "...": "same shape" },
      "all":   { "...": "same shape" }
    }
  },
  "summary": {
    "today":       { "hectares": 32.1, "flightHours": 1.53, "haPerHour": 21.0, "avgSpeedKmh": 45.2, "sprayVolumeLiters": 256.0 },
    "yesterday":   { "...": "same shape" },
    "todayHasData": true,
    "deltas":      { "hectaresPct": 15, "flightHoursPct": 10, "haPerHourPct": null, "avgSpeedPct": null, "sprayVolumePct": null }
  },
  "activeJobs": {
    "jobs": [
      { "jobId": 42, "name": "North Block", "status": 3, "displayStatus": "IN_PROGRESS", "createdDate": "2026-06-01T14:22:00.000Z", "progressPct": 75.00, "haTotal": 250.0, "haSprayed": 187.5, "volumeAppliedLiters": 3750.0 }
    ]
  },
  "performance": {
    "startDate": "2026-05-19", "endDate": "2026-05-25",
    "avgXtError": 2.82, "xtThreshold": { "good": 1.0, "monitor": 3.0 }, "hasXtData": true,
    "avgSprayAltitudeMeters": 3.62, "altitudeSource": "sprayHeight",
    "altThreshold": { "target": 3.7, "goodBand": 0.15, "monitorBand": 0.46 },
    "hasAltitudeData": true, "sampleSize": 4
  },
  "trend": {
    "labels":        ["2026-05-19", "2026-05-20", "2026-05-21"],
    "hoursFlown":    [1.53, 2.1, 0],
    "hectaresPerDay":[32.1, 45.5, 0]
  }
}

Selective fetch examples:

GET /api/dashboard/pilot/snapshot?include=kpi                                          → only kpi
GET /api/dashboard/pilot/snapshot?include=kpi,summary,activeJobs                       → 3 modules (recommended for periodic polling)
GET /api/dashboard/pilot/snapshot?include=kpi,summary,activeJobs&period=week           → activeJobs haSprayed/volume scoped to current week
GET /api/dashboard/pilot/snapshot?include=performance,trend&startDate=2026-05-01&endDate=2026-05-14

Error Cases

Condition Status Error tag
Missing/invalid JWT 401 not_authorized
All include values are unrecognised 409 invalid_param
Date range > 90 days (when trend/performance included) 409 invalid_param
Bad date format 409 invalid_param
Invalid period value (not day/week/month/year) 409 invalid_param

6 Error Responses

All errors follow the standard AgMission error format:

{
  "error": {
    ".tag": "error_constant_value",
    "message": "Human-readable detail (development mode only)"
  }
}
HTTP Status .tag value When it occurs
401 not_authorized Missing/invalid JWT, or caller not the job owner
409 invalid_param Bad date format, range > 90 days, invalid job_id, all include modules unrecognised
409 job_not_found Job does not exist
409 status_job_invalid Job is not in required status for transition

7 Data Model Notes

Pilot Scoping

All Application metrics are scoped via the Job's operator field, not byUser on Application:

Job.operator = pilotId  →  collect jobIds  →  Application.jobId IN jobIds

Application.byUser is the master Applicator account — it is not the pilot. Never use it to scope pilot metrics.

Application UTC fields

Time-window filtering uses Application.startDateTimeUTC (the canonical spray-time timestamp), with Application.endDateTimeUTC available for end-of-flight queries and utcOffset available for display. The legacy startDateTime / endDateTime strings remain display-only and are not used for dashboard date arithmetic.

Job._id is a Number

Job._id is an auto-incrementing Number (via mongoose-sequence), not a MongoDB ObjectId. Frontend must treat jobId values as integers, not hex strings.

Vehicle and Client are User discriminators

Both Vehicle (kind=DEVICE) and Client (kind=CLIENT) are stored in the users MongoDB collection. The $lookup in activeJobs joins against users for both.

Models and Collections

Model name in code Collection Key dashboard fields
Job jobs operator, byPuid, ttSprArea, status, client, vehicle
App (Application) applications jobId, status, startDateTimeUTC, endDateTimeUTC, utcOffset, totalSprayed, totalFlightTime, totalSprayTime, totalSprLength, totalFlightLength, totalSprayMat, avgSpraySpeed, appRate, avgHdop, flowAccuracyPct
AppFile appfiles appId
AppDetail application_details fileId, xTrack, sprayHeight, radarAlt
User users operator, byPuid (scoping only — no dashboard-specific fields)
Setting settings dashboard (custom gauge thresholds per pilot)

Setting.dashboard

Each pilot's custom gauge thresholds are stored as an optional nested object in their Setting document (the settings collection, linked by userId). All fields default to undefined (not set) — the system constants are used when the field is absent.

Setting.dashboard: {
  xtGood:         Number | undefined   // XT ideal threshold (m)
  xtMonitor:      Number | undefined   // XT caution threshold (m)
  altTarget:      Number | undefined   // Altitude target (m)
  altGoodBand:    Number | undefined   // ±band for green zone (m)
  altMonitorBand: Number | undefined   // ±band for yellow zone (m)
}

System defaults (used when the field is absent):

Field Default
xtGood 1.0
xtMonitor 3.0
altTarget 3.7
altGoodBand 0.15
altMonitorBand 0.46

8 Frontend Integration Guide

Suggested Fetch Strategy

Load the dashboard in two tiers to keep the initial paint fast:

Tier 1 — above the fold (load in parallel on mount)

GET /api/dashboard/pilot/kpi?tz=...
GET /api/dashboard/pilot/summary?tz=...
GET /api/dashboard/pilot/activeJobs?tz=...&period=week

Tier 2 — charts and gauges (load after Tier 1 resolves or in parallel)

GET /api/dashboard/pilot/trend?tz=...&startDate=...&endDate=...
GET /api/dashboard/pilot/performance?tz=...

When the user saves custom thresholds

PUT /api/dashboard/pilot/performance/thresholds   { xtGood: 7, xtMonitor: 13 }
→ use the response directly to update the gauge (no extra GET needed)

Period filter — when the user switches Day / Week / Month / Year tab

GET /api/dashboard/pilot/activeJobs?tz=...&period=day
GET /api/dashboard/pilot/activeJobs?tz=...&period=week
GET /api/dashboard/pilot/activeJobs?tz=...&period=month
GET /api/dashboard/pilot/activeJobs?tz=...&period=year

Timezone Snippet

// Read once and reuse
const tz = Intl.DateTimeFormat().resolvedOptions().timeZone; // e.g. "America/Sao_Paulo"

const params = new URLSearchParams({ tz });
fetch(`/api/dashboard/pilot/kpi?${params}`, { headers: { Authorization: `Bearer ${token}` } });

Empty State Handling

Every endpoint returns meaningful zero-filled data when the pilot has no jobs:

  • Number fields return 0
  • Null fields (avgXtError, altitudeSource) return null
  • Boolean guard fields (hasXtData, hasAltitudeData) return false
  • Array fields (labels, hoursFlown, hectaresPerDay) return empty []

The frontend should check hasXtData and hasAltitudeData before rendering gauges, and render a "No data yet" placeholder instead of a gauge at null.

Progress Bar Colour Logic

Use displayStatus from activeJobs for badge colour, and progressPct for bar width:

displayStatus = "NEW"         → grey badge, hide progress bar
displayStatus = "IN_PROGRESS" → blue badge, show progress bar at progressPct%
displayStatus = "COMPLETED"   → green badge, show bar at 100%

Mark as Completed Button Visibility

The Complete action is restricted to the Applicator (Job.byPuid), not the Pilot. If the frontend is used by an Applicator who is reviewing a pilot's job, show the button only when the raw status === 3 (SPRAYED). After a successful PATCH, update the job in local state to status = 4, displayStatus = "COMPLETED".

Trend Chart Date Pickers

  • Pass startDate/endDate as YYYY-MM-DD strings in the user's local timezone.
  • Maximum selectable range: 90 days.
  • Default range on first render: current week (omit both parameters and let the API default).

Filter Parameters Quick Reference

Scenario Parameters to use
User switches Day/Week/Month/Year tab period=day|week|month|year on /activeJobs
User picks a date range in the trend startDate + endDate on /trend and /performance
Dashboard reset to today Omit period — all endpoints revert to live defaults

Polling Strategy

Dashboard widgets show live data that changes as jobs progress and applications are uploaded. The recommended approach is interval-based pull polling — no WebSockets needed for Phase 1.

Data group Changes when Recommended interval
KPI · Summary · ActiveJobs Job status changes, apps uploaded Every 60 s
Trend Application uploaded on the day Every 5 min
Performance Heavy aggregation, historical On demand / 5 min

Guards against excessive requests:

  1. Page Visibility API — pause polling when the tab is hidden, restart on focus:

    Plain JS (framework-agnostic):

    document.addEventListener('visibilitychange', () => {
      document.hidden ? clearInterval(pollTimer) : (pollTimer = startPolling());
    });
    

    Angular 9.x with NgRx + RxJS (@ngrx/effects):

    import { Injectable, NgZone } from '@angular/core';
    import { Actions, createEffect, ofType } from '@ngrx/effects';
    import { Store } from '@ngrx/store';
    import { fromEvent, merge, of, timer, EMPTY } from 'rxjs';
    import {
      switchMap, map, exhaustMap, takeUntil, filter
    } from 'rxjs/operators';
    import { HttpClient } from '@angular/common/http';
    import * as DashboardActions from './dashboard.actions';
    
    @Injectable()
    export class DashboardPollEffects {
    
      /** Emits true when the tab is visible, false when hidden. */
      private readonly visibility$ = merge(
        of(!document.hidden),                                   // initial state
        fromEvent(document, 'visibilitychange').pipe(
          map(() => !document.hidden)
        )
      );
    
      /** Core poll stream: 60 s interval, paused while tab is hidden. */
      readonly pollSnapshot$ = createEffect(() =>
        this.actions$.pipe(
          ofType(DashboardActions.startPolling),
          switchMap(() =>
            this.visibility$.pipe(
              switchMap(visible =>
                visible
                  ? timer(0, 60_000).pipe(               // tick immediately, then every 60 s
                      exhaustMap(() =>
                        this.http.get<SnapshotResponse>('/api/dashboard/pilot/snapshot')
                          .pipe(map(data => DashboardActions.snapshotLoaded({ data })))
                      )
                    )
                  : EMPTY                                // tab hidden — no requests
              ),
              takeUntil(this.actions$.pipe(ofType(DashboardActions.stopPolling)))
            )
          )
        )
      );
    
      /** Stop polling on component destroy (dispatched from ngOnDestroy). */
      readonly stopPolling$ = createEffect(() =>
        this.actions$.pipe(
          ofType(DashboardActions.stopPolling),
          map(() => DashboardActions.pollingStopped())
        )
      );
    
      constructor(
        private actions$: Actions,
        private http: HttpClient,
        private ngZone: NgZone
      ) {}
    }
    

    Corresponding actions (dashboard.actions.ts):

    import { createAction, props } from '@ngrx/store';
    
    export const startPolling   = createAction('[Dashboard] Start Polling');
    export const stopPolling    = createAction('[Dashboard] Stop Polling');
    export const pollingStopped = createAction('[Dashboard] Polling Stopped');
    export const snapshotLoaded = createAction(
      '[Dashboard] Snapshot Loaded',
      props<{ data: SnapshotResponse }>()
    );
    

    Component wiring:

    // dashboard.component.ts
    ngOnInit()    { this.store.dispatch(DashboardActions.startPolling()); }
    ngOnDestroy() { this.store.dispatch(DashboardActions.stopPolling()); }
    

    Angular 9 note: createEffect requires @ngrx/effects ≥ 9.x (ships with Angular 9 LTS). The exhaustMap inside the timer prevents a slow response from queuing duplicate requests — equivalent to the clearInterval guard in the plain-JS version.

  2. Batch the live-update group — see proposed GET /snapshot in §9.4. Three concurrent requests per tick become one, reducing server load 3× at scale.

  3. Skip Tier 2 on polls/trend and /performance require heavy aggregation. Only re-fetch them when the user changes the date range or when the tab regains focus after more than 5 min of inactivity.


Gauge Rendering Reference

XT Error gauge (lower is better):
  0 m ──────── 1.0 m ──────────── 3.0 m ──────────→
       green          yellow              red
       ↑ good          ↑ monitor

Altitude gauge (target 3.7 m):
  ← red ── 3.24 ── yellow ── 3.55 ── [3.7] ── 3.85 ── yellow ── 4.16 ── red →
           ↑ -0.46          ↑ -0.15   target   ↑ +0.15           ↑ +0.46

9 Backend Architecture Notes

9.1 Pilot Scope Data Flow Diagram

flowchart LR
  A[Authenticated User req uid] --> B[Job Query operator equals uid]
  B --> C[Collect Job IDs]
  C --> D[Application Query jobId in Job IDs and status 3]
  D --> E[KPI Summary Trend Active Jobs Aggregates]

9.2 Endpoint Interaction Diagram

flowchart TD
  FE[Frontend Pilot Dashboard] --> K[GET pilot kpi]
  FE --> S[GET pilot summary]
  FE --> T[GET pilot trend]
  FE --> A[GET pilot activeJobs]
  FE --> P[GET pilot performance]
  FE --> TH[PUT pilot performance thresholds]
  FE --> C[PATCH jobs job_id complete]

  K --> J[(jobs)]
  K --> AP[(applications)]
  S --> AP
  T --> AP
  A --> J
  A --> AP
  P --> AP
  P --> AF[(appfiles)]
  P --> AD[(application_details)]
  TH --> ST[(settings)]
  C --> J

9.3 Performance Query Safety Diagram

flowchart LR
  A[Scoped Job IDs] --> B[Processed Applications within date range]
  B --> C[AppFile Lookup by appId]
  C --> D[ApplicationDetail Match by fileId IN list]
  D --> E[Aggregate spray-on records only sprayStat eq 1]
  E --> F[Return XT error and altitude averages]

9.4 Periodic Polling Sequence

The sequence below covers the full client lifecycle: two-tier initial load, periodic polling, and user-driven drilldown. The proposed /snapshot endpoint replaces three parallel Tier-1 poll requests with one, cutting polling overhead by 3×.

sequenceDiagram
  participant FE as Frontend
  participant API as Dashboard API
  participant DB as MongoDB

  Note over FE,DB: 1 — Initial load
  par Tier 1 (parallel)
    FE->>API: GET /kpi?tz=...
  and
    FE->>API: GET /summary?tz=...
  and
    FE->>API: GET /activeJobs?tz=...&period=week
  end
  API->>DB: Aggregate jobs + applications
  DB-->>API: Results
  API-->>FE: KPI · Summary · ActiveJobs

  par Tier 2 (parallel)
    FE->>API: GET /trend?tz=...
  and
    FE->>API: GET /performance?tz=...
  end
  API->>DB: Aggregate application + detail records
  DB-->>API: Results
  API-->>FE: Trend · Performance gauge

  Note over FE,DB: 2 — Periodic refresh (tab visible, every 60 s)
  loop Poll interval
    FE->>API: GET /snapshot?include=kpi,summary,activeJobs&tz=...
    API->>DB: Aggregate jobs + applications (lightweight)
    DB-->>API: Batch results
    API-->>FE: { kpi, summary, activeJobs }
  end
  Note over FE,DB: trend + performance refreshed on date filter change or interval change or refresh button click only

GET /api/dashboard/pilot/snapshot

A single endpoint that returns any combination of KPI, Summary, ActiveJobs, Performance, and Trend in one round-trip, eliminating the N+1 overhead during periodic polling.

Query parameters: include (module list) + tz + startDate/endDate (for trend/performance modules). See §5.8 for full documentation.

Response shape (all modules):

{
  "kpi":        { },
  "summary":    { },
  "activeJobs": { },
  "performance":{ },
  "trend":      { }
}

Each nested object has the same shape as the corresponding individual endpoint response.

Recommended polling payload: ?include=kpi,summary,activeJobs — see § Why trend and performance are excluded from periodic polling for the rationale.

Why trend and performance are excluded from periodic polling

The 60 s poll uses ?include=kpi,summary,activeJobs deliberately. trend and performance are omitted for two independent reasons:

1. Query cost

  • kpi, summary, and activeJobs aggregate over applications and jobs only — collections that are indexed on operator/jobId and contain O(thousands) of documents per pilot.
  • performance additionally fans out into application_details (potentially billions of rows) via { fileId: 1 } index lookups — one query per AppFile in the date range.
  • trend runs a per-day aggregation across the full applications collection for the chosen date window. Wider windows compound the cost.
  • Running these every 60 s for every open browser tab creates avoidable DB pressure.

2. Data-change frequency

  • KPI counters, today's operational totals, and active-job progress change continuously throughout the working day — sub-minute freshness is meaningful.
  • Trend charts (daily hectares/hours) and performance gauges (cross-track error, spray height) are computed from uploaded flight records. They only change when a new Application is uploaded, which happens at most a few times per day. There is no value in re-fetching them every 60 s.

When to refresh trend and performance

  • On initial page load (already in Tier 2 of the sequence diagram above).
  • When the user changes the date filter.
  • On tab re-focus after a long absence (visibilitychange event + staleness check).
  • After the user manually triggers a refresh (for example, by clicking a refresh button or changing the refresh interval from a dropdown list (5/10/30/60 minutes)).

Custom startDate/endDate parameters are a secondary consideration: the frontend would need to remember the current filter state to include them in a poll, which adds complexity; but the cost and staleness arguments above are the primary reason for the separation.


Why Job.operator and not Application.byUser

Application.byUser is the master Applicator account that uploaded the file — not the pilot assigned to fly the job. Using it would mix data from all jobs the Applicator manages, not just those assigned to this pilot. The correct field is Job.operator (set when a pilot is assigned to a job).

Why Application.startDateTimeUTC and not the legacy string fields

Application.startDateTime and endDateTime are legacy display fields. They are still computed during file import by two different code paths, and the two paths produce incompatible values in both format and timezone semantics. The dashboard now queries the UTC companion fields instead: startDateTimeUTC, endDateTimeUTC, and utcOffset.

AgNav binary (.nt) files — workers/job_worker.js: computeStartEndDate()

AgNav filename      →  YYYYMMDD  (LOCAL mission date assigned by the device)
GPS seconds-of-day →  HHmmss    (GPS UTC time-of-day — already UTC, not local)

combined as-is  →  "20250522T000632"

Result: legacy hybrid string: datePart = local mission date, timePart = GPS UTC time-of-day. Both parts carry independent semantics — the date is local, the time is UTC. Converting to a proper UTC Date (startDateTimeUTC) requires a UTC offset derived from the flight location; see helpers/application_datetime.js: toUtcDateFromAppDateTime() for the date-shift logic that handles timezone crossings correctly.

SatLoc (.log) files — helpers/satloc_application_processor.js

`

record.gpsTime  ←  Unix epoch seconds (UTC)
new Date(gpsTime * 1000).toISOString()  →  "2020-07-29T00:15:38.030Z"

Result: UTC time as ISO 8601 string with Z suffix. The new startDateTimeUTC and endDateTimeUTC fields store the canonical UTC Date values directly.

Why this makes range queries impossible

A pilot spraying at 10:06 local time (UTC+10) produces two entirely different stored values depending on which file type was uploaded:

Source Stored value Meaning
AgNav "20250729T100634" legacy hybrid string (local date + UTC time-of-day)
SatLoc "2025-07-29T00:06:34.000Z" UTC 00:06 (same moment)
Both startDateTimeUTC / endDateTimeUTC canonical UTC Date values for query/filter use
Both utcOffset minutes east of UTC, derived from flight location/timezone

MongoDB string-range queries ($gte/$lt) compare lexicographically. Across these two formats, the comparison is meaningless — the strings are ordered differently and represent different timezones. Additionally, String fields cannot use a date index.

createdDate remains the stable server-side ingest timestamp for upload/ingest operational reports. For dashboard filtering, charting, and per-job application totals, use startDateTimeUTC / endDateTimeUTC with utcOffset rather than the legacy string fields.

For AgNav records the local timezone is never stored in the legacy string fields, so there is no way to convert "20250522T000632" to UTC without external context (pilot's location/timezone at time of flight). The new utcOffset and UTC companion fields address that gap for new and backfilled applications.

Frontend display guidance

Recommended client-side rule set:

  1. Use startDateTimeUTC / endDateTimeUTC as the canonical values for all filtering, sorting, chart bucketing, and API query parameters.
  2. Display local pilot time by taking the UTC field and shifting it by utcOffset minutes.
  3. If the UI needs to show both, render the UTC value as the primary canonical timestamp and a secondary localized label such as 22 May 2025, 10:06 using the offset.
  4. If utcOffset is missing, fall back to the browser timezone only for display. Do not use the browser timezone for server-side filtering or export logic.
  5. For day-based summaries, derive the day boundary from startDateTimeUTC + utcOffset so the displayed day matches the pilot's working day instead of the browser's locale.

Correct field to use

createdDate is still useful when you want to group by upload time instead of spray time. It is a Date field set by the server at upload time — consistent, UTC, and independent of file type.

Known tradeoff: createdDate is the upload date, not the spray date. A pilot who completes fieldwork on Monday but uploads on Friday will have those records attributed to Friday if you filter by upload time. The UTC companion fields are the correct choice for spray window queries.

ApplicationDetail Query Safety

application_details is a very large collection (potentially billions of rows). The only index available for dashboard use is { fileId: 1 }. All performance queries follow a three-step pattern to avoid collection scans:

1. App.find({ jobId: { $in: jobIds }, startDateTimeUTC: { $gte: start, $lt: end } })  → appIds in range
2. AppFile.find({ appId: { $in: appIds } })                                       → fileIds
3. AppDetail.aggregate([{ $match: { fileId: { $in: fileIds } } }, ...])           → spray-on metrics

Never add additional ApplicationDetail queries without scoping by fileId first.

Why Spray-On Records Only (sprayStat === 3 || sprayStat === 1)

ApplicationDetail records cover the entire flight including taxi, transit, and ferry legs. During these phases xTrack can be millions of metres from the spray line and altitude can be hundreds of metres AGL — both meaningless for agronomic gauges. Filtering to sprayStat === 3 || sprayStat === 1 (pump active or spray active) isolates only the records where the aircraft was actually spraying, producing accurate XT error and altitude metrics.

Unit Conversions (done server-side)

Raw storage Conversion API field
totalFlightTime seconds → hours ÷3600 flightHours, hoursFlown
totalFlightLength metres → km ÷1000 operations.distanceTravelledKm
totalSprLength metres → km ÷1000 operations.distanceSprayedKm
avgSpraySpeed m/s → km/h ×3.6 avgSpeedKmh

All other fields (totalSprayed, totalSprayMat, xTrack, sprayHeight, radarAlt) are stored and returned in their natural units (ha, L, m).


10 Open Decisions

These items affect backend behaviour but have not been finalised by the Product Owner. They are Phase 2 scope and the current implementation uses the defaults noted.

ID Question Current default
KPI-1 Does "Assigned Jobs" count all-time or just open/current-season jobs? Resolved: open jobs only (NEW/READY/DOWNLOADED/SPRAYED)
ACT-1 Should INVOICED (5) jobs appear in the Active Jobs panel? Resolved: Excluded. Only NEW/READY/DOWNLOADED/SPRAYED/COMPLETED shown.
ACT-Q1 Should the period filter on Active Jobs also filter the job list (by createdAt), or only the Application totals per job? Resolved: both. period filters the job list by createdAt AND scopes Application totals by startDateTimeUTC. tz still controls the calendar boundary calculation.
Q11 Can the Pilot trigger the complete action, or Applicator-only? Resolved: Any user under the same Applicator account may complete the job, except users with role inspector or client. The check uses req.userInfo.puid (the caller's root Applicator ID), which resolves correctly for the Applicator themselves and all their sub-users (Pilot, co-pilot, etc).
PERF-1 When no altitude sensor exists, should the gauge show N/A or be hidden? Returns hasAltitudeData: false, value null
JOBS-1 Are jobs with operator = null (unassigned) relevant to any view? Not included in any pilot-scoped query

11 Changelog

All revisions target #3054 — Operational Analytics - Pilot Dashboard unless noted.

Quick-Reference Table

Version Date SVN Rev Task# Summary
2.5 2026-06-15 #3054 Endpoint URLs added to §5.1§5.7; parseDateRange 90-day cap now DST-immune with invalid-date guard
2.4 2026-06-08 #3054 New operations metrics: sprayEfficiencyPct, ferryTimePct, flowAccuracyPct, avgHdop; new Application fields avgHdop/flowAccuracyPct; migration script updated
2.3 2026-06-05 r1227 #3054 todayHasData flag in summary; progressPct float precision; createdDate in activeJobs; COMPLETED in KPI assignedJobs; fix avgXtErrorMetersavgXtError in §5.5 docs
2.2 2026-06-03 #3054 Datetime root-cause fix: AgNav hybrid UTC conversion corrected; backfill --force; docs sync
2.1 2026-06-02 #3054 Distance aggregates: inline streaming calc + 120s/1000m gates; docs/tests alignment
2.0 2026-05-28 #3054 Add GET /snapshot; rename avgXtErrorMetersavgXtError; resolve Q11; update KPI tests
1.9 2026-05-26 #3054 Remove selectedDate drill-down from all endpoints; remove parseDateWindow helper
1.8 2026-05-25 #3054 Move dashboardSettingsSetting.dashboard; fix completeJob auth (allow sub-users); move completeJob to job controller
1.7 2026-05-22 #3054 Bug fixes: displayStatus underscore (code not applied in v1.3); threshold null reset + cross-field validation
1.6 2026-05-22 r1181 #3054 KPI: rename sprayedsprayedHectares
1.5 2026-05-21 r1176 #3054 New PUT /performance/thresholds endpoint; User.dashboardSettings per-pilot threshold store
1.4 2026-05-20 r1173 #3054 Performance: date-range mode, spray-on records only, startDate/endDate in response
1.3 2026-05-15 r1155 #3054 Breaking: KPI response restructure; period filter on activeJobs; displayStatus typo fix
1.2 2026-05-14 #3054 External baseline (frontend-distributed copy; equivalent to server v1.1)
1.1 2026-05-14 r1144 #3054 selectedDate drilldown param on all endpoints; frontend drilldown integration guide
1.0 2026-04-29 r1081 #3064 Initial release

v2.5 — 2026-06-15 (—, #3054)

Endpoint URLs added to §5.1§5.7

  • Every endpoint section now opens with a **URL**: line (GET/PUT/PATCH + full path).
  • Previously only §5.6 and §5.8 had an explicit URL line; §5.1§5.5 and §5.7 were missing it.

parseDateRange — 90-day cap made DST-immune

  • Replaced the millisecond-arithmetic diffDays (Math.round((endExcl startUTC) / 86400000)) with a direct calendar-day count from the date strings: (endDay startDay) / 86400000 + 1 using UTC midnight (T00:00:00Z).
  • This eliminates the theoretical off-by-one risk in DST-observing timezones where the UTC offset of startDate and endDate can differ by up to 1 hour, causing Math.round to produce the wrong integer.
  • Added an explicit invalid-date guard (isNaN check on startDay/endDay): date strings that pass the YYYY-MM-DD regex but represent non-existent dates (e.g. 2025-02-30) now return 409 invalid_param instead of producing NaN-based DB queries that bypass the cap silently.
  • Applies to both GET /pilot/trend and GET /pilot/performance (both call parseDateRange).

§8 formatting fix

  • Tier 2 fetch example: the two endpoint URLs were incorrectly merged onto a single line; restored as two separate lines.

v2.4 — 2026-06-08 (—, #3054)

KPI Cards (/kpi) — three new operations metrics

Added to the operations block (today-scoped) in the KPI response:

Field Formula Notes
sprayEfficiencyPct SUM(totalSprayTime) / SUM(totalFlightTime) × 100 null when no flight time
ferryTimePct (SUM(totalFlightTime) SUM(totalSprayTime)) / SUM(totalFlightTime) × 100 Complement of sprayEfficiencyPct; both sum to 100
flowAccuracyPct AVG(Application.flowAccuracyPct) null when no sessions with a prescribed rate
avgHdop AVG(Application.avgHdop) null when no HDOP data uploaded today

New Application schema fields

  • avgHdop — average HDOP across spray-on (sprayStat > 0) records, computed at import time in job_worker.js importDataFiles(). Lower is better (< 1 excellent, 12 good, > 5 poor).
  • flowAccuracyPct(totalSprayMat / totalSprayed / appRate) × 100, computed in job_worker.js work() after all three source fields are available. null when any is zero/absent.

Migration script updates (now in scripts/migrate_applications.js)

  • processFile() selects stdHdop and accumulates hdopSum/hdopCount during spray-on records within valid segments.
  • processApplication() aggregates per-file HDOP into avgHdop and writes it to Application.
  • backfillFlowAccuracy() runs as a second pass: a server-side aggregation pipeline update that computes flowAccuracyPct for all Application documents already having totalSprayMat, totalSprayed, and appRate.
  • --force and --dry-run flags both apply to the new pass.

emptyOps fallbacksprayEfficiencyPct, ferryTimePct, flowAccuracyPct, and avgHdop default to null (not 0) when there are no Application records for today, distinguishing "no data" from a genuine zero.


v2.3 — 2026-06-05 (r1227, #3054)

Daily Summary (/summary) — todayHasData flag

  • Added todayHasData: boolean to the summary response.
  • When false (no Application records uploaded for today), all deltas.*Pct fields are returned as null instead of computing a misleading -100%.
  • When true, delta calculation is unchanged: round((today yesterday) / yesterday × 100).
  • Why: today values of 0 because no data was uploaded are indistinguishable from a genuine zero-activity day at the aggregation level. A -100% delta in that state is alarming and misleading. Frontend should check todayHasData before rendering colored arrows.

Active Jobs (/activeJobs) — createdDate field

  • Added createdDate (ISO 8601 UTC string, null if absent) to each job object in the response.
  • Sourced from Job.createdAt.

Active Jobs (/activeJobs) — progressPct precision fix

  • progressPct is now a float with up to 2 decimal places (e.g. 0.47) instead of an integer rounded value.
  • Previously round() truncated any value below 0.5% to 0, hiding real progress on large jobs with small completions.
  • Cap and floor behaviour unchanged: clamped to [0, 100].

KPI Cards (/kpi) — assignedJobs now includes COMPLETED

  • periods.<p>.assignedJobs now counts jobs with status NEW / READY / DOWNLOADED / SPRAYED / COMPLETED.
  • Previously COMPLETED jobs were excluded, causing the count to drop when a job was marked done within the period.

Performance (/performance) — docs align avgXtError field name

  • §5.5 response examples and field notes updated from avgXtErrorMetersavgXtError to match the actual API response (renamed in code at v2.0 / r1227).

v2.2 — 2026-06-03 (—, #3054)

Application Datetime Conversion — Root-Cause Fix

  • Fixed helpers/application_datetime.js: toUtcDateFromAppDateTime() for AgNav hybrid strings where:
    • datePart is local mission date (YYYYMMDD), and
    • timePart is GPS UTC time-of-day (HHmmss).
  • Replaced prior dayShift = floor((utcSecondsOfDay + offsetSeconds) / 86400) logic with explicit local-day rollover handling:
    • localSecondsOfDay < 0 → shift UTC date -1 day
    • localSecondsOfDay >= 86400 → shift UTC date +1 day
    • otherwise no date shift
  • This removes the systematic +1-day start-time drift seen on western timezones with early UTC start times.

Safety Guard Behavior

  • Kept the startDateTimeUTC > endDateTimeUTC guard in buildApplicationDateFields() as a fallback safety net for genuinely bad/corrupt data, not as a primary correction path.

Backfill Script Improvements

  • Updated datetime backfill (now in scripts/migrate_applications.js --skip-aggregates):
    • Added --force mode to recompute datetime fields for all apps with legacy startDateTime.
    • Retained targeted mode for missing/zero/inverted UTC fields.
    • Updated script usage/help comments accordingly.
  • Re-ran backfill with --force so existing records are recomputed with the corrected formula.

Documentation Updates

  • Updated §Overview Backfill Script usage/selection criteria in this document.
  • Updated AgNav datetime semantics section to explicitly document hybrid format and date-shift conversion rationale.

v2.1 — 2026-06-02 (—, #3054)

Distance Aggregate Calculation — Efficiency + Correctness Alignment

  • totalSprLength and totalFlightLength are now computed inline during file-read loops in job_worker (readNTFile, readShapeDataFile, readSatLogAsc) instead of relying on a post-read full-array rescan.
  • Cross-file boundary segments are now included when shape spray-on/spray-off data are merged, so join-point distance is not dropped.
  • Distance validity gates are aligned and explicitly documented:
    • 0 < dt <= 120s (with midnight rollover handling)
    • dist <= 1000m
    • spray-distance additionally requires spray-on segment (prev.sprayStat > 0 || curr.sprayStat > 0)
  • _computeSprLength / _computeFlightLength remain as fallback helpers and now apply the same gate logic for consistency.

Migration Script Alignment

  • Distance calculation in scripts/migrate_applications.js uses the same time+distance gates as runtime worker logic.

Documentation + Test Alignment

  • Updated metric definitions in this document and AGGREGATED_FIELDS_CALCULATION.md to reflect gate rules and inline streaming aggregation.
  • Dashboard test suite re-run after updates: 66 passing. (with RUN_COMPLETE_TEST=1, DASHBOARD_TEST_JOB_ID set to a job with a long spray leg to confirm the new logic is working as intended)

v2.0 — 2026-05-28 (—, #3054, #3064, #3063)

Add GET /snapshot composite endpoint

  • New endpoint GET /api/dashboard/pilot/snapshot returns any combination of kpi, summary, activeJobs, performance, trend in one request
  • ?include= param selects modules (comma-separated); unrecognised module names silently ignored; default returns all modules
  • ?startDate/?endDate apply to performance and trend modules; 90-day cap enforced per module
  • Shared job/app data fetched once internally — no N+1 database calls
  • §5.8 added; §9.4 updated from "Proposed" to "Implemented"

Rename avgXtErrorMetersavgXtError (§5.5 Performance)

  • Field name no longer carries unit suffix for consistency with other fields (avgSpraySpeed, avgSprayAltitudeMeters are unaffected)
  • Schema field Application.avgXtError updated; migration script updated
  • Performance tests confirm new field name working

Resolve Q11 — complete-job access

  • Confirmed: any user under the same Applicator account may complete a job, except inspector and client role users
  • §5.7 Authorization table already reflects this; Q11 row struck through in §10

v1.9 — 2026-05-26 (—, #3054)

Remove selectedDate drill-down (all affected endpoints)

  • Removed selectedDate query parameter from /kpi, /summary, /activeJobs, and /performance
  • Removed internal parseDateWindow() helper — it existed solely to serve selectedDate
  • /kpi: always returns live period windows (day / week / month / year / all); periods no longer collapse to a single day
  • /summary: always compares today vs. yesterday; "today" is no longer overridable via query param
  • /activeJobs: period param retained; selectedDate priority chain removed — new priority: period > all-time
  • /performance: startDate/endDate and the current-week default retained; selectedDate override removed
  • §8 Frontend Integration Guide: removed "Drilldown" fetch pattern, removed Drilldown Snippet code example, renamed "Drilldown vs. Date Range" table to "Filter Parameters Quick Reference", removed selectedDate row
  • §9.4 Sequence diagram: removed "3 — User drilldown" sequence block; removed selectedDate from proposed /snapshot query params

v1.8 — 2026-05-25 (—, #3054)

Architecture: dashboardSettings renamed to Setting.dashboard; moved from User (no API change)

  • User.dashboardSettings subdocument removed from model/user.js. The five threshold fields (xtGood, xtMonitor, altTarget, altGoodBand, altMonitorBand) are now stored as Setting.dashboard on the pilot's Setting document in the settings collection.
  • Field renamed from dashboardSettings to dashboard — the Settings suffix is redundant given the document already lives in the Setting collection.
  • All per-user preferences (measurement units, spray-path options, map colours, etc.) are now consolidated in the Setting collection. Reading the User document for dashboard purposes is no longer required.
  • controllers/dashboard.js now uses Setting.findOne/findOneAndUpdate (with upsert: true) instead of User.findById/findByIdAndUpdate for threshold reads and writes.
  • The API contract, endpoint paths, request/response shapes, and error codes are unchanged.

completeJob auth fix — sub-users allowed

  • The ownership check was job.byPuid.toString() !== req.uid, which rejected Pilot sub-users because their own _idbyPuid. Changed to job.byPuid.toString() !== req.userInfo.puid so both the Applicator and any sub-user under that Applicator can complete the job.

completeJob moved to controllers/job.js (no API change)

  • Handler relocated from controllers/dashboard.js to controllers/job.js for cohesion. Route unchanged: PATCH /api/jobs/:job_id/complete.

v1.7 — 2026-05-22 (—, #3054)

Active Jobs (/activeJobs) — Bug Fix

  • displayStatus for READY/DOWNLOADED/SPRAYED jobs was being returned as "IN PROGRESS" (with a space) despite the v1.3 changelog documenting the fix to "IN_PROGRESS" (underscore). The code was not updated in v1.3. Fixed now — this is a breaking change for any consumer that string-compared against "IN PROGRESS".

Save Performance Thresholds (PUT /performance/thresholds) — Bug Fixes

  • Passing null to reset a threshold field to the system default was silently ignored: $set with a JavaScript undefined value is stripped by the MongoDB driver, leaving the stored value unchanged. The endpoint now uses $unset for null fields so the stored custom value is correctly removed.
  • Cross-field validation (xtMonitor > xtGood, altMonitorBand > altGoodBand) was resolved against system defaults when a field was absent from the request, not the user's currently stored values. A partial update such as {xtMonitor: 3} could pass validation while leaving a stored combination of xtGood=5, xtMonitor=3 (invalid). The endpoint now fetches the stored dashboardSettings before validation and uses the stored values as the fallback.

Documentation

  • §5.3 Trend: corrected the note about partial startDate/endDate pairs — both are silently defaulted (no 409)
  • §5.6 Thresholds: updated constraint description to reflect the improved cross-field validation fallback chain

v1.6 — 2026-05-22 (r1181, #3054)

KPI Cards (/kpi)

  • Renamed periods.<p>.sprayedperiods.<p>.sprayedHectares across all period objects (day, week, month, year, all) for clarity and consistency with the field's unit

v1.5 — 2026-05-21 (r1176, #3054)

New Endpoint

  • Added PUT /api/dashboard/pilot/performance/thresholds (§5.6): persists custom XT error and altitude gauge thresholds per pilot; returns effective thresholds after save; supports null to reset individual fields to system defaults
  • Mark Job as Completed moved from §5.6 → §5.7 to accommodate new thresholds endpoint

Data Model

  • Added User.dashboardSettings optional subdocument (xtGood, xtMonitor, altTarget, altGoodBand, altMonitorBand) for per-pilot threshold overrides; system defaults used when fields absent
  • Added model/user.js to the Implemented In list (§1)

Frontend Guide

  • Added "When the user saves custom thresholds" fetch pattern — response from PUT /thresholds can be used directly to update the gauge without a follow-up GET /performance

Diagrams

  • Updated endpoint interaction diagram: added TH[PUT pilot performance thresholds] node with TH → U[(users)] edge

v1.4 — 2026-05-20 (r1173, #3054)

Performance Gauges (/performance) — Breaking Change

  • Replaced last-10-files static sample with a date-range query
  • Added startDate and endDate query params (default: current calendar week MonSun); max range 90 days
  • Date resolution priority: selectedDatestartDate+endDate → current week default
  • Response now includes startDate and endDate fields reflecting the effective analysis window
  • Metrics now computed from spray-on records only (sprayStat === 1) — transit, taxi, and ferry-flight records are excluded; this eliminates spurious XT error spikes (millions of metres off-line during turns) and AMSL altitude readings from non-spray legs
  • sampleSize now reflects the number of AppFile records in the date window (was: "up to ~20" files cap)
  • Updated hasAltitudeData field note: now means "no altitude sensor data in sample" (was: "aircraft has no height sensor")

Backend Notes (§9)

  • Added "Why Spray-On Records Only (sprayStat === 1)" explanatory section
  • Updated ApplicationDetail query pattern: step 1 now uses createdDate range filter instead of .limit(10)
  • Updated Performance Query Safety diagram to reflect date-range scoping and spray-on filter step

v1.3 — 2026-05-15 (r1155, #3054)

KPI Cards (/kpi) — Breaking Response Shape Change

Old flat top-level fields removed: assignedJobs, assignedHectares, totalSprayed, totalFlightHours, jobCounts, and the historical block.

New structure:

  • operations block (today-scoped): missionsFlown, distanceTravelledKm, distanceSprayedKm
    • distanceTravelledKm replaces distanceKm and now uses Application.totalFlightLength (all GPS segments including turns) instead of totalSprLength
    • distanceSprayedKm is new — spray-on segments only (Application.totalSprLength / 1000)
    • sprayVolumeLiters removed from operations
  • periods block: day, week, month, year, all sub-objects each containing assignedJobs, assignedHectares, sprayed, flightHours
    • jobCounts (new, inProgress, completed) present on day, week, month only — not on year or all
    • all period added (no time boundary — covers all records for this pilot)
  • Added totalFlightLength field tracking to Application model, AppFile, and job_worker

Active Jobs (/activeJobs)

  • Added period query param (day | week | month | year): filters both the job list (by Job.createdAt) and Application sub-totals (by Application.startDateTimeUTC) to the selected window; tz continues to control the boundary math
  • Priority chain: selectedDate > period > all-time (no filter)
  • Updated field notes for haSprayed and volumeAppliedLiters to clarify they are scoped to the active window

Status / Display

  • Fixed displayStatus value: "IN PROGRESS" (with space) → "IN_PROGRESS" (underscore) — breaking change for frontend consumers

Unit Conversions (§9)

  • Added totalFlightLength (metres → km ÷ 1000) → operations.distanceTravelledKm to the server-side conversion table

Frontend Guide (§8)

  • Added "Period filter" fetch pattern for Day / Week / Month / Year tab switching on /activeJobs
  • Expanded Drilldown vs. Date Range table with "User switches Day/Week/Month/Year tab" scenario
  • Updated Tier 1 fetch example: /activeJobs?tz=...&period=week

Open Decisions

  • ACT-Q1 resolved: period filter applies to both the job list and Application totals

v1.1 — 2026-05-14 (r1144, #3054)

  • Added selectedDate query param (YYYY-MM-DD) to /kpi, /summary, /activeJobs, /performance for single-day drilldown filtering from chart interactions
  • Added tz param documentation to /activeJobs and /performance (previously these endpoints accepted tz but it was undocumented)
  • Added frontend Drilldown Snippet JavaScript code example
  • Added Drilldown vs. Date Range reference table
  • Fixed DST-safe yesterday derivation in /summary (no longer a simple 24-hour subtraction)

v1.0 — 2026-04-29 (r1081, #3064)

Initial release implementing all Pilot Dashboard endpoints:

  • GET /api/dashboard/pilot/kpi — KPI cards with historical breakdowns
  • GET /api/dashboard/pilot/summary — today vs. yesterday metrics with deltas
  • GET /api/dashboard/pilot/trend — daily hours flown and hectares for date range
  • GET /api/dashboard/pilot/activeJobs — active job list with progress
  • GET /api/dashboard/pilot/performance — XT error and spray altitude gauges
  • PATCH /api/jobs/:job_id/complete — transition job SPRAYED → COMPLETED
  • Postman collection: docs/Pilot_Dashboard_API.postman_collection.json
  • Mocha/Chai integration test script: tests/test_pilot_dashboard_api.js