Devin Major f5670712f6 analytics dashboard init commit (may 8 2026)

2026-05-08 13:34:01 -04:00

26 KiB

Raw Blame History

Pilot Analytics Dashboard — API Design Reference

Version: 1.0 — Phase 1 (2026-04-29) Scope: Backend API contract for the Pilot Analytics Dashboard. This document is the single source of truth for both backend and frontend/client development.

1 Overview
2 Authentication
3 Status Constants
4 Timezone Handling
5 Endpoints
6 Error Responses
7 Data Model Notes
8 Frontend Integration Guide
9 Backend Architecture Notes
10 Open Decisions

1 Overview

The Pilot Dashboard is a read-only analytics surface (plus one write action) scoped to the currently authenticated Pilot user. All endpoints derive the pilot identity from the JWT token — no client-side user ID parameter is accepted or trusted.

Base path: /api/dashboard/pilot

Implemented in:

controllers/dashboard.js
routes/dashboard.js
routes/job.js (for the complete action)

Testing resources:

Postman collection: docs/Pilot_Dashboard_API.postman_collection.json
Node.js test script: tests/test_pilot_dashboard_api.js

All dashboard endpoints share the same scoping logic:

pilot → Job.operator = req.uid → jobIds → Application.jobId IN jobIds

2 Authentication

All routes require a valid JWT bearer token. The checkUser middleware is applied globally in server.js before any route is mounted.

Authorization: Bearer <jwt>

The token payload sets req.uid (user ID string) and req.userInfo on every request. Dashboard endpoints check req.uid and throw 401 Not Authorized if absent.

The dashboard route group (/api/dashboard) does not require a subscription package (checkRqPkgSubscription is not applied there). The complete-job endpoint lives under /api/jobs which does apply that middleware.

3 Status Constants

Job Status (`helpers/job_constants.js`)

Constant	Value	Meaning
`NEW`	`0`	Job created, not yet prepared
`READY`	`1`	Prepared and available for aircraft download
`DOWNLOADED`	`2`	Downloaded to aircraft
`SPRAYED`	`3`	Application data uploaded, not yet reviewed
`COMPLETED`	`4`	Reviewed and marked done by Applicator
`INVOICED`	`5`	Invoice issued
`ARCHIVED`	`9`	Archived

Display Status (frontend label mapping)

The activeJobs endpoint returns a displayStatus string derived from the raw numeric status:

`status` value(s)	`displayStatus`	Suggested UI treatment
`0` (NEW)	`"NEW"`	Grey badge, no progress bar
`1`, `2`, `3`	`"IN_PROGRESS"`	Blue/active badge, show progress
`4` (COMPLETED)	`"COMPLETED"`	Green badge, full bar

INVOICED (5) and ARCHIVED (9) are excluded from the active jobs panel entirely.

Application Status

Only status: 3 (processed) applications are counted in all aggregations. Uploading (1), in-progress (2), and error (0) applications are excluded automatically.

4 Timezone Handling

All time-windowed endpoints accept an optional ?tz=<IANA> query parameter.

Default: UTC
Example: ?tz=America/Sao_Paulo, ?tz=America/Chicago, ?tz=Australia/Brisbane
Invalid values silently fall back to UTC — the frontend should always send a valid tz.

What timezone affects:

"Today" and "yesterday" day boundaries
Current week (Mon–Sun) boundaries
Current month and year boundaries
Date labels in the trend chart (YYYY-MM-DD)

What timezone does NOT affect:

Raw UTC timestamps stored in MongoDB
Job createdAt (always UTC)

Recommended frontend behaviour: read the browser's timezone once (Intl.DateTimeFormat().resolvedOptions().timeZone) and pass it on every dashboard request.

5 Endpoints

5 1 KPI Cards

Returns top-level KPI cards and historical breakdowns for the authenticated pilot.

Query Parameters

Parameter	Type	Default	Description
`tz`	String	`UTC`	IANA timezone for period windows

Response `200 OK`

{
  "assignedJobs":     42,
  "assignedHectares": 1850.40,
  "sprayedToday":     120.50,
  "flightHoursToday": 3.25,
  "operations": {
    "distanceKm":        345.80,
    "sprayVolumeLiters": 2400.00
  },
  "historical": {
    "jobs": {
      "year":  38,
      "month": 12,
      "week":  4,
      "day":   42
    },
    "hectares": {
      "year":  1600.00,
      "month": 420.00,
      "week":  180.00,
      "day":   120.50
    },
    "flightHours": {
      "year":  210.50,
      "month": 52.75,
      "week":  18.25,
      "day":   3.25
    }
  }
}

Field Notes

Field	Unit	Notes
`assignedJobs`	count	All-time total jobs where `Job.operator = uid`
`assignedHectares`	ha	Sum of `Job.ttSprArea` across all assigned jobs
`sprayedToday`	ha	Sum of `Application.totalSprayed` today
`flightHoursToday`	hours	Sum of `Application.totalFlightTime` / 3600
`operations.distanceKm`	km	Sum of `Application.totalSprLength` / 1000
`operations.sprayVolumeLiters`	litres	Sum of `Application.totalSprayMat`
`historical.jobs.day`	count	Same as `assignedJobs` (all-time)
`historical.jobs.week/month/year`	count	Jobs created within the period (by `createdAt`)

All numeric values are rounded to 2 decimal places.

5 2 Daily Summary

Returns today vs. yesterday operational metrics with percentage change deltas.

Query Parameters

Parameter	Type	Default	Description
`tz`	String	`UTC`	IANA timezone for day boundaries

Response `200 OK`

{
  "today": {
    "hectares":          120.50,
    "flightHours":       3.25,
    "haPerHour":         37.08,
    "avgSpeedKmh":       28.50,
    "sprayVolumeLiters": 2400.00
  },
  "yesterday": {
    "hectares":          95.00,
    "flightHours":       2.80,
    "haPerHour":         33.93,
    "avgSpeedKmh":       26.10,
    "sprayVolumeLiters": 1900.00
  },
  "deltas": {
    "hectaresPct":    27,
    "flightHoursPct": 16,
    "haPerHourPct":   9,
    "avgSpeedPct":    9,
    "sprayVolumePct": 26
  }
}

Field Notes

Field	Unit	Notes
`today.haPerHour`	ha/hr	Derived: `hectares / flightHours` (0 if no flight time)
`today.avgSpeedKmh`	km/h	`Application.avgSpraySpeed` (m/s) × 3.6, averaged
`deltas.*Pct`	%	`round((today − yesterday) / yesterday × 100)`
`deltas.*Pct` = `null`	—	yesterday value was 0 (division by zero avoided)

Positive delta = improvement today vs. yesterday. Negative = drop.

5 3 Trend Charts

Returns daily hours flown and hectares sprayed for a date range. Default range is the current calendar week (Monday–Sunday) in the given timezone.

Query Parameters

Parameter	Type	Default	Description
`tz`	String	`UTC`	IANA timezone for day grouping and boundaries
`startDate`	String	Monday of this week	Start date inclusive. Format: `YYYY-MM-DD`
`endDate`	String	Sunday of this week	End date inclusive. Format: `YYYY-MM-DD`

Maximum range: 90 days. Returns 409 if exceeded.
Both startDate and endDate must be provided together, or both omitted.

Response `200 OK`

{
  "labels":        ["2026-04-27", "2026-04-28", "2026-04-29", "2026-04-30", "2026-05-01", "2026-05-02", "2026-05-03"],
  "hoursFlown":    [2.50, 3.25, 0,    1.80, 0,    0,    0],
  "hectaresPerDay":[80.0, 120.5, 0,   65.0, 0,    0,    0]
}

Field Notes

Field	Type	Notes
`labels`	`String[]`	One entry per calendar day in `YYYY-MM-DD`, tz-adjusted
`hoursFlown`	`Number[]`	Parallel array. Days with no activity = `0`
`hectaresPerDay`	`Number[]`	Parallel array. Days with no activity = `0`

Array contract: all three arrays are always the same length and in the same order. Frontend can zip them: labels[i] ↔ hoursFlown[i] ↔ hectaresPerDay[i].

5 4 Active Jobs Panel

Returns the pilot's active-status jobs with per-job progress and applied totals.

Statuses included: NEW (0), READY (1), DOWNLOADED (2), SPRAYED (3), COMPLETED (4)
Statuses excluded: INVOICED (5), ARCHIVED (9)
Maximum 50 most recent jobs returned (sorted by createdAt descending)

Query Parameters

None.

Response `200 OK`

{
  "jobs": [
    {
      "jobId":               1042,
      "name":                "North Block — Canola",
      "clientName":          "Sunrise Farms Ltd.",
      "aircraftReg":         "C-FABM",
      "status":              3,
      "displayStatus":       "IN_PROGRESS",
      "haTotal":             250.00,
      "haSprayed":           187.50,
      "progressPct":         75,
      "volumeAppliedLiters": 3750.00
    }
  ]
}

Field Notes

Field	Type	Notes
`jobId`	Number	Numeric job ID (`Job._id`, not a MongoDB ObjectId)
`name`	String	Job name. Empty string if not set
`clientName`	String	From linked `Client` user record. Empty if not linked
`aircraftReg`	String	`Vehicle.tailNumber` or `Vehicle.unitId`. Empty if not linked
`status`	Number	Raw numeric status (see constants table)
`displayStatus`	String	`"NEW"` \| `"IN_PROGRESS"` \| `"COMPLETED"`
`haTotal`	Number	`Job.ttSprArea` — planned area in ha. `0` if not set
`haSprayed`	Number	Sum of `Application.totalSprayed` for this job (processed only)
`progressPct`	Number	`0–100`. `0` if `haTotal` is 0 or no applications
`volumeAppliedLiters`	Number	Sum of `Application.totalSprayMat` for this job

progressPct formula: min(100, max(0, round(haSprayed / haTotal × 100))) This can exceed 100% theoretically if haSprayed > haTotal; the backend caps it at 100.

5 5 Performance Gauges

Returns average XT cross-track error and spray altitude gauges based on the pilot's last 10 processed application files.

Note

: ApplicationDetail is a billion-row collection. Queries are strictly bounded to a set of fileId values to use the collection's primary index. Never query this collection without a fileId filter.

Query Parameters

None.

Response `200 OK` — data available

{
  "avgXtErrorMeters":       1.85,
  "xtThreshold": {
    "good":    1.0,
    "monitor": 3.0
  },
  "hasXtData":              true,

  "avgSprayAltitudeMeters": 3.62,
  "altitudeSource":         "sprayHeight",
  "altThreshold": {
    "target":      3.7,
    "goodBand":    0.15,
    "monitorBand": 0.46
  },
  "hasAltitudeData":        true,
  "sampleSize":             24
}

Response `200 OK` — no data (new pilot, no apps uploaded)

{
  "avgXtErrorMeters":       null,
  "xtThreshold": { "good": 1.0, "monitor": 3.0 },
  "hasXtData":              false,

  "avgSprayAltitudeMeters": null,
  "altitudeSource":         null,
  "altThreshold": { "target": 3.7, "goodBand": 0.15, "monitorBand": 0.46 },
  "hasAltitudeData":        false,
  "sampleSize":             0
}

Field Notes

Field	Unit	Notes
`avgXtErrorMeters`	metres	Average of `abs(ApplicationDetail.xTrack)`. `null` if no XT data
`xtThreshold.good`	metres	Below this → green zone
`xtThreshold.monitor`	metres	Above this → red zone; between good and monitor → yellow
`hasXtData`	bool	`false` = no valid xTrack readings in sample (show "No data")
`avgSprayAltitudeMeters`	metres	Average spray height from best available sensor. `null` if none
`altitudeSource`	String	`"sprayHeight"` (FM dedicated sensor) or `"radarAlt"` (AGL). `null` if no data
`altThreshold.target`	metres	Ideal spray height (~3.7 m)
`altThreshold.goodBand`	metres	±0.15 m of target → green
`altThreshold.monitorBand`	metres	±0.46 m of target → yellow; outside → red
`hasAltitudeData`	bool	`false` = aircraft has no height sensor (show "No data")
`sampleSize`	count	Number of AppFile records in the analysis window (up to ~20)

Threshold Gauge Logic (frontend)

XT Cross-track error:
  value < good          → green  (acceptable precision)
  good ≤ value < monitor → yellow (monitor)
  value ≥ monitor       → red   (needs attention)

Spray Altitude:
  |value - target| < goodBand    → green
  |value - target| < monitorBand → yellow
  |value - target| ≥ monitorBand → red

Altitude Source Priority

The backend selects the best available altitude sensor in this order:

sprayHeight — dedicated FM spray height sensor (most accurate)
radarAlt — radar altimeter (AGL, good fallback)
No data — hasAltitudeData: false, avgSprayAltitudeMeters: null

gpsAlt (AMSL) is not used for spray height — it measures altitude above sea level, not above the crop canopy.

xTrack = 0 values are excluded from the XT average because 0 means "no reading", not "perfectly on track".

5 6 Mark Job as Completed

Transitions a job from SPRAYED (3) to COMPLETED (4).

This endpoint lives under /api/jobs (not /api/dashboard) and requires an active subscription package (enforced by checkRqPkgSubscription middleware).

URL Parameters

Parameter	Type	Description
`job_id`	Number	Numeric job ID

Request Body

None.

Response `200 OK`

Returns the full updated job document (with client, operator, and vehicle populated).

{
  "_id":    1042,
  "name":   "North Block — Canola",
  "status": 4,
  "client": { "_id": "...", "name": "Sunrise Farms Ltd." },
  "operator": { "_id": "...", "name": "Jane Pilot" },
  "vehicle":  { "_id": "...", "name": "Agri-One", "tailNumber": "C-FABM" },
  "..."
}

Authorization

Only the Applicator who owns the job (Job.byPuid === req.uid) may complete it. The Pilot (Job.operator) cannot call this endpoint.

Error Cases

Condition	Status	Error tag
`job_id` is not a valid positive number	`409`	`invalid_param`
Job not found	`409`	`job_not_found`
Caller is not the job owner (`byPuid`)	`401`	`not_authorized`
Job status is not `SPRAYED (3)`	`409`	`status_job_invalid`

6 Error Responses

All errors follow the standard AgMission error format:

{
  "error": {
    ".tag": "error_constant_value",
    "message": "Human-readable detail (development mode only)"
  }
}

HTTP Status	`.tag` value	When it occurs
`401`	`not_authorized`	Missing/invalid JWT, or caller not the job owner
`409`	`invalid_param`	Bad date format, range > 90 days, invalid job_id
`409`	`job_not_found`	Job does not exist
`409`	`status_job_invalid`	Job is not in required status for transition

7 Data Model Notes

Pilot Scoping

All Application metrics are scoped via the Job's operator field, not byUser on Application:

Job.operator = pilotId  →  collect jobIds  →  Application.jobId IN jobIds

Application.byUser is the master Applicator account — it is not the pilot. Never use it to scope pilot metrics.

Application.createdDate

Time-window filtering uses Application.createdDate (the upload timestamp, a real MongoDB Date), not Application.endDateTime. The endDateTime field is a raw device string with no guaranteed format and is unreliable for date arithmetic.

Job._id is a Number

Job._id is an auto-incrementing Number (via mongoose-sequence), not a MongoDB ObjectId. Frontend must treat jobId values as integers, not hex strings.

Vehicle and Client are User discriminators

Both Vehicle (kind=DEVICE) and Client (kind=CLIENT) are stored in the users MongoDB collection. The $lookup in activeJobs joins against users for both.

Models and Collections

Model name in code	Collection	Key dashboard fields
`Job`	`jobs`	`operator`, `byPuid`, `ttSprArea`, `status`, `client`, `vehicle`
`App` (Application)	`applications`	`jobId`, `status`, `createdDate`, `totalSprayed`, `totalFlightTime`, `totalSprLength`, `totalSprayMat`, `avgSpraySpeed`
`AppFile`	`appfiles`	`appId`
`AppDetail`	`application_details`	`fileId`, `xTrack`, `sprayHeight`, `radarAlt`

8 Frontend Integration Guide

Suggested Fetch Strategy

Load the dashboard in two tiers to keep the initial paint fast:

Tier 1 — above the fold (load in parallel on mount)

GET /api/dashboard/pilot/kpi?tz=...
GET /api/dashboard/pilot/summary?tz=...
GET /api/dashboard/pilot/activeJobs

Tier 2 — charts and gauges (load after Tier 1 resolves or in parallel)

GET /api/dashboard/pilot/trend?tz=...&startDate=...&endDate=...
GET /api/dashboard/pilot/performance

Timezone Snippet

// Read once and reuse
const tz = Intl.DateTimeFormat().resolvedOptions().timeZone; // e.g. "America/Sao_Paulo"

const params = new URLSearchParams({ tz });
fetch(`/api/dashboard/pilot/kpi?${params}`, { headers: { Authorization: `Bearer ${token}` } });

Empty State Handling

Every endpoint returns meaningful zero-filled data when the pilot has no jobs:

Number fields return 0
Null fields (avgXtErrorMeters, altitudeSource) return null
Boolean guard fields (hasXtData, hasAltitudeData) return false
Array fields (labels, hoursFlown, hectaresPerDay) return empty []

The frontend should check hasXtData and hasAltitudeData before rendering gauges, and render a "No data yet" placeholder instead of a gauge at null.

Progress Bar Colour Logic

Use displayStatus from activeJobs for badge colour, and progressPct for bar width:

displayStatus = "NEW"         → grey badge, hide progress bar
displayStatus = "IN_PROGRESS" → blue badge, show progress bar at progressPct%
displayStatus = "COMPLETED"   → green badge, show bar at 100%

Mark as Completed Button Visibility

The Complete action is restricted to the Applicator (Job.byPuid), not the Pilot. If the frontend is used by an Applicator who is reviewing a pilot's job, show the button only when the raw status === 3 (SPRAYED). After a successful PATCH, update the job in local state to status = 4, displayStatus = "COMPLETED".

Trend Chart Date Pickers

Pass startDate/endDate as YYYY-MM-DD strings in the user's local timezone.
Maximum selectable range: 90 days.
Default range on first render: current week (omit both parameters and let the API default).

Gauge Rendering Reference

XT Error gauge (lower is better):
  0 m ──────── 1.0 m ──────────── 3.0 m ──────────→
       green          yellow              red
       ↑ good          ↑ monitor

Altitude gauge (target 3.7 m):
  ← red ── 3.24 ── yellow ── 3.55 ── [3.7] ── 3.85 ── yellow ── 4.16 ── red →
           ↑ -0.46          ↑ -0.15   target   ↑ +0.15           ↑ +0.46

9 Backend Architecture Notes

9 1 Pilot Scope Data Flow Diagram

flowchart LR
  A[Authenticated User req uid] --> B[Job Query operator equals uid]
  B --> C[Collect Job IDs]
  C --> D[Application Query jobId in Job IDs and status 3]
  D --> E[KPI Summary Trend Active Jobs Aggregates]

9 2 Endpoint Interaction Diagram

flowchart TD
  FE[Frontend Pilot Dashboard] --> K[GET pilot kpi]
  FE --> S[GET pilot summary]
  FE --> T[GET pilot trend]
  FE --> A[GET pilot activeJobs]
  FE --> P[GET pilot performance]
  FE --> C[PATCH jobs job_id complete]

  K --> J[(jobs)]
  K --> AP[(applications)]
  S --> AP
  T --> AP
  A --> J
  A --> AP
  P --> AP
  P --> AF[(appfiles)]
  P --> AD[(application_details)]
  C --> J

9 3 Performance Query Safety Diagram

flowchart LR
  A[Scoped Job IDs] --> B[Recent Processed Applications limit 10]
  B --> C[AppFile Lookup by appId]
  C --> D[ApplicationDetail Match by fileId in list]
  D --> E[Aggregate XT and Altitude Metrics]

Why Job.operator and not Application.byUser

Application.byUser is the master Applicator account that uploaded the file — not the pilot assigned to fly the job. Using it would mix data from all jobs the Applicator manages, not just those assigned to this pilot. The correct field is Job.operator (set when a pilot is assigned to a job).

Why Application.createdDate and not endDateTime

Application.endDateTime is a raw string as received from the aircraft firmware. Its format varies by firmware version and is not reliable for MongoDB date range queries. createdDate is a Date field set by the server at upload time and is consistent across all records.

ApplicationDetail Query Safety

application_details is a very large collection (potentially billions of rows). The only index available for dashboard use is { fileId: 1 }. All performance queries follow a three-step pattern to avoid collection scans:

1. App.find({ jobId: { $in: jobIds } }).limit(10)   → recent appIds
2. AppFile.find({ appId: { $in: appIds } })          → fileIds
3. AppDetail.aggregate([{ $match: { fileId: { $in: fileIds } } }, ...])

Never add additional ApplicationDetail queries without scoping by fileId first.

Unit Conversions (done server-side)

Raw storage	Conversion	API field
`totalFlightTime`	seconds → hours ÷3600	`flightHours`, `hoursFlown`
`totalSprLength`	metres → km ÷1000	`operations.distanceKm`
`avgSpraySpeed`	m/s → km/h ×3.6	`avgSpeedKmh`

All other fields (totalSprayed, totalSprayMat, xTrack, sprayHeight, radarAlt) are stored and returned in their natural units (ha, L, m).

10 Open Decisions

These items affect backend behaviour but have not been finalised by the Product Owner. They are Phase 2 scope and the current implementation uses the defaults noted.

ID	Question	Current default
KPI-1	Does "Assigned Jobs" count all-time or just open/current-season jobs?	All-time
ACT-1	Should INVOICED (5) jobs appear in the Active Jobs panel?	Excluded
Q11	Can the Pilot trigger the complete action, or Applicator-only?	Applicator (`byPuid`) only
PERF-1	When no altitude sensor exists, should the gauge show N/A or be hidden?	Returns `hasAltitudeData: false`, value null
JOBS-1	Are jobs with `operator = null` (unassigned) relevant to any view?	Not included in any pilot-scoped query

26 KiB Raw Blame History Unescape Escape

Pilot Analytics Dashboard — API Design Reference

Table of Contents

1 Overview

2 Authentication

3 Status Constants

Job Status (helpers/job_constants.js)

Display Status (frontend label mapping)

Application Status

4 Timezone Handling

5 Endpoints

5 1 KPI Cards

Query Parameters

Response 200 OK

Field Notes

5 2 Daily Summary

Query Parameters

Response 200 OK

Field Notes

5 3 Trend Charts

Query Parameters

Response 200 OK

Field Notes

5 4 Active Jobs Panel

Query Parameters

Response 200 OK

Field Notes

5 5 Performance Gauges

Query Parameters

Response 200 OK — data available

Response 200 OK — no data (new pilot, no apps uploaded)

Field Notes

Threshold Gauge Logic (frontend)

Altitude Source Priority

5 6 Mark Job as Completed

URL Parameters

Request Body

Response 200 OK

Authorization

Error Cases

6 Error Responses

7 Data Model Notes

Pilot Scoping

Application.createdDate

Job._id is a Number

Vehicle and Client are User discriminators

Models and Collections

8 Frontend Integration Guide

Suggested Fetch Strategy

Timezone Snippet

Empty State Handling

Progress Bar Colour Logic

Mark as Completed Button Visibility

Trend Chart Date Pickers

Gauge Rendering Reference

9 Backend Architecture Notes

9 1 Pilot Scope Data Flow Diagram

9 2 Endpoint Interaction Diagram

9 3 Performance Query Safety Diagram

Why Job.operator and not Application.byUser

Why Application.createdDate and not endDateTime

ApplicationDetail Query Safety

Unit Conversions (done server-side)

10 Open Decisions

26 KiB

Raw Blame History

Job Status (`helpers/job_constants.js`)

Response `200 OK`

Response `200 OK`

Response `200 OK`

Response `200 OK`

Response `200 OK` — data available

Response `200 OK` — no data (new pilot, no apps uploaded)

Response `200 OK`