1754 lines
85 KiB
Markdown
1754 lines
85 KiB
Markdown
# Pilot Analytics Dashboard — API Design Reference
|
||
|
||
**Version**: 2.5 — (2026-06-15)
|
||
**Scope**: Backend API contract for the Pilot Analytics Dashboard.
|
||
This document is the single source of truth for **both backend and frontend/client** development.
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
- [1 Overview](#1-overview)
|
||
- [2 Authentication](#2-authentication)
|
||
- [3 Status Constants](#3-status-constants)
|
||
- [4 Timezone Handling](#4-timezone-handling)
|
||
- [5 Endpoints](#5-endpoints)
|
||
- [5.1 KPI Cards](#51-kpi-cards)
|
||
- [5.2 Daily Summary](#52-daily-summary)
|
||
- [5.3 Trend Charts](#53-trend-charts)
|
||
- [5.4 Active Jobs Panel](#54-active-jobs-panel)
|
||
- [5.5 Performance Gauges](#55-performance-gauges)
|
||
- [5.6 Save Performance Thresholds](#56-save-performance-thresholds)
|
||
- [5.7 Mark Job as Completed](#57-mark-job-as-completed)
|
||
- [5.8 Snapshot (Composite Dashboard)](#58-snapshot-composite-dashboard)
|
||
- [6 Error Responses](#6-error-responses)
|
||
- [7 Data Model Notes](#7-data-model-notes)
|
||
- [8 Frontend Integration Guide](#8-frontend-integration-guide)
|
||
- [9 Backend Architecture Notes](#9-backend-architecture-notes)
|
||
- [9.1 Pilot Scope Data Flow Diagram](#91-pilot-scope-data-flow-diagram)
|
||
- [9.2 Endpoint Interaction Diagram](#92-endpoint-interaction-diagram)
|
||
- [9.3 Performance Query Safety Diagram](#93-performance-query-safety-diagram)
|
||
- [9.4 Periodic Polling Sequence](#94-periodic-polling-sequence)
|
||
- [10 Open Decisions](#10-open-decisions)
|
||
- [11 Changelog](#11-changelog)
|
||
|
||
---
|
||
|
||
## 1 Overview
|
||
|
||
The Pilot Dashboard is a read-only analytics surface (plus one write action) scoped to the
|
||
**currently authenticated Pilot user**. All endpoints derive the pilot identity from the
|
||
JWT token — no client-side user ID parameter is accepted or trusted.
|
||
|
||
**Base path**: `/api/dashboard/pilot`
|
||
|
||
**Implemented in**:
|
||
|
||
- `controllers/dashboard.js`
|
||
- `routes/dashboard.js`
|
||
- `model/setting.js` (for `dashboard` threshold storage)
|
||
- `routes/job.js` (for the complete action)
|
||
|
||
**Testing resources**:
|
||
|
||
- Postman collection: `docs/Pilot_Dashboard_API.postman_collection.json`
|
||
- Node.js test script: `tests/test_pilot_dashboard_api.js`
|
||
- Test command: `npm run test:dashboard`
|
||
|
||
### Dashboard Test Suite Requirements
|
||
|
||
`npm run test:dashboard` runs **live integration tests** (not mocked/unit-only). To execute all non-optional checks reliably:
|
||
|
||
1. Server must be running and reachable at `https://localhost:4100` (or set `PILOT_DASHBOARD_BASE_URL`).
|
||
2. `DASHBOARD_TEST_TOKEN` must be set to a valid Pilot JWT.
|
||
3. For complete-job tests, set:
|
||
- `RUN_COMPLETE_TEST=1`
|
||
- `DASHBOARD_TEST_JOB_ID=<jobId in SPRAYED(3) status>`
|
||
|
||
Optional toggles:
|
||
|
||
- `RUN_SNAPSHOT_TESTS=0` skips `/snapshot` endpoint tests.
|
||
- `RUN_COMPLETE_TEST=0` (default) skips complete-job mutation tests.
|
||
|
||
If required env vars are missing, Mocha reports `pending` (skipped) tests by design.
|
||
|
||
### Migration Script: `scripts/migrate_applications.js`
|
||
|
||
Purpose:
|
||
- **All-in-one replacement** for the former `backfill_application_datetimes.js` and `migrate_app_aggregates.js` scripts (both kept as deprecated stubs).
|
||
- Pass 1A — Aggregate metrics: `Application.avgSpraySpeed`, `totalSprLength`, `totalFlightLength`, `avgXtError`, `avgHdop`; and `AppFile.totalSprLength`, `AppFile.totalFlightLength`.
|
||
- Pass 1B — Datetime fields (for apps that have legacy `startDateTime`): `Application.utcOffset`, `startDateTimeUTC`, `endDateTimeUTC`.
|
||
- First valid lat/lon coordinate captured during the existing Pass 1A AppDetail stream — no extra DB query needed.
|
||
- Pass 2 — `Application.flowAccuracyPct` (Application-level only, no AppDetail streaming required).
|
||
|
||
Selection criteria (non-force mode):
|
||
- Union of both former scripts' `$or` conditions:
|
||
- **Datetime**: `utcOffset` missing or 0, `startDateTimeUTC` / `endDateTimeUTC` missing, or `startDateTimeUTC > endDateTimeUTC`.
|
||
- **Aggregates**: any of `avgSpraySpeed`, `totalSprLength`, `totalFlightLength`, `avgXtError`, `avgHdop` missing/null/0.
|
||
- Use `--skip-datetime` or `--skip-aggregates` to restrict to one set of conditions.
|
||
|
||
Why `Apps to process: 0` can happen:
|
||
- All eligible apps are already backfilled, OR
|
||
- Remaining records missing UTC companions do not have legacy `startDateTime` (script skips those for datetime, but still processes aggregates).
|
||
|
||
Usage:
|
||
```bash
|
||
node scripts/migrate_applications.js
|
||
node scripts/migrate_applications.js --dry-run
|
||
node scripts/migrate_applications.js --env ./environment.env --dry-run
|
||
node scripts/migrate_applications.js --missing-limit 200
|
||
# After a formula fix: reprocess ALL apps to correct previously stored values
|
||
node scripts/migrate_applications.js --force
|
||
# Tier 1 — most recent 90 days only
|
||
node scripts/migrate_applications.js --tier-days=90
|
||
# Datetime fields only (fast, uses lightweight findOne)
|
||
node scripts/migrate_applications.js --skip-aggregates
|
||
# Aggregate metrics only (skip datetime computation)
|
||
node scripts/migrate_applications.js --skip-datetime
|
||
```
|
||
|
||
Operational diagnostics:
|
||
- When `Apps to process: 0` and legacy datetime is missing/null on records, prints each affected app and related job context:
|
||
- `appId`, `App-jobId`, `jobId`, `jobStatus`, `appStatus`
|
||
- `startDateTime`, `endDateTime`
|
||
- `appFiles`, `appDetails`, `likelyNoDataFiles`
|
||
- This makes "zip has no data files" cases directly visible from script output.
|
||
|
||
**All dashboard endpoints share the same scoping logic**:
|
||
|
||
```
|
||
pilot → Job.operator = req.uid → jobIds → Application.jobId IN jobIds
|
||
```
|
||
|
||
---
|
||
|
||
## 2 Authentication
|
||
|
||
All routes require a valid JWT bearer token. The `checkUser` middleware is applied globally
|
||
in `server.js` before any route is mounted.
|
||
|
||
```
|
||
Authorization: Bearer <jwt>
|
||
```
|
||
|
||
The token payload sets `req.uid` (user ID string) and `req.userInfo` on every request.
|
||
Dashboard endpoints check `req.uid` and throw `401 Not Authorized` if absent.
|
||
|
||
The dashboard route group (`/api/dashboard`) does **not** require a subscription package
|
||
(`checkRqPkgSubscription` is not applied there). The complete-job endpoint lives under
|
||
`/api/jobs` which **does** apply that middleware.
|
||
|
||
---
|
||
|
||
## 3 Status Constants
|
||
|
||
### Job Status (`helpers/job_constants.js`)
|
||
|
||
| Constant | Value | Meaning |
|
||
|--------------|-------|----------------------------------------------|
|
||
| `NEW` | `0` | Job created, not yet prepared |
|
||
| `READY` | `1` | Prepared and available for aircraft download |
|
||
| `DOWNLOADED` | `2` | Downloaded to aircraft |
|
||
| `SPRAYED` | `3` | Application data uploaded, not yet reviewed |
|
||
| `COMPLETED` | `4` | Reviewed and marked done by Applicator |
|
||
| `INVOICED` | `5` | Invoice issued |
|
||
| `ARCHIVED` | `9` | Archived |
|
||
|
||
### Display Status (frontend label mapping)
|
||
|
||
The `activeJobs` endpoint returns a `displayStatus` string derived from the raw numeric status:
|
||
|
||
| `status` value(s) | `displayStatus` | Suggested UI treatment |
|
||
|--------------------|-----------------|-----------------------------------|
|
||
| `0` (NEW) | `"NEW"` | Grey badge, no progress bar |
|
||
| `1`, `2`, `3` | `"IN_PROGRESS"` | Blue/active badge, show progress |
|
||
| `4` (COMPLETED) | `"COMPLETED"` | Green badge, full bar |
|
||
|
||
> INVOICED (`5`) and ARCHIVED (`9`) are excluded from the active jobs panel entirely.
|
||
|
||
### Application Status
|
||
|
||
Only `status: 3` (processed) applications are counted in all aggregations. Uploading (`1`),
|
||
in-progress (`2`), and error (`0`) applications are excluded automatically.
|
||
|
||
---
|
||
|
||
## 4 Timezone Handling
|
||
|
||
All time-windowed endpoints accept an optional `?tz=<IANA>` query parameter.
|
||
|
||
- **Default**: `UTC`
|
||
- **Example**: `?tz=America/Sao_Paulo`, `?tz=America/Chicago`, `?tz=Australia/Brisbane`
|
||
- **Invalid values** silently fall back to `UTC` — the frontend should always send a valid tz.
|
||
|
||
**What timezone affects**:
|
||
- "Today" and "yesterday" day boundaries
|
||
- Current week (Mon–Sun) boundaries
|
||
- Current month and year boundaries
|
||
- Date labels in the trend chart (`YYYY-MM-DD`)
|
||
|
||
**What timezone does NOT affect**:
|
||
- Raw UTC timestamps stored in MongoDB
|
||
- Job `createdAt` (always UTC)
|
||
|
||
**Recommended frontend behaviour**: read the browser's timezone once (`Intl.DateTimeFormat().resolvedOptions().timeZone`) and pass it on every dashboard request. The API still needs `tz` to derive calendar boundaries and day labels, even though the Application filters now use the stored UTC fields.
|
||
|
||
---
|
||
|
||
## 5 Endpoints
|
||
|
||
### 5.1 KPI Cards
|
||
|
||
**URL**: `GET /api/dashboard/pilot/kpi`
|
||
|
||
Returns top-level KPI cards and historical breakdowns for the authenticated pilot.
|
||
|
||
#### Query Parameters
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|--------|---------|----------------------------------|
|
||
| `tz` | String | `UTC` | IANA timezone for period windows |
|
||
|
||
#### Response `200 OK`
|
||
|
||
```json
|
||
{
|
||
"operations": {
|
||
"missionsFlown": 1,
|
||
"distanceTravelledKm": 256.22,
|
||
"distanceSprayedKm": 50.82,
|
||
"sprayEfficiencyPct": 68.40,
|
||
"ferryTimePct": 31.60,
|
||
"flowAccuracyPct": 97.50,
|
||
"avgHdop": 1.20
|
||
},
|
||
"periods": {
|
||
"day": {
|
||
"assignedJobs": 1,
|
||
"assignedHectares": 939.02,
|
||
"sprayedHectares": 148.49,
|
||
"flightHours": 1.53,
|
||
"sprayEfficiencyPct": 68.40,
|
||
"ferryTimePct": 31.60,
|
||
"flowAccuracyPct": 97.50,
|
||
"avgHdop": 1.20,
|
||
"jobCounts": { "new": 0, "inProgress": 1, "completed": 0 }
|
||
},
|
||
"week": {
|
||
"assignedJobs": 4,
|
||
"assignedHectares": 9807.32,
|
||
"sprayedHectares": 15762.43,
|
||
"flightHours": 18.37,
|
||
"sprayEfficiencyPct": 71.00,
|
||
"ferryTimePct": 29.00,
|
||
"flowAccuracyPct": 96.25,
|
||
"avgHdop": 1.15,
|
||
"jobCounts": { "new": 1, "inProgress": 3, "completed": 1 }
|
||
},
|
||
"month": {
|
||
"assignedJobs": 7,
|
||
"assignedHectares": 27908.71,
|
||
"sprayedHectares": 22838.75,
|
||
"flightHours": 21.73,
|
||
"sprayEfficiencyPct": 70.50,
|
||
"ferryTimePct": 29.50,
|
||
"flowAccuracyPct": 98.10,
|
||
"avgHdop": 1.08,
|
||
"jobCounts": { "new": 1, "inProgress": 6, "completed": 4 }
|
||
},
|
||
"year": {
|
||
"assignedJobs": 7,
|
||
"assignedHectares": 27908.71,
|
||
"sprayedHectares": 22838.75,
|
||
"flightHours": 21.73,
|
||
"sprayEfficiencyPct": 69.80,
|
||
"ferryTimePct": 30.20,
|
||
"flowAccuracyPct": 97.50,
|
||
"avgHdop": 1.18,
|
||
"jobCounts": { "new": 2, "inProgress": 3, "completed": 2 }
|
||
},
|
||
"all": {
|
||
"assignedJobs": 7,
|
||
"assignedHectares": 27908.71,
|
||
"sprayedHectares": 22838.75,
|
||
"flightHours": 21.71,
|
||
"sprayEfficiencyPct": 69.80,
|
||
"ferryTimePct": 30.20,
|
||
"flowAccuracyPct": 97.50,
|
||
"avgHdop": 1.18,
|
||
"jobCounts": { "new": 2, "inProgress": 3, "completed": 2 }
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
> **New in v2.4**: All efficiency and GPS quality metrics now available in each period, not just today's operations. Pilots can track historical trends in spray efficiency, ferry percentage, flow control accuracy, and GPS health across day/week/month/year/all-time windows.
|
||
|
||
> `jobCounts` is present on **all** periods: `day`, `week`, `month`, `year`, and `all`. New metrics (`sprayEfficiencyPct`, `ferryTimePct`, `flowAccuracyPct`, `avgHdop`) are also present on all periods.
|
||
|
||
#### Field Notes
|
||
|
||
| Field | Unit | Notes |
|
||
|-----------------------------------------|-------|-------------------------------------------------------------------------------------------------|
|
||
| `operations.missionsFlown` | count | Number of Application records uploaded **today** (scoped to current day window) |
|
||
| `operations.distanceTravelledKm` | km | Sum of `Application.totalFlightLength` / 1000 for today — all GPS segments including turns (segments with time gap > 120 s or distance > 1000 m excluded) |
|
||
| `operations.distanceSprayedKm` | km | Sum of `Application.totalSprLength` / 1000 for today — spray-on segments only (same time/distance gates as above) |
|
||
| `operations.sprayEfficiencyPct` | % | `SUM(totalSprayTime) / SUM(totalFlightTime) × 100` for today. `null` when no flight time. Higher = more time actively spraying vs turning/ferrying |
|
||
| `operations.ferryTimePct` | % | `(SUM(totalFlightTime) − SUM(totalSprayTime)) / SUM(totalFlightTime) × 100` for today. Complement of `sprayEfficiencyPct`; both sum to 100 when non-null. `null` when no flight time |
|
||
| `operations.flowAccuracyPct` | % | Average of `Application.flowAccuracyPct` for today — `(actual rate / prescribed appRate) × 100` per session. `null` when no sessions with prescribed rate |
|
||
| `operations.avgHdop` | — | Average of `Application.avgHdop` for today — mean HDOP across spray-on records. `null` when no data. `< 1` excellent; `1–2` good; `2–5` moderate; `> 5` poor |
|
||
| `periods.<p>.assignedJobs` | count | Open jobs (NEW / READY / DOWNLOADED / SPRAYED / COMPLETED) created within period `<p>` |
|
||
| `periods.<p>.assignedHectares` | ha | Sum of `Job.ttSprArea` for jobs created within period `<p>` |
|
||
| `periods.<p>.sprayedHectares` | ha | Sum of `Application.totalSprayed` for applications uploaded within period `<p>` |
|
||
| `periods.<p>.flightHours` | hours | Sum of `Application.totalFlightTime` / 3600 for applications uploaded within period `<p>` (all flight records, same validity rules as `totalFlightTime`) |
|
||
| `periods.<p>.sprayEfficiencyPct` | % | `SUM(totalSprayTime) / SUM(totalFlightTime) × 100` for period `<p>`. `null` when no flight time. Higher = more time actively spraying vs turning/ferrying |
|
||
| `periods.<p>.ferryTimePct` | % | `(SUM(totalFlightTime) − SUM(totalSprayTime)) / SUM(totalFlightTime) × 100` for period `<p>`. Complement of spray efficiency; both sum to 100 when non-null. `null` when no flight time |
|
||
| `periods.<p>.flowAccuracyPct` | % | Average of `Application.flowAccuracyPct` for applications in period `<p>`. `null` when no sessions with prescribed rate |
|
||
| `periods.<p>.avgHdop` | — | Average of `Application.avgHdop` for period `<p>`. `null` when no data. `< 1` excellent; `1–2` good; `2–5` moderate; `> 5` poor |
|
||
| `periods.<p>.jobCounts.new` | count | Jobs with status NEW (0) created within that period |
|
||
| `periods.<p>.jobCounts.inProgress` | count | Jobs with status READY (1), DOWNLOADED (2), or SPRAYED (3) created within that period |
|
||
| `periods.<p>.jobCounts.completed` | count | Jobs with status COMPLETED (4) created within that period |
|
||
|
||
`<p>` = `day` \| `week` \| `month` \| `year` \| `all`. The `all` period has no time boundary — it covers all records for this pilot.
|
||
|
||
All numeric values are rounded to 2 decimal places.
|
||
|
||
---
|
||
|
||
### 5.2 Daily Summary
|
||
|
||
**URL**: `GET /api/dashboard/pilot/summary`
|
||
|
||
Returns today vs. yesterday operational metrics with percentage change deltas.
|
||
|
||
#### Query Parameters
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|--------|---------|-----------------------------------|
|
||
| `tz` | String | `UTC` | IANA timezone for day boundaries |
|
||
|
||
#### Response `200 OK`
|
||
|
||
```json
|
||
{
|
||
"today": {
|
||
"hectares": 120.50,
|
||
"flightHours": 3.25,
|
||
"haPerHour": 37.08,
|
||
"avgSpeedKmh": 28.50,
|
||
"sprayVolumeLiters": 2400.00
|
||
},
|
||
"yesterday": {
|
||
"hectares": 95.00,
|
||
"flightHours": 2.80,
|
||
"haPerHour": 33.93,
|
||
"avgSpeedKmh": 26.10,
|
||
"sprayVolumeLiters": 1900.00
|
||
},
|
||
"todayHasData": true,
|
||
"deltas": {
|
||
"hectaresPct": 27,
|
||
"flightHoursPct": 16,
|
||
"haPerHourPct": 9,
|
||
"avgSpeedPct": 9,
|
||
"sprayVolumePct": 26
|
||
}
|
||
}
|
||
```
|
||
|
||
#### Field Notes
|
||
|
||
| Field | Unit | Notes |
|
||
|---------------------------------|---------|--------------------------------------------------------------------------------------------|
|
||
| `today.haPerHour` | ha/hr | Derived: `hectares / flightHours` (0 if no flight time) |
|
||
| `today.avgSpeedKmh` | km/h | `Application.avgSpraySpeed` (m/s) × 3.6, averaged |
|
||
| `todayHasData` | bool | `false` when no Application records exist for today. Use this to distinguish "no upload yet" from a real operational drop |
|
||
| `deltas.*Pct` | % | `round((today − yesterday) / yesterday × 100)`. All fields are `null` when `todayHasData` is `false` |
|
||
| `deltas.*Pct` = `null` | — | Either no data uploaded today (`todayHasData: false`), or yesterday value was 0 (division by zero avoided) |
|
||
|
||
Positive delta = improvement today vs. yesterday. Negative = drop.
|
||
|
||
> **Frontend guidance**: only render colored delta arrows (red/green) when `todayHasData` is `true` and the delta value is non-null. When `todayHasData` is `false`, show a neutral "—" or "Awaiting data" state — a `-100%` would be misleading when the cause is a missing upload, not a real performance drop.
|
||
|
||
---
|
||
|
||
### 5.3 Trend Charts
|
||
|
||
**URL**: `GET /api/dashboard/pilot/trend`
|
||
|
||
Returns daily hours flown and hectares sprayed for a date range.
|
||
Default range is the current calendar week (Monday–Sunday) in the given timezone.
|
||
|
||
#### Query Parameters
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-------------|--------|--------------------|--------------------------------------------------|
|
||
| `tz` | String | `UTC` | IANA timezone for day grouping and boundaries |
|
||
| `startDate` | String | Monday of this week| Start date inclusive. Format: `YYYY-MM-DD` |
|
||
| `endDate` | String | Sunday of this week| End date inclusive. Format: `YYYY-MM-DD` |
|
||
|
||
- Maximum range: **90 days**. Returns `409` if exceeded.
|
||
- If either `startDate` or `endDate` is omitted, **both** are silently defaulted to the current week (Mon–Sun). No `409` is returned for a one-sided pair — the partial value is discarded.
|
||
|
||
#### Response `200 OK`
|
||
|
||
```json
|
||
{
|
||
"labels": ["2026-04-27", "2026-04-28", "2026-04-29", "2026-04-30", "2026-05-01", "2026-05-02", "2026-05-03"],
|
||
"hoursFlown": [2.50, 3.25, 0, 1.80, 0, 0, 0],
|
||
"hectaresPerDay":[80.0, 120.5, 0, 65.0, 0, 0, 0]
|
||
}
|
||
```
|
||
|
||
#### Field Notes
|
||
|
||
| Field | Type | Notes |
|
||
|-----------------|-----------------|----------------------------------------------------------|
|
||
| `labels` | `String[]` | One entry per calendar day in `YYYY-MM-DD`, tz-adjusted |
|
||
| `hoursFlown` | `Number[]` | Parallel array. Days with no activity = `0` |
|
||
| `hectaresPerDay`| `Number[]` | Parallel array. Days with no activity = `0` |
|
||
|
||
**Array contract**: all three arrays are always the same length and in the same order.
|
||
Frontend can zip them: `labels[i]` ↔ `hoursFlown[i]` ↔ `hectaresPerDay[i]`.
|
||
|
||
---
|
||
|
||
### 5.4 Active Jobs Panel
|
||
|
||
**URL**: `GET /api/dashboard/pilot/activeJobs`
|
||
|
||
Returns the pilot's active-status jobs with per-job progress and applied totals.
|
||
|
||
- Statuses included: `NEW (0)`, `READY (1)`, `DOWNLOADED (2)`, `SPRAYED (3)`, `COMPLETED (4)`
|
||
- Statuses excluded: `INVOICED (5)`, `ARCHIVED (9)`
|
||
- Maximum 50 most recent jobs returned (sorted by `createdAt` descending)
|
||
|
||
#### Query Parameters
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|--------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||
| `tz` | String | `UTC` | IANA timezone used to compute all period boundaries |
|
||
| `period` | String | — | Time window for both job list and Application totals: `day` \| `week` \| `month` \| `year`. Filters jobs by `createdAt` and Application totals by `startDateTimeUTC`. The `tz` parameter still controls how those calendar windows are derived. |
|
||
|
||
**Parameter priority**: `period` > *(none — all-time)*
|
||
|
||
> When a `period` is provided, only jobs created within that window are returned **and** the `haSprayed`/`volumeAppliedLiters` totals are scoped to applications whose `startDateTimeUTC` falls in the same calendar window. `haTotal` and `progressPct` always reflect the job's planned area regardless of the filter. The legacy string fields remain display-only; `tz` is still needed to translate the requested calendar period into UTC boundaries.
|
||
|
||
#### Response `200 OK`
|
||
|
||
```json
|
||
{
|
||
"jobs": [
|
||
{
|
||
"jobId": 1042,
|
||
"name": "North Block — Canola",
|
||
"clientName": "Sunrise Farms Ltd.",
|
||
"aircraftReg": "C-FABM",
|
||
"status": 3,
|
||
"displayStatus": "IN_PROGRESS",
|
||
"createdDate": "2026-06-01T14:22:00.000Z",
|
||
"haTotal": 250.00,
|
||
"haSprayed": 187.50,
|
||
"progressPct": 75.00,
|
||
"volumeAppliedLiters": 3750.00
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
#### Field Notes
|
||
|
||
| Field | Type | Notes |
|
||
|-----------------------|---------|--------------------------------------------------------------------|
|
||
| `jobId` | Number | Numeric job ID (`Job._id`, not a MongoDB ObjectId) |
|
||
| `name` | String | Job name. Empty string if not set |
|
||
| `clientName` | String | From linked `Client` user record. Empty if not linked |
|
||
| `aircraftReg` | String | `Vehicle.tailNumber` or `Vehicle.unitId`. Empty if not linked |
|
||
| `status` | Number | Raw numeric status (see constants table) |
|
||
| `displayStatus` | String | `"NEW"` \| `"IN_PROGRESS"` \| `"COMPLETED"` |
|
||
| `createdDate` | Date | `Job.createdAt` — UTC ISO 8601 timestamp of job creation. `null` if not set |
|
||
| `haTotal` | Number | `Job.ttSprArea` — planned area in ha. `0` if not set |
|
||
| `haSprayed` | Number | Sum of `Application.totalSprayed` for this job within the active window (processed apps only) |
|
||
| `progressPct` | Number | `0–100` (2 decimal places). `0` if `haTotal` is 0 or no applications |
|
||
| `volumeAppliedLiters` | Number | Sum of `Application.totalSprayMat` for this job within the active window |
|
||
|
||
**`progressPct` formula**: `parseFloat(min(100, max(0, haSprayed / haTotal × 100)).toFixed(2))`
|
||
Returns a float with up to 2 decimal places (e.g. `0.47` for a job under 1% complete). The backend caps at 100 if `haSprayed > haTotal`.
|
||
|
||
---
|
||
|
||
### 5.5 Performance Gauges
|
||
|
||
**URL**: `GET /api/dashboard/pilot/performance`
|
||
|
||
Returns average XT cross-track error and spray altitude gauges for all processed
|
||
application files within the requested date range.
|
||
|
||
**All values are calculated using spray-on records only** (`sprayStat === 1`),
|
||
giving meaningful agronomic metrics free from transit/ferry-flight pollution.
|
||
The average is record-weighted across all individual GPS data points (one row
|
||
per second of flight) in the period — days with more spray-on time have more
|
||
influence than days with fewer records.
|
||
|
||
> **Note**: ApplicationDetail is a billion-row collection. Queries are strictly
|
||
> bounded to a set of `fileId` values to use the collection's primary index.
|
||
> Never query this collection without a `fileId` filter.
|
||
|
||
#### Query Parameters
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-------------|--------|------------------------|----------------------------------------------------------------------|
|
||
| `tz` | String | `UTC` | IANA timezone string for date boundary calculations |
|
||
| `startDate` | String | Monday of current week | Range start date `YYYY-MM-DD` (inclusive) |
|
||
| `endDate` | String | Sunday of current week | Range end date `YYYY-MM-DD` (inclusive). Max range: 90 days |
|
||
|
||
**Date resolution**: `startDate`+`endDate` → current week default
|
||
|
||
#### Response `200 OK` — data available
|
||
|
||
```json
|
||
{
|
||
"startDate": "2026-05-19",
|
||
"endDate": "2026-05-25",
|
||
|
||
"avgXtError": 2.82,
|
||
"xtThreshold": {
|
||
"good": 1.0,
|
||
"monitor": 3.0
|
||
},
|
||
"hasXtData": true,
|
||
|
||
"avgSprayAltitudeMeters": 3.62,
|
||
"altitudeSource": "sprayHeight",
|
||
"altThreshold": {
|
||
"target": 3.7,
|
||
"goodBand": 0.15,
|
||
"monitorBand": 0.46
|
||
},
|
||
"hasAltitudeData": true,
|
||
"sampleSize": 4
|
||
}
|
||
```
|
||
|
||
#### Response `200 OK` — no data (new pilot, no apps uploaded)
|
||
|
||
```json
|
||
{
|
||
"startDate": "2026-05-19",
|
||
"endDate": "2026-05-25",
|
||
|
||
"avgXtError": null,
|
||
"xtThreshold": { "good": 1.0, "monitor": 3.0 },
|
||
"hasXtData": false,
|
||
|
||
"avgSprayAltitudeMeters": null,
|
||
"altitudeSource": null,
|
||
"altThreshold": { "target": 3.7, "goodBand": 0.15, "monitorBand": 0.46 },
|
||
"hasAltitudeData": false,
|
||
"sampleSize": 0
|
||
}
|
||
```
|
||
|
||
#### Field Notes
|
||
|
||
| Field | Unit | Notes |
|
||
|---------------------------|--------|--------------------------------------------------------------------------------------------|
|
||
| `startDate` | String | Effective start of the analysis window (`YYYY-MM-DD`) |
|
||
| `endDate` | String | Effective end of the analysis window (`YYYY-MM-DD`) |
|
||
| `avgXtError` | metres | Mean `abs(xTrack)` across all spray-on GPS records. `null` if no data |
|
||
| `xtThreshold.good` | metres | Below this → green zone |
|
||
| `xtThreshold.monitor` | metres | Above this → red zone; between good and monitor → yellow |
|
||
| `hasXtData` | bool | `false` = no valid xTrack readings in sample (show "No data") |
|
||
| `avgSprayAltitudeMeters` | metres | Mean spray height from best available sensor, spray-on records only. `null` if none |
|
||
| `altitudeSource` | String | `"sprayHeight"` (FM dedicated sensor) or `"radarAlt"` (AGL fallback). `null` if no data |
|
||
| `altThreshold.target` | metres | Ideal spray height (~3.7 m) |
|
||
| `altThreshold.goodBand` | metres | ±0.15 m of target → green |
|
||
| `altThreshold.monitorBand`| metres | ±0.46 m of target → yellow; outside → red |
|
||
| `hasAltitudeData` | bool | `false` = no altitude sensor data in sample (show "No data") |
|
||
| `sampleSize` | count | Number of AppFile records included in the analysis window |
|
||
|
||
#### Threshold Gauge Logic (frontend)
|
||
|
||
```
|
||
XT Cross-track error:
|
||
value < good → green (acceptable precision)
|
||
good ≤ value < monitor → yellow (monitor)
|
||
value ≥ monitor → red (needs attention)
|
||
|
||
Spray Altitude:
|
||
|value - target| < goodBand → green
|
||
|value - target| < monitorBand → yellow
|
||
|value - target| ≥ monitorBand → red
|
||
```
|
||
|
||
#### Altitude Source Priority
|
||
|
||
The backend selects the best available altitude sensor in this order:
|
||
1. **`sprayHeight`** — dedicated FM spray height sensor (most accurate)
|
||
2. **`radarAlt`** — radar altimeter (AGL, good fallback)
|
||
3. **No data** — `hasAltitudeData: false`, `avgSprayAltitudeMeters: null`
|
||
|
||
> `gpsAlt` (AMSL) is **not** used for spray height — it measures altitude above sea level,
|
||
> not above the crop canopy.
|
||
|
||
`xTrack = 0` values are excluded from the XT average because `0` means "no reading",
|
||
not "perfectly on track".
|
||
|
||
---
|
||
|
||
### 5.6 Save Performance Thresholds
|
||
|
||
Persists custom XT error and altitude gauge thresholds for the authenticated pilot.
|
||
Values are stored per-user in `Setting.dashboard` and are automatically applied
|
||
the next time `GET /pilot/performance` is called.
|
||
|
||
All fields are optional — send only the fields you want to change. Pass `null` for a
|
||
field to reset it to the system default.
|
||
|
||
**URL**: `PUT /api/dashboard/pilot/performance/thresholds`
|
||
|
||
#### Request Body (JSON)
|
||
|
||
| Field | Type | Default (system) | Description |
|
||
|------------------|---------------|------------------|--------------------------------------------------------|
|
||
| `xtGood` | Number \| null | `1.0` | XT ideal threshold in metres (top of green zone) |
|
||
| `xtMonitor` | Number \| null | `3.0` | XT caution threshold in metres (top of yellow zone) |
|
||
| `altTarget` | Number \| null | `3.7` | Altitude target in metres (centre of altitude gauge) |
|
||
| `altGoodBand` | Number \| null | `0.15` | ±band from target for green zone |
|
||
| `altMonitorBand` | Number \| null | `0.46` | ±band from target for yellow zone |
|
||
|
||
**Constraints**:
|
||
- All supplied values must be positive finite numbers (`> 0`).
|
||
- `xtMonitor` must be greater than `xtGood`.
|
||
- `altMonitorBand` must be greater than `altGoodBand`.
|
||
- Constraints are checked against the **effective post-save value** for each field: the value being saved in this request → if being reset (`null`), the system default → the currently stored custom value → the system default. This means partial updates (sending only some fields) are correctly validated against the real in-DB state.
|
||
|
||
#### Example Request Body
|
||
|
||
```json
|
||
{
|
||
"xtGood": 7.0,
|
||
"xtMonitor": 13.0
|
||
}
|
||
```
|
||
|
||
> Only the two XT fields are sent — altitude thresholds remain unchanged.
|
||
|
||
#### Response `200 OK`
|
||
|
||
Returns the **effective** thresholds after saving (saved value, or system default if not customised):
|
||
|
||
```json
|
||
{
|
||
"xtThreshold": {
|
||
"good": 7.0,
|
||
"monitor": 13.0
|
||
},
|
||
"altThreshold": {
|
||
"target": 3.7,
|
||
"goodBand": 0.15,
|
||
"monitorBand": 0.46
|
||
}
|
||
}
|
||
```
|
||
|
||
> The frontend can use this response to immediately update the gauge without a separate `GET /performance` call.
|
||
|
||
#### Reset to Defaults
|
||
|
||
Pass `null` to clear a custom value and revert to the system default:
|
||
|
||
```json
|
||
{ "xtGood": null, "xtMonitor": null }
|
||
```
|
||
|
||
#### Error Cases
|
||
|
||
| Condition | Status | Error tag |
|
||
|---------------------------------------------------|--------|-----------------|
|
||
| Missing or invalid JWT | `401` | `not_authorized`|
|
||
| Non-positive or non-finite value | `409` | `invalid_param` |
|
||
| `xtMonitor` ≤ `xtGood` (effective values) | `409` | `invalid_param` |
|
||
| `altMonitorBand` ≤ `altGoodBand` (effective) | `409` | `invalid_param` |
|
||
|
||
---
|
||
|
||
### 5.7 Mark Job as Completed
|
||
|
||
**URL**: `PATCH /api/jobs/:job_id/complete`
|
||
|
||
Transitions a job from **SPRAYED (3)** to **COMPLETED (4)**.
|
||
|
||
This endpoint lives under `/api/jobs` (not `/api/dashboard`) and requires an active
|
||
subscription package (enforced by `checkRqPkgSubscription` middleware).
|
||
|
||
#### URL Parameters
|
||
|
||
| Parameter | Type | Description |
|
||
|-----------|--------|----------------------------------------|
|
||
| `job_id` | Number | Numeric job ID |
|
||
|
||
#### Request Body
|
||
|
||
None.
|
||
|
||
#### Response `200 OK`
|
||
|
||
Returns the full updated job document (with `client`, `operator`, and `vehicle` populated).
|
||
|
||
```json
|
||
{
|
||
"_id": 1042,
|
||
"name": "North Block — Canola",
|
||
"status": 4,
|
||
"client": { "_id": "...", "name": "Sunrise Farms Ltd." },
|
||
"operator": { "_id": "...", "name": "Jane Pilot" },
|
||
"vehicle": { "_id": "...", "name": "Agri-One", "tailNumber": "C-FABM" },
|
||
"..."
|
||
}
|
||
```
|
||
|
||
#### Authorization
|
||
|
||
The **Applicator** who owns the job (`Job.byPuid`) may complete it, as may any **sub-user
|
||
(Pilot)** operating under that Applicator. The check compares `Job.byPuid` against
|
||
`req.userInfo.puid` (the caller's root Applicator ID), which resolves correctly for both
|
||
the Applicator themselves and their sub-users.
|
||
|
||
| Caller type | `req.userInfo.puid` | Allowed? |
|
||
|-------------|---------------------|----------|
|
||
| Applicator | same as `req.uid` | ✅ Yes |
|
||
| Pilot sub-user under the owning Applicator | parent's `_id` | ✅ Yes |
|
||
| Any other user | different from `Job.byPuid` | ❌ 401 |
|
||
|
||
#### Error Cases
|
||
|
||
| Condition | Status | Error tag |
|
||
|----------------------------------------------|--------|-----------------------|
|
||
| `job_id` is not a valid positive number | `409` | `invalid_param` |
|
||
| Job not found | `409` | `job_not_found` |
|
||
| Caller is not the job owner (`byPuid`) | `401` | `not_authorized` |
|
||
| Job status is not `SPRAYED (3)` | `409` | `status_job_invalid` |
|
||
|
||
---
|
||
|
||
### 5.8 Snapshot (Composite Dashboard)
|
||
|
||
Returns multiple dashboard modules in a single request. Eliminates N+1 API calls on
|
||
initial page load and during periodic polling.
|
||
|
||
**URL**: `GET /api/dashboard/pilot/snapshot`
|
||
|
||
All sub-modules share the same pilot-scoped job/app data fetch internally — no redundant
|
||
database queries. Each module uses its own query parameter validation (date-range modules
|
||
independently validate `startDate`/`endDate`; `kpi`, `summary`, and `activeJobs` use `tz`;
|
||
`activeJobs` additionally accepts `period` for time-scoped sub-totals).
|
||
|
||
#### Query Parameters
|
||
|
||
| Parameter | Type | Default | Applies to module(s) | Description |
|
||
|-------------|--------|-------------------------------|--------------------------------|-------------------------------------------------------------|
|
||
| `include` | String | `kpi,summary,activeJobs,performance,trend` | — | Comma-separated list of modules to return. Unknown names are silently ignored. |
|
||
| `tz` | String | `UTC` | all | IANA timezone string |
|
||
| `period` | String | *(omit = all-time)* | `activeJobs` | Time window for both job list and Application totals: `day` \| `week` \| `month` \| `year`. Filters jobs by `createdAt` and Application totals by `startDateTimeUTC`. The `tz` parameter still controls the period boundaries. |
|
||
| `startDate` | String | Monday of current week | `performance`, `trend` | `YYYY-MM-DD` |
|
||
| `endDate` | String | Sunday of current week | `performance`, `trend` | `YYYY-MM-DD`. Max range: 90 days |
|
||
|
||
Valid `include` values: `kpi` · `summary` · `activeJobs` · `performance` · `trend`
|
||
|
||
> **Note on `invalid_param` from snapshot with trend**: The 90-day cap applies inside the
|
||
> snapshot just as it does on the standalone `/trend` endpoint. If `startDate`→`endDate`
|
||
> spans more than 90 days and `trend` (or `performance`) is in the `include` list, the
|
||
> entire request returns `409 invalid_param`. To avoid this, keep the date range ≤ 90 days.
|
||
|
||
#### Response `200 OK`
|
||
|
||
Only requested modules are present in the response object. Example with all modules:
|
||
|
||
```json
|
||
{
|
||
"kpi": {
|
||
"operations": {
|
||
"missionsFlown": 2,
|
||
"distanceTravelledKm": 45.2,
|
||
"distanceSprayedKm": 32.1,
|
||
"sprayEfficiencyPct": 71.00,
|
||
"ferryTimePct": 29.00,
|
||
"flowAccuracyPct": 98.50,
|
||
"avgHdop": 1.10
|
||
},
|
||
"periods": {
|
||
"day": { "assignedJobs": 3, "assignedHectares": 45.5, "sprayedHectares": 32.1, "flightHours": 1.53, "jobCounts": { "new": 1, "inProgress": 1, "completed": 1 } },
|
||
"week": { "...": "same shape" },
|
||
"month": { "...": "same shape" },
|
||
"year": { "...": "same shape" },
|
||
"all": { "...": "same shape" }
|
||
}
|
||
},
|
||
"summary": {
|
||
"today": { "hectares": 32.1, "flightHours": 1.53, "haPerHour": 21.0, "avgSpeedKmh": 45.2, "sprayVolumeLiters": 256.0 },
|
||
"yesterday": { "...": "same shape" },
|
||
"todayHasData": true,
|
||
"deltas": { "hectaresPct": 15, "flightHoursPct": 10, "haPerHourPct": null, "avgSpeedPct": null, "sprayVolumePct": null }
|
||
},
|
||
"activeJobs": {
|
||
"jobs": [
|
||
{ "jobId": 42, "name": "North Block", "status": 3, "displayStatus": "IN_PROGRESS", "createdDate": "2026-06-01T14:22:00.000Z", "progressPct": 75.00, "haTotal": 250.0, "haSprayed": 187.5, "volumeAppliedLiters": 3750.0 }
|
||
]
|
||
},
|
||
"performance": {
|
||
"startDate": "2026-05-19", "endDate": "2026-05-25",
|
||
"avgXtError": 2.82, "xtThreshold": { "good": 1.0, "monitor": 3.0 }, "hasXtData": true,
|
||
"avgSprayAltitudeMeters": 3.62, "altitudeSource": "sprayHeight",
|
||
"altThreshold": { "target": 3.7, "goodBand": 0.15, "monitorBand": 0.46 },
|
||
"hasAltitudeData": true, "sampleSize": 4
|
||
},
|
||
"trend": {
|
||
"labels": ["2026-05-19", "2026-05-20", "2026-05-21"],
|
||
"hoursFlown": [1.53, 2.1, 0],
|
||
"hectaresPerDay":[32.1, 45.5, 0]
|
||
}
|
||
}
|
||
```
|
||
|
||
**Selective fetch examples**:
|
||
|
||
```
|
||
GET /api/dashboard/pilot/snapshot?include=kpi → only kpi
|
||
GET /api/dashboard/pilot/snapshot?include=kpi,summary,activeJobs → 3 modules (recommended for periodic polling)
|
||
GET /api/dashboard/pilot/snapshot?include=kpi,summary,activeJobs&period=week → activeJobs haSprayed/volume scoped to current week
|
||
GET /api/dashboard/pilot/snapshot?include=performance,trend&startDate=2026-05-01&endDate=2026-05-14
|
||
```
|
||
|
||
#### Error Cases
|
||
|
||
| Condition | Status | Error tag |
|
||
|--------------------------------------------------|--------|------------------|
|
||
| Missing/invalid JWT | `401` | `not_authorized` |
|
||
| All `include` values are unrecognised | `409` | `invalid_param` |
|
||
| Date range > 90 days (when `trend`/`performance` included) | `409` | `invalid_param` |
|
||
| Bad date format | `409` | `invalid_param` |
|
||
| Invalid `period` value (not day/week/month/year) | `409` | `invalid_param` |
|
||
|
||
---
|
||
|
||
## 6 Error Responses
|
||
|
||
All errors follow the standard AgMission error format:
|
||
|
||
```json
|
||
{
|
||
"error": {
|
||
".tag": "error_constant_value",
|
||
"message": "Human-readable detail (development mode only)"
|
||
}
|
||
}
|
||
```
|
||
|
||
| HTTP Status | `.tag` value | When it occurs |
|
||
|-------------|------------------------|------------------------------------------------------|
|
||
| `401` | `not_authorized` | Missing/invalid JWT, or caller not the job owner |
|
||
| `409` | `invalid_param` | Bad date format, range > 90 days, invalid job_id, all `include` modules unrecognised |
|
||
| `409` | `job_not_found` | Job does not exist |
|
||
| `409` | `status_job_invalid` | Job is not in required status for transition |
|
||
|
||
---
|
||
|
||
## 7 Data Model Notes
|
||
|
||
### Pilot Scoping
|
||
|
||
All Application metrics are scoped via the Job's `operator` field, not `byUser` on Application:
|
||
|
||
```
|
||
Job.operator = pilotId → collect jobIds → Application.jobId IN jobIds
|
||
```
|
||
|
||
`Application.byUser` is the master Applicator account — it is **not** the pilot. Never use it
|
||
to scope pilot metrics.
|
||
|
||
### Application UTC fields
|
||
|
||
Time-window filtering uses `Application.startDateTimeUTC` (the canonical spray-time timestamp),
|
||
with `Application.endDateTimeUTC` available for end-of-flight queries and `utcOffset` available
|
||
for display. The legacy `startDateTime` / `endDateTime` strings remain display-only and are not
|
||
used for dashboard date arithmetic.
|
||
|
||
### Job._id is a Number
|
||
|
||
`Job._id` is an auto-incrementing Number (via `mongoose-sequence`), not a MongoDB ObjectId.
|
||
Frontend must treat `jobId` values as integers, not hex strings.
|
||
|
||
### Vehicle and Client are User discriminators
|
||
|
||
Both `Vehicle` (kind=`DEVICE`) and `Client` (kind=`CLIENT`) are stored in the `users` MongoDB
|
||
collection. The `$lookup` in `activeJobs` joins against `users` for both.
|
||
|
||
### Models and Collections
|
||
|
||
| Model name in code | Collection | Key dashboard fields |
|
||
|--------------------|-----------------------|-------------------------------------------------------------|
|
||
| `Job` | `jobs` | `operator`, `byPuid`, `ttSprArea`, `status`, `client`, `vehicle` |
|
||
| `App` (Application)| `applications` | `jobId`, `status`, `startDateTimeUTC`, `endDateTimeUTC`, `utcOffset`, `totalSprayed`, `totalFlightTime`, `totalSprayTime`, `totalSprLength`, `totalFlightLength`, `totalSprayMat`, `avgSpraySpeed`, `appRate`, `avgHdop`, `flowAccuracyPct` |
|
||
| `AppFile` | `appfiles` | `appId` |
|
||
| `AppDetail` | `application_details` | `fileId`, `xTrack`, `sprayHeight`, `radarAlt` |
|
||
| `User` | `users` | `operator`, `byPuid` (scoping only — no dashboard-specific fields) |
|
||
| `Setting` | `settings` | `dashboard` (custom gauge thresholds per pilot) |
|
||
|
||
### Setting.dashboard
|
||
|
||
Each pilot's custom gauge thresholds are stored as an optional nested object in their `Setting`
|
||
document (the `settings` collection, linked by `userId`). All fields default to `undefined`
|
||
(not set) — the system constants are used when the field is absent.
|
||
|
||
```
|
||
Setting.dashboard: {
|
||
xtGood: Number | undefined // XT ideal threshold (m)
|
||
xtMonitor: Number | undefined // XT caution threshold (m)
|
||
altTarget: Number | undefined // Altitude target (m)
|
||
altGoodBand: Number | undefined // ±band for green zone (m)
|
||
altMonitorBand: Number | undefined // ±band for yellow zone (m)
|
||
}
|
||
```
|
||
|
||
System defaults (used when the field is absent):
|
||
|
||
| Field | Default |
|
||
|------------------|---------|
|
||
| `xtGood` | `1.0` |
|
||
| `xtMonitor` | `3.0` |
|
||
| `altTarget` | `3.7` |
|
||
| `altGoodBand` | `0.15` |
|
||
| `altMonitorBand` | `0.46` |
|
||
|
||
---
|
||
|
||
## 8 Frontend Integration Guide
|
||
|
||
### Suggested Fetch Strategy
|
||
|
||
Load the dashboard in two tiers to keep the initial paint fast:
|
||
|
||
**Tier 1 — above the fold (load in parallel on mount)**
|
||
```
|
||
GET /api/dashboard/pilot/kpi?tz=...
|
||
GET /api/dashboard/pilot/summary?tz=...
|
||
GET /api/dashboard/pilot/activeJobs?tz=...&period=week
|
||
```
|
||
|
||
**Tier 2 — charts and gauges (load after Tier 1 resolves or in parallel)**
|
||
```
|
||
GET /api/dashboard/pilot/trend?tz=...&startDate=...&endDate=...
|
||
GET /api/dashboard/pilot/performance?tz=...
|
||
```
|
||
|
||
**When the user saves custom thresholds**
|
||
```
|
||
PUT /api/dashboard/pilot/performance/thresholds { xtGood: 7, xtMonitor: 13 }
|
||
→ use the response directly to update the gauge (no extra GET needed)
|
||
```
|
||
|
||
**Period filter — when the user switches Day / Week / Month / Year tab**
|
||
```
|
||
GET /api/dashboard/pilot/activeJobs?tz=...&period=day
|
||
GET /api/dashboard/pilot/activeJobs?tz=...&period=week
|
||
GET /api/dashboard/pilot/activeJobs?tz=...&period=month
|
||
GET /api/dashboard/pilot/activeJobs?tz=...&period=year
|
||
```
|
||
|
||
### Timezone Snippet
|
||
|
||
```javascript
|
||
// Read once and reuse
|
||
const tz = Intl.DateTimeFormat().resolvedOptions().timeZone; // e.g. "America/Sao_Paulo"
|
||
|
||
const params = new URLSearchParams({ tz });
|
||
fetch(`/api/dashboard/pilot/kpi?${params}`, { headers: { Authorization: `Bearer ${token}` } });
|
||
```
|
||
|
||
### Empty State Handling
|
||
|
||
Every endpoint returns meaningful zero-filled data when the pilot has no jobs:
|
||
- Number fields return `0`
|
||
- Null fields (`avgXtError`, `altitudeSource`) return `null`
|
||
- Boolean guard fields (`hasXtData`, `hasAltitudeData`) return `false`
|
||
- Array fields (`labels`, `hoursFlown`, `hectaresPerDay`) return empty `[]`
|
||
|
||
The frontend should check `hasXtData` and `hasAltitudeData` before rendering gauges, and
|
||
render a "No data yet" placeholder instead of a gauge at `null`.
|
||
|
||
### Progress Bar Colour Logic
|
||
|
||
Use `displayStatus` from `activeJobs` for badge colour, and `progressPct` for bar width:
|
||
|
||
```
|
||
displayStatus = "NEW" → grey badge, hide progress bar
|
||
displayStatus = "IN_PROGRESS" → blue badge, show progress bar at progressPct%
|
||
displayStatus = "COMPLETED" → green badge, show bar at 100%
|
||
```
|
||
|
||
### Mark as Completed Button Visibility
|
||
|
||
The **Complete** action is restricted to the **Applicator** (`Job.byPuid`), not the Pilot.
|
||
If the frontend is used by an Applicator who is reviewing a pilot's job, show the button only
|
||
when the raw `status === 3` (SPRAYED). After a successful `PATCH`, update the job in local
|
||
state to `status = 4`, `displayStatus = "COMPLETED"`.
|
||
|
||
### Trend Chart Date Pickers
|
||
|
||
- Pass `startDate`/`endDate` as `YYYY-MM-DD` strings in the user's local timezone.
|
||
- Maximum selectable range: 90 days.
|
||
- Default range on first render: current week (omit both parameters and let the API default).
|
||
|
||
### Filter Parameters Quick Reference
|
||
|
||
| Scenario | Parameters to use |
|
||
|--------------------------------------|--------------------------------------------------------------------------|
|
||
| User switches Day/Week/Month/Year tab | `period=day\|week\|month\|year` on `/activeJobs` |
|
||
| User picks a date range in the trend | `startDate` + `endDate` on `/trend` and `/performance` |
|
||
| Dashboard reset to today | Omit `period` — all endpoints revert to live defaults |
|
||
|
||
### Polling Strategy
|
||
|
||
Dashboard widgets show live data that changes as jobs progress and applications are uploaded.
|
||
The recommended approach is **interval-based pull polling** — no WebSockets needed for Phase 1.
|
||
|
||
| Data group | Changes when | Recommended interval |
|
||
|-----------------------------|--------------------------------------|----------------------|
|
||
| KPI · Summary · ActiveJobs | Job status changes, apps uploaded | Every 60 s |
|
||
| Trend | Application uploaded on the day | Every 5 min |
|
||
| Performance | Heavy aggregation, historical | On demand / 5 min |
|
||
|
||
**Guards against excessive requests:**
|
||
|
||
1. **Page Visibility API** — pause polling when the tab is hidden, restart on focus:
|
||
|
||
*Plain JS (framework-agnostic):*
|
||
```javascript
|
||
document.addEventListener('visibilitychange', () => {
|
||
document.hidden ? clearInterval(pollTimer) : (pollTimer = startPolling());
|
||
});
|
||
```
|
||
|
||
*Angular 9.x with NgRx + RxJS (`@ngrx/effects`):*
|
||
```typescript
|
||
import { Injectable, NgZone } from '@angular/core';
|
||
import { Actions, createEffect, ofType } from '@ngrx/effects';
|
||
import { Store } from '@ngrx/store';
|
||
import { fromEvent, merge, of, timer, EMPTY } from 'rxjs';
|
||
import {
|
||
switchMap, map, exhaustMap, takeUntil, filter
|
||
} from 'rxjs/operators';
|
||
import { HttpClient } from '@angular/common/http';
|
||
import * as DashboardActions from './dashboard.actions';
|
||
|
||
@Injectable()
|
||
export class DashboardPollEffects {
|
||
|
||
/** Emits true when the tab is visible, false when hidden. */
|
||
private readonly visibility$ = merge(
|
||
of(!document.hidden), // initial state
|
||
fromEvent(document, 'visibilitychange').pipe(
|
||
map(() => !document.hidden)
|
||
)
|
||
);
|
||
|
||
/** Core poll stream: 60 s interval, paused while tab is hidden. */
|
||
readonly pollSnapshot$ = createEffect(() =>
|
||
this.actions$.pipe(
|
||
ofType(DashboardActions.startPolling),
|
||
switchMap(() =>
|
||
this.visibility$.pipe(
|
||
switchMap(visible =>
|
||
visible
|
||
? timer(0, 60_000).pipe( // tick immediately, then every 60 s
|
||
exhaustMap(() =>
|
||
this.http.get<SnapshotResponse>('/api/dashboard/pilot/snapshot')
|
||
.pipe(map(data => DashboardActions.snapshotLoaded({ data })))
|
||
)
|
||
)
|
||
: EMPTY // tab hidden — no requests
|
||
),
|
||
takeUntil(this.actions$.pipe(ofType(DashboardActions.stopPolling)))
|
||
)
|
||
)
|
||
)
|
||
);
|
||
|
||
/** Stop polling on component destroy (dispatched from ngOnDestroy). */
|
||
readonly stopPolling$ = createEffect(() =>
|
||
this.actions$.pipe(
|
||
ofType(DashboardActions.stopPolling),
|
||
map(() => DashboardActions.pollingStopped())
|
||
)
|
||
);
|
||
|
||
constructor(
|
||
private actions$: Actions,
|
||
private http: HttpClient,
|
||
private ngZone: NgZone
|
||
) {}
|
||
}
|
||
```
|
||
|
||
*Corresponding actions (`dashboard.actions.ts`):*
|
||
```typescript
|
||
import { createAction, props } from '@ngrx/store';
|
||
|
||
export const startPolling = createAction('[Dashboard] Start Polling');
|
||
export const stopPolling = createAction('[Dashboard] Stop Polling');
|
||
export const pollingStopped = createAction('[Dashboard] Polling Stopped');
|
||
export const snapshotLoaded = createAction(
|
||
'[Dashboard] Snapshot Loaded',
|
||
props<{ data: SnapshotResponse }>()
|
||
);
|
||
```
|
||
|
||
*Component wiring:*
|
||
```typescript
|
||
// dashboard.component.ts
|
||
ngOnInit() { this.store.dispatch(DashboardActions.startPolling()); }
|
||
ngOnDestroy() { this.store.dispatch(DashboardActions.stopPolling()); }
|
||
```
|
||
|
||
> **Angular 9 note**: `createEffect` requires `@ngrx/effects` ≥ 9.x (ships with Angular 9
|
||
> LTS). The `exhaustMap` inside the timer prevents a slow response from queuing duplicate
|
||
> requests — equivalent to the `clearInterval` guard in the plain-JS version.
|
||
|
||
2. **Batch the live-update group** — see proposed `GET /snapshot` in §9.4. Three concurrent
|
||
requests per tick become one, reducing server load 3× at scale.
|
||
3. **Skip Tier 2 on polls** — `/trend` and `/performance` require heavy aggregation. Only
|
||
re-fetch them when the user changes the date range or when the tab regains focus after
|
||
more than 5 min of inactivity.
|
||
|
||
---
|
||
|
||
### Gauge Rendering Reference
|
||
|
||
```
|
||
XT Error gauge (lower is better):
|
||
0 m ──────── 1.0 m ──────────── 3.0 m ──────────→
|
||
green yellow red
|
||
↑ good ↑ monitor
|
||
|
||
Altitude gauge (target 3.7 m):
|
||
← red ── 3.24 ── yellow ── 3.55 ── [3.7] ── 3.85 ── yellow ── 4.16 ── red →
|
||
↑ -0.46 ↑ -0.15 target ↑ +0.15 ↑ +0.46
|
||
```
|
||
|
||
---
|
||
|
||
## 9 Backend Architecture Notes
|
||
|
||
### 9.1 Pilot Scope Data Flow Diagram
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
A[Authenticated User req uid] --> B[Job Query operator equals uid]
|
||
B --> C[Collect Job IDs]
|
||
C --> D[Application Query jobId in Job IDs and status 3]
|
||
D --> E[KPI Summary Trend Active Jobs Aggregates]
|
||
```
|
||
|
||
### 9.2 Endpoint Interaction Diagram
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
FE[Frontend Pilot Dashboard] --> K[GET pilot kpi]
|
||
FE --> S[GET pilot summary]
|
||
FE --> T[GET pilot trend]
|
||
FE --> A[GET pilot activeJobs]
|
||
FE --> P[GET pilot performance]
|
||
FE --> TH[PUT pilot performance thresholds]
|
||
FE --> C[PATCH jobs job_id complete]
|
||
|
||
K --> J[(jobs)]
|
||
K --> AP[(applications)]
|
||
S --> AP
|
||
T --> AP
|
||
A --> J
|
||
A --> AP
|
||
P --> AP
|
||
P --> AF[(appfiles)]
|
||
P --> AD[(application_details)]
|
||
TH --> ST[(settings)]
|
||
C --> J
|
||
```
|
||
|
||
### 9.3 Performance Query Safety Diagram
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
A[Scoped Job IDs] --> B[Processed Applications within date range]
|
||
B --> C[AppFile Lookup by appId]
|
||
C --> D[ApplicationDetail Match by fileId IN list]
|
||
D --> E[Aggregate spray-on records only sprayStat eq 1]
|
||
E --> F[Return XT error and altitude averages]
|
||
```
|
||
|
||
---
|
||
|
||
### 9.4 Periodic Polling Sequence
|
||
|
||
The sequence below covers the full client lifecycle: two-tier initial load, periodic
|
||
polling, and user-driven drilldown. The proposed `/snapshot` endpoint replaces three
|
||
parallel Tier-1 poll requests with one, cutting polling overhead by 3×.
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant FE as Frontend
|
||
participant API as Dashboard API
|
||
participant DB as MongoDB
|
||
|
||
Note over FE,DB: 1 — Initial load
|
||
par Tier 1 (parallel)
|
||
FE->>API: GET /kpi?tz=...
|
||
and
|
||
FE->>API: GET /summary?tz=...
|
||
and
|
||
FE->>API: GET /activeJobs?tz=...&period=week
|
||
end
|
||
API->>DB: Aggregate jobs + applications
|
||
DB-->>API: Results
|
||
API-->>FE: KPI · Summary · ActiveJobs
|
||
|
||
par Tier 2 (parallel)
|
||
FE->>API: GET /trend?tz=...
|
||
and
|
||
FE->>API: GET /performance?tz=...
|
||
end
|
||
API->>DB: Aggregate application + detail records
|
||
DB-->>API: Results
|
||
API-->>FE: Trend · Performance gauge
|
||
|
||
Note over FE,DB: 2 — Periodic refresh (tab visible, every 60 s)
|
||
loop Poll interval
|
||
FE->>API: GET /snapshot?include=kpi,summary,activeJobs&tz=...
|
||
API->>DB: Aggregate jobs + applications (lightweight)
|
||
DB-->>API: Batch results
|
||
API-->>FE: { kpi, summary, activeJobs }
|
||
end
|
||
Note over FE,DB: trend + performance refreshed on date filter change or interval change or refresh button click only
|
||
|
||
```
|
||
|
||
#### `GET /api/dashboard/pilot/snapshot`
|
||
|
||
A single endpoint that returns any combination of KPI, Summary, ActiveJobs, Performance,
|
||
and Trend in one round-trip, eliminating the N+1 overhead during periodic polling.
|
||
|
||
**Query parameters**: `include` (module list) + `tz` + `startDate`/`endDate` (for `trend`/`performance` modules). See §5.8 for full documentation.
|
||
|
||
**Response shape** (all modules):
|
||
|
||
```json
|
||
{
|
||
"kpi": { },
|
||
"summary": { },
|
||
"activeJobs": { },
|
||
"performance":{ },
|
||
"trend": { }
|
||
}
|
||
```
|
||
|
||
Each nested object has the same shape as the corresponding individual endpoint response.
|
||
|
||
> **Recommended polling payload**: `?include=kpi,summary,activeJobs` — see [§ Why trend and performance are excluded from periodic polling](#why-trend-and-performance-are-excluded-from-periodic-polling) for the rationale.
|
||
|
||
### Why trend and performance are excluded from periodic polling
|
||
|
||
The 60 s poll uses `?include=kpi,summary,activeJobs` deliberately. `trend` and `performance`
|
||
are omitted for two independent reasons:
|
||
|
||
**1. Query cost**
|
||
- `kpi`, `summary`, and `activeJobs` aggregate over `applications` and `jobs` only — collections
|
||
that are indexed on `operator`/`jobId` and contain O(thousands) of documents per pilot.
|
||
- `performance` additionally fans out into `application_details` (potentially billions of rows)
|
||
via `{ fileId: 1 }` index lookups — one query per AppFile in the date range.
|
||
- `trend` runs a per-day aggregation across the full `applications` collection for the chosen
|
||
date window. Wider windows compound the cost.
|
||
- Running these every 60 s for every open browser tab creates avoidable DB pressure.
|
||
|
||
**2. Data-change frequency**
|
||
- KPI counters, today's operational totals, and active-job progress change continuously
|
||
throughout the working day — sub-minute freshness is meaningful.
|
||
- Trend charts (daily hectares/hours) and performance gauges (cross-track error, spray height)
|
||
are computed from uploaded flight records. They only change when a new Application is
|
||
uploaded, which happens at most a few times per day. There is no value in re-fetching
|
||
them every 60 s.
|
||
|
||
**When to refresh trend and performance**
|
||
- On initial page load (already in Tier 2 of the sequence diagram above).
|
||
- When the user changes the date filter.
|
||
- On tab re-focus after a long absence (`visibilitychange` event + staleness check).
|
||
- After the user manually triggers a refresh (for example, by clicking a refresh button or changing the refresh interval from a dropdown list (5/10/30/60 minutes)).
|
||
|
||
Custom `startDate`/`endDate` parameters are a secondary consideration: the frontend would
|
||
need to remember the current filter state to include them in a poll, which adds complexity;
|
||
but the cost and staleness arguments above are the primary reason for the separation.
|
||
|
||
---
|
||
|
||
### Why Job.operator and not Application.byUser
|
||
|
||
`Application.byUser` is the master Applicator account that uploaded the file — not the
|
||
pilot assigned to fly the job. Using it would mix data from all jobs the Applicator manages,
|
||
not just those assigned to this pilot. The correct field is `Job.operator` (set when a pilot
|
||
is assigned to a job).
|
||
|
||
### Why Application.startDateTimeUTC and not the legacy string fields
|
||
|
||
`Application.startDateTime` and `endDateTime` are legacy display fields. They are still computed
|
||
during file import by two different code paths, and the two paths produce **incompatible values**
|
||
in both format and timezone semantics. The dashboard now queries the UTC companion fields
|
||
instead: `startDateTimeUTC`, `endDateTimeUTC`, and `utcOffset`.
|
||
|
||
#### AgNav binary (`.nt`) files — `workers/job_worker.js: computeStartEndDate()`
|
||
|
||
```
|
||
AgNav filename → YYYYMMDD (LOCAL mission date assigned by the device)
|
||
GPS seconds-of-day → HHmmss (GPS UTC time-of-day — already UTC, not local)
|
||
|
||
combined as-is → "20250522T000632"
|
||
```
|
||
|
||
Result: **legacy hybrid string**: `datePart` = local mission date, `timePart` = GPS UTC
|
||
time-of-day. **Both parts carry independent semantics** — the date is local, the time is UTC.
|
||
Converting to a proper UTC Date (`startDateTimeUTC`) requires a UTC offset derived from the
|
||
flight location; see `helpers/application_datetime.js: toUtcDateFromAppDateTime()` for the
|
||
date-shift logic that handles timezone crossings correctly.
|
||
|
||
#### SatLoc (`.log`) files — `helpers/satloc_application_processor.js`
|
||
`
|
||
```
|
||
record.gpsTime ← Unix epoch seconds (UTC)
|
||
new Date(gpsTime * 1000).toISOString() → "2020-07-29T00:15:38.030Z"
|
||
```
|
||
|
||
Result: **UTC time** as ISO 8601 string with `Z` suffix. The new `startDateTimeUTC` and
|
||
`endDateTimeUTC` fields store the canonical UTC Date values directly.
|
||
|
||
#### Why this makes range queries impossible
|
||
|
||
A pilot spraying at 10:06 local time (UTC+10) produces two entirely different stored values
|
||
depending on which file type was uploaded:
|
||
|
||
| Source | Stored value | Meaning |
|
||
|--------|-------------|---------|
|
||
| AgNav | `"20250729T100634"` | legacy hybrid string (local date + UTC time-of-day) |
|
||
| SatLoc | `"2025-07-29T00:06:34.000Z"` | UTC 00:06 (same moment) |
|
||
| Both | `startDateTimeUTC` / `endDateTimeUTC` | canonical UTC Date values for query/filter use |
|
||
| Both | `utcOffset` | minutes east of UTC, derived from flight location/timezone |
|
||
|
||
MongoDB string-range queries (`$gte`/`$lt`) compare lexicographically. Across these two formats,
|
||
the comparison is meaningless — the strings are ordered differently and represent different
|
||
timezones. Additionally, `String` fields cannot use a date index.
|
||
|
||
`createdDate` remains the stable server-side ingest timestamp for upload/ingest operational
|
||
reports. For dashboard filtering, charting, and per-job application totals, use
|
||
`startDateTimeUTC` / `endDateTimeUTC` with `utcOffset` rather than the legacy string fields.
|
||
|
||
For AgNav records the local timezone is never stored in the legacy string fields, so there is no
|
||
way to convert `"20250522T000632"` to UTC without external context (pilot's location/timezone at
|
||
time of flight). The new `utcOffset` and UTC companion fields address that gap for new and
|
||
backfilled applications.
|
||
|
||
#### Frontend display guidance
|
||
|
||
Recommended client-side rule set:
|
||
|
||
1. Use `startDateTimeUTC` / `endDateTimeUTC` as the canonical values for all filtering, sorting,
|
||
chart bucketing, and API query parameters.
|
||
2. Display local pilot time by taking the UTC field and shifting it by `utcOffset` minutes.
|
||
3. If the UI needs to show both, render the UTC value as the primary canonical timestamp and a
|
||
secondary localized label such as `22 May 2025, 10:06` using the offset.
|
||
4. If `utcOffset` is missing, fall back to the browser timezone only for display. Do not use the
|
||
browser timezone for server-side filtering or export logic.
|
||
5. For day-based summaries, derive the day boundary from `startDateTimeUTC` + `utcOffset` so the
|
||
displayed day matches the pilot's working day instead of the browser's locale.
|
||
|
||
#### Correct field to use
|
||
|
||
`createdDate` is still useful when you want to group by upload time instead of spray time. It is a
|
||
`Date` field set by the server at upload time — consistent, UTC, and independent of file type.
|
||
|
||
**Known tradeoff**: `createdDate` is the *upload* date, not the *spray* date. A pilot who
|
||
completes fieldwork on Monday but uploads on Friday will have those records attributed to
|
||
Friday if you filter by upload time. The UTC companion fields are the correct choice for spray
|
||
window queries.
|
||
|
||
### ApplicationDetail Query Safety
|
||
|
||
`application_details` is a very large collection (potentially billions of rows). The only
|
||
index available for dashboard use is `{ fileId: 1 }`. All performance queries follow a
|
||
three-step pattern to avoid collection scans:
|
||
|
||
```
|
||
1. App.find({ jobId: { $in: jobIds }, startDateTimeUTC: { $gte: start, $lt: end } }) → appIds in range
|
||
2. AppFile.find({ appId: { $in: appIds } }) → fileIds
|
||
3. AppDetail.aggregate([{ $match: { fileId: { $in: fileIds } } }, ...]) → spray-on metrics
|
||
```
|
||
|
||
Never add additional `ApplicationDetail` queries without scoping by `fileId` first.
|
||
|
||
### Why Spray-On Records Only (sprayStat === 3 || sprayStat === 1)
|
||
|
||
ApplicationDetail records cover the entire flight including taxi, transit, and ferry legs.
|
||
During these phases `xTrack` can be millions of metres from the spray line and altitude
|
||
can be hundreds of metres AGL — both meaningless for agronomic gauges. Filtering to
|
||
`sprayStat === 3 || sprayStat === 1` (pump active or spray active) isolates only the records where the aircraft was
|
||
actually spraying, producing accurate XT error and altitude metrics.
|
||
|
||
### Unit Conversions (done server-side)
|
||
|
||
| Raw storage | Conversion | API field |
|
||
|----------------------|-----------------------|----------------------------------------|
|
||
| `totalFlightTime` | seconds → hours ÷3600 | `flightHours`, `hoursFlown` |
|
||
| `totalFlightLength` | metres → km ÷1000 | `operations.distanceTravelledKm` |
|
||
| `totalSprLength` | metres → km ÷1000 | `operations.distanceSprayedKm` |
|
||
| `avgSpraySpeed` | m/s → km/h ×3.6 | `avgSpeedKmh` |
|
||
|
||
All other fields (`totalSprayed`, `totalSprayMat`, `xTrack`, `sprayHeight`, `radarAlt`)
|
||
are stored and returned in their natural units (ha, L, m).
|
||
|
||
---
|
||
|
||
## 10 Open Decisions
|
||
|
||
These items affect backend behaviour but have not been finalised by the Product Owner.
|
||
They are Phase 2 scope and the current implementation uses the defaults noted.
|
||
|
||
| ID | Question | Current default |
|
||
|--------|--------------------------------------------------------------------------|----------------------------------------------|
|
||
| ~~KPI-1~~ | ~~Does "Assigned Jobs" count all-time or just open/current-season jobs?~~ | **Resolved**: open jobs only (NEW/READY/DOWNLOADED/SPRAYED) |
|
||
| ~~ACT-1~~ | ~~Should INVOICED (5) jobs appear in the Active Jobs panel?~~ | **Resolved**: Excluded. Only NEW/READY/DOWNLOADED/SPRAYED/COMPLETED shown. |
|
||
| ACT-Q1 | Should the period filter on Active Jobs also filter the job list (by `createdAt`), or only the Application totals per job? | **Resolved**: both. `period` filters the job list by `createdAt` AND scopes Application totals by `startDateTimeUTC`. `tz` still controls the calendar boundary calculation. |
|
||
| ~~Q11~~ | ~~Can the Pilot trigger the complete action, or Applicator-only?~~ | **Resolved**: Any user under the same Applicator account may complete the job, **except** users with role `inspector` or `client`. The check uses `req.userInfo.puid` (the caller's root Applicator ID), which resolves correctly for the Applicator themselves and all their sub-users (Pilot, co-pilot, etc). |
|
||
| PERF-1 | When no altitude sensor exists, should the gauge show N/A or be hidden? | Returns `hasAltitudeData: false`, value null |
|
||
| JOBS-1 | Are jobs with `operator = null` (unassigned) relevant to any view? | Not included in any pilot-scoped query |
|
||
|
||
---
|
||
|
||
## 11 Changelog
|
||
|
||
> All revisions target `#3054 — Operational Analytics - Pilot Dashboard` unless noted.
|
||
|
||
### Quick-Reference Table
|
||
|
||
| Version | Date | SVN Rev | Task# | Summary |
|
||
|---------|------------|---------|--------|------------------------------------------------------------------------------------------------|
|
||
| 2.5 | 2026-06-15 | — | #3054 | Endpoint URLs added to §5.1–§5.7; `parseDateRange` 90-day cap now DST-immune with invalid-date guard |
|
||
| 2.4 | 2026-06-08 | — | #3054 | New operations metrics: `sprayEfficiencyPct`, `ferryTimePct`, `flowAccuracyPct`, `avgHdop`; new Application fields `avgHdop`/`flowAccuracyPct`; migration script updated |
|
||
| 2.3 | 2026-06-05 | r1227 | #3054 | `todayHasData` flag in summary; `progressPct` float precision; `createdDate` in activeJobs; COMPLETED in KPI assignedJobs; fix `avgXtErrorMeters`→`avgXtError` in §5.5 docs |
|
||
| 2.2 | 2026-06-03 | — | #3054 | Datetime root-cause fix: AgNav hybrid UTC conversion corrected; backfill `--force`; docs sync |
|
||
| 2.1 | 2026-06-02 | — | #3054 | Distance aggregates: inline streaming calc + 120s/1000m gates; docs/tests alignment |
|
||
| 2.0 | 2026-05-28 | — | #3054 | Add `GET /snapshot`; rename `avgXtErrorMeters`→`avgXtError`; resolve Q11; update KPI tests |
|
||
| 1.9 | 2026-05-26 | — | #3054 | Remove `selectedDate` drill-down from all endpoints; remove `parseDateWindow` helper |
|
||
| 1.8 | 2026-05-25 | — | #3054 | Move `dashboardSettings` → `Setting.dashboard`; fix `completeJob` auth (allow sub-users); move `completeJob` to job controller |
|
||
| 1.7 | 2026-05-22 | — | #3054 | Bug fixes: `displayStatus` underscore (code not applied in v1.3); threshold null reset + cross-field validation |
|
||
| 1.6 | 2026-05-22 | r1181 | #3054 | KPI: rename `sprayed` → `sprayedHectares` |
|
||
| 1.5 | 2026-05-21 | r1176 | #3054 | New `PUT /performance/thresholds` endpoint; `User.dashboardSettings` per-pilot threshold store |
|
||
| 1.4 | 2026-05-20 | r1173 | #3054 | Performance: date-range mode, spray-on records only, `startDate`/`endDate` in response |
|
||
| 1.3 | 2026-05-15 | r1155 | #3054 | Breaking: KPI response restructure; `period` filter on activeJobs; `displayStatus` typo fix |
|
||
| 1.2 | 2026-05-14 | — | #3054 | External baseline (frontend-distributed copy; equivalent to server v1.1) |
|
||
| 1.1 | 2026-05-14 | r1144 | #3054 | `selectedDate` drilldown param on all endpoints; frontend drilldown integration guide |
|
||
| 1.0 | 2026-04-29 | r1081 | #3064 | Initial release |
|
||
|
||
---
|
||
|
||
### v2.5 — 2026-06-15 (—, #3054)
|
||
|
||
**Endpoint URLs added to §5.1–§5.7**
|
||
- Every endpoint section now opens with a `**URL**:` line (`GET`/`PUT`/`PATCH` + full path).
|
||
- Previously only §5.6 and §5.8 had an explicit URL line; §5.1–§5.5 and §5.7 were missing it.
|
||
|
||
**`parseDateRange` — 90-day cap made DST-immune**
|
||
- Replaced the millisecond-arithmetic `diffDays` (`Math.round((endExcl − startUTC) / 86400000)`) with a direct calendar-day count from the date strings: `(endDay − startDay) / 86400000 + 1` using UTC midnight (`T00:00:00Z`).
|
||
- This eliminates the theoretical off-by-one risk in DST-observing timezones where the UTC offset of `startDate` and `endDate` can differ by up to 1 hour, causing `Math.round` to produce the wrong integer.
|
||
- Added an explicit invalid-date guard (`isNaN` check on `startDay`/`endDay`): date strings that pass the `YYYY-MM-DD` regex but represent non-existent dates (e.g. `2025-02-30`) now return `409 invalid_param` instead of producing `NaN`-based DB queries that bypass the cap silently.
|
||
- Applies to both `GET /pilot/trend` and `GET /pilot/performance` (both call `parseDateRange`).
|
||
|
||
**§8 formatting fix**
|
||
- Tier 2 fetch example: the two endpoint URLs were incorrectly merged onto a single line; restored as two separate lines.
|
||
|
||
---
|
||
|
||
### v2.4 — 2026-06-08 (—, #3054)
|
||
|
||
**KPI Cards (`/kpi`) — three new `operations` metrics**
|
||
|
||
Added to the `operations` block (today-scoped) in the KPI response:
|
||
|
||
| Field | Formula | Notes |
|
||
|---|---|---|
|
||
| `sprayEfficiencyPct` | `SUM(totalSprayTime) / SUM(totalFlightTime) × 100` | `null` when no flight time |
|
||
| `ferryTimePct` | `(SUM(totalFlightTime) − SUM(totalSprayTime)) / SUM(totalFlightTime) × 100` | Complement of `sprayEfficiencyPct`; both sum to 100 |
|
||
| `flowAccuracyPct` | `AVG(Application.flowAccuracyPct)` | `null` when no sessions with a prescribed rate |
|
||
| `avgHdop` | `AVG(Application.avgHdop)` | `null` when no HDOP data uploaded today |
|
||
|
||
**New `Application` schema fields**
|
||
- `avgHdop` — average HDOP across spray-on (`sprayStat > 0`) records, computed at import time in `job_worker.js importDataFiles()`. Lower is better (< 1 excellent, 1–2 good, > 5 poor).
|
||
- `flowAccuracyPct` — `(totalSprayMat / totalSprayed / appRate) × 100`, computed in `job_worker.js work()` after all three source fields are available. `null` when any is zero/absent.
|
||
|
||
**Migration script updates (now in `scripts/migrate_applications.js`)**
|
||
- `processFile()` selects `stdHdop` and accumulates `hdopSum`/`hdopCount` during spray-on records within valid segments.
|
||
- `processApplication()` aggregates per-file HDOP into `avgHdop` and writes it to `Application`.
|
||
- `backfillFlowAccuracy()` runs as a second pass: a server-side aggregation pipeline update that computes `flowAccuracyPct` for all Application documents already having `totalSprayMat`, `totalSprayed`, and `appRate`.
|
||
- `--force` and `--dry-run` flags both apply to the new pass.
|
||
|
||
**`emptyOps` fallback** — `sprayEfficiencyPct`, `ferryTimePct`, `flowAccuracyPct`, and `avgHdop` default to `null` (not `0`) when there are no Application records for today, distinguishing "no data" from a genuine zero.
|
||
|
||
---
|
||
|
||
### v2.3 — 2026-06-05 (r1227, #3054)
|
||
|
||
**Daily Summary (`/summary`) — `todayHasData` flag**
|
||
- Added `todayHasData: boolean` to the summary response.
|
||
- When `false` (no Application records uploaded for today), all `deltas.*Pct` fields are returned as `null` instead of computing a misleading `-100%`.
|
||
- When `true`, delta calculation is unchanged: `round((today − yesterday) / yesterday × 100)`.
|
||
- **Why**: today values of `0` because no data was uploaded are indistinguishable from a genuine zero-activity day at the aggregation level. A `-100%` delta in that state is alarming and misleading. Frontend should check `todayHasData` before rendering colored arrows.
|
||
|
||
**Active Jobs (`/activeJobs`) — `createdDate` field**
|
||
- Added `createdDate` (ISO 8601 UTC string, `null` if absent) to each job object in the response.
|
||
- Sourced from `Job.createdAt`.
|
||
|
||
**Active Jobs (`/activeJobs`) — `progressPct` precision fix**
|
||
- `progressPct` is now a float with up to 2 decimal places (e.g. `0.47`) instead of an integer rounded value.
|
||
- Previously `round()` truncated any value below `0.5%` to `0`, hiding real progress on large jobs with small completions.
|
||
- Cap and floor behaviour unchanged: clamped to `[0, 100]`.
|
||
|
||
**KPI Cards (`/kpi`) — `assignedJobs` now includes COMPLETED**
|
||
- `periods.<p>.assignedJobs` now counts jobs with status NEW / READY / DOWNLOADED / SPRAYED / **COMPLETED**.
|
||
- Previously COMPLETED jobs were excluded, causing the count to drop when a job was marked done within the period.
|
||
|
||
**Performance (`/performance`) — docs align `avgXtError` field name**
|
||
- §5.5 response examples and field notes updated from `avgXtErrorMeters` → `avgXtError` to match the actual API response (renamed in code at v2.0 / r1227).
|
||
|
||
---
|
||
|
||
### v2.2 — 2026-06-03 (—, #3054)
|
||
|
||
**Application Datetime Conversion — Root-Cause Fix**
|
||
- Fixed `helpers/application_datetime.js: toUtcDateFromAppDateTime()` for AgNav hybrid strings where:
|
||
- `datePart` is local mission date (`YYYYMMDD`), and
|
||
- `timePart` is GPS UTC time-of-day (`HHmmss`).
|
||
- Replaced prior `dayShift = floor((utcSecondsOfDay + offsetSeconds) / 86400)` logic with explicit local-day rollover handling:
|
||
- `localSecondsOfDay < 0` → shift UTC date `-1` day
|
||
- `localSecondsOfDay >= 86400` → shift UTC date `+1` day
|
||
- otherwise no date shift
|
||
- This removes the systematic +1-day start-time drift seen on western timezones with early UTC start times.
|
||
|
||
**Safety Guard Behavior**
|
||
- Kept the `startDateTimeUTC > endDateTimeUTC` guard in `buildApplicationDateFields()` as a fallback safety net for genuinely bad/corrupt data, not as a primary correction path.
|
||
|
||
**Backfill Script Improvements**
|
||
- Updated datetime backfill (now in `scripts/migrate_applications.js --skip-aggregates`):
|
||
- Added `--force` mode to recompute datetime fields for all apps with legacy `startDateTime`.
|
||
- Retained targeted mode for missing/zero/inverted UTC fields.
|
||
- Updated script usage/help comments accordingly.
|
||
- Re-ran backfill with `--force` so existing records are recomputed with the corrected formula.
|
||
|
||
**Documentation Updates**
|
||
- Updated §Overview Backfill Script usage/selection criteria in this document.
|
||
- Updated AgNav datetime semantics section to explicitly document hybrid format and date-shift conversion rationale.
|
||
|
||
---
|
||
|
||
### v2.1 — 2026-06-02 (—, #3054)
|
||
|
||
**Distance Aggregate Calculation — Efficiency + Correctness Alignment**
|
||
- `totalSprLength` and `totalFlightLength` are now computed inline during file-read loops in `job_worker` (`readNTFile`, `readShapeDataFile`, `readSatLogAsc`) instead of relying on a post-read full-array rescan.
|
||
- Cross-file boundary segments are now included when shape spray-on/spray-off data are merged, so join-point distance is not dropped.
|
||
- Distance validity gates are aligned and explicitly documented:
|
||
- `0 < dt <= 120s` (with midnight rollover handling)
|
||
- `dist <= 1000m`
|
||
- spray-distance additionally requires spray-on segment (`prev.sprayStat > 0 || curr.sprayStat > 0`)
|
||
- `_computeSprLength` / `_computeFlightLength` remain as fallback helpers and now apply the same gate logic for consistency.
|
||
|
||
**Migration Script Alignment**
|
||
- Distance calculation in `scripts/migrate_applications.js` uses the same time+distance gates as runtime worker logic.
|
||
|
||
**Documentation + Test Alignment**
|
||
- Updated metric definitions in this document and `AGGREGATED_FIELDS_CALCULATION.md` to reflect gate rules and inline streaming aggregation.
|
||
- Dashboard test suite re-run after updates: `66 passing`. (with RUN_COMPLETE_TEST=1, DASHBOARD_TEST_JOB_ID set to a job with a long spray leg to confirm the new logic is working as intended)
|
||
|
||
---
|
||
|
||
### v2.0 — 2026-05-28 (—, #3054, #3064, #3063)
|
||
|
||
**Add `GET /snapshot` composite endpoint**
|
||
- New endpoint `GET /api/dashboard/pilot/snapshot` returns any combination of `kpi`, `summary`, `activeJobs`, `performance`, `trend` in one request
|
||
- `?include=` param selects modules (comma-separated); unrecognised module names silently ignored; default returns all modules
|
||
- `?startDate`/`?endDate` apply to `performance` and `trend` modules; 90-day cap enforced per module
|
||
- Shared job/app data fetched once internally — no N+1 database calls
|
||
- §5.8 added; §9.4 updated from "Proposed" to "Implemented"
|
||
|
||
**Rename `avgXtErrorMeters` → `avgXtError` (§5.5 Performance)**
|
||
- Field name no longer carries unit suffix for consistency with other fields (`avgSpraySpeed`, `avgSprayAltitudeMeters` are unaffected)
|
||
- Schema field `Application.avgXtError` updated; migration script updated
|
||
- Performance tests confirm new field name working
|
||
|
||
**Resolve Q11 — complete-job access**
|
||
- Confirmed: any user under the same Applicator account may complete a job, **except** `inspector` and `client` role users
|
||
- §5.7 Authorization table already reflects this; Q11 row struck through in §10
|
||
|
||
---
|
||
|
||
### v1.9 — 2026-05-26 (—, #3054)
|
||
|
||
**Remove `selectedDate` drill-down (all affected endpoints)**
|
||
- Removed `selectedDate` query parameter from `/kpi`, `/summary`, `/activeJobs`, and `/performance`
|
||
- Removed internal `parseDateWindow()` helper — it existed solely to serve `selectedDate`
|
||
- `/kpi`: always returns live period windows (day / week / month / year / all); periods no longer collapse to a single day
|
||
- `/summary`: always compares today vs. yesterday; "today" is no longer overridable via query param
|
||
- `/activeJobs`: `period` param retained; `selectedDate` priority chain removed — new priority: `period` > all-time
|
||
- `/performance`: `startDate`/`endDate` and the current-week default retained; `selectedDate` override removed
|
||
- §8 Frontend Integration Guide: removed "Drilldown" fetch pattern, removed `Drilldown Snippet` code example, renamed "Drilldown vs. Date Range" table to "Filter Parameters Quick Reference", removed `selectedDate` row
|
||
- §9.4 Sequence diagram: removed "3 — User drilldown" sequence block; removed `selectedDate` from proposed `/snapshot` query params
|
||
|
||
---
|
||
|
||
### v1.8 — 2026-05-25 (—, #3054)
|
||
|
||
**Architecture: `dashboardSettings` renamed to `Setting.dashboard`; moved from `User` (no API change)**
|
||
- `User.dashboardSettings` subdocument removed from `model/user.js`. The five threshold fields
|
||
(`xtGood`, `xtMonitor`, `altTarget`, `altGoodBand`, `altMonitorBand`) are now stored as
|
||
`Setting.dashboard` on the pilot's `Setting` document in the `settings` collection.
|
||
- Field renamed from `dashboardSettings` to `dashboard` — the `Settings` suffix is redundant
|
||
given the document already lives in the `Setting` collection.
|
||
- All per-user preferences (measurement units, spray-path options, map colours, etc.) are now
|
||
consolidated in the `Setting` collection. Reading the `User` document for dashboard purposes
|
||
is no longer required.
|
||
- `controllers/dashboard.js` now uses `Setting.findOne`/`findOneAndUpdate` (with `upsert: true`)
|
||
instead of `User.findById`/`findByIdAndUpdate` for threshold reads and writes.
|
||
- The API contract, endpoint paths, request/response shapes, and error codes are **unchanged**.
|
||
|
||
**`completeJob` auth fix — sub-users allowed**
|
||
- The ownership check was `job.byPuid.toString() !== req.uid`, which rejected Pilot sub-users
|
||
because their own `_id` ≠ `byPuid`. Changed to `job.byPuid.toString() !== req.userInfo.puid`
|
||
so both the Applicator and any sub-user under that Applicator can complete the job.
|
||
|
||
**`completeJob` moved to `controllers/job.js` (no API change)**
|
||
- Handler relocated from `controllers/dashboard.js` to `controllers/job.js` for cohesion.
|
||
Route unchanged: `PATCH /api/jobs/:job_id/complete`.
|
||
|
||
---
|
||
|
||
### v1.7 — 2026-05-22 (—, #3054)
|
||
|
||
**Active Jobs (`/activeJobs`) — Bug Fix**
|
||
- `displayStatus` for READY/DOWNLOADED/SPRAYED jobs was being returned as `"IN PROGRESS"` (with a space) despite the v1.3 changelog documenting the fix to `"IN_PROGRESS"` (underscore). The code was not updated in v1.3. Fixed now — **this is a breaking change for any consumer that string-compared against `"IN PROGRESS"`**.
|
||
|
||
**Save Performance Thresholds (`PUT /performance/thresholds`) — Bug Fixes**
|
||
- Passing `null` to reset a threshold field to the system default was silently ignored: `$set` with a JavaScript `undefined` value is stripped by the MongoDB driver, leaving the stored value unchanged. The endpoint now uses `$unset` for `null` fields so the stored custom value is correctly removed.
|
||
- Cross-field validation (`xtMonitor > xtGood`, `altMonitorBand > altGoodBand`) was resolved against system defaults when a field was absent from the request, not the user's currently stored values. A partial update such as `{xtMonitor: 3}` could pass validation while leaving a stored combination of `xtGood=5`, `xtMonitor=3` (invalid). The endpoint now fetches the stored `dashboardSettings` before validation and uses the stored values as the fallback.
|
||
|
||
**Documentation**
|
||
- §5.3 Trend: corrected the note about partial `startDate`/`endDate` pairs — both are silently defaulted (no `409`)
|
||
- §5.6 Thresholds: updated constraint description to reflect the improved cross-field validation fallback chain
|
||
|
||
---
|
||
|
||
### v1.6 — 2026-05-22 (r1181, #3054)
|
||
|
||
**KPI Cards (`/kpi`)**
|
||
- Renamed `periods.<p>.sprayed` → `periods.<p>.sprayedHectares` across all period objects (`day`, `week`, `month`, `year`, `all`) for clarity and consistency with the field's unit
|
||
|
||
---
|
||
|
||
### v1.5 — 2026-05-21 (r1176, #3054)
|
||
|
||
**New Endpoint**
|
||
- Added `PUT /api/dashboard/pilot/performance/thresholds` (§5.6): persists custom XT error and altitude gauge thresholds per pilot; returns effective thresholds after save; supports `null` to reset individual fields to system defaults
|
||
- Mark Job as Completed moved from §5.6 → §5.7 to accommodate new thresholds endpoint
|
||
|
||
**Data Model**
|
||
- Added `User.dashboardSettings` optional subdocument (`xtGood`, `xtMonitor`, `altTarget`, `altGoodBand`, `altMonitorBand`) for per-pilot threshold overrides; system defaults used when fields absent
|
||
- Added `model/user.js` to the Implemented In list (§1)
|
||
|
||
**Frontend Guide**
|
||
- Added "When the user saves custom thresholds" fetch pattern — response from `PUT /thresholds` can be used directly to update the gauge without a follow-up `GET /performance`
|
||
|
||
**Diagrams**
|
||
- Updated endpoint interaction diagram: added `TH[PUT pilot performance thresholds]` node with `TH → U[(users)]` edge
|
||
|
||
---
|
||
|
||
### v1.4 — 2026-05-20 (r1173, #3054)
|
||
|
||
**Performance Gauges (`/performance`) — Breaking Change**
|
||
- Replaced last-10-files static sample with a **date-range query**
|
||
- Added `startDate` and `endDate` query params (default: current calendar week Mon–Sun); max range 90 days
|
||
- Date resolution priority: `selectedDate` → `startDate`+`endDate` → current week default
|
||
- Response now includes `startDate` and `endDate` fields reflecting the effective analysis window
|
||
- Metrics now computed from **spray-on records only** (`sprayStat === 1`) — transit, taxi, and ferry-flight records are excluded; this eliminates spurious XT error spikes (millions of metres off-line during turns) and AMSL altitude readings from non-spray legs
|
||
- `sampleSize` now reflects the number of `AppFile` records in the date window (was: "up to ~20" files cap)
|
||
- Updated `hasAltitudeData` field note: now means "no altitude sensor data in sample" (was: "aircraft has no height sensor")
|
||
|
||
**Backend Notes (§9)**
|
||
- Added "Why Spray-On Records Only (`sprayStat === 1`)" explanatory section
|
||
- Updated ApplicationDetail query pattern: step 1 now uses `createdDate` range filter instead of `.limit(10)`
|
||
- Updated Performance Query Safety diagram to reflect date-range scoping and spray-on filter step
|
||
|
||
---
|
||
|
||
### v1.3 — 2026-05-15 (r1155, #3054)
|
||
|
||
**KPI Cards (`/kpi`) — Breaking Response Shape Change**
|
||
|
||
Old flat top-level fields **removed**: `assignedJobs`, `assignedHectares`, `totalSprayed`, `totalFlightHours`, `jobCounts`, and the `historical` block.
|
||
|
||
New structure:
|
||
- `operations` block (today-scoped): `missionsFlown`, `distanceTravelledKm`, `distanceSprayedKm`
|
||
- `distanceTravelledKm` replaces `distanceKm` and now uses `Application.totalFlightLength` (all GPS segments including turns) instead of `totalSprLength`
|
||
- `distanceSprayedKm` is new — spray-on segments only (`Application.totalSprLength / 1000`)
|
||
- `sprayVolumeLiters` removed from `operations`
|
||
- `periods` block: `day`, `week`, `month`, `year`, `all` sub-objects each containing `assignedJobs`, `assignedHectares`, `sprayed`, `flightHours`
|
||
- `jobCounts` (`new`, `inProgress`, `completed`) present on `day`, `week`, `month` only — not on `year` or `all`
|
||
- `all` period added (no time boundary — covers all records for this pilot)
|
||
- Added `totalFlightLength` field tracking to `Application` model, `AppFile`, and `job_worker`
|
||
|
||
**Active Jobs (`/activeJobs`)**
|
||
- Added `period` query param (`day` | `week` | `month` | `year`): filters both the job list (by `Job.createdAt`) and Application sub-totals (by `Application.startDateTimeUTC`) to the selected window; `tz` continues to control the boundary math
|
||
- Priority chain: `selectedDate` > `period` > all-time (no filter)
|
||
- Updated field notes for `haSprayed` and `volumeAppliedLiters` to clarify they are scoped to the active window
|
||
|
||
**Status / Display**
|
||
- Fixed `displayStatus` value: `"IN PROGRESS"` (with space) → `"IN_PROGRESS"` (underscore) — **breaking change for frontend consumers**
|
||
|
||
**Unit Conversions (§9)**
|
||
- Added `totalFlightLength` (metres → km ÷ 1000) → `operations.distanceTravelledKm` to the server-side conversion table
|
||
|
||
**Frontend Guide (§8)**
|
||
- Added "Period filter" fetch pattern for Day / Week / Month / Year tab switching on `/activeJobs`
|
||
- Expanded Drilldown vs. Date Range table with "User switches Day/Week/Month/Year tab" scenario
|
||
- Updated Tier 1 fetch example: `/activeJobs?tz=...&period=week`
|
||
|
||
**Open Decisions**
|
||
- `ACT-Q1` resolved: `period` filter applies to both the job list and Application totals
|
||
|
||
---
|
||
|
||
### v1.1 — 2026-05-14 (r1144, #3054)
|
||
|
||
- Added `selectedDate` query param (`YYYY-MM-DD`) to `/kpi`, `/summary`, `/activeJobs`, `/performance` for single-day drilldown filtering from chart interactions
|
||
- Added `tz` param documentation to `/activeJobs` and `/performance` (previously these endpoints accepted `tz` but it was undocumented)
|
||
- Added frontend **Drilldown Snippet** JavaScript code example
|
||
- Added **Drilldown vs. Date Range** reference table
|
||
- Fixed DST-safe yesterday derivation in `/summary` (no longer a simple 24-hour subtraction)
|
||
|
||
---
|
||
|
||
### v1.0 — 2026-04-29 (r1081, #3064)
|
||
|
||
Initial release implementing all Pilot Dashboard endpoints:
|
||
|
||
- `GET /api/dashboard/pilot/kpi` — KPI cards with historical breakdowns
|
||
- `GET /api/dashboard/pilot/summary` — today vs. yesterday metrics with deltas
|
||
- `GET /api/dashboard/pilot/trend` — daily hours flown and hectares for date range
|
||
- `GET /api/dashboard/pilot/activeJobs` — active job list with progress
|
||
- `GET /api/dashboard/pilot/performance` — XT error and spray altitude gauges
|
||
- `PATCH /api/jobs/:job_id/complete` — transition job SPRAYED → COMPLETED
|
||
- Postman collection: `docs/Pilot_Dashboard_API.postman_collection.json`
|
||
- Mocha/Chai integration test script: `tests/test_pilot_dashboard_api.js`
|