agnav/agmission
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
045ce34398
agmission/server/docs/ADVANCED_REPORT_METRICS_VERIFICATION.md

52 KiB
Raw Blame History

Advanced Application Report — Metrics Calculation & Verification

Purpose: trace every displayed number on the Advanced Application Report (pages 1–3) back to its raw source field, document the exact formula and unit conversions at each step, and record whether it was verified correct, fixed, or left open for a product decision.

Scope: helpers/report_util.js (D1 analytics engine — pure computation over ApplicationDetail points), controllers/advanced_report.js (D2 datasource builder — formatting/localization/overrides), plus the shared conversion helpers in helpers/utils.js and helpers/geo_util.js.

Report layout referenced:

  • Page 1 — Mission Information, KPI tiles, Mission Statistics, Products table, Weather box
  • Page 2 — Mission Coverage grid (coverageCards, client-side page, suppressed for single-zone jobs)
  • Page 2/3 — Zone Detail (one page per zone: Flight Statistics box + zone map)
  • Page 3 — Flight Line Statistics table (per-line rows, nested under each zone)

Legacy Consistency Matrix

Every calculation reviewed, checked against the actual legacy code (not documentation prose — one documented legacy rule turned out to be wrong when checked against the real code, see §5 item 3), with an explicit verdict on whether a legacy equivalent exists and whether this report's logic matches it. Legacy source: workers/job_worker.js (import pipeline), controllers/job.js (legacy report datasource, makeJobAppDataSource/preAppReport_post), helpers/job_util.js.

Metric Legacy equivalent? Same logic? Verdict
Point-to-point distance Yes — geoUtil.distance(), literally the same function ✅ identical (not a copy, the same call) Match
Midnight-wrap time diff (todDiff) Yes — inline in 3 places in job_worker.js, same 80000/86400 constants ✅ byte-for-byte same formula Match
Planned area (incl. exclusion-zone netting) Yes — jobUtil.calcTTSprayAreas() ✅ mirrors exactly, including its shared double-subtraction bug on overlapping exclusions (§5 item 11) Match (fixed to match, this session)
Spray-pass segmentation (line breaks) Yes — getSprayOnSegments(), controllers/job.js:604 ⚠️ partial — shares the core rules (llnum change, marker, ≥1km jump) but legacy also has a satellite-quality (satsIn) edge trim and a fuller endSegChecker() transition table this engine doesn't replicate Needs a decision — see below
Turn time (state machine) Yes — inline in job_worker.js:1486-1516 ✅ same core pattern, plus a deliberate correctness addition (atFresh guard) legacy lacks Match + intentional improvement
Avg Speed (per-point rule) Yes — totalSpeedAcc/spraySpeedCount, gated on sprayStat>0 only ✅ fixed this session to include the line-start marker, matching legacy exactly (previously wrongly excluded it based on a doc error) Match (fixed)
Avg XT Error (per-point rule) Yes — gated on sprayStat===1||===3 ✅ matches exactly Match
Avg Height (per-point rule) No legacy field at all — New (unavoidable)
Total Flight Time / Duration / Ferry Time Yes — sum of consecutive-point deltas, each capped ≤120s ✅ fixed this session (was wall-clock last-minus-first; now matches legacy's capped-delta sum exactly) Match (fixed)
Volume/material integration Partial — legacy's getAppliedRate() 3-way priority (configured rate / flow-derived rate / lhaReq fallback), area × rate per record ⚠️ different method — this engine always integrates flow over time directly when flow data exists, with no per-record priority scheme; mathematically equivalent to legacy's flow-derived branch, but legacy prefers the configured-rate branch by default. The configured-rate branch itself is no longer used as a mission-level fallback here at all — see §5 item 10 Match where flow data exists; intentionally diverges where it doesn't (dashes instead of a configured-rate guess) — see §5 item 10
Volume integration's outlier gap Legacy caps every time accumulator at 120s ✅ fixed this session — this engine's volume loop had no upper gap bound at all until now Match (fixed)
Avg Speed / Avg XT / Avg Height — mission-level weighting Legacy: one flat mean across every qualifying record in the whole job (no zones, no lines — just count-weighted) ⚠️ different — this engine derives the mission figure from a zone roll-up that's point-count-weighted pass→line, then time-weighted line→zone and zone→mission. Quantified divergence: a constructed case with uneven GPS logging density across zones showed 49.1 vs. legacy's-equivalent 18.2 (2.7×) for the same underlying readings Open — flagging for a decision, see below. Not fixed without confirmation: reconciling this changes core architecture (mission would no longer be a pure function of the zone breakdown for these three fields specifically)
Avg Speed / Avg XT / Avg Height — zone/line-level weighting No legacy equivalent — legacy never had a per-zone or per-line breakdown of these at all — New (unavoidable) — the weighting choice here (point-count then time) is a reasonable original design, not a legacy deviation, since there's nothing in legacy to deviate from
Coverage % (any level) No legacy equivalent — legacy's "coverage" is a raw area figure (hectares), never a percentage — New (unavoidable)
Zone assignment (majorityZone/nearestZone) No legacy equivalent — legacy never attributes a GPS point to a specific zone — New (unavoidable) — nearestZone()'s no-sanity-check fallback fixed this session after being found misattributing an entire unrelated real flight on live data, see §5 item 9
Products table Rate/Total Rate Yes — controllers/job.js:933,940 ✅ byte-for-byte same pattern (bare unit, no /ac suffix) Match
Weather (wind/temp/humidity units) Yes — same raw field units (windSpd m/s, temp °C) ✅ same conversions Match
Mission Information passthrough fields (name, customer, pilot, aircraft, dates, etc.) Yes — makeJobAppDataSource() ✅ same source fields; two intentional additions (License Number, Flight # tail-number fallback — the latter since reverted to match legacy exactly) Match

Two items above need your explicit call before any further code change, since fixing them "properly" means non-trivial architecture work, not a quick patch:

  1. Spray-pass segmentation's missing satellite-quality trim and transition table. Legacy trims low-satellite-count (satsIn<99) points from a segment's edges, and has a more detailed endSegChecker() covering specific sprayStat transition pairs. Replicating this would require adding satsIn to DETAIL_PROJECTION (controllers/advanced_report.js:47) and porting the trim logic — a real change, not a one-line fix. Is this in scope, or is the current simplified rule (llnum change / marker / ≥1km jump) acceptable given it covers the common cases?

  2. Mission-level Avg Speed/XT/Height weighting. Matching legacy exactly here means tracking a separate, flat, count-weighted accumulator across the whole mission (mirroring legacy's job-level number) specifically for the mission's own tile, independent of the zone breakdown underneath it — which means the mission figure would no longer be mathematically derivable from the zone rows for these three fields, unlike every other reconciled metric in this report (area, time, distance, volume). That's a deliberate trade-off: closer to legacy, but a new kind of inconsistency within this report. Want this changed, or is the current zone-derived approach acceptable given there was never a zone-level precedent to begin with?

0. Foundational primitives

Three primitive functions do all the raw geometry/time work; every other figure in the report is built by combining their outputs. They are not the same function, and don't all feed the same metrics — being precise about which figure depends on which matters here.

Function Formula File Verdict
geoUtil.distance(latLng1, latLng2) turf.distance() (haversine great-circle, spherical earth ~6371km radius), after reordering [lat,lon] → [lon,lat] for turf helpers/geo_util.js:176-181 ✅ Correct — accurate for mission-scale distances, returns km
turf.area(feature) (called directly, no geoUtil wrapper) Polygon surface area from its boundary shape — a completely separate calculation, no point-to-point distance summing involved report_util.js:49,51-52 (plannedAreaM2()) ✅ Correct
todDiff(t2, t1) Seconds-of-day difference, corrected for the midnight rollover: if the raw difference is negative and ≥80,000s (~22.2h), recompute as (86400 − t1) + t2 report_util.js:33-37 ✅ Correct — one narrow theoretical edge case at the WRAP_GUARD_S threshold (see below), not practically triggerable given ~1Hz sampling within a single continuous file

todDiff() edge case (low risk, not a live bug): the 80000-second cutoff creates a narrow "dead zone" — a same-file gap of 79999s returns a nonsensical -79999 (not treated as a wrap) while 80000s correctly flips to being treated as one. Triggering this on real data would require two consecutive points in the same file roughly 1.8–22 hours apart landing near midnight — not realistic for ~1Hz GPS sampling within one continuous flight file. If it ever did occur, the result degrades gracefully (a negative/zero duration renders as a dash per the formatters' ≤0 → DASH rule) rather than showing a wrong number. Verified every gpsTime difference in the engine routes through this function — no raw subtraction bypasses it.

What actually depends on which:

  • Distance (Total/Spray/Ferry Distance) and Length (a flight line's length) — built directly by summing geoUtil.distance() over consecutive GPS points (report_util.js:151-152,225). No other function is involved.
  • A flight line's Area Covered — indirect: Length × Swath Width (report_util.js:243). Needs geoUtil.distance()'s output plus a separate, unrelated input (the sprayer's configured swath).
  • A zone's Planned Area — does not use geoUtil.distance() at all. It comes from turf.area(), measuring the drawn polygon's shape directly — a different calculation entirely.
  • Avg Speed — not built from either function in the normal case; it comes straight from the aircraft's own GPS speed sensor reading (grSpeed, recorded per point). geoUtil.distance() only enters as a fallback when no valid sensor speed exists at all: Length ÷ Time (report_util.js:238).

1. Page 1 — Mission Information

Simple passthrough fields (no calculation), verified against the actual Mongoose schema (not just assumed from the field name):

Field Source File Verdict
Mission Name job.name advanced_report.js:378 ✅
Job Type job.appType advanced_report.js:379 ✅
Crop job.crop.name (populated ref) or raw job.crop (legacy string fallback) advanced_report.js:380, schema model/job.js:74 ✅
Date - Planned moment(job.startDate) – moment(job.endDate) advanced_report.js:362,381-382, schema model/job.js:66-67 ✅
Date/Time - Actual first apps[0].startDateTime → last apps[N-1].endDateTime, apps pre-sorted by startDateTime asc advanced_report.js:366-374, query sort at advanced_report.js:146 ✅ correct for sequential flights — ⚠️ see note below
Customer / Customer Address job.client.name (ref UserTypes.CLIENT) / getFormattedAddress(job.client) advanced_report.js:385-386, schema model/job.js:70 ✅ — confirmed client and the separately-fetched applicator (via job.byPuid) are genuinely two different entities, not a naming collision
Pilot / Operator job.operator.name (ref UserTypes.PILOT) advanced_report.js:387, schema model/job.js:69 ✅
License Number job.operator.licence advanced_report.js:388, schema model/pilot.js:9 ✅ (spelling matches schema exactly)
Aircraft job.vehicle.name + job.vehicle.model advanced_report.js:389, schema model/vehicle.js:20 ✅
Flight # job.flightNumber (dash if absent) advanced_report.js:390, schema model/job.js:73, set from imported data in workers/job_worker.js:714 ✅ — reverted to match legacy exactly (controllers/job.js:1229); previously fell back to vehicle.tailNumber, which was removed since a tail number identifies the aircraft permanently, not this specific flight — see below
Applicator / Applicator Address Customer.findOne({_id: job.byPuid}) advanced_report.js:168-171, schema comment model/job.js:161 ("Applicator userId") ✅
Remark job.remark advanced_report.js:411 ✅

⚠️ Note (minor, low risk): "Actual Dates" assumes the app with the latest startDateTime also has the latest endDateTime (apps[apps.length-1].endDateTime). True for normal sequential single-aircraft flights; could misorder only if two flight files genuinely overlap in time.

2. Page 1 — KPI Tiles

Tile Formula (source → engine → display) Engine file:line Display file:line Verdict
Coverage min(Σzone min(sprayedAreaM2, plannedAreaM2) ÷ Σzone plannedAreaM2(net of excludedAreas) × 100, 100) — each zone's contribution capped at its own plan before summing report_util.js:45-58 (per-zone planned area), :385-392 (per-zone-capped mission roll-up) advanced_report.js:344-352,400 ✅ correct (fixed — see §5.1 and §5.6)
Avg Speed mean grSpeed per pass (m/s, includes sprayStat==3 marker) → weighted by point-count to line → weighted by spray-time to zone → weighted by spray-time to mission → ×2.23694 (mph) / ×3.6 (km/h) report_util.js:232,238 (pass), :267,289 (line), :316,326 (zone), :361 (mission) advanced_report.js:319 ✅ correct — matches the actual legacy avgSpraySpeed code (see §5, item 3 for the correction history)
Avg Height mean sprayHeight per pass (m, >0 only, null if no FM sensor) → same weighted cascade → ×3.28084 (ft) report_util.js:234,240,269,291,318,328,363 advanced_report.js:321 ✅
Avg XT Error mean |xTrack| per pass (m, ≠0, includes sprayStat==3) → same weighted cascade → ×3.28084 (ft) report_util.js:233,239,268,290,317,327,362 advanced_report.js:321 ✅ — confirmed this correctly includes the marker point, matching the legacy avgXtError gate (sprayStat===1 || sprayStat===3, workers/job_worker.js:1478); Avg Speed also includes the marker (both now confirmed to use the same inclusive rule — see §5, item 3)
Total Volume Σ trapezoidal lminApp integration (L) across sprayed zones; if null (no flow-meter data): dash, same as zones/lines report_util.js:227-228,241,315,350,360 (measured) advanced_report.js:359-365 ✅ fixed — mission-level estimate fallback removed, see §5 item 10
Zones Sprayed count(zones with lineCount>0) / count(all zones) report_util.js:340,365-366 advanced_report.js:399 ✅ trivial, no arithmetic risk

3. Page 1 — Mission Statistics

Field Formula File:line Verdict
Planned Area areaStr(plannedM2) — manual Report-Settings override if set, else mission.plannedAreaM2 (net of exclusion zones) advanced_report.js:346-347,400; engine report_util.js:352 ✅ (fixed — see §5.1)
Sprayed Area areaStr(sprayedM2) — manual override if set, else mission.sprayedAreaM2 advanced_report.js:348-349,401 ✅
Total Flight Time / Total Duration (top box) hm(mission.totalFlightS); totalFlightS = Σ consecutive-point deltas, each excluded entirely if ≤0 or >120s — matches legacy exactly (workers/job_worker.js:1447-1455) report_util.js:142-158,372 ✅ correct (fixed — see §5, item 9)
Total Spray Time hm(mission.sprayTimeS) = Σ sprayed-zone sprayTimeS report_util.js:348,354; advanced_report.js:403 ✅ — reconciliation to zone sum unit-tested (test_report_util.js, NFR-3.3 case)
Ferry Time hm(mission.ferryTimeS) = max(totalFlightS − sprayTimeS, 0) report_util.js:373 ✅ correct (inherits the fix to totalFlightS — see §5, item 9)
Total Distance distStr(mission.totalDistanceM) = max(Σ all-point-pair distances gated at <1km jump, sprayDistanceM) report_util.js:140-154,357 ✅
Spray Distance distStr(mission.sprayDistanceM) = Σ line lengthM report_util.js:225,349,358 ✅
Ferry Distance distStr(mission.ferryDistanceM) = max(totalDistanceM − sprayDistanceM, 0) report_util.js:359 ✅
Avg App Rate rateStr(volumeL, sprayedM2) = (volume in job units) ÷ (area in job units) advanced_report.js:335-338,408 ✅ fixed — now correctly dashes with no flow data, consistent with zones/lines — see §5 item 10
Avg Flow Rate flowStr(mission.avgFlowLmin) = mission.volumeL ÷ (sprayTimeS/60), measured only, no estimate fallback report_util.js:369; advanced_report.js:409 ✅ correctly stays dash when unmeasured (consistent, unlike Avg App Rate)
Swath Width job.swathWidth (job config value, not the analytics-derived per-line swath used in area math) advanced_report.js:410 ✅ — intentional: shows the job setting, distinct from pass.swathM used in area calc (report_util.js:242)

4. Page 1 — Products Table

Field Formula File:line Verdict
Product Name / Restricted / EPA Reg# / Type passthrough of populated job.products[].product fields advanced_report.js:505-508, populate at :102 ✅
Rate ${rate} ${getProdUnit(unit)} — bare unit string ("gal", "lb", "lit"...), no per-area suffix advanced_report.js:509, utils.getProdUnit() at helpers/utils.js:717-747 ✅ matches legacy exactly (controllers/job.js:933,940) — API doc example was wrong, now fixed, see §5 item 8
Total Volume Used rate × (sprayedArea in job units), with oz→gal conversion when unit===OZ advanced_report.js:511-513, utils.ozToGal() at helpers/utils.js:411-413 ✅ math correct — inherits sprayedM2 override behavior; this is a config rate × area, always known and always labeled correctly, unlike the mission/zone rate fields (see §5 item 17)

Verified: job.products[].rate/unit schema (model/job.js:51-57) matches field access exactly; APTypes.CARRIER (helpers/constants.js:36-40) and Units/getProdUnit mapping (helpers/constants.js:5, helpers/utils.js:717-747) are correctly aligned — 0=oz, 1=gal, 2=lb, 3=lit, 4=kg, matching both enums.

5. Page 1 — Weather Box

Field Formula File:line Verdict
Wind Speed Manual: wi.windSpd (already knots, per schema) shown as-is. Aggregated: mpSecToKnot(avgWindSpd) = ×1.94384 (application_detail.windSpd is m/s) advanced_report.js:536,547; schema model/job.js weatherInfo (knots), model/application_detail.js:37 (m/s); utils.mpSecToKnot at helpers/utils.js:371-374 ✅ both paths verified against actual field units in the schema, not assumed
Wind Direction Manual: raw compass string. Aggregated: round(avgWindDir)° + deg2Compass(avgWindDir) (16-point compass) advanced_report.js:537,548; utils.deg2Compass at helpers/utils.js:219-231; source field model/application_detail.js:38 (degrees) ✅ correct, though the two paths render differently in style (plain string vs "270° WNW") — cosmetic only
Temperature Manual: (wi.temp−32)×5/9 if US (assumes °F stored), else as-is. Aggregated: avgTemp direct (already °C per schema) advanced_report.js:534,538,549; model/application_detail.js:39 (Celsius) ✅
Humidity direct passthrough, rounded to 0 decimals advanced_report.js:539,550; model/application_detail.js:40 ✅
Aggregation query AppDetail.aggregate() — $avg over windSpd>0, windDir∈[0,360], temp∈[5,60], humid∈[9,90] helpers/job_util.js:325-338 (getDataWeatherInfo) ✅ sane outlier gates

6. Page 2 — Mission Coverage Grid (coverageCards)

Always populated for every zone regardless of Report Contents filtering (§6 of the API contract), even if the client suppresses the page for single-zone jobs.

Field Formula File:line Verdict
Sprayed / Planned ${sprayedArea} / ${plannedArea} (dash on sprayed side if lineCount==0) advanced_report.js:433-435 ✅ — reuses already-verified zone sprayedAreaM2/plannedAreaM2
Coverage % zs.coveragePct (dash if unsprayed) — uncapped, can show >100% on real overspray advanced_report.js:436; engine report_util.js:349 ✅ (fixed — see §5, item 6 companion fix)

7. Page 2/3 — Zone Detail

All zone-level fields reuse the same accumulator logic already verified for the mission-level KPIs (§2–3), rolled up to per-zone instead of per-mission. Zone-specific formulas:

Field Formula File:line Verdict
Planned/Sprayed Area, Coverage %, Avg Speed/Height/XT Error/Flow Rate same formulas as mission-level, applied per zone (Coverage % uncapped — see §5, item 6) advanced_report.js:453-464 ✅
Volume Applied / Avg App Rate zs.volumeL — measured only advanced_report.js:456-457 ✅ — mission-level tile now matches this behavior too, see §5 item 10
Product comma-joined names of the job's active-ingredient products (job.products, filtered to exclude APTypes.CARRIER) — mission-wide, identical value on every zone, same convention as crop: mission.crop advanced_report.js:576-592 ✅ correct by construction — not a per-zone calculation, deliberately mission-wide
Start Time / End Time (zone-level) todStr(zs.startTimeS) / todStr(zs.endTimeS) — the zone's own first/last spray-on timestamp, engine-side _firstT/_lastT now also exposed as startTimeS/endTimeS instead of being discarded after deriving flightTimeS advanced_report.js:598-599; engine report_util.js:524-525 (set right after flightTimeS at line 521, before the _firstT/_lastT cleanup at line 533) ✅ same source timestamps flightTimeS already derives from, no new engine logic
Flight Time hm(zs.flightTimeS) = todDiff(last spray-end+turn, first spray-start) within that zone only — a different definition than mission totalFlightS (not gap-tolerant the same way, and doesn't include ferry to/from the zone) advanced_report.js:458; engine report_util.js:308-309,318,331 ✅ well-defined as its own metric — just not directly summable to the mission Total Flight Time field (different scope by design)
Avg Turn Time secStr(zs.avgTurnTimeS) = unweighted mean of each line's own (already-averaged) turn time advanced_report.js:460; engine report_util.js:285,297,306,316,329 ✅ underlying turnTimeS values fixed (§5.4) — ⚠️ minor residual note: still an unweighted double-average across lines; could skew if a zone's lines have very uneven turn counts (not fixed, low-impact)
Map image zone capture .jpg, falls back to mission map.jpg placeholder on capture failure advanced_report.js:465 ✅ (capture pipeline, not a calculation)

8. Page 3 — Flight Line Statistics Table

Column Formula File:line Verdict
Start Time secondsToHMS(startTimeS % 86400, format=1) → HH:MM:SS advanced_report.js:340,479; helpers/utils.js:804-816 ✅
Spray Time secStr(sprayTimeS) advanced_report.js:480 ✅
Length lenStr(lengthM) = ×3.28084 ft advanced_report.js:481 ✅
Avg Speed same formula as §2 advanced_report.js:482 ✅
Area Covered toArea(areaM2, isUS) (ac/ha), 2 decimals advanced_report.js:483 ✅
Rate rateStr(l.volumeL, l.areaM2) — measured only, dash when no flow data (the common case, matches the screenshot) advanced_report.js:484 ✅ — mission-level tile now consistent with this, see §5 item 10
Avg XT Error same formula as §2 advanced_report.js:485 ✅
Turn Time secStr(l.turnTimeS) — mean of measured 5–120s off→on gaps before this line, null if none measured advanced_report.js:486; engine report_util.js:168-184,285,297 ✅ (fixed — see §5.4)
Unsprayed-zone placeholder row single all-dash row per zone with lineCount===0 advanced_report.js:488-495 ✅ matches FR-4.6

5. Findings Summary

Fixed this pass

  1. §2, §3 — Planned Area ignored exclusion zones. plannedAreaM2() now nets out intersecting job.excludedAreas (mirrors jobUtil.calcTTSprayAreas), matching Job.ttSprArea. Fixed in helpers/report_util.js:45-58,97,103,301, wired in controllers/advanced_report.js:141. Tests added in tests/test_report_util.js.

  2. Duration formatting rounding bug. hm() displayed "60m"/"1h 60m" instead of carrying into the hour. Fixed in controllers/advanced_report.js:327-334.

  3. Avg Speed marker-exclusion — fixed, then found to be wrong, then reverted. Originally excluded sprayStat==3 from the speed average, citing AGGREGATED_FIELDS_CALCULATION.md's claim that legacy avgSpraySpeed excludes the marker. That claim turned out to be false — reading the actual legacy code (workers/job_worker.js:1459-1483) shows the sprayStat!==3 exclusion there applies to the spray-time accumulator, not speed; the speed accumulator (totalSpeedAcc) fires for every sprayStat>0 record, marker included, with no exclusion at all. The doc's prose conflated the two rules. Reverted the exclusion in helpers/report_util.js:236 back to including the marker (matching the real legacy code), inverted the test in tests/test_report_util.js to assert inclusion instead of exclusion, and corrected both this doc and ADVANCED_REPORTS_IMPLEMENTATION_PLAN.md §3 to stop citing that doc as authoritative on this point. Net effect: current code now matches legacy exactly. Lesson: verify claims against the referenced code directly, not against a doc's prose description of that code.

  4. Turn Time leaked across zones when a line number was reused in a different zone (e.g. Zone A's line 1 and Zone B's line 1 both flown, but only Zone A's line 1→2 transition had a measured turn). turnGaps were keyed only by bare llnum, and zone assignment happens after the turn-time state machine runs (it only knows zone membership once finish() does point-in-polygon assignment on the completed passes) — so both line rows pulled from the same bucket and Zone B's line incorrectly showed a 20s turn that was actually Zone A's. Reproduced and confirmed empirically before fixing. Fixed by tagging each recorded gap with the passes index of the pass it follows (helpers/report_util.js:126-131,176), then resolving that pass's assigned zone once zone assignment is done and keying the lookup by zoneIdx:llnum (helpers/report_util.js:277-285) — matching how line rows themselves are already keyed. Two regression tests added (tests/test_report_util.js) using the existing "typical multi-zone mission" fixture, which already contained this exact scenario but never asserted on it.

  5. Flight Line table order broke across a midnight wrap. The final .sort((a,b) => a.startTimeS - b.startTimeS) on the lines array used plain numeric subtraction — every other time comparison in this file deliberately uses the wrap-safe todDiff() instead. Reproduced directly: a line flown right before midnight (startTimeS: 86390) and one flown right after (startTimeS: 10, chronologically later) came out in reverse order (llnum 2 before llnum 1). Fixed by removing the sort entirely (helpers/report_util.js:287-303) — lineMap's Map insertion order already reflects true chronological order (a pass for a given zone+llnum is always first encountered at its true start time, since passes is built strictly in stream order and a pass can't start until the previous one ends), so no timestamp math was needed at all. Regression test added (tests/test_report_util.js).

  6. Coverage % could show 100% while zones were still completely untouched. The mission-level ratio summed each sprayed zone's raw, uncapped area before dividing by total planned area — so an overlapped/oversprayed zone (overlap is never deduplicated, see §2) could numerically exceed its own planned size and mask an entirely different zone that was never sprayed at all. Reported directly against real data (Job #96: "Coverage: 100%" alongside "Zones Sprayed: 22/27" — 5 zones never touched, yet the raw sprayed-area sum exceeded the full 27-zone planned total once overlap was counted). Reproduced with a minimal case: one zone oversprayed at 14,357% of its own plan, one zone completely untouched — old formula: 100%; correct answer: 50% (one zone fully done, one zone at zero). Fixed by capping each zone's contribution at min(zone.sprayedAreaM2, zone.plannedAreaM2) before summing, both in the analytics engine (helpers/report_util.js:385-392) and in the datasource builder's independent recomputation, which now reuses the engine's corrected value when no manual Report-Settings override is active (controllers/advanced_report.js:344-352,400). mission.sprayedAreaM2 itself (the true, uncapped swept-area total shown as "Sprayed Area") is unchanged — only the coverage percentage is capped per zone (at the mission level only — see below). Regression test added (tests/test_report_util.js).

    Companion fix, same root question: a zone's own coveragePct (shown on its Zone Detail page and its Mission Coverage Grid card) was also capped at 100% (report_util.js:349, now removed). Real overspray of a single zone — swath overlap, turns, re-flown sections, all normal in actual spraying — is legitimate, useful information (e.g. how much extra product went down), and capping it at "100%" threw that away, showing an oversprayed zone identically to an exactly-matched one. Confirmed against real data (Job #96's Mission Coverage Grid: zones showing 367/366.3 ac ≈ 100.2% and 433.1/412.3 ac ≈ 105.0%, both displayed as a flat "100%"). Removed the cap (helpers/report_util.js:349) so the zone's own card/detail page now shows the true percentage, which can exceed 100%. This does not affect the mission-level fix above — the mission calculation caps each zone's contribution to the sum independently, using the raw area values directly, not this field. pctStr() (controllers/advanced_report.js:325) has no capping of its own, so it renders whatever value it's given correctly. Regression test added.

    Second companion fix — the manual Report-Settings override path had the same bug via a different mechanism. Job #96's report kept showing "Coverage: 100%" even after the fix above, because Job.rptOp.printArea/areaSize/coverage had a manual override active (confirmed directly via MongoDB: rptOp.areaSize and rptOp.coverage convert to exactly the report's displayed 37,335.7 ac / 42,341.8 ac). The datasource builder's override-fallback branch — used precisely because an override has no per-zone breakdown to cap against — still did Math.min(ratio, 100), so it reproduced the identical bug through a path the zone-capping fix doesn't touch at all. And the same contradiction persisted: "Zones Sprayed: 22/27" is always computed from real per-zone data regardless of any override, so it kept correctly showing 5 untouched zones right next to a now-doubly-explained "100%". Verified 42,341.8/37,335.7 = 113.41% — the override itself claims more was sprayed than planned; capping that hides real information the same way the original bug did. Removed the cap on this fallback branch too (controllers/advanced_report.js:353-355) — it now shows the true, possibly >100%, ratio. No test file exists for advanced_report.js (only report_util.js has unit test coverage); verified manually against Job #96's exact rptOp values, confirmed the formula now returns 113.41% instead of 100%.

  7. Total Flight Time / Duration / Ferry Time used a wall-clock definition instead of legacy's 120-second-gap-capped sum. Re-verified directly (not just re-stated from before): the existing test_report_util.js "multiple files" test only proved inter-file gaps are excluded (a gap between two uploaded files) — it said nothing about a gap within one file. Tested that separately: one file, a spray segment, a 600-second internal gap (GPS dropout / idle) with no fileBreak() in between, then more spraying — mission.totalFlightS came back as 609 (the full last-minus-first span, gap included), where legacy's convention (sum of per-record deltas each capped at ≤120s, workers/job_worker.js:1447-1455) would have dropped that single 600s delta entirely, landing near 8s instead. Per explicit instruction to follow legacy's existing logic rather than invent new behavior: fixed — totalFlightS is now accumulated the same way, summing consecutive-point deltas and excluding any single gap that's ≤0 or >120s (helpers/report_util.js:142-158). This also let fileFirstT/fileLastT/closeFileTime() be removed entirely — the per-point accumulation naturally resets at fileBreak() since prev is already nulled there, so no separate file-boundary bookkeeping was needed. Verified: the 600s-gap case now returns 8 instead of 609. Regression test added (tests/test_report_util.js); the existing "multiple files" test still passes with the same expected value (both mechanisms agree when there's no internal gap), its description updated to stop citing the now-removed mechanism.

  8. Products table "Rate" column is missing its per-area unit suffix. Resolved — not a bug. Checked legacy's own product-rate formatting directly (controllers/job.js:933,940, makeJobAppDataSource's product loop): rateStr: utils.toLocaleStr(rate,2,lang) + ' ' + utils.getProdUnit(unit) — byte-for-byte the same bare-unit pattern ("0.50 gal", no /ac) as advanced_report.js:509. The advanced report's Products table faithfully mirrors an established legacy convention; it was the API contract doc's example that was wrong (docs/ ADVANCED_REPORTS_API.md, showed "rateStr": "0.50 gal/ac"). Fixed the doc example to "0.50 gal" to match both the real legacy behavior and the actual implementation. No code change needed.

  9. nearestZone() could attribute an entire, unrelated real flight to whatever zone happened to be "least wrong" — no matter how implausibly far away it actually was. Found on real production data: Job #95's "Spray_09" zone showed a full, convincing Zone Detail page — 55 flight lines, "Sprayed Area: 643.7 ac," "Coverage: 6.6%" — for ground that was never actually flown over. Traced the underlying GPS points: they're at 44.35°N, 44.36°N (southern Ontario) while Spray_09's actual polygon sits at 31.56–31.61°N (coastal Georgia) — over 1,400 km away. Confirmed via the job's own file metadata that this data (test-June23-01.zip, evidently a generic test recording) genuinely belongs to Job #95 — it's not a cross-job data mix-up — it simply has no real geographic relationship to any of the job's 9 zones. majorityZone() correctly found zero matches (as it should), which triggered the nearestZone() fallback — but that fallback had no distance sanity check at all, so it forced an assignment regardless of plausibility. This subsumes the old low-confidence longitude-compression note (previously item 11 in this section): fixed by rewriting nearestZone() to measure real great-circle distance (geoUtil.distance()) instead of raw lat/lon-degree math, and by adding NEAREST_ZONE_MAX_KM = 50 — a line whose closest zone is still farther than that is now genuinely unassignable (zoneIdx = -1), rather than forced onto the "least far" zone (helpers/report_util.js). Unassigned lines are excluded from every zone and zone-derived mission total (area, spray time, distance) the same way a truly-outside-every-zone line already was, and are now summarized separately in a new mission.unassigned field (lineCount, sprayTimeS, lengthM, areaM2) so this activity is visible in the datasource rather than silently vanishing or being mislabeled — surfacing it in the actual report template is a follow-up (not touched here, out of reach without the Stimulsoft designer). Verified end-to-end against Job #95's real zones and all 58,200 of its real GPS points, not just a synthetic case: Spray_09's lineCount went from 55 to 0, its sprayedAreaM2 from 643.7 ac to 0, and mission.unassigned now correctly reports the 55 lines, ~30.7 minutes of spray time (matching the original report's "Spray Time: 31m" almost exactly) that used to be fabricated into Spray_09's page. Three regression tests added (tests/test_report_util.js), including one reproducing this exact real-world scenario.

  10. Total Volume / Avg App Rate estimate broke mission ≡ Σ(zones) and turned out not to be measurement-free. When there's no flow-meter data, the mission-level tile silently substituted job.appRate × sprayedArea (a legacy-derived estimate, controllers/job.js:850-858, "Total used volume, estimated"), while the zone/line breakdown underneath stayed dashed (measured-only) — visible directly on Job #104/#96 screenshots and reproduced cleanly on real Job #97 data (single zone, all 292.6 ac of it sprayed, no flow-controller data): the mission tile would show "Total Volume: 2,926 gal, Avg App Rate: 10.00 gal/ac" while that job's only zone — which covers the exact same 292.6 ac — showed "Volume Applied: –, Avg App Rate: –" on its own Zone Detail page. Examined what the estimate actually represents: job.appRate is a single static value configured before the flight ever happened — a target, not anything measured during or after it — so the formula assumes uniform delivery at that exact rate everywhere the aircraft flew, with no allowance for rate changes, drift, or real variation. Legacy's own one-page Application Report has shown this same assumption-based number for years, but legacy has no zone/line breakdown to contradict — the Advanced Report's zone hierarchy is what turns "a labeled estimate" into "a number that visibly disagrees with the zone that contains it." Decided (explicit product direction): since the estimate isn't measurement-free, drop it from the mission level rather than push it down to zones/lines — mission-level Total Volume/Avg App Rate now dash under the same no-flow-data condition zones/lines already do, restoring mission ≡ Σ(zones). The manual actual-volume override (rptOp.useActualVol/rptOp.actualVol) is unaffected — that's a user-entered figure, not an assumption-based guess. Fixed by removing the job.appRate × sprayedArea fallback block entirely (controllers/advanced_report.js:359-365); no test file exists for advanced_report.js (only report_util.js has unit coverage — see D1/D2 split in the implementation plan), verified manually against Job #97's real mission.volumeL === null engine output and the resulting dashed tile. The deeper question of whether "Avg App Rate" should mean a measured rate or a planned/configured rate remains — legacy's own Application.appRate field is itself defined as mean(lhaReq) (the requested rate recorded per GPS point, workers/job_worker.js:1643-1687), a third definition this engine doesn't currently read at all (lhaReq isn't in DETAIL_PROJECTION, controllers/advanced_report.js:47) — noted for a future pass, not blocking this fix.

  11. majorityZone()'s "whole line to one zone" rule let a small zone lose most of its coverage credit to a much larger neighbor sharing a boundary. Found on real production data: Job #106's Zone 4 (18 ac planned) showed 12.4% coverage — its map thumbnail was almost entirely empty except one thin sliver — while Zone 3 (45.6 ac) showed only 68.8% with just half its polygon painted, and the adjacent, much larger Zone 2 (412.3 ac) showed 107.7%, over its own plan. Traced the cause: a pass whose GPS points straddle the boundary between two zones was being handed to whichever zone held the sampled majority of its points — for a pass mostly inside Zone 2 but genuinely also covering part of Zone 3/4, Zone 2 took 100% of the credit, Zone 3/4 got none of theirs. Confirmed via the underlying line numbers: 74/83–91 (the boundary-crossing passes) appeared in Zone 2's, Zone 3's, and Zone 4's Flight Line tables simultaneously — proof the same physical passes were being fragmented and miscredited across zone boundaries, not a rendering artifact. Fixed by adding a cheap sampled pre-check (straddlesMultipleZones, reuses majorityZone's existing sample — no added cost for the overwhelming majority of lines that stay inside one zone) that only escalates to a full, unsampled per-point zone lookup (splitPassByZone) for lines that actually touch more than one real zone; each resulting per-zone segment gets its own stats via computeSegmentStats (extracted, byte-identical to the old per-pass loop — DRY, zero behavior change for the non-split path). Turn-gap attribution (passLastZone) and the map-drawing draw.spray segments were updated to key off the new per-segment zone tags instead of the old per-pass one. Documented, deliberate trade-off: the single GPS interval that actually crosses the boundary isn't counted in either segment's length/area/volume — a per-crossing discrepancy of one GPS interval (a few meters), traded for not needing to split that edge's distance between two zones. Verified end-to-end against Job #106's real zones and all 40,156 of its real GPS points: Zone 4 went from 12.4% to 96.9% coverage, Zone 3 from 68.8% to 112%, Zone 2 correctly came back down from 107.7% to 98.1%, while Zone 1 — which doesn't border any other zone in this job — stayed byte-for-byte identical (99.8% both before and after), confirming the fix is scoped to boundary-sharing zones only. Two regression tests added (tests/test_report_util.js): one confirming the pre-existing majority-zone case (a line leaving a zone into empty space) is untouched, one reproducing the genuine cross-zone-boundary split and checking mission totals still reconcile exactly with the split zone roll-ups (NFR-3.3).

Confirmed bug — fix deferred

  1. Ounce-configured jobs (job.appRateUnit === RateUnits.OZ_PER_ACRE) get mislabeled rate fields for measured data. None of the mission/zone/line volume or rate formatters have an oz-aware conversion path, unlike the Products table (which already does this correctly — see §4 above). rateStr() (advanced_report.js:335-338) always labels the number with the raw rateUnitString(job.appRateUnit, ...) ("oz/ac"), but the number itself always comes from toVolume(volL, isLiquid, isUS), which for isLiquid===true unconditionally means gallons, never ounces. So for measured flow data on an oz-configured job, "Avg App Rate"/"Rate" shows a real gal/ac figure under an "oz/ac" label — the number is correct, the unit suffix isn't. volStr has the same gap (hardcodes a "gal" label, no oz path). Legacy already solves this — controllers/job.js:853-858 converts with utils.ozToGal() and swaps the displayed rate-unit to GAL_PER_ACRE so label and number never disagree (the same pattern already applied correctly in the Products table's unit === Units.OZ branch, advanced_report.js:520) — but the Advanced Report has no equivalent of either step in the mission/zone/line paths.

Originally found alongside a second sub-bug: the mission-level estimate (job.appRate × area) fed an oz-denominated number into the same gallon-only conversion, producing a volume 128× too large (reproduced on Job 13603: 1,716.85 oz true volume computed and displayed as 1,717 gal). That sub-bug is now moot — item 10's fix removed the estimate-fallback block entirely (controllers/advanced_report.js:359-365 no longer computes an estimate at all), so there's nothing left to mislabel on that path. The mislabeling above, for genuinely measured flow data on an oz-configured job, is unaffected by that fix and remains open.

  • Fix, not yet applied. Recommended approach mirrors legacy exactly: thread a "display rate unit" (raw job.appRateUnit, or GAL_PER_ACRE when it was oz) through rateStr/volStr instead of always assuming gallons for any isLiquid unit.

Open — deferred pending a product decision

  1. Legacy's "Actual Spray Volume" fills in a guessed rate whenever the flow sensor reads zero; the Advanced Report's volumeL never does. Both engines agree exactly on every segment where the flow sensor (lminApp) actually reported something — confirmed on real Job 105 data (JBI - AutoCal, 21,824 real points, 3 files): a faithful reconstruction of legacy's own per-record formula (getAppliedRate(), workers/job_worker.js:1640-1662 + helpers/utils.js:1795-1820), using the real stored utmX/utmY (not an approximation), lands at 120.16 gal from flow-derived segments alone — matching the engine's real mission.volumeL (117.43 gal) to within rounding. The two methods are mathematically equivalent when both are looking at real sensor data; this is not a calculation-method difference.
  • The gap to the DB's actual stored App.totalSprayMat (154.07 gal — the "Mat Sprayed"/"Actual Spray Volume" figure) is fully explained, and reproduces to the decimal: getAppliedRate() (helpers/utils.js:154-176,1798) reads a per-file configured rate from the imported file's own Q-file metadata (fileMeta.appRate — 0.16 gal/ac for this file, an equipment setting, not job.appRate) and substitutes it whenever record.lminApp is falsy for that GPS record, instead of treating a zero reading as zero volume. For Job 105, 4,078 of the ~18,634 contributing spray segments (≈22%) had lminApp === 0 — legacy added 33.91 gal for those moments using the substitute rate; the Advanced Report added 0. 120.16 + 33.91 = 154.07 — an exact match to the stored value, confirming this is the complete and only mechanism, not one contributing factor among several.
  • The real, unresolved question is which behavior is more correct, and it can't be settled from the data alone: if a lminApp === 0 reading reflects a genuine sensor dropout/glitch while the nozzle kept running, legacy's fill-in is the more accurate total. If it reflects the nozzle genuinely being off at that instant (boom-section cycling, swath-overlap avoidance switching sections off, a tank running dry), the Advanced Report's "trust the zero" is more accurate and legacy has been quietly overstating volume by padding every such moment. Given how large and systematic the share is on this file (≈1 in 5 segments, not an occasional blip), a genuine zero-flow condition looks more likely than sensor noise, but this is a judgment call about equipment behavior, not something resolvable by re-reading either codebase.
  • Not fixed. Needs a decision: leave the Advanced Report trusting real zero readings (current behavior, arguably the more defensible default absent evidence the sensor is unreliable), or add an equivalent "assume nominal rate on a zero reading" fallback to match legacy's number exactly.

Pre-existing platform bug — found here, but out of scope to fix in this report

  1. Overlapping exclusion zones double-subtract their shared overlap in plannedAreaM2(), and can drive the planned area negative. Reproduced directly: a zone with two exclusions covering 60% and 60% (overlapping each other, together covering 90%) should net 10% remaining — instead returns a negative area, because each exclusion's overlap is subtracted independently instead of first unioning the exclusions together. Confirmed this is not new — jobUtil.calcTTSprayAreas() (helpers/job_util.js:117-149) has the byte-identical flaw and produces the exact same negative number on the same input, since plannedAreaM2() was deliberately written to mirror it. This affects the live Job.ttSprArea field today, not just this report — fixing it only here would create a new mismatch rather than remove one. Flagging as a separate platform issue, not fixing in report_util.js. Only triggers when a user draws two exclusion zones that overlap each other inside the same spray zone.

Low-confidence / low-impact notes (not fixed)

  1. majorityZone() gives all point-in-polygon credit to the first matching zone only (break on match) — confirmed directly: two overlapping zones with 10 test points inside both produced counts: [10, 0], denying the second zone any credit for points genuinely inside it too. Matters only if spray zones (not just exclusion zones, which are already known to sometimes overlap) are ever drawn overlapping each other (report_util.js:63-77).
  2. Zone-level avgTurnTimeS is an unweighted mean of each line's own (already-averaged) turn time — a double average, separate from the cross-zone leak fixed in item 4 above (report_util.js:329).
  3. Manual Report Settings overrides (rptOp.areaSize/coverage) affect the mission summary (advanced_report.js:346-349) but not coverageCards/zonesDS, which always show computed values — pre-existing legacy-pattern behavior, not something newly introduced.
  4. "Actual Dates" assumes the last app sorted by startDateTime also has the latest endDateTime (advanced_report.js:366-374) — true for normal sequential flights only.
  5. A self-intersecting ("bowtie") zone polygon silently computes a near-zero area (shoelace-formula signed-area cancellation) rather than erroring — an inherent limitation of this class of area algorithm for any invalid, non-simple polygon, not specific to this code. Only matters if the zone-drawing tool ever allows saving such a shape.
  6. Turn-time state machine has no branch for an off-period whose llnum doesn't match the line just being tracked (only reachable with anomalous data that doesn't carry over the previous line's number during ferry/off travel — not the convention this codebase's own test fixtures assume). Reproduced directly: a genuine 8s turn got silently dropped (turnTimeS: null) when the off-period in between carried an unexpected llnum. Fails safe — degrades to "unmeasured," never to a wrong value — consistent with every other edge case found in this engine (report_util.js:171-181).

Test coverage

tests/test_report_util.js — 40 passing cases covering: midnight-wrap time math, planned-area exclusion-zone netting (new), multi-zone/multi-file reconciliation (NFR-3.3), no-flow-controller degradation, SatLoc-style missing-sensor degradation, unsprayed-zone dash handling, boundary-straddling majority-zone assignment (a line leaving a zone into empty space, stays whole), genuine cross-zone- boundary splitting (new — a line actually crossing into a different zone becomes one segment per zone, reproduces the exact Job #106 real-world case, item 11), a degenerate zero-length/zero-time line dropped from the report instead of showing a "0 ft / 0 ac" row (new), nearest-zone fallback, the line-start-marker speed inclusion (matching the real legacy code — see §5, item 3), the cross-zone turn-time leak on a reused line number (new), line ordering across a midnight wrap (new), mission Coverage % never reaching 100% while a zone remains untouched (new), a zone's own Coverage % correctly showing >100% on real overspray instead of capping (new), totalFlightS excluding an internal >120s gap the same way legacy does (new), volume integration excluding the same gap (new), and a line implausibly far from every zone landing in mission.unassigned instead of being force-assigned (new — reproduces the exact Job #95 real-world case).

Run: npx mocha --exit --require tests/setup.js tests/test_report_util.js