agmission/Development/server/tests/DATA_EXPORT_TEST_SUITE_README.md

Data Export API - Test Suite

Overview

This test suite verifies all endpoints of the Data Export API against real database data. The tests check that:

1. ✅ All endpoints return proper responses
2. ✅ API response fields match database values exactly
3. ✅ No fields are filled with wrong or assumed data
4. ✅ CSV and GeoJSON formats are valid
5. ✅ Unit conversions (metric ↔ US) are accurate
6. ✅ sprayStat=3 records are properly filtered
7. ✅ appRateApplied computation is correct
8. ✅ Authorization and validation work properly

---

Test Scripts

1. test_export_verify_endpoints.js

Purpose: Basic endpoint verification with real data

Tests:

• GET /api/v1/jobs/:jobId/sessions - Session summary
• GET /api/v1/jobs/:jobId/sessions/:fileId/records - GPS trace records
• GET /api/v1/jobs/:jobId/areas - Spray area GeoJSON
• POST /api/v1/jobs/:jobId/export - Trigger export
• GET /api/v1/exports/:exportId - Poll export status
• GET /api/v1/exports/:exportId/download - Download export file
• Authorization - API key validation

What it verifies:

• All endpoints are accessible with valid API key
• Responses contain expected fields
• sprayStat=3 is filtered correctly
• API key authorization works
• Test data flows through the pipeline correctly

Run:

# Terminal 1: Start the server
npm run dev
# or
DEBUG=agm:* node --inspect server.js

# Terminal 2: Run tests
npm test -- tests/test_export_verify_endpoints.js --timeout 60000

Sample Output:

  Data Export API - Endpoint Verification
  ✅ Sessions endpoint: 1 session(s)
     - totalFlightTime_s: 3600s
     - avgSpraySpeed_ms: 40 m/s
  ✅ Records endpoint: 9 records
     - sprayStat=3 properly filtered
  ✅ Areas endpoint: 1 features
  ✅ Export triggered: [exportId]
  ✅ Export status: ready
  ✅ Downloaded: [N] lines, [N] columns
  ✅ Invalid key rejected (401)

---

2. test_data_export_api_all_endpoints.js

Purpose: Comprehensive endpoint testing with field-level validation

Tests:

• User and data setup
• Session summary endpoint with db comparison
• Raw GPS trace endpoint with sprayStat=3 filtering
• Spray areas GeoJSON endpoint
• CSV export with metric units
• GeoJSON export
• US units export
• Interval thinning
• Authorization validation
• Data integrity checks (appRateApplied computation)

What it verifies:

• Every response field matches the database source
• appRateApplied is computed correctly
• All computed fields are accurate
• Multiple export formats work
• Unit conversion is available

Run:

npm test -- tests/test_data_export_api_all_endpoints.js --timeout 60000

---

3. test_data_export_formats.js

Purpose: CSV and GeoJSON format validation

Tests:

• CSV generation (headers, data rows)
• CSV metric unit headers
• CSV US unit headers
• CSV US unit value conversion
• GeoJSON validity (valid JSON, FeatureCollection structure)
• GeoJSON geometry validation (Point, coordinates, altitude)
• sprayStat=3 exclusion in exports
• Feature properties in GeoJSON
• Interval thinning in exports

What it verifies:

• CSV files are well-formed and properly escaped
• Unit conversion factors are applied correctly
• GeoJSON is valid RFC 7946 format
• All records are included (or thinned by interval)
• sprayStat=3 segment markers are excluded from exports

Run:

npm test -- tests/test_data_export_formats.js --timeout 60000

---

Running All Tests

# Start server in one terminal
DEBUG=agm:* npm run dev

# In another terminal, run all tests
npm test -- tests/test_export_verify_endpoints.js --timeout 60000 && \
npm test -- tests/data_export_api_all_endpoints.js --timeout 60000 && \
npm test -- tests/test_data_export_formats.js --timeout 60000

---

What Issues These Tests Can Identify

1. Wrong/Assumed Data

✅ Tests verify that every response field exactly matches the database

• If a field is missing from the response, test fails
• If a field has wrong value, test fails with expected vs. actual
• If a field is computed incorrectly, test fails

2. sprayStat=3 Filtering

✅ Tests verify that segment START markers (sprayStat=3) are excluded

• If any sprayStat=3 appears in CSV/records endpoint, test fails
• If GeoJSON contains sprayStat=3, test fails

3. appRateApplied Computation

✅ Tests verify the formula is correct: lminApp / (grSpeed × swath) × 10000

• If computation is wrong, test fails with tolerance check

4. Unit Conversion

✅ Tests verify metric-to-US conversions are accurate

• Alt: m × 3.28084 → ft
• Speed: m/s × 2.23694 → mph
• Temp: °C × 9/5 + 32 → °F
• Flow: L/min × 0.264172 → gal/min
• App rate: L/ha × 0.10694 → gal/ac

5. Format Validity

✅ Tests verify files are well-formed

• CSV: proper escaping, consistent column count
• GeoJSON: valid JSON, proper structure, valid coordinates

6. Authorization

✅ Tests verify API key authentication works

• Invalid/missing keys are rejected (401)
• Valid keys are accepted

---

Test Data

Each test automatically creates:

• 1 Admin user (owner)
• 1 Client user (required by Job model)
• 1 Pilot
• 1 Vehicle/Aircraft
• 1 Job (with spray areas)
• 1 App (session)
• 1 AppFile
• 10 AppDetail records (GPS points, mix of spray states)
• 1 API key with DATA_EXPORT service

All data is cleaned up after tests complete.

---

Key Fields Tested

Sessions Endpoint

totalFlightTime_s, totalSprayTime_s, totalTurnTime_s
totalSprayed_ha, totalSprayMat, totalSprayMatUnit, avgSpraySpeed_ms
sprayZoneName, sprayZoneArea_ha, appRate, appRateUnit
matType, flowController, sprayOnLag_s, sprayOffLag_s, pulsesPerLitre
sessionPilotName, pilotId, pilotName, aircraftName, aircraftTailNumber

Records Endpoint

GPS: gpsTime, lat, lon, utmX, utmY, alt, groundSpeed, heading, crossTrackError
Quality: lockedLine, hdop, satsInView, correctionId, waasId
Application: flowRateApplied, flowRateRequired, appRateRequired, appRateApplied, swathWidth, boomPressure_psi, sprayStat
MET: windSpeed, windDir, temp, humidity
Session metadata: sprayOnLag_s, sprayOffLag_s, pulsesPerLitre

Areas Endpoint

GeoJSON Feature properties:
name, appRate, area_ha, type
geometry: Polygon coordinates

---

Troubleshooting

ECONNREFUSED on port 3000

Problem: Tests fail with connection refused Solution: Start the server first

npm run dev  # Terminal 1
npm test     # Terminal 2 (after server starts)

Timeout errors

Problem: Tests timeout waiting for async export Solution: Increase timeout or check if background workers are running

npm test -- --timeout 120000  # 2 minute timeout

Field mismatch errors

Problem: Test says API field doesn't match database value Solution: Check the actual vs. expected values in test output

• For numeric fields: tolerance is usually 0.01
• For string fields: must be exact match
• For null fields: check if field should exist

sprayStat=3 not filtered

Problem: Test finds sprayStat=3 in response Solution: Verify the filter in controllers/api_pub.js and api_export.js

// Should be:
{ sprayStat: { $ne: 3 } }  // exclude 3

---

Performance Notes

• Each test suite takes ~1-2 minutes (waiting for async exports)
• Tests create isolated test data (no interference between runs)
• All cleanup is automatic (no manual database cleanup needed)
• Tests are safe to run repeatedly on production-like databases
• No modifications to existing data (read-only for queries, isolated test data for creation)

---

Next Steps

1. Run the tests: Execute scripts to identify any issues
2. Fix any failures: Use error messages to locate incorrect data mappings
3. Add more tests: Extend with additional validation scenarios
4. Integrate with CI/CD: Add to your test pipeline (npm test)
5. Monitor: Keep tests passing as you modify endpoints

---

Related Files

• Endpoints: controllers/api_pub.js, controllers/api_export.js
• Models: model/application_detail.js, model/export_job.js
• Routes: routes/export.js, routes/api_pub.js
• Design Doc: docs/DATA_EXPORT_API_DESIGN.md