agnav/agmission
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
67ee8b86b8
agmission/server/tests/DATA_EXPORT_TEST_SUITE_README.md

9.6 KiB
Raw Blame History

Data Export API - Test Suite

Overview

This test suite verifies all endpoints of the Data Export API against real database data. The tests check that:

  1. ✅ All endpoints return proper responses
  2. ✅ API response fields match database values exactly
  3. ✅ No fields are filled with wrong or assumed data
  4. ✅ CSV and GeoJSON formats are valid
  5. ✅ Unit conversions (metric ↔ US) are accurate
  6. ✅ sprayStat marker values are preserved in public records/exports
  7. ✅ appRateApplied computation is correct
  8. ✅ Authorization and validation work properly

Test Scripts

1. test_export_verify_endpoints.js

Purpose: Basic endpoint verification with real data

Tests:

  • GET /api/v1/jobs/:jobId/sessions - Session summary
  • GET /api/v1/jobs/:jobId/sessions/:fileId/records - GPS trace records
  • GET /api/v1/jobs/:jobId/areas - Spray area GeoJSON
  • POST /api/v1/jobs/:jobId/export - Trigger export
  • GET /api/v1/exports/:exportId - Poll export status
  • GET /api/v1/exports/:exportId/download - Download export file
  • Authorization - API key validation

What it verifies:

  • All endpoints are accessible with valid API key
  • Responses contain expected fields
  • sprayStat values are returned as stored (including marker states)
  • API key authorization works
  • Test data flows through the pipeline correctly

Run:

# Terminal 1: Start the server on dedicated test port
AGM_PORT=4107 node -r dotenv/config server.js dotenv_config_path=./environment.env

# Terminal 2: Run tests
AGM_PORT=4107 npx mocha --exit tests/test_export_verify_endpoints.js

Sample Output:

  Data Export API - Endpoint Verification
  ✅ Sessions endpoint: 1 session(s)
     - totalFlightTime_s: 3600s
     - avgSpraySpeed_ms: 40 m/s
  ✅ Records endpoint: 10 records
     - spray-state markers preserved
  ✅ Areas endpoint: 1 features
  ✅ Export triggered: [exportId]
  ✅ Export status: ready
  ✅ Downloaded: [N] lines, [N] columns
  ✅ Invalid key rejected (401)

2. test_data_export_api_all_endpoints.js

Purpose: Comprehensive endpoint testing with field-level validation

Tests:

  • User and data setup
  • Session summary endpoint with db comparison
  • Raw GPS trace endpoint with spray-state preservation
  • Spray areas GeoJSON endpoint
  • CSV export with metric units
  • GeoJSON export
  • US units export
  • Interval thinning
  • Authorization validation
  • Data integrity checks (appRateApplied computation)

What it verifies:

  • Every response field matches the database source
  • appRateApplied is computed correctly
  • All computed fields are accurate
  • Multiple export formats work
  • Unit conversion is available

Run:

# Start server (if not already running in another terminal)
AGM_PORT=4107 node -r dotenv/config server.js dotenv_config_path=./environment.env

# Run suite
AGM_PORT=4107 npx mocha --exit tests/test_data_export_api_all_endpoints.js

3. test_data_export_formats.js

Purpose: CSV and GeoJSON format validation

Tests:

  • CSV generation (headers, data rows)
  • CSV metric unit headers
  • CSV US unit headers
  • CSV US unit value conversion
  • GeoJSON validity (valid JSON, FeatureCollection structure)
  • GeoJSON geometry validation (Point, coordinates, altitude)
  • spray-state preservation in exports
  • Feature properties in GeoJSON
  • Interval thinning in exports

What it verifies:

  • CSV files are well-formed and properly escaped
  • Unit conversion factors are applied correctly
  • GeoJSON is valid RFC 7946 format
  • All records are included (or thinned by interval)
  • spray-state marker rows are preserved in exports

Run:

# Start server (if not already running in another terminal)
AGM_PORT=4107 node -r dotenv/config server.js dotenv_config_path=./environment.env

# Run suite
AGM_PORT=4107 npx mocha --exit tests/test_data_export_formats.js

Running All Tests

# Terminal 1: start server (recommended dedicated test port)
AGM_PORT=4107 node -r dotenv/config server.js dotenv_config_path=./environment.env

# Terminal 2: run all Data Export API suites
AGM_PORT=4107 npx mocha --exit tests/test_export_verify_endpoints.js && \
AGM_PORT=4107 npx mocha --exit tests/test_data_export_api_all_endpoints.js && \
AGM_PORT=4107 npx mocha --exit tests/test_data_export_formats.js

Branch Endpoint Coverage (This Feature Branch)

Use these suites when validating endpoints added/updated in this branch:

  1. tests/test_data_export_api_all_endpoints.js (primary regression)

    • GET /api/v1/jobs/:jobId/sessions
    • GET /api/v1/jobs/:jobId/sessions/:fileId/records
    • GET /api/v1/jobs/:jobId/areas
    • POST /api/v1/jobs/:jobId/export
    • GET /api/v1/exports/:exportId
    • GET /api/v1/exports/:exportId/download

  2. tests/test_data_export_formats.js

    • CSV/GeoJSON format checks
    • metric/us unit validation
    • export content validation

  3. tests/test_export_verify_endpoints.js

    • smoke verification for end-to-end endpoint availability

Quick run (single command):

AGM_PORT=4107 npx mocha --exit \
  tests/test_data_export_api_all_endpoints.js \
  tests/test_data_export_formats.js \
  tests/test_export_verify_endpoints.js

What Issues These Tests Can Identify

1. Wrong/Assumed Data

✅ Tests verify that every response field exactly matches the database

  • If a field is missing from the response, test fails
  • If a field has wrong value, test fails with expected vs. actual
  • If a field is computed incorrectly, test fails

2. sprayStat Marker Preservation

✅ Tests verify spray-state values are preserved (including marker states such as 3)

  • If marker rows are unexpectedly removed from CSV/records endpoint, test fails
  • If GeoJSON omits marker rows present in source data, test fails

3. appRateApplied Computation

✅ Tests verify the formula is correct: lminApp / (grSpeed × swath) × 10000

  • If computation is wrong, test fails with tolerance check

4. Unit Conversion

✅ Tests verify metric-to-US conversions are accurate

  • Alt: m × 3.28084 → ft
  • Speed: m/s × 2.23694 → mph
  • Temp: °C × 9/5 + 32 → °F
  • Flow: L/min × 0.264172 → gal/min
  • App rate: L/ha × 0.10694 → gal/ac

5. Format Validity

✅ Tests verify files are well-formed

  • CSV: proper escaping, consistent column count
  • GeoJSON: valid JSON, proper structure, valid coordinates

6. Authorization

✅ Tests verify API key authentication works

  • Invalid/missing keys are rejected (401)
  • Valid keys are accepted

Test Data

Each test automatically creates:

  • 1 Admin user (owner)
  • 1 Client user (required by Job model)
  • 1 Pilot
  • 1 Vehicle/Aircraft
  • 1 Job (with spray areas)
  • 1 App (session)
  • 1 AppFile
  • 10 AppDetail records (GPS points, mix of spray states)
  • 1 API key with DATA_EXPORT service

All data is cleaned up after tests complete.

Key Fields Tested

Sessions Endpoint

totalFlightTime_s, totalSprayTime_s, totalTurnTime_s
totalSprayed_ha, totalSprayMat, totalSprayMatUnit, avgSpraySpeed_ms
sprayZoneName, sprayZoneArea_ha, appRate, appRateUnit
flowController, sprayOnLag_s, sprayOffLag_s, pulsesPerLitre
sessionPilotName, pilotId, pilotName, aircraftName, aircraftTailNumber

Records Endpoint

GPS: gpsTime, lat, lon, utmX, utmY, alt, groundSpeed, heading, crossTrackError
Quality: lockedLine, hdop, satsIn, tslu, calcodeFreq
Application: flowRateApplied, flowRateRequired, appRateRequired, appRateApplied, swathWidth, boomPressure_psi, sprayStat
MET: windSpeed, windDir, temp, humidity
Session metadata: sprayOnLag_s, sprayOffLag_s, pulsesPerLitre

Areas Endpoint

GeoJSON Feature properties:
name, appRate, area_ha, type
geometry: Polygon coordinates

Troubleshooting

ECONNREFUSED on port 3000

Problem: Tests fail with connection refused Solution: Start the server first

npm run dev  # Terminal 1
npm test     # Terminal 2 (after server starts)

Timeout errors

Problem: Tests timeout waiting for async export Solution: Increase timeout or check if background workers are running

npm test -- --timeout 120000  # 2 minute timeout

Field mismatch errors

Problem: Test says API field doesn't match database value Solution: Check the actual vs. expected values in test output

  • For numeric fields: tolerance is usually 0.01
  • For string fields: must be exact match
  • For null fields: check if field should exist

sprayStat marker mismatch

Problem: Response row count or marker values differ from source AppDetail data Solution: Verify endpoint/export queries are not applying sprayStat exclusion filters

Performance Notes

  • Each test suite takes ~1-2 minutes (waiting for async exports)
  • Tests create isolated test data (no interference between runs)
  • All cleanup is automatic (no manual database cleanup needed)
  • Tests are safe to run repeatedly on production-like databases
  • No modifications to existing data (read-only for queries, isolated test data for creation)

Next Steps

  1. Run the tests: Execute scripts to identify any issues
  2. Fix any failures: Use error messages to locate incorrect data mappings
  3. Add more tests: Extend with additional validation scenarios
  4. Integrate with CI/CD: Add to your test pipeline (npm test)
  5. Monitor: Keep tests passing as you modify endpoints

Related Files