agmission/server/tests/DATA_EXPORT_TEST_SUITE_README.md

# Data Export API - Test Suite

## Overview

This test suite verifies all endpoints of the Data Export API against real database data. The tests check that:

1. ✅ All endpoints return proper responses
2. ✅ API response fields match database values exactly
3. ✅ No fields are filled with wrong or assumed data
4. ✅ CSV and GeoJSON formats are valid
5. ✅ Unit conversions (metric ↔ US) are accurate
6. ✅ sprayStat marker values are preserved in public records/exports
7. ✅ appRateApplied computation is correct
8. ✅ Authorization and validation work properly

---

## Test Scripts

### 1. `test_export_verify_endpoints.js`
**Purpose**: Basic endpoint verification with real data

**Tests**:
- GET `/api/v1/jobs/:jobId/sessions` - Session summary
- GET `/api/v1/jobs/:jobId/sessions/:fileId/records` - GPS trace records  
- GET `/api/v1/jobs/:jobId/areas` - Spray area GeoJSON
- POST `/api/v1/jobs/:jobId/export` - Trigger export
- GET `/api/v1/exports/:exportId` - Poll export status
- GET `/api/v1/exports/:exportId/download` - Download export file
- Authorization - API key validation

**What it verifies**:
- All endpoints are accessible with valid API key
- Responses contain expected fields
- sprayStat values are returned as stored (including marker states)
- API key authorization works
- Test data flows through the pipeline correctly

**Run**:
```bash
# Terminal 1: Start the server on dedicated test port
AGM_PORT=4107 node -r dotenv/config server.js dotenv_config_path=./environment.env

# Terminal 2: Run tests
AGM_PORT=4107 npx mocha --exit tests/test_export_verify_endpoints.js
```

**Sample Output**:
```
  Data Export API - Endpoint Verification
  ✅ Sessions endpoint: 1 session(s)
     - totalFlightTime_s: 3600s
     - avgSpraySpeed_ms: 40 m/s
  ✅ Records endpoint: 10 records
     - spray-state markers preserved
  ✅ Areas endpoint: 1 features
  ✅ Export triggered: [exportId]
  ✅ Export status: ready
  ✅ Downloaded: [N] lines, [N] columns
  ✅ Invalid key rejected (401)
```

---

### 2. `test_data_export_api_all_endpoints.js`
**Purpose**: Comprehensive endpoint testing with field-level validation

**Tests**:
- User and data setup
- Session summary endpoint with db comparison
- Raw GPS trace endpoint with spray-state preservation
- Spray areas GeoJSON endpoint
- CSV export with metric units
- GeoJSON export
- US units export
- Interval thinning
- Authorization validation
- Data integrity checks (appRateApplied computation)

**What it verifies**:
- Every response field matches the database source
- appRateApplied is computed correctly
- All computed fields are accurate
- Multiple export formats work
- Unit conversion is available

**Run**:
```bash
# Start server (if not already running in another terminal)
AGM_PORT=4107 node -r dotenv/config server.js dotenv_config_path=./environment.env

# Run suite
AGM_PORT=4107 npx mocha --exit tests/test_data_export_api_all_endpoints.js
```

---

### 3. `test_data_export_formats.js`
**Purpose**: CSV and GeoJSON format validation

**Tests**:
- CSV generation (headers, data rows)
- CSV metric unit headers
- CSV US unit headers
- CSV US unit value conversion
- GeoJSON validity (valid JSON, FeatureCollection structure)
- GeoJSON geometry validation (Point, coordinates, altitude)
- spray-state preservation in exports
- Feature properties in GeoJSON
- Interval thinning in exports

**What it verifies**:
- CSV files are well-formed and properly escaped
- Unit conversion factors are applied correctly
- GeoJSON is valid RFC 7946 format
- All records are included (or thinned by interval)
- spray-state marker rows are preserved in exports

**Run**:
```bash
# Start server (if not already running in another terminal)
AGM_PORT=4107 node -r dotenv/config server.js dotenv_config_path=./environment.env

# Run suite
AGM_PORT=4107 npx mocha --exit tests/test_data_export_formats.js
```

---

## Running All Tests

```bash
# Terminal 1: start server (recommended dedicated test port)
AGM_PORT=4107 node -r dotenv/config server.js dotenv_config_path=./environment.env

# Terminal 2: run all Data Export API suites
AGM_PORT=4107 npx mocha --exit tests/test_export_verify_endpoints.js && \
AGM_PORT=4107 npx mocha --exit tests/test_data_export_api_all_endpoints.js && \
AGM_PORT=4107 npx mocha --exit tests/test_data_export_formats.js
```

## Branch Endpoint Coverage (This Feature Branch)

Use these suites when validating endpoints added/updated in this branch:

1. `tests/test_data_export_api_all_endpoints.js` (primary regression)
   - `GET /api/v1/jobs/:jobId/sessions`
   - `GET /api/v1/jobs/:jobId/sessions/:fileId/records`
   - `GET /api/v1/jobs/:jobId/areas`
   - `POST /api/v1/jobs/:jobId/export`
   - `GET /api/v1/exports/:exportId`
   - `GET /api/v1/exports/:exportId/download`

2. `tests/test_data_export_formats.js`
   - CSV/GeoJSON format checks
   - metric/us unit validation
   - export content validation

3. `tests/test_export_verify_endpoints.js`
   - smoke verification for end-to-end endpoint availability

Quick run (single command):

```bash
AGM_PORT=4107 npx mocha --exit \
  tests/test_data_export_api_all_endpoints.js \
  tests/test_data_export_formats.js \
  tests/test_export_verify_endpoints.js
```

---

## What Issues These Tests Can Identify

### 1. **Wrong/Assumed Data**
✅ Tests verify that every response field exactly matches the database
- If a field is missing from the response, test fails
- If a field has wrong value, test fails with expected vs. actual
- If a field is computed incorrectly, test fails

### 2. **sprayStat Marker Preservation**
✅ Tests verify spray-state values are preserved (including marker states such as 3)
- If marker rows are unexpectedly removed from CSV/records endpoint, test fails
- If GeoJSON omits marker rows present in source data, test fails

### 3. **appRateApplied Computation**
✅ Tests verify the formula is correct: lminApp / (grSpeed × swath) × 10000
- If computation is wrong, test fails with tolerance check

### 4. **Unit Conversion**
✅ Tests verify metric-to-US conversions are accurate
- Alt: m × 3.28084 → ft
- Speed: m/s × 2.23694 → mph  
- Temp: °C × 9/5 + 32 → °F
- Flow: L/min × 0.264172 → gal/min
- App rate: L/ha × 0.10694 → gal/ac

### 5. **Format Validity**
✅ Tests verify files are well-formed
- CSV: proper escaping, consistent column count
- GeoJSON: valid JSON, proper structure, valid coordinates

### 6. **Authorization**
✅ Tests verify API key authentication works
- Invalid/missing keys are rejected (401)
- Valid keys are accepted

---

## Test Data

Each test automatically creates:
- 1 Admin user (owner)
- 1 Client user (required by Job model)
- 1 Pilot
- 1 Vehicle/Aircraft
- 1 Job (with spray areas)
- 1 App (session)
- 1 AppFile
- 10 AppDetail records (GPS points, mix of spray states)
- 1 API key with DATA_EXPORT service

All data is cleaned up after tests complete.

---

## Key Fields Tested

### Sessions Endpoint
```
totalFlightTime_s, totalSprayTime_s, totalTurnTime_s
totalSprayed_ha, totalSprayMat, totalSprayMatUnit, avgSpraySpeed_ms
sprayZoneName, sprayZoneArea_ha, appRate, appRateUnit
flowController, sprayOnLag_s, sprayOffLag_s, pulsesPerLitre
sessionPilotName, pilotId, pilotName, aircraftName, aircraftTailNumber
```

### Records Endpoint
```
GPS: gpsTime, lat, lon, utmX, utmY, alt, groundSpeed, heading, crossTrackError
Quality: lockedLine, hdop, satsIn, tslu, calcodeFreq
Application: flowRateApplied, flowRateRequired, appRateRequired, appRateApplied, swathWidth, boomPressure_psi, sprayStat
MET: windSpeed, windDir, temp, humidity
Session metadata: sprayOnLag_s, sprayOffLag_s, pulsesPerLitre
```

### Areas Endpoint
```
GeoJSON Feature properties:
name, appRate, area_ha, type
geometry: Polygon coordinates
```

---

## Troubleshooting

### ECONNREFUSED on port 3000
**Problem**: Tests fail with connection refused
**Solution**: Start the server first
```bash
npm run dev  # Terminal 1
npm test     # Terminal 2 (after server starts)
```

### Timeout errors
**Problem**: Tests timeout waiting for async export
**Solution**: Increase timeout or check if background workers are running
```bash
npm test -- --timeout 120000  # 2 minute timeout
```

### Field mismatch errors
**Problem**: Test says API field doesn't match database value
**Solution**: Check the actual vs. expected values in test output
- For numeric fields: tolerance is usually 0.01
- For string fields: must be exact match
- For null fields: check if field should exist

### sprayStat marker mismatch
**Problem**: Response row count or marker values differ from source AppDetail data
**Solution**: Verify endpoint/export queries are not applying sprayStat exclusion filters

---

## Performance Notes

- Each test suite takes ~1-2 minutes (waiting for async exports)
- Tests create isolated test data (no interference between runs)
- All cleanup is automatic (no manual database cleanup needed)
- Tests are safe to run repeatedly on production-like databases
- No modifications to existing data (read-only for queries, isolated test data for creation)

---

## Next Steps

1. **Run the tests**: Execute scripts to identify any issues
2. **Fix any failures**: Use error messages to locate incorrect data mappings
3. **Add more tests**: Extend with additional validation scenarios
4. **Integrate with CI/CD**: Add to your test pipeline (npm test)
5. **Monitor**: Keep tests passing as you modify endpoints

---

## Related Files

- Endpoints: [controllers/api_pub.js](../controllers/api_pub.js), [controllers/api_export.js](../controllers/api_export.js)
- Models: [model/application_detail.js](../model/application_detail.js), [model/export_job.js](../model/export_job.js)
- Routes: [routes/export.js](../routes/export.js), [routes/api_pub.js](../routes/api_pub.js)
- Design Doc: [docs/DATA_EXPORT_API_DESIGN.md](../docs/DATA_EXPORT_API_DESIGN.md)