agnav/agmission
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
df31b2080d
agmission/Development/server/docs/DATA_EXPORT_DOCUMENTATION_INDEX.md
Devin Major df31b2080d
All checks were successful
Server Tests / Mocha – Unit & Utility Tests (push) Successful in 42s
Details
-(#3013) Data Export - Implement Data Export API BE (Cont.) 
+ Added public data export API enhancements, tests, and customer documentation
  + Extended /api/v1 data export endpoints with richer session, records, area, and async export output
  + Added confirmed/fallback report values, client metadata, mapped area, over-spray, volume/apprate (string) units, and weather blocks
  + Normalized flowController to "No FC" and align record field names with playback output
  + Converted record wind speed output to knots, add Fligh Mater only record/export fields behind fm=true, and persist fm on export jobs
  + Added export status/area constants, HTTP 202 support, route-level API docs, and per-account export rate limiting support
  + Added comprehensive endpoint, format, and verification test coverage plus test-suite README
  + Added customer-facing data export design, integration, rate-limit, and documentation index guides
  + Updated README/DLQ docs and related documentation links to current HTTPS dashboard paths
2026-04-24 09:05:55 -04:00

236 lines
9.9 KiB
Markdown
Raw Blame History

# AgMission Data Export API — Documentation Index
Complete reference documentation for the Data Export API, including customer integration, implementation details, and operational guides.
---
## 📖 For Different Audiences
### 👥 Customer Technical Teams & Integrators
**Start here**: [DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md](DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md)
- Quick start with authentication
- All 6 API endpoints with examples
- Use cases (Power BI, ArcGIS, data warehousing)
- Error handling best practices
- Code examples (cURL, Python, JavaScript)
**For rate limiting questions**: [DATA_EXPORT_API_RATE_LIMITING.md](DATA_EXPORT_API_RATE_LIMITING.md)
- How per-account rate limiting works
- Request deduplication explained
- File lifecycle and TTL
- 10+ detailed scenarios with code
- Batch workflow optimization
---
### 🔨 Internal Engineering & DevOps
**API Implementation**: [docs/API_SPECIFICATION.md](API_SPECIFICATION.md)
- Data contracts
- Database schema
- Error codes and status mappings
**Configuration Reference**: [docs/APPLICATION_DETAIL_SCHEMA_CHANGES.md](APPLICATION_DETAIL_SCHEMA_CHANGES.md)
- Environment variables (EXPORT_TTL_HOURS, EXPORT_RATE_LIMIT_MAX, etc.)
- Database migrations
- Field mappings
**Operational Monitoring**: [docs/MONITORING_GUIDE.md](MONITORING_GUIDE.md)
- Health checks
- Performance metrics
- Alert thresholds
- Debug configuration
**Architecture & Design**:
- [ARCHITECTURE_SUMMARY.md](ARCHITECTURE_SUMMARY.md) — System design overview
- [DATABASE_DESIGN.md](DATABASE_DESIGN.md) — MongoDB schema, indexes, TTL
- [PARTNER_INTEGRATION_ARCHITECTURE.md](PARTNER_INTEGRATION_ARCHITECTURE.md) — Partner API integration
---
### 📊 Sales & Account Managers
**For rate limit discussions with customers**:
- Use [DATA_EXPORT_API_RATE_LIMITING.md](DATA_EXPORT_API_RATE_LIMITING.md) scenarios to explain limits
- Default: 20 exports/60 minutes (1 export every 3 minutes)
- Deduplication means identical requests don't consume quota
- TTL = 24 hours (configurable per enterprise agreement)
**For SLA / support discussions**:
- See [DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md](DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md#support--slas)
- 99.5% monthly uptime SLA
- 4-hour email support response, 1-hour phone support
- Data accuracy: ±0.5% for area/volume
---
## 📚 Complete Documentation Map
### 1. API Documentation
| Document | Purpose | Audience |
|---|---|---|
| [DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md](DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md) | Full API reference with examples | Customers, integrators |
| [DATA_EXPORT_API_RATE_LIMITING.md](DATA_EXPORT_API_RATE_LIMITING.md) | Rate limiting deep dive + scenarios | Everyone (10+ examples) |
| [API_SPECIFICATION.md](API_SPECIFICATION.md) | Data contracts, error codes | Engineering, integrators |
| [EXPORT_USAGE_DETAIL.md](EXPORT_USAGE_DETAIL.md) | CSV/GeoJSON field reference | Data analysts |
| [CURSOR_PAGINATION_GUIDE.md](CURSOR_PAGINATION_GUIDE.md) | Records pagination details | Power BI, data warehouse engineers |
### 2. Implementation Details
| Document | Purpose | Audience |
|---|---|---|
| [APPLICATION_DETAIL_SCHEMA_CHANGES.md](APPLICATION_DETAIL_SCHEMA_CHANGES.md) | Data model & env config | Engineering |
| [DATABASE_DESIGN.md](DATABASE_DESIGN.md) | MongoDB schema, indexes, TTL | DBA, backend engineers |
| [IMPLEMENTATION_GUIDE.md](IMPLEMENTATION_GUIDE.md) | Feature implementation checklist | Engineering leads |
### 3. Architecture & Design
| Document | Purpose | Audience |
|---|---|---|
| [ARCHITECTURE_SUMMARY.md](ARCHITECTURE_SUMMARY.md) | System design overview | Architects, senior engineers |
| [PARTNER_INTEGRATION_ARCHITECTURE.md](PARTNER_INTEGRATION_ARCHITECTURE.md) | Partner API integration | Integration engineers |
| [DLQ_ARCHITECTURE_DIAGRAMS.md](DLQ_ARCHITECTURE_DIAGRAMS.md) | Error handling flow (Mermaid diagrams) | Troubleshooting, monitoring |
### 4. Operational Guides
| Document | Purpose | Audience |
|---|---|---|
| [MONITORING_GUIDE.md](MONITORING_GUIDE.md) | Health checks, metrics, alerts | DevOps, SRE |
| [DEBUG_CONFIGURATION_GUIDE.md](DEBUG_CONFIGURATION_GUIDE.md) | Debug logging setup | Engineering |
| [PINO_MODULE_FILTERING_GUIDE.md](PINO_MODULE_FILTERING_GUIDE.md) | Logger module filtering | Debugging, troubleshooting |
| [FATAL_ERROR_HANDLING.md](FATAL_ERROR_HANDLING.md) | Crash handling, error reporting | DevOps, on-call engineers |
### 5. Data Format Reference
| Document | Purpose | Audience |
|---|---|---|
| [DATA_FORMAT_NOTES.md](DATA_FORMAT_NOTES.md) | Field types, units, nullable fields | Data analysts, integrators |
| [EXPORT_USAGE_DETAIL.md](EXPORT_USAGE_DETAIL.md) | CSV/GeoJSON column reference | BI engineers, data warehouse |
| [LOGFileFormat_Air_3_77_COMPLETE.md](LOGFileFormat_Air_3_77_COMPLETE.md) | Aircraft log file parsing | Log processing engineers |
---
## 🎯 Quick Navigation by Task
### "I'm integrating with AgMission API for the first time"
1. Read: [DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md](DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md) — Authentication, quick start, all endpoints
2. Copy: Code examples from Appendix (cURL, Python, JavaScript)
3. Reference: [DATA_EXPORT_API_RATE_LIMITING.md](DATA_EXPORT_API_RATE_LIMITING.md) for rate limit handling
### "I need to set up Power BI incremental refresh"
1. Use: [CURSOR_PAGINATION_GUIDE.md](CURSOR_PAGINATION_GUIDE.md) for cursor-based polling
2. Copy: Use Case #1 from [DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md](DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md#use-case-1-power-bi-incremental-refresh)
3. Handle: 429 responses per [DATA_EXPORT_API_RATE_LIMITING.md](DATA_EXPORT_API_RATE_LIMITING.md)
### "I need to export data to ArcGIS"
1. Use: [DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md](DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md#3-get-spray-areas) `/api/v1/jobs/:jobId/areas` endpoint
2. Copy: Use Case #2 code sample
3. Reference: [EXPORT_USAGE_DETAIL.md](EXPORT_USAGE_DETAIL.md) for GeoJSON field meanings
### "I need to do a nightly bulk load to our data warehouse"
1. Plan: [DATA_EXPORT_API_RATE_LIMITING.md](DATA_EXPORT_API_RATE_LIMITING.md#best-practices) (batch workflow optimization)
2. Implement: Use Case #3 shell script for nightly batch export
3. Handle: Deduplication + rate limits per [DATA_EXPORT_API_RATE_LIMITING.md](DATA_EXPORT_API_RATE_LIMITING.md)
### "I'm experiencing rate limit 429 errors"
1. Understand: [DATA_EXPORT_API_RATE_LIMITING.md](DATA_EXPORT_API_RATE_LIMITING.md#1-per-account-rate-limiting)
2. Check: Are you deduplicated? Look for `"reused": true` in response
3. Optimize: See Best Practices section
4. Request upgrade: Contact sales for higher rate limit tier
### "I'm debugging an export job failure"
1. Check: [API_SPECIFICATION.md](API_SPECIFICATION.md) error codes
2. Verify: Environment config in [APPLICATION_DETAIL_SCHEMA_CHANGES.md](APPLICATION_DETAIL_SCHEMA_CHANGES.md)
3. Log: Enable debug via [DEBUG_CONFIGURATION_GUIDE.md](DEBUG_CONFIGURATION_GUIDE.md)
4. Monitor: [MONITORING_GUIDE.md](MONITORING_GUIDE.md) for health checks
### "I need to understand the data model"
1. Start: [DATABASE_DESIGN.md](DATABASE_DESIGN.md) for MongoDB schema
2. Understand: [DATA_FORMAT_NOTES.md](DATA_FORMAT_NOTES.md) for field types/units
3. Reference: [APPLICATION_DETAIL_SCHEMA_CHANGES.md](APPLICATION_DETAIL_SCHEMA_CHANGES.md) for field mappings
---
## 🔑 Key Concepts
### Authentication
- API key format: `ak_test_xxx` (test) or `ak_live_xxx` (production)
- Header: `X-API-Key: <key>` (NOT `Authorization: Bearer`)
- Manage at: https://agmission.agnav.com/api-keys
### Rate Limiting
- **Per-account** (not IP-based)
- **Default**: 20 exports per 60 minutes (1 export every 3 minutes)
- **Deduplication**: Identical requests within 5 mins reuse existing export (no quota consumed)
- **Response headers**: `RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`, `Retry-After`
### File Lifecycle
- **TTL**: 24 hours (configurable via `EXPORT_TTL_HOURS`)
- **Persistence**: File kept on disk until TTL expires
- **Auto-cleanup**: Expired files deleted automatically
- **Download**: Can be downloaded multiple times before expiry
### Deduplication
- Identical requests within 5 mins (configurable) reuse existing export
- No rate limit consumed on reused exports
- Response includes `"reused": true` flag
- Includes: same job, format, units, interval, owner account
### Data Units
- **Metric** (default): `lit/ha`, `m/s`, `°C`, `kg`
- **US**: `gal/ac`, `mph`, `°F`, `lbs`
- Field examples: `appRateUnit: "lit/ha"`, `volumeUnit: "lit"`, `windSpeed_kt`
---
## 📞 Support & Escalation
### Documentation Issues
- Found a mistake or gap? File issue in GitHub repo
- Improvements welcome: Pull requests to `docs/` folder
### API Usage Questions
- Email: `technical-support@agnav.com`
- Response: 4 hours (business hours)
- Slack: Dedicated channel (enterprise customers)
### Rate Limit Exceptions
- Contact: Your account manager or `sales@agnav.com`
- Options: Increase rate limit tier, extend TTL, adjust dedup window
### Bug Reports / Incidents
- Severity 1 (outage): Phone 1-800-AGNAV-11
- Severity 2 (major issue): Email + phone escalation
- Severity 3 (minor): Email support
---
## 📋 Version History
| Version | Date | Changes |
|---|---|---|
| 1.0 | April 2026 | Initial release: Sessions, Records, Areas, Export endpoints; per-account rate limiting; request deduplication; file TTL |
---
## 🚀 Getting Started Checklist
- [ ] Read [DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md](DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md) (15 min)
- [ ] Get API key from https://agmission.agnav.com/api-keys
- [ ] Test `/api/v1/jobs/:jobId/sessions` endpoint with cURL
- [ ] Review [DATA_EXPORT_API_RATE_LIMITING.md](DATA_EXPORT_API_RATE_LIMITING.md) for your use case (5 min)
- [ ] Implement retry + backoff for 429 responses
- [ ] Design your polling/export workflow
- [ ] Load test against rate limits
- [ ] Enable monitoring/alerting for 429s
- [ ] Go live!
---
**Last Updated**: April 22, 2026
**Maintained By**: AgMission Technical Team
**Contact**: `technical-support@agnav.com`
Reference in New Issue View Git Blame Copy Permalink