agmission/Development/server/docs/archived/DATA_EXPORT_API_DOCUMENTATION_COMPLETE.md

📚 Data Export API — Complete Documentation Package

Executive Summary

Comprehensive documentation for the AgMission Data Export API has been created and all existing documentation has been updated. This package includes:

• ✅ Customer Integration Guide — Full API reference for external teams
• ✅ Rate Limiting & Deduplication Guide — 10+ detailed scenarios
• ✅ Documentation Index — Navigation hub for all audiences
• ✅ JSDoc API Comments — Ready for apidoc generation
• ✅ Updated Main Index — Cross-references to new docs

Total documentation created: 2,700+ lines across 4 new/updated files

---

📖 Documentation Files

1. DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md (PRIMARY ENTRY POINT)

For: Customers, integrators, BI teams, data warehouse engineers
Length: ~1,500 lines
Time to read: 30-45 minutes

Contains:

• Architecture overview with diagram
• Authentication & API key management
• Quick Start (3-minute setup)
• All 6 API Endpoints documented:
  • GET /api/v1/jobs/:jobId/sessions — Session summary
  • GET /api/v1/jobs/:jobId/sessions/:fileId/records — Paginated GPS trace (with cursor)
  • GET /api/v1/jobs/:jobId/areas — GeoJSON spray areas
  • POST /api/v1/jobs/:jobId/export — Trigger async export
  • GET /api/v1/exports/:exportId — Poll status
  • GET /api/v1/exports/:exportId/download — Stream file
• Complete parameter/response documentation
• 3 Real Use Cases with code:
  1. Power BI Incremental Refresh (Python)
  2. ArcGIS Map Automation (JavaScript)
  3. Data Warehouse Nightly Load (Bash)
• Error handling guide
• SLA commitments (99.5% uptime, 24h TTL)
• Support channels & response times
• Code examples (cURL, Python, JavaScript/Node.js)

Key differentiators:

• NOT a technical spec — written for business users
• Includes actual working code samples
• Real-world use cases from customer workflows
• Security best practices (API key rotation, TLS, env vars)

---

2. DATA_EXPORT_API_RATE_LIMITING.md (DETAILED REFERENCE)

For: Everyone (customers, engineers, sales)
Length: ~800 lines
Time to read: 20-30 minutes

Contains:

• Overview of 3 protection mechanisms:

  1. Per-account rate limiting (not IP-based)
  2. Request deduplication (reuse within time window)
  3. File TTL/lifecycle management

• Per-Account Rate Limiting section:

  • Configuration (20 req/60min default)
  • HTTP response format (429, rate-limit headers)
  • 5 Scenarios:
    • ✅ Within limit (multiple requests over time)
    • ❌ Rate limit exceeded (429 response)
    • ✅ Reuse ready export (cached, no wait)
    • ❌ Different params = new job
    • ✅ Reuse in-progress (within window)

• Request Deduplication section:

  • How it works (query logic explained)
  • Benefits (rate limit not consumed)
  • 3 Scenarios with outcomes

• File Lifecycle section:

  • TTL configuration (24 hours default)
  • Timeline (request → ready → download → delete)
  • Multi-download support
  • Auto-cleanup on expiry

• Best Practices:

  • Dedup-aware workflow patterns
  • Batch request optimization
  • Rate limit planning for 100-job exports
  • Graceful 429 error handling with backoff

• Monitoring & Troubleshooting:

  • Checking remaining rate limit quota
  • Detecting deduplicated requests
  • Unix timestamp conversion

• Reference section:

  • Pseudo-code for dedup query logic

Key differentiators:

• Each scenario shows request/response pairs
• Includes time-based progression
• Shows rate-limit headers for each example
• Covers both happy path and error cases

---

3. DATA_EXPORT_DOCUMENTATION_INDEX.md (NAVIGATION HUB)

For: Internal and external teams finding their way
Length: ~400 lines

Contains:

• For Different Audiences:

  • Customer Technical Teams (start with integration guide + rate limiting)
  • Internal Engineering (implementation, config, monitoring)
  • Sales & Account Managers (rate limit tiers, SLA, upgrade paths)

• Complete Documentation Map:

  • All 20+ export-related documents
  • One-sentence descriptions
  • Organized by purpose (API docs, implementation, architecture, operations)

• Quick Navigation by Task (8 scenarios):

  • "I'm integrating for the first time"
  • "I need to set up Power BI incremental refresh"
  • "I need to export data to ArcGIS"
  • "I need nightly bulk loads to data warehouse"
  • "I'm experiencing rate limit 429 errors"
  • "I'm debugging an export job failure"
  • "I need to understand the data model"
  • And more...

• Key Concepts (reference):

  • Authentication (API key format, NOT Bearer token!)
  • Rate limiting (per-account, 20/60min default)
  • Deduplication (same request within 5 mins)
  • File lifecycle (24-hour TTL)
  • Data units (metric vs US)

• Support & Escalation:

  • Issue types (doc issues, API questions, rate limit, bugs)
  • Contact info and response times
  • GitHub repo issues for docs

• Getting Started Checklist:

  • 8-step setup from first read to production

---

4. routes/api_pub.js (JSOC COMMENTS - FOR APIDOC)

For: API documentation generation
Lines added: 200+

Includes JSDoc for all 6 endpoints:

• @api — HTTP method and path
• @apiVersion — 1.0.0
• @apiName — Unique name
• @apiGroup — Endpoint grouping
• @apiDescription — Detailed explanation
• @apiParam — Path, query, body parameters
• @apiHeader — Required headers (X-API-Key, Content-Type)
• @apiSuccess — Success response structure
• @apiError — Error conditions
• @apiErrorExample — Example error responses
• @apiExample — cURL example commands
• @apiHeader — Response headers (RateLimit-*, Retry-After)

Endpoints documented:

1. GET /api/v1/jobs/:jobId/sessions
2. GET /api/v1/jobs/:jobId/sessions/:fileId/records
3. GET /api/v1/jobs/:jobId/areas
4. POST /api/v1/jobs/:jobId/export
5. GET /api/v1/exports/:exportId
6. GET /api/v1/exports/:exportId/download

Generated by: npm run docs → outputs to public/apidoc/

---

5. DOCUMENTATION_INDEX.md (UPDATED)

What changed:

• Added new "Data Export API" section after DLQ section
• 4 new doc links with descriptions
• Cross-references to related documentation

New section links:

• DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md ★
• DATA_EXPORT_API_RATE_LIMITING.md
• EXPORT_USAGE_DETAIL.md
• CURSOR_PAGINATION_GUIDE.md

---

6. DATA_EXPORT_DOCUMENTATION_UPDATES.md (SUMMARY)

Purpose: Document what was created and why
Contains:

• Overview of new documents
• Before/after improvements
• Content breakdown for each doc
• Usage metrics (2,700+ lines, 15+ examples)
• Quick navigation links
• Impact summary

---

🎯 Rate Limiting Examples

Example 1: Within Limit ✅

# Request 1 (14:00 UTC)
curl -X POST https://api.agmission.com/api/v1/jobs/12345/export \
  -H "X-API-Key: ak_test_..." \
  -d '{"format":"csv"}'

Response (202 Accepted):
{
  "exportId": "66f4a8c1...",
  "status": "pending"
}
Headers: RateLimit-Remaining: 19

# Request 2 (14:05 UTC) — still OK
Response: 202 Accepted, RateLimit-Remaining: 18

Example 2: Rate Limit Exceeded ❌

# Assume 20 requests already made in past 60 minutes

curl -X POST https://api.agmission.com/api/v1/jobs/12347/export \
  -H "X-API-Key: ak_test_..." \
  -d '{"format":"csv"}'

Response (429 Too Many Requests):
RateLimit-Remaining: 0
Retry-After: 1800  # Wait 30 minutes

{
  "error": "Export rate limit exceeded. Please wait before requesting another export."
}

Example 3: Deduplication (Reused Export) ✅

# Request 1 (14:00) — trigger
curl -X POST https://api.agmission.com/api/v1/jobs/12345/export \
  -H "X-API-Key: ak_test_..." \
  -d '{"format":"csv","units":"metric"}'

Response (202 Accepted): exportId: 66f4a8c1

# Request 2 (14:05, same params) — DEDUPLICATED
curl -X POST https://api.agmission.com/api/v1/jobs/12345/export \
  -H "X-API-Key: ak_test_..." \
  -d '{"format":"csv","units":"metric"}'

Response (200 OK — reused!):
{
  "exportId": "66f4a8c1",  # SAME ID
  "status": "ready",
  "reused": true           # Flag indicates dedup
}
RateLimit-Remaining: 19  # NOT consumed!

---

💡 Use Case Examples

Power BI Incremental Refresh

import requests
from datetime import datetime

def sync_to_powerbi(job_id, api_key):
    # Get sessions
    sessions = requests.get(
        f'https://api.agmission.com/api/v1/jobs/{job_id}/sessions',
        headers={'X-API-Key': api_key}
    ).json()
    
    for session in sessions['data']:
        file_id = session['sessionId']
        
        # Paginate records with cursor
        cursor = None
        while True:
            params = {'limit': 2000}
            if cursor:
                params['startingAfter'] = cursor
            
            page = requests.get(
                f'https://api.agmission.com/api/v1/jobs/{job_id}/sessions/{file_id}/records',
                params=params,
                headers={'X-API-Key': api_key}
            ).json()
            
            # Push to Power BI...
            
            if not page.get('hasMore'):
                break
            cursor = page.get('nextCursor')

ArcGIS Map Layer Update

const areas = await fetch(
  `https://api.agmission.com/api/v1/jobs/12345/areas`,
  { headers: { 'X-API-Key': apiKey } }
).then(r => r.json());

const features = areas.features.map(f => ({
  geometry: f.geometry,
  attributes: {
    name: f.properties.name,
    type: f.properties.type,
    area_ha: f.properties.area_ha
  }
}));

// Add to ArcGIS feature service...

Nightly Data Warehouse Load

#!/bin/bash
for job_id in 12345 12346 12347; do
  # Trigger export
  export_id=$(curl -s -X POST ".../jobs/${job_id}/export" \
    -H "X-API-Key: ${API_KEY}" \
    -d '{"format":"csv"}' | jq -r '.exportId')
  
  # Poll until ready...
  while [ "$(curl -s ".../exports/${export_id}?key=${API_KEY}" | jq -r '.status')" != "ready" ]; do
    sleep 5
  done
  
  # Download to S3
  curl -X GET ".../exports/${export_id}/download" \
    -H "X-API-Key: ${API_KEY}" \
    | aws s3 cp - "s3://bucket/spray_data/job${job_id}.csv"
done

---

🔑 Key Configuration Reference

Setting	Default	Location
Rate limit max	20	EXPORT_RATE_LIMIT_MAX env var
Rate limit window	60 min	EXPORT_RATE_LIMIT_WINDOW_MINS env var
Dedup window	5 min	EXPORT_DEDUP_MINS env var
File TTL	24 hours	EXPORT_TTL_HOURS env var
Uptime SLA	99.5% monthly	Customer agreement
Email support	4 hours	Business hours only
Phone support	1 hour	9am-5pm ET

---

📊 Documentation Statistics

Metric	Value
New files created	4
Existing files updated	2
Total lines written	2,700+
Code examples	15+
Scenarios documented	10+ (rate limiting + dedup)
API endpoints	6
Use cases	3 (with working code)
JSDoc lines	200+
Audience groups	3 (customers, engineers, sales)
Navigation paths	8 ("I need to..." tasks)
Quick start time	3 minutes
Full integration guide time	30-45 minutes

---

🚀 Getting Started

For Customers (First-time integration)

1. Read docs/DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md (30 min)
2. Get API key from https://agmission.agnav.com/api-keys
3. Test with quick start example (cURL)
4. Check docs/DATA_EXPORT_API_RATE_LIMITING.md for your use case
5. Implement retry logic for 429 responses
6. Go live!

For Internal Teams

1. Find your doc via docs/DATA_EXPORT_DOCUMENTATION_INDEX.md
2. For engineers: Check docs/APPLICATION_DETAIL_SCHEMA_CHANGES.md for config
3. For monitoring: See docs/MONITORING_GUIDE.md
4. For debugging: Enable via docs/DEBUG_CONFIGURATION_GUIDE.md

For Sales/Account Management

1. Reference docs/DATA_EXPORT_API_RATE_LIMITING.md scenarios
2. Explain limits to customers (20/60min default, upgradeable)
3. Point to SLA section for commitments
4. Discuss deduplication benefits

---

✅ Completeness Checklist

• ✅ Rate limiting fully documented (config, behavior, scenarios)
• ✅ Deduplication logic explained (query, benefits, examples)
• ✅ All 6 endpoints documented (parameters, responses, errors)
• ✅ Code examples for all use cases (Power BI, ArcGIS, data warehouse)
• ✅ Error handling guide (status codes, recovery)
• ✅ SLA commitments documented (uptime, TTL, support)
• ✅ Authentication guide (API key format, security)
• ✅ JSDoc for apidoc generation (200+ lines)
• ✅ Quick navigation index (8 task-based paths)
• ✅ Getting started checklist (8 steps)

---

📞 Support & Feedback

Questions about the API?
→ docs/DATA_EXPORT_CUSTOMER_INTEGRATION_GUIDE.md#support--slas

Need to understand rate limits?
→ docs/DATA_EXPORT_API_RATE_LIMITING.md

Looking for specific docs?
→ docs/DATA_EXPORT_DOCUMENTATION_INDEX.md

Found a doc issue?
→ GitHub issues (see docs/DATA_EXPORT_DOCUMENTATION_INDEX.md#support--escalation)

---

Last Updated: April 22, 2026
Audience: Customers, Engineers, Sales, Account Managers
Status: ✅ Complete and ready for production use