agmission/Development/server/scripts/MIGRATION_SUMMARY.md

Customer Data Migration - Implementation Summary

Overview

A comprehensive customer data migration solution has been implemented to support migrating data from multiple source customer master accounts to a single destination account.

Implementation Components

1. Main Migration Script

File: scripts/migrateCustomerData.js

Key Features:

• ✅ Multiple source customer support
• ✅ Robust conflict detection with abort-by-default behavior
• ✅ Detailed preview mode showing all changes
• ✅ Option to convert source customers to admin accounts (kind='2')
• ✅ Transaction safety using mongo_enhanced.js
• ✅ Complete migration history in JSON format
• ✅ Conflict merging capability (when explicitly requested)

Lines of Code: ~1,200 lines with comprehensive error handling

2. Documentation Files

README_CUSTOMER_MIGRATION.md

Complete reference guide covering:

• Detailed feature descriptions
• Usage examples
• Migration process explanation
• Safety features
• Troubleshooting guide
• Best practices

MIGRATION_QUICK_REFERENCE.md

Quick reference guide with:

• 6 common migration scenarios
• Decision tree for choosing approach
• Conflict resolution strategies
• Verification steps
• Validation queries
• Pre/post migration checklists

Technical Architecture

Database Collections Affected

Collection	Update Type	Field
Customer	Mark deleted	markedDelete, active, username
User (Sub-accounts)	Parent update or merge	parent
User (Clients)	Parent update or merge	parent
User (Pilots)	Parent update or merge	parent
User (Vehicles)	Parent update or merge	parent
User (Officers)	Parent update or merge	parent
User (Inspectors)	Parent update or merge	parent
Job	Reference update	byPuid, client, operator, vehicle
Product	Reference update	byPuid
Crop	Reference update	byPuid
CostingItem	Reference update	byPuid
Invoice	Reference update	customer
JobAssign	Account merge updates	pilot, vehicle
JobLog	Via job relationship	(follows job)

Transaction Management

Uses mongo_enhanced.js for:

• Automatic retry with exponential backoff
• Proper session lifecycle management
• Transaction abort on errors
• 60-second transaction timeout
• Robust error handling

Conflict Detection

Detects conflicts across:

• Client accounts (kind='3'): username conflicts
• Pilot accounts (kind='5'): username conflicts
• Vehicle accounts (kind='9'): username conflicts
• Officer accounts (kind='4'): username conflicts
• Inspector accounts (kind='6'): username conflicts

Default: Abort migration if any conflicts found Option: Use --merge-conflicts to force merge (marks old accounts as deleted)

Migration Process Flow

1. VALIDATION
   ├─ Verify all source customers exist
   ├─ Verify destination customer exists
   ├─ Check none are deleted
   └─ Gather statistics

2. CONFLICT DETECTION
   ├─ Check client username conflicts
   ├─ Check pilot username conflicts
   ├─ Check vehicle username conflicts
   └─ ABORT if conflicts found (unless --merge-conflicts)

3. PREVIEW
   ├─ Display source customer details
   ├─ Display destination customer details
   ├─ Show migration summary
   ├─ Show all conflicts
   └─ EXIT if --preview mode

4. EXECUTION (Single Transaction)
   ├─ Convert customers to admin (if --convert-to-admin)
   ├─ Migrate sub-accounts (merge if conflicts)
   ├─ Migrate entities (products, crops, costing items)
   ├─ Migrate jobs (all related data follows)
   ├─ Migrate invoices (if not --skip-invoices)
   └─ Deactivate source customers

5. HISTORY LOGGING
   ├─ Record all details
   ├─ Append to JSON file
   └─ Include success/failure status

Usage Examples

Basic Migration

node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea

Multiple Sources with Admin Conversion

node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011,507f1f77bcf86cd799439012 \
  --destination 507f191e810c19729de860ea \
  --convert-to-admin

Preview Mode (Always Recommended First)

node scripts/migrateCustomerData.js \
  --preview \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea

Safety Features

1. Transaction Atomicity

• All operations in single MongoDB transaction
• Automatic rollback on any error
• No partial migrations

2. Conflict Abort

• Default behavior: abort on conflicts
• Prevents accidental data loss
• Forces explicit decision with --merge-conflicts

3. Preview Mode

• See all changes before execution
• Review conflicts and statistics
• No database modifications

4. History Tracking

• Every migration logged to JSON
• Includes timestamp, options, stats, errors
• Complete audit trail

5. Validation

• Pre-flight checks on all customers
• Verify data integrity
• Check for deleted accounts

Migration History Format

{
  "timestamp": "2025-10-21T10:30:00.000Z",
  "sourceCustomerIds": ["..."],
  "targetCustomerId": "...",
  "options": { ... },
  "status": "completed|failed|aborted_due_to_conflicts|preview_completed",
  "completedAt": "2025-10-21T10:30:45.000Z",
  "stats": {
    "sourceCustomers": { ... },
    "totalJobs": 150,
    "totalClients": 25,
    "conflicts": [],
    "errors": []
  },
  "details": {
    "sources": [ ... ],
    "conflicts": [ ... ],
    "changes": [ ... ]
  }
}

Testing Recommendations

1. Test on Development Environment

Run full migration on dev/staging before production

2. Use Preview Mode

Always preview before executing

3. Test with Small Dataset First

Start with customer having minimal data

4. Verify with Queries

Run validation queries after migration (see Quick Reference)

5. Backup Before Migration

Ensure recent backup exists

Known Limitations

Not Automatically Migrated

• Stripe subscriptions: Require manual Stripe API calls
• Bill periods: Historical billing cycles
• Payment logs: Transaction history
• Authentication tokens: User sessions
• Email/notification preferences: User settings

Manual Steps Required

1. Stripe subscription transfer (if applicable)
2. Notify affected users of account changes
3. Update any external integrations
4. Verify custom reports/dashboards

Performance Considerations

Transaction Timeout

• Default: 60 seconds
• For large migrations: Consider batching
• Monitor migration_history.json for timing

Batch Size

• Recommend: 2-5 source customers per run
• Very large customers: Migrate individually
• Monitor database load

Database Load

• Migration runs in single transaction
• Locks affected documents
• Schedule during low-traffic periods

Rollback Strategy

Automatic Rollback

• Transaction automatically rolls back on error
• No partial state persists

Manual Rollback

If migration completed but needs reversal:

1. Restore from backup (RECOMMENDED)
2. Manual reversal using migration_history.json details
3. Use database administrator

Support and Debugging

Enable Debug Logging

DEBUG=agm:migrate-customer-data node scripts/migrateCustomerData.js ...

Full Debug Output

DEBUG=agm:* node scripts/migrateCustomerData.js ...

Review History

cat migration_history.json | jq '.[-1]'

Check Errors

cat migration_history.json | jq '.[-1].stats.errors'

Related Files

• scripts/migrateCustomerData.js - Main script
• README_CUSTOMER_MIGRATION.md - Full documentation
• MIGRATION_QUICK_REFERENCE.md - Quick reference guide
• migration_history.json - Migration history (created on first run)
• helpers/mongo_enhanced.js - Transaction utilities
• model/job.js - Job model with byPuid reference

Future Enhancements

Potential improvements:

• Web UI for migration management
• Email notifications on completion
• Automatic Stripe subscription handling
• Pre-migration validation report
• Post-migration verification report
• Scheduled migration support
• Migration rollback automation

Version History

• v1.0 (2025-10-21): Initial implementation
  • Multiple source support
  • Conflict detection and abort
  • Preview mode
  • Admin conversion option
  • History tracking
  • Transaction safety

Approval Checklist

Before using in production:

• Code review completed
• Tested on development environment
• Tested on staging environment
• Database backup procedure verified
• Rollback strategy documented
• Team trained on usage
• Documentation reviewed
• Edge cases tested

---

Status: ✅ Ready for Testing
Next Step: Test on development environment with sample data
Maintainer: Database Administration Team