agnav/agmission
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
release/3.2.1
agmission/Development/server/scripts/README_CUSTOMER_MIGRATION.md

17 KiB
Raw Permalink Blame History

Customer Data Migration Script

Overview

This script provides a comprehensive solution for migrating data from multiple source customer master accounts to a single destination customer account. It includes robust conflict detection, detailed preview capabilities, transaction safety, and complete migration history tracking.

Features

✅ Multiple Source Support - Migrate from multiple source customer accounts simultaneously
✅ Conflict Detection - Automatically detects username conflicts and aborts by default
✅ Detailed Preview - Visualizes all changes before execution
✅ Transaction Safety - Uses mongo_enhanced.js for robust transaction handling
✅ Admin Conversion - Optionally convert source customers to admin accounts
✅ Merge Capability - Can merge conflicting accounts when explicitly requested
✅ History Tracking - Maintains complete migration history in JSON format
✅ Comprehensive Migration - Handles all related data including:

  • Sub-accounts (clients, pilots, vehicles)
  • Jobs and applications
  • Products and crops
  • Invoices and billing data
  • Costing items

Usage

Preview Migration (Recommended First Step)

Always run a preview first to see what will be migrated and detect conflicts:

# Using customer usernames (recommended - easier to read)
node scripts/migrateCustomerData.js \
  --preview \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag

# Using customer IDs (also supported)
node scripts/migrateCustomerData.js \
  --preview \
  --sources SOURCE_CUSTOMER_ID1,SOURCE_CUSTOMER_ID2 \
  --destination DESTINATION_CUSTOMER_ID

# Mixed: usernames and IDs can be combined
node scripts/migrateCustomerData.js \
  --preview \
  --sources acme_aviation,507f1f77bcf86cd799439012 \
  --destination consolidated_ag

# With custom environment file (e.g., production)
node scripts/migrateCustomerData.js \
  --env ../environment_prod.env \
  --preview \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag

Execute Migration

After reviewing the preview, execute the migration:

# Using usernames (source customers converted to admin by default)
node scripts/migrateCustomerData.js \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag

# Using IDs
node scripts/migrateCustomerData.js \
  --sources SOURCE_CUSTOMER_ID1,SOURCE_CUSTOMER_ID2 \
  --destination DESTINATION_CUSTOMER_ID

# With entity reuse to avoid duplicates
node scripts/migrateCustomerData.js \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag \
  --reuse-existing-entities

# With production environment
node scripts/migrateCustomerData.js \
  --env ../environment_prod.env \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag \
  --reuse-existing-entities

Convert Source Customers to Admins

Source customers are converted to admin accounts by default. To disable this:

# Deactivate source customers instead of converting to admin
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag \
  --deactivate-source

# Explicit admin conversion (this is the default behavior)
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag \
  --convert-to-admin

Reuse Existing Entities

To avoid creating duplicate products/crops with the same names:

node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag \
  --reuse-existing-entities

This will:

  • Compare product/crop names between source and destination
  • Reuse destination entities when names match
  • Delete source entities that were matched
  • Only migrate entities with unique names

Command-Line Options

Option Description Required Default
--sources Comma-separated list of source customer IDs or usernames ✅ Yes -
--destination Destination customer ID or username ✅ Yes -
--preview Show migration plan without executing No false
--convert-to-admin Convert source customers to admin accounts No true
--deactivate-source Deactivate source customers instead of converting to admin No false
--reuse-existing-entities Reuse destination's products/crops with matching names No false
--skip-invoices Don't migrate invoice references No false
--output-file Custom output file path No ./migration_history.json
--env Custom environment file path No ../environment.env
--help, -h Show help message No -

Note: You can use either customer usernames (e.g., acme_aviation) or MongoDB ObjectIds (e.g., 507f1f77bcf86cd799439011). Usernames are recommended as they're easier to read and remember.

Migration Process

1. Validation Phase

The script first validates:

  • All source customer IDs exist
  • None of the source customers are already deleted
  • Destination customer exists and is active
  • Gathers statistics for all customers

2. Conflict Detection Phase

The script checks for username conflicts between:

  • Source clients vs destination clients
  • Source pilots vs destination pilots
  • Source vehicles vs destination vehicles

Default Behavior: If conflicts are found, the migration ABORTS immediately.

With --merge-conflicts: Conflicting accounts are merged:

  • References updated to point to existing accounts
  • Old accounts marked as deleted
  • Username changed to prevent future conflicts

3. Preview Phase

Shows a detailed breakdown of:

  • Source customer information and statistics
  • Destination customer information
  • All conflicts (if any)
  • Summary of data to be migrated
  • Actions that will be taken

4. Execution Phase (if not preview mode)

Within a single MongoDB transaction:

  1. Convert Customer Accounts (if --convert-to-admin)

    • Creates new admin user from customer data
    • Marks original customer as migrated

  2. Migrate Sub-Accounts

    • Clients, pilots, vehicles
    • Updates parent references or merges conflicts

  3. Migrate Entities

    • Products
    • Crops
    • Costing items

  4. Migrate Jobs

    • Updates byPuid references
    • All related data (logs, assignments, applications) automatically follow

  5. Migrate Billing Data

    • Updates invoice customer references

  6. Deactivate Source Customers (unless converted to admin)

    • Marks as deleted
    • Changes username to prevent conflicts

5. History Logging

All migrations are logged to a JSON file with:

  • Timestamp
  • Source and destination IDs
  • Complete statistics
  • Detailed changes list
  • Conflict information
  • Success/failure status
  • Error details (if failed)

Migration History File

The script maintains a complete history of all migrations in migration_history.json (or custom path via --output-file).

Structure

[
  {
    "timestamp": "2025-10-21T10:30:00.000Z",
    "sourceCustomerIds": ["507f1f77bcf86cd799439011"],
    "targetCustomerId": "507f191e810c19729de860ea",
    "options": {
      "preview": false,
      "convertToAdmin": false,
      "mergeConflicts": false,
      "updateInvoices": true
    },
    "status": "completed",
    "completedAt": "2025-10-21T10:30:45.000Z",
    "stats": {
      "sourceCustomers": { ... },
      "totalJobs": 150,
      "totalClients": 25,
      "totalPilots": 10,
      "totalVehicles": 8,
      "totalProducts": 45,
      "totalCrops": 12,
      "totalApplications": 890,
      "totalInvoices": 75,
      "conflicts": [],
      "skipped": [],
      "errors": []
    },
    "details": {
      "sources": [ ... ],
      "conflicts": [],
      "changes": [ ... ]
    }
  }
]

Examples

Example 1: Simple Migration (Single Source)

# Preview first (using username)
node scripts/migrateCustomerData.js \
  --preview \
  --sources acme_aviation \
  --destination consolidated_ag

# Execute after reviewing
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag

# Alternative: Using IDs
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea

Example 2: Multiple Sources with Admin Conversion

# Using usernames (recommended)
node scripts/migrateCustomerData.js \
  --sources acme_west,acme_east,acme_south \
  --destination acme_corporate \
  --convert-to-admin

# Using IDs
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011,507f1f77bcf86cd799439012,507f1f77bcf86cd799439013 \
  --destination 507f191e810c19729de860ea \
  --convert-to-admin

Example 3: Migration with Conflict Merging

# Preview to see conflicts
node scripts/migrateCustomerData.js \
  --preview \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea

# Execute with merge if conflicts are acceptable
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea \
  --merge-conflicts

Example 4: Custom Output File

node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea \
  --output-file /path/to/custom/migration_log.json

Preview Output Example

================================================================================
MIGRATION PREVIEW
================================================================================

Source Customers:
  • acme_aviation (507f1f77bcf86cd799439011)
    Email: contact@acme.com
    Country: US
    Clients: 25
    Pilots: 10
    Vehicles: 8
    Jobs: 150
    Products: 45
    Crops: 12
    Applications: 890
    Invoices: 75
    ⚠️  Will be DEACTIVATED after migration

Destination Customer:
  • consolidated_ag (507f191e810c19729de860ea)
    Email: admin@consolidated.com
    Country: US

Migration Summary:
  Total Clients: 25
  Total Pilots: 10
  Total Vehicles: 8
  Total Jobs: 150
  Total Products: 45
  Total Crops: 12
  Total Applications: 890
  Total Invoices: 75

Preview mode - no changes made

Preview saved to: ./migration_history.json

Conflict Output Example

When conflicts are detected:

⚠️  MIGRATION ABORTED - CONFLICTS DETECTED

The following conflicts were found:

From customer: acme_aviation
  • Client: john_farmer
    Source ID: 507f1f77bcf86cd799439015
    Conflicts with existing ID: 507f191e810c19729de86100
    Email: john@farm.com
  • Pilot: mike_pilot
    Source ID: 507f1f77bcf86cd799439016
    Conflicts with existing ID: 507f191e810c19729de86101
    Email: mike@pilots.com

To proceed with merging conflicts, use --merge-conflicts flag
WARNING: Merging will link old accounts to existing ones and mark old accounts as deleted

Safety Features

1. Transaction Rollback

If any error occurs during migration, the entire transaction is rolled back. No partial migrations.

2. Validation Before Execution

  • Verifies all customers exist
  • Checks for deleted customers
  • Validates data integrity

3. Conflict Detection

  • Detects username conflicts across all sub-account types
  • Aborts by default to prevent data loss

4. History Tracking

  • Every migration attempt is logged
  • Includes success/failure status
  • Complete audit trail

5. Preview Mode

  • Test migration without making changes
  • See exactly what will happen
  • Review conflicts before deciding

What Gets Migrated

✅ Migrated Collections

Collection Migration Method Notes
Clients Parent reference updated Merged if conflicts
Pilots Parent reference updated Merged if conflicts
Vehicles Parent reference updated Merged if conflicts
Jobs byPuid reference updated All related data follows
Applications Via job relationship No direct update needed
App Files Via application relationship No direct update needed
App Details Via app file relationship No direct update needed
Job Logs Via job relationship No direct update needed
Job Assigns Via job relationship Updated if account merge
Products byPuid reference updated -
Crops byPuid reference updated -
Costing Items byPuid reference updated -
Invoices Customer reference updated Optional via --skip-invoices

❌ Not Migrated

  • Stripe subscriptions (requires manual handling)
  • Bill periods
  • Payment logs
  • User authentication tokens
  • Session data

Troubleshooting

Issue: Transaction Timeout

Symptom: Migration fails with transaction timeout error

Solution: The script uses enhanced transaction handling with 60-second timeout. For very large migrations:

  • Ensure database connection is stable
  • Consider migrating in smaller batches (fewer source customers at once)

Issue: Conflicts Detected

Symptom: Migration aborted with conflict list

Solutions:

  1. Recommended: Manually resolve conflicts by renaming accounts in source or destination
  2. Use --merge-conflicts flag (USE WITH CAUTION)
  3. Migrate source customers without conflicting accounts first

Issue: Validation Failed

Symptom: "Source or target customer not found"

Solutions:

  • Verify customer IDs are correct
  • Ensure customers are not already deleted
  • Check database connection

Issue: Permission Denied

Symptom: MongoDB permission errors

Solutions:

  • Ensure database user has write permissions
  • Check that user can create transactions
  • Verify user has access to all collections

Best Practices

1. Always Preview First

# ALWAYS run preview before execution
node scripts/migrateCustomerData.js --preview --sources ... --destination ...

2. Backup Database

Before running any migration, ensure you have a recent database backup.

3. Test on Non-Production First

Test the migration process on a development or staging environment first.

4. Review Conflicts Carefully

If conflicts are detected, review them carefully. Consider manually resolving rather than using --merge-conflicts.

5. Monitor History File

Regularly review migration_history.json to track all migrations.

6. Plan Stripe Subscriptions

Customer subscriptions cannot be automatically migrated and require manual intervention through Stripe.

7. Use Custom Environment Files

For production migrations, use the --env flag to specify the production environment file:

# Production migration
node scripts/migrateCustomerData.js \
  --env ../environment_prod.env \
  --sources acme_aviation \
  --destination consolidated_ag \
  --preview

# Development/testing (default)
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag \
  --preview

Rollback Migrations

If you need to undo a migration, use the rollback script:

# Rollback the last migration (development environment)
node scripts/rollbackMigration.js

# Rollback with production environment
node scripts/rollbackMigration.js --env ../environment_prod.env

What Rollback Does

  • ✅ Restores all migrated entities (byPuid references)
  • ✅ Restores deleted products/crops (if entity reuse was used)
  • ✅ Restores invoice references
  • ✅ Restores sub-account parent references
  • ✅ Restores source customer from admin back to customer
  • ✅ Reactivates deactivated source customers

Rollback Limitations

  • ⚠️ Can only rollback the last completed migration
  • ⚠️ Must be run against the same database as the migration
  • ⚠️ Cannot rollback if database has been manually modified since migration
  • ⚠️ Check entities exist before restoring (handles already-rolled-back state)

Database Schema Impact

Updated Fields

  • User.parent - Updated for sub-accounts
  • Job.byPuid - Updated to destination customer
  • Product.byPuid - Updated to destination customer
  • Crop.byPuid - Updated to destination customer
  • CostingItem.byPuid - Updated to destination customer
  • Invoice.customer - Updated to destination customer
  • Job.client/operator/vehicle - Updated if account merged

Marked for Deletion

  • Customer.markedDelete - Source customers (unless converted to admin)
  • Customer.username - Suffixed with timestamp
  • User.markedDelete - Merged sub-accounts
  • User.username - Suffixed with timestamp

Support and Debugging

Enable debug output:

DEBUG=agm:migrate-customer-data node scripts/migrateCustomerData.js ...

For detailed transaction debugging:

DEBUG=agm:migrate-customer-data,agm:mongo_enhanced node scripts/migrateCustomerData.js ...

Related Files

  • scripts/migrateCustomerData.js - Main migration script
  • scripts/rollbackMigration.js - Rollback script for undoing migrations
  • helpers/mongo_enhanced.js - Transaction utilities
  • helpers/constants.js - User types and constants
  • model/job.js - Job model with byPuid reference
  • migration_history.json - Migration history log (tracks all migrations and rollbacks)

License

Internal use only - AgMission Customer Data Management