594 lines
17 KiB
Markdown
594 lines
17 KiB
Markdown
# Customer Data Migration Script
|
|
|
|
## Overview
|
|
|
|
This script provides a comprehensive solution for migrating data from multiple source customer master accounts to a single destination customer account. It includes robust conflict detection, detailed preview capabilities, transaction safety, and complete migration history tracking.
|
|
|
|
## Features
|
|
|
|
✅ **Multiple Source Support** - Migrate from multiple source customer accounts simultaneously
|
|
✅ **Conflict Detection** - Automatically detects username conflicts and aborts by default
|
|
✅ **Detailed Preview** - Visualizes all changes before execution
|
|
✅ **Transaction Safety** - Uses `mongo_enhanced.js` for robust transaction handling
|
|
✅ **Admin Conversion** - Optionally convert source customers to admin accounts
|
|
✅ **Merge Capability** - Can merge conflicting accounts when explicitly requested
|
|
✅ **History Tracking** - Maintains complete migration history in JSON format
|
|
✅ **Comprehensive Migration** - Handles all related data including:
|
|
- Sub-accounts (clients, pilots, vehicles)
|
|
- Jobs and applications
|
|
- Products and crops
|
|
- Invoices and billing data
|
|
- Costing items
|
|
|
|
## Usage
|
|
|
|
### Preview Migration (Recommended First Step)
|
|
|
|
Always run a preview first to see what will be migrated and detect conflicts:
|
|
|
|
```bash
|
|
# Using customer usernames (recommended - easier to read)
|
|
node scripts/migrateCustomerData.js \
|
|
--preview \
|
|
--sources acme_aviation,regional_ag \
|
|
--destination consolidated_ag
|
|
|
|
# Using customer IDs (also supported)
|
|
node scripts/migrateCustomerData.js \
|
|
--preview \
|
|
--sources SOURCE_CUSTOMER_ID1,SOURCE_CUSTOMER_ID2 \
|
|
--destination DESTINATION_CUSTOMER_ID
|
|
|
|
# Mixed: usernames and IDs can be combined
|
|
node scripts/migrateCustomerData.js \
|
|
--preview \
|
|
--sources acme_aviation,507f1f77bcf86cd799439012 \
|
|
--destination consolidated_ag
|
|
|
|
# With custom environment file (e.g., production)
|
|
node scripts/migrateCustomerData.js \
|
|
--env ../environment_prod.env \
|
|
--preview \
|
|
--sources acme_aviation,regional_ag \
|
|
--destination consolidated_ag
|
|
```
|
|
|
|
### Execute Migration
|
|
|
|
After reviewing the preview, execute the migration:
|
|
|
|
```bash
|
|
# Using usernames (source customers converted to admin by default)
|
|
node scripts/migrateCustomerData.js \
|
|
--sources acme_aviation,regional_ag \
|
|
--destination consolidated_ag
|
|
|
|
# Using IDs
|
|
node scripts/migrateCustomerData.js \
|
|
--sources SOURCE_CUSTOMER_ID1,SOURCE_CUSTOMER_ID2 \
|
|
--destination DESTINATION_CUSTOMER_ID
|
|
|
|
# With entity reuse to avoid duplicates
|
|
node scripts/migrateCustomerData.js \
|
|
--sources acme_aviation,regional_ag \
|
|
--destination consolidated_ag \
|
|
--reuse-existing-entities
|
|
|
|
# With production environment
|
|
node scripts/migrateCustomerData.js \
|
|
--env ../environment_prod.env \
|
|
--sources acme_aviation,regional_ag \
|
|
--destination consolidated_ag \
|
|
--reuse-existing-entities
|
|
```
|
|
|
|
### Convert Source Customers to Admins
|
|
|
|
Source customers are converted to admin accounts by default. To disable this:
|
|
|
|
```bash
|
|
# Deactivate source customers instead of converting to admin
|
|
node scripts/migrateCustomerData.js \
|
|
--sources acme_aviation \
|
|
--destination consolidated_ag \
|
|
--deactivate-source
|
|
|
|
# Explicit admin conversion (this is the default behavior)
|
|
node scripts/migrateCustomerData.js \
|
|
--sources acme_aviation \
|
|
--destination consolidated_ag \
|
|
--convert-to-admin
|
|
```
|
|
|
|
### Reuse Existing Entities
|
|
|
|
To avoid creating duplicate products/crops with the same names:
|
|
|
|
```bash
|
|
node scripts/migrateCustomerData.js \
|
|
--sources acme_aviation \
|
|
--destination consolidated_ag \
|
|
--reuse-existing-entities
|
|
```
|
|
|
|
This will:
|
|
- Compare product/crop names between source and destination
|
|
- Reuse destination entities when names match
|
|
- Delete source entities that were matched
|
|
- Only migrate entities with unique names
|
|
|
|
## Command-Line Options
|
|
|
|
| Option | Description | Required | Default |
|
|
|--------|-------------|----------|---------|
|
|
| `--sources` | Comma-separated list of source customer IDs or usernames | ✅ Yes | - |
|
|
| `--destination` | Destination customer ID or username | ✅ Yes | - |
|
|
| `--preview` | Show migration plan without executing | No | false |
|
|
| `--convert-to-admin` | Convert source customers to admin accounts | No | true |
|
|
| `--deactivate-source` | Deactivate source customers instead of converting to admin | No | false |
|
|
| `--reuse-existing-entities` | Reuse destination's products/crops with matching names | No | false |
|
|
| `--skip-invoices` | Don't migrate invoice references | No | false |
|
|
| `--output-file` | Custom output file path | No | `./migration_history.json` |
|
|
| `--env` | Custom environment file path | No | `../environment.env` |
|
|
| `--help`, `-h` | Show help message | No | - |
|
|
|
|
**Note**: You can use either customer **usernames** (e.g., `acme_aviation`) or **MongoDB ObjectIds** (e.g., `507f1f77bcf86cd799439011`). Usernames are recommended as they're easier to read and remember.
|
|
|
|
## Migration Process
|
|
|
|
### 1. Validation Phase
|
|
|
|
The script first validates:
|
|
- All source customer IDs exist
|
|
- None of the source customers are already deleted
|
|
- Destination customer exists and is active
|
|
- Gathers statistics for all customers
|
|
|
|
### 2. Conflict Detection Phase
|
|
|
|
The script checks for username conflicts between:
|
|
- Source clients vs destination clients
|
|
- Source pilots vs destination pilots
|
|
- Source vehicles vs destination vehicles
|
|
|
|
**Default Behavior**: If conflicts are found, the migration **ABORTS** immediately.
|
|
|
|
**With `--merge-conflicts`**: Conflicting accounts are merged:
|
|
- References updated to point to existing accounts
|
|
- Old accounts marked as deleted
|
|
- Username changed to prevent future conflicts
|
|
|
|
### 3. Preview Phase
|
|
|
|
Shows a detailed breakdown of:
|
|
- Source customer information and statistics
|
|
- Destination customer information
|
|
- All conflicts (if any)
|
|
- Summary of data to be migrated
|
|
- Actions that will be taken
|
|
|
|
### 4. Execution Phase (if not preview mode)
|
|
|
|
Within a single MongoDB transaction:
|
|
|
|
1. **Convert Customer Accounts** (if `--convert-to-admin`)
|
|
- Creates new admin user from customer data
|
|
- Marks original customer as migrated
|
|
|
|
2. **Migrate Sub-Accounts**
|
|
- Clients, pilots, vehicles
|
|
- Updates parent references or merges conflicts
|
|
|
|
3. **Migrate Entities**
|
|
- Products
|
|
- Crops
|
|
- Costing items
|
|
|
|
4. **Migrate Jobs**
|
|
- Updates `byPuid` references
|
|
- All related data (logs, assignments, applications) automatically follow
|
|
|
|
5. **Migrate Billing Data**
|
|
- Updates invoice customer references
|
|
|
|
6. **Deactivate Source Customers** (unless converted to admin)
|
|
- Marks as deleted
|
|
- Changes username to prevent conflicts
|
|
|
|
### 5. History Logging
|
|
|
|
All migrations are logged to a JSON file with:
|
|
- Timestamp
|
|
- Source and destination IDs
|
|
- Complete statistics
|
|
- Detailed changes list
|
|
- Conflict information
|
|
- Success/failure status
|
|
- Error details (if failed)
|
|
|
|
## Migration History File
|
|
|
|
The script maintains a complete history of all migrations in `migration_history.json` (or custom path via `--output-file`).
|
|
|
|
### Structure
|
|
|
|
```json
|
|
[
|
|
{
|
|
"timestamp": "2025-10-21T10:30:00.000Z",
|
|
"sourceCustomerIds": ["507f1f77bcf86cd799439011"],
|
|
"targetCustomerId": "507f191e810c19729de860ea",
|
|
"options": {
|
|
"preview": false,
|
|
"convertToAdmin": false,
|
|
"mergeConflicts": false,
|
|
"updateInvoices": true
|
|
},
|
|
"status": "completed",
|
|
"completedAt": "2025-10-21T10:30:45.000Z",
|
|
"stats": {
|
|
"sourceCustomers": { ... },
|
|
"totalJobs": 150,
|
|
"totalClients": 25,
|
|
"totalPilots": 10,
|
|
"totalVehicles": 8,
|
|
"totalProducts": 45,
|
|
"totalCrops": 12,
|
|
"totalApplications": 890,
|
|
"totalInvoices": 75,
|
|
"conflicts": [],
|
|
"skipped": [],
|
|
"errors": []
|
|
},
|
|
"details": {
|
|
"sources": [ ... ],
|
|
"conflicts": [],
|
|
"changes": [ ... ]
|
|
}
|
|
}
|
|
]
|
|
```
|
|
|
|
## Examples
|
|
|
|
### Example 1: Simple Migration (Single Source)
|
|
|
|
```bash
|
|
# Preview first (using username)
|
|
node scripts/migrateCustomerData.js \
|
|
--preview \
|
|
--sources acme_aviation \
|
|
--destination consolidated_ag
|
|
|
|
# Execute after reviewing
|
|
node scripts/migrateCustomerData.js \
|
|
--sources acme_aviation \
|
|
--destination consolidated_ag
|
|
|
|
# Alternative: Using IDs
|
|
node scripts/migrateCustomerData.js \
|
|
--sources 507f1f77bcf86cd799439011 \
|
|
--destination 507f191e810c19729de860ea
|
|
```
|
|
|
|
### Example 2: Multiple Sources with Admin Conversion
|
|
|
|
```bash
|
|
# Using usernames (recommended)
|
|
node scripts/migrateCustomerData.js \
|
|
--sources acme_west,acme_east,acme_south \
|
|
--destination acme_corporate \
|
|
--convert-to-admin
|
|
|
|
# Using IDs
|
|
node scripts/migrateCustomerData.js \
|
|
--sources 507f1f77bcf86cd799439011,507f1f77bcf86cd799439012,507f1f77bcf86cd799439013 \
|
|
--destination 507f191e810c19729de860ea \
|
|
--convert-to-admin
|
|
```
|
|
|
|
### Example 3: Migration with Conflict Merging
|
|
|
|
```bash
|
|
# Preview to see conflicts
|
|
node scripts/migrateCustomerData.js \
|
|
--preview \
|
|
--sources 507f1f77bcf86cd799439011 \
|
|
--destination 507f191e810c19729de860ea
|
|
|
|
# Execute with merge if conflicts are acceptable
|
|
node scripts/migrateCustomerData.js \
|
|
--sources 507f1f77bcf86cd799439011 \
|
|
--destination 507f191e810c19729de860ea \
|
|
--merge-conflicts
|
|
```
|
|
|
|
### Example 4: Custom Output File
|
|
|
|
```bash
|
|
node scripts/migrateCustomerData.js \
|
|
--sources 507f1f77bcf86cd799439011 \
|
|
--destination 507f191e810c19729de860ea \
|
|
--output-file /path/to/custom/migration_log.json
|
|
```
|
|
|
|
## Preview Output Example
|
|
|
|
```
|
|
================================================================================
|
|
MIGRATION PREVIEW
|
|
================================================================================
|
|
|
|
Source Customers:
|
|
• acme_aviation (507f1f77bcf86cd799439011)
|
|
Email: contact@acme.com
|
|
Country: US
|
|
Clients: 25
|
|
Pilots: 10
|
|
Vehicles: 8
|
|
Jobs: 150
|
|
Products: 45
|
|
Crops: 12
|
|
Applications: 890
|
|
Invoices: 75
|
|
⚠️ Will be DEACTIVATED after migration
|
|
|
|
Destination Customer:
|
|
• consolidated_ag (507f191e810c19729de860ea)
|
|
Email: admin@consolidated.com
|
|
Country: US
|
|
|
|
Migration Summary:
|
|
Total Clients: 25
|
|
Total Pilots: 10
|
|
Total Vehicles: 8
|
|
Total Jobs: 150
|
|
Total Products: 45
|
|
Total Crops: 12
|
|
Total Applications: 890
|
|
Total Invoices: 75
|
|
|
|
Preview mode - no changes made
|
|
|
|
Preview saved to: ./migration_history.json
|
|
```
|
|
|
|
## Conflict Output Example
|
|
|
|
When conflicts are detected:
|
|
|
|
```
|
|
⚠️ MIGRATION ABORTED - CONFLICTS DETECTED
|
|
|
|
The following conflicts were found:
|
|
|
|
From customer: acme_aviation
|
|
• Client: john_farmer
|
|
Source ID: 507f1f77bcf86cd799439015
|
|
Conflicts with existing ID: 507f191e810c19729de86100
|
|
Email: john@farm.com
|
|
• Pilot: mike_pilot
|
|
Source ID: 507f1f77bcf86cd799439016
|
|
Conflicts with existing ID: 507f191e810c19729de86101
|
|
Email: mike@pilots.com
|
|
|
|
To proceed with merging conflicts, use --merge-conflicts flag
|
|
WARNING: Merging will link old accounts to existing ones and mark old accounts as deleted
|
|
```
|
|
|
|
## Safety Features
|
|
|
|
### 1. Transaction Rollback
|
|
|
|
If any error occurs during migration, the **entire transaction is rolled back**. No partial migrations.
|
|
|
|
### 2. Validation Before Execution
|
|
|
|
- Verifies all customers exist
|
|
- Checks for deleted customers
|
|
- Validates data integrity
|
|
|
|
### 3. Conflict Detection
|
|
|
|
- Detects username conflicts across all sub-account types
|
|
- Aborts by default to prevent data loss
|
|
|
|
### 4. History Tracking
|
|
|
|
- Every migration attempt is logged
|
|
- Includes success/failure status
|
|
- Complete audit trail
|
|
|
|
### 5. Preview Mode
|
|
|
|
- Test migration without making changes
|
|
- See exactly what will happen
|
|
- Review conflicts before deciding
|
|
|
|
## What Gets Migrated
|
|
|
|
### ✅ Migrated Collections
|
|
|
|
| Collection | Migration Method | Notes |
|
|
|------------|------------------|-------|
|
|
| **Clients** | Parent reference updated | Merged if conflicts |
|
|
| **Pilots** | Parent reference updated | Merged if conflicts |
|
|
| **Vehicles** | Parent reference updated | Merged if conflicts |
|
|
| **Jobs** | `byPuid` reference updated | All related data follows |
|
|
| **Applications** | Via job relationship | No direct update needed |
|
|
| **App Files** | Via application relationship | No direct update needed |
|
|
| **App Details** | Via app file relationship | No direct update needed |
|
|
| **Job Logs** | Via job relationship | No direct update needed |
|
|
| **Job Assigns** | Via job relationship | Updated if account merge |
|
|
| **Products** | `byPuid` reference updated | - |
|
|
| **Crops** | `byPuid` reference updated | - |
|
|
| **Costing Items** | `byPuid` reference updated | - |
|
|
| **Invoices** | Customer reference updated | Optional via `--skip-invoices` |
|
|
|
|
### ❌ Not Migrated
|
|
|
|
- Stripe subscriptions (requires manual handling)
|
|
- Bill periods
|
|
- Payment logs
|
|
- User authentication tokens
|
|
- Session data
|
|
|
|
## Troubleshooting
|
|
|
|
### Issue: Transaction Timeout
|
|
|
|
**Symptom**: Migration fails with transaction timeout error
|
|
|
|
**Solution**: The script uses enhanced transaction handling with 60-second timeout. For very large migrations:
|
|
- Ensure database connection is stable
|
|
- Consider migrating in smaller batches (fewer source customers at once)
|
|
|
|
### Issue: Conflicts Detected
|
|
|
|
**Symptom**: Migration aborted with conflict list
|
|
|
|
**Solutions**:
|
|
1. **Recommended**: Manually resolve conflicts by renaming accounts in source or destination
|
|
2. Use `--merge-conflicts` flag (USE WITH CAUTION)
|
|
3. Migrate source customers without conflicting accounts first
|
|
|
|
### Issue: Validation Failed
|
|
|
|
**Symptom**: "Source or target customer not found"
|
|
|
|
**Solutions**:
|
|
- Verify customer IDs are correct
|
|
- Ensure customers are not already deleted
|
|
- Check database connection
|
|
|
|
### Issue: Permission Denied
|
|
|
|
**Symptom**: MongoDB permission errors
|
|
|
|
**Solutions**:
|
|
- Ensure database user has write permissions
|
|
- Check that user can create transactions
|
|
- Verify user has access to all collections
|
|
|
|
## Best Practices
|
|
|
|
### 1. Always Preview First
|
|
|
|
```bash
|
|
# ALWAYS run preview before execution
|
|
node scripts/migrateCustomerData.js --preview --sources ... --destination ...
|
|
```
|
|
|
|
### 2. Backup Database
|
|
|
|
Before running any migration, ensure you have a recent database backup.
|
|
|
|
### 3. Test on Non-Production First
|
|
|
|
Test the migration process on a development or staging environment first.
|
|
|
|
### 4. Review Conflicts Carefully
|
|
|
|
If conflicts are detected, review them carefully. Consider manually resolving rather than using `--merge-conflicts`.
|
|
|
|
### 5. Monitor History File
|
|
|
|
Regularly review `migration_history.json` to track all migrations.
|
|
|
|
### 6. Plan Stripe Subscriptions
|
|
|
|
Customer subscriptions cannot be automatically migrated and require manual intervention through Stripe.
|
|
|
|
### 7. Use Custom Environment Files
|
|
|
|
For production migrations, use the `--env` flag to specify the production environment file:
|
|
|
|
```bash
|
|
# Production migration
|
|
node scripts/migrateCustomerData.js \
|
|
--env ../environment_prod.env \
|
|
--sources acme_aviation \
|
|
--destination consolidated_ag \
|
|
--preview
|
|
|
|
# Development/testing (default)
|
|
node scripts/migrateCustomerData.js \
|
|
--sources acme_aviation \
|
|
--destination consolidated_ag \
|
|
--preview
|
|
```
|
|
|
|
## Rollback Migrations
|
|
|
|
If you need to undo a migration, use the rollback script:
|
|
|
|
```bash
|
|
# Rollback the last migration (development environment)
|
|
node scripts/rollbackMigration.js
|
|
|
|
# Rollback with production environment
|
|
node scripts/rollbackMigration.js --env ../environment_prod.env
|
|
```
|
|
|
|
### What Rollback Does
|
|
|
|
- ✅ Restores all migrated entities (`byPuid` references)
|
|
- ✅ Restores deleted products/crops (if entity reuse was used)
|
|
- ✅ Restores invoice references
|
|
- ✅ Restores sub-account parent references
|
|
- ✅ Restores source customer from admin back to customer
|
|
- ✅ Reactivates deactivated source customers
|
|
|
|
### Rollback Limitations
|
|
|
|
- ⚠️ Can only rollback the **last** completed migration
|
|
- ⚠️ Must be run against the same database as the migration
|
|
- ⚠️ Cannot rollback if database has been manually modified since migration
|
|
- ⚠️ Check entities exist before restoring (handles already-rolled-back state)
|
|
|
|
## Database Schema Impact
|
|
|
|
### Updated Fields
|
|
|
|
- `User.parent` - Updated for sub-accounts
|
|
- `Job.byPuid` - Updated to destination customer
|
|
- `Product.byPuid` - Updated to destination customer
|
|
- `Crop.byPuid` - Updated to destination customer
|
|
- `CostingItem.byPuid` - Updated to destination customer
|
|
- `Invoice.customer` - Updated to destination customer
|
|
- `Job.client/operator/vehicle` - Updated if account merged
|
|
|
|
### Marked for Deletion
|
|
|
|
- `Customer.markedDelete` - Source customers (unless converted to admin)
|
|
- `Customer.username` - Suffixed with timestamp
|
|
- `User.markedDelete` - Merged sub-accounts
|
|
- `User.username` - Suffixed with timestamp
|
|
|
|
## Support and Debugging
|
|
|
|
Enable debug output:
|
|
|
|
```bash
|
|
DEBUG=agm:migrate-customer-data node scripts/migrateCustomerData.js ...
|
|
```
|
|
|
|
For detailed transaction debugging:
|
|
|
|
```bash
|
|
DEBUG=agm:migrate-customer-data,agm:mongo_enhanced node scripts/migrateCustomerData.js ...
|
|
```
|
|
|
|
## Related Files
|
|
|
|
- `scripts/migrateCustomerData.js` - Main migration script
|
|
- `scripts/rollbackMigration.js` - Rollback script for undoing migrations
|
|
- `helpers/mongo_enhanced.js` - Transaction utilities
|
|
- `helpers/constants.js` - User types and constants
|
|
- `model/job.js` - Job model with `byPuid` reference
|
|
- `migration_history.json` - Migration history log (tracks all migrations and rollbacks)
|
|
|
|
## License
|
|
|
|
Internal use only - AgMission Customer Data Management
|