# Customer Data Migration Script

## Overview

This script provides a comprehensive solution for migrating data from multiple source customer master accounts to a single destination customer account. It includes robust conflict detection, detailed preview capabilities, transaction safety, and complete migration history tracking.

## Features

✅ **Multiple Source Support** - Migrate from multiple source customer accounts simultaneously  
✅ **Conflict Detection** - Automatically detects username conflicts and aborts by default  
✅ **Detailed Preview** - Visualizes all changes before execution  
✅ **Transaction Safety** - Uses `mongo_enhanced.js` for robust transaction handling  
✅ **Admin Conversion** - Optionally convert source customers to admin accounts  
✅ **Merge Capability** - Can merge conflicting accounts when explicitly requested  
✅ **History Tracking** - Maintains complete migration history in JSON format  
✅ **Comprehensive Migration** - Handles all related data including:
  - Sub-accounts (clients, pilots, vehicles)
  - Jobs and applications
  - Products and crops
  - Invoices and billing data
  - Costing items

## Usage

### Preview Migration (Recommended First Step)

Always run a preview first to see what will be migrated and detect conflicts:

```bash
# Using customer usernames (recommended - easier to read)
node scripts/migrateCustomerData.js \
  --preview \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag

# Using customer IDs (also supported)
node scripts/migrateCustomerData.js \
  --preview \
  --sources SOURCE_CUSTOMER_ID1,SOURCE_CUSTOMER_ID2 \
  --destination DESTINATION_CUSTOMER_ID

# Mixed: usernames and IDs can be combined
node scripts/migrateCustomerData.js \
  --preview \
  --sources acme_aviation,507f1f77bcf86cd799439012 \
  --destination consolidated_ag

# With custom environment file (e.g., production)
node scripts/migrateCustomerData.js \
  --env ../environment_prod.env \
  --preview \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag
```

### Execute Migration

After reviewing the preview, execute the migration:

```bash
# Using usernames (source customers converted to admin by default)
node scripts/migrateCustomerData.js \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag

# Using IDs
node scripts/migrateCustomerData.js \
  --sources SOURCE_CUSTOMER_ID1,SOURCE_CUSTOMER_ID2 \
  --destination DESTINATION_CUSTOMER_ID

# With entity reuse to avoid duplicates
node scripts/migrateCustomerData.js \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag \
  --reuse-existing-entities

# With production environment
node scripts/migrateCustomerData.js \
  --env ../environment_prod.env \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag \
  --reuse-existing-entities
```

### Convert Source Customers to Admins

Source customers are converted to admin accounts by default. To disable this:

```bash
# Deactivate source customers instead of converting to admin
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag \
  --deactivate-source

# Explicit admin conversion (this is the default behavior)
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag \
  --convert-to-admin
```

### Reuse Existing Entities

To avoid creating duplicate products/crops with the same names:

```bash
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag \
  --reuse-existing-entities
```

This will:
- Compare product/crop names between source and destination
- Reuse destination entities when names match
- Delete source entities that were matched
- Only migrate entities with unique names

## Command-Line Options

| Option | Description | Required | Default |
|--------|-------------|----------|---------|
| `--sources` | Comma-separated list of source customer IDs or usernames | ✅ Yes | - |
| `--destination` | Destination customer ID or username | ✅ Yes | - |
| `--preview` | Show migration plan without executing | No | false |
| `--convert-to-admin` | Convert source customers to admin accounts | No | true |
| `--deactivate-source` | Deactivate source customers instead of converting to admin | No | false |
| `--reuse-existing-entities` | Reuse destination's products/crops with matching names | No | false |
| `--skip-invoices` | Don't migrate invoice references | No | false |
| `--output-file` | Custom output file path | No | `./migration_history.json` |
| `--env` | Custom environment file path | No | `../environment.env` |
| `--help`, `-h` | Show help message | No | - |

**Note**: You can use either customer **usernames** (e.g., `acme_aviation`) or **MongoDB ObjectIds** (e.g., `507f1f77bcf86cd799439011`). Usernames are recommended as they're easier to read and remember.

## Migration Process

### 1. Validation Phase

The script first validates:
- All source customer IDs exist
- None of the source customers are already deleted
- Destination customer exists and is active
- Gathers statistics for all customers

### 2. Conflict Detection Phase

The script checks for username conflicts between:
- Source clients vs destination clients
- Source pilots vs destination pilots  
- Source vehicles vs destination vehicles

**Default Behavior**: If conflicts are found, the migration **ABORTS** immediately.

**With `--merge-conflicts`**: Conflicting accounts are merged:
- References updated to point to existing accounts
- Old accounts marked as deleted
- Username changed to prevent future conflicts

### 3. Preview Phase

Shows a detailed breakdown of:
- Source customer information and statistics
- Destination customer information
- All conflicts (if any)
- Summary of data to be migrated
- Actions that will be taken

### 4. Execution Phase (if not preview mode)

Within a single MongoDB transaction:

1. **Convert Customer Accounts** (if `--convert-to-admin`)
   - Creates new admin user from customer data
   - Marks original customer as migrated

2. **Migrate Sub-Accounts**
   - Clients, pilots, vehicles
   - Updates parent references or merges conflicts

3. **Migrate Entities**
   - Products
   - Crops
   - Costing items

4. **Migrate Jobs**
   - Updates `byPuid` references
   - All related data (logs, assignments, applications) automatically follow

5. **Migrate Billing Data**
   - Updates invoice customer references

6. **Deactivate Source Customers** (unless converted to admin)
   - Marks as deleted
   - Changes username to prevent conflicts

### 5. History Logging

All migrations are logged to a JSON file with:
- Timestamp
- Source and destination IDs
- Complete statistics
- Detailed changes list
- Conflict information
- Success/failure status
- Error details (if failed)

## Migration History File

The script maintains a complete history of all migrations in `migration_history.json` (or custom path via `--output-file`).

### Structure

```json
[
  {
    "timestamp": "2025-10-21T10:30:00.000Z",
    "sourceCustomerIds": ["507f1f77bcf86cd799439011"],
    "targetCustomerId": "507f191e810c19729de860ea",
    "options": {
      "preview": false,
      "convertToAdmin": false,
      "mergeConflicts": false,
      "updateInvoices": true
    },
    "status": "completed",
    "completedAt": "2025-10-21T10:30:45.000Z",
    "stats": {
      "sourceCustomers": { ... },
      "totalJobs": 150,
      "totalClients": 25,
      "totalPilots": 10,
      "totalVehicles": 8,
      "totalProducts": 45,
      "totalCrops": 12,
      "totalApplications": 890,
      "totalInvoices": 75,
      "conflicts": [],
      "skipped": [],
      "errors": []
    },
    "details": {
      "sources": [ ... ],
      "conflicts": [],
      "changes": [ ... ]
    }
  }
]
```

## Examples

### Example 1: Simple Migration (Single Source)

```bash
# Preview first (using username)
node scripts/migrateCustomerData.js \
  --preview \
  --sources acme_aviation \
  --destination consolidated_ag

# Execute after reviewing
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag

# Alternative: Using IDs
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea
```

### Example 2: Multiple Sources with Admin Conversion

```bash
# Using usernames (recommended)
node scripts/migrateCustomerData.js \
  --sources acme_west,acme_east,acme_south \
  --destination acme_corporate \
  --convert-to-admin

# Using IDs
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011,507f1f77bcf86cd799439012,507f1f77bcf86cd799439013 \
  --destination 507f191e810c19729de860ea \
  --convert-to-admin
```

### Example 3: Migration with Conflict Merging

```bash
# Preview to see conflicts
node scripts/migrateCustomerData.js \
  --preview \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea

# Execute with merge if conflicts are acceptable
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea \
  --merge-conflicts
```

### Example 4: Custom Output File

```bash
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea \
  --output-file /path/to/custom/migration_log.json
```

## Preview Output Example

```
================================================================================
MIGRATION PREVIEW
================================================================================

Source Customers:
  • acme_aviation (507f1f77bcf86cd799439011)
    Email: contact@acme.com
    Country: US
    Clients: 25
    Pilots: 10
    Vehicles: 8
    Jobs: 150
    Products: 45
    Crops: 12
    Applications: 890
    Invoices: 75
    ⚠️  Will be DEACTIVATED after migration

Destination Customer:
  • consolidated_ag (507f191e810c19729de860ea)
    Email: admin@consolidated.com
    Country: US

Migration Summary:
  Total Clients: 25
  Total Pilots: 10
  Total Vehicles: 8
  Total Jobs: 150
  Total Products: 45
  Total Crops: 12
  Total Applications: 890
  Total Invoices: 75

Preview mode - no changes made

Preview saved to: ./migration_history.json
```

## Conflict Output Example

When conflicts are detected:

```
⚠️  MIGRATION ABORTED - CONFLICTS DETECTED

The following conflicts were found:

From customer: acme_aviation
  • Client: john_farmer
    Source ID: 507f1f77bcf86cd799439015
    Conflicts with existing ID: 507f191e810c19729de86100
    Email: john@farm.com
  • Pilot: mike_pilot
    Source ID: 507f1f77bcf86cd799439016
    Conflicts with existing ID: 507f191e810c19729de86101
    Email: mike@pilots.com

To proceed with merging conflicts, use --merge-conflicts flag
WARNING: Merging will link old accounts to existing ones and mark old accounts as deleted
```

## Safety Features

### 1. Transaction Rollback

If any error occurs during migration, the **entire transaction is rolled back**. No partial migrations.

### 2. Validation Before Execution

- Verifies all customers exist
- Checks for deleted customers
- Validates data integrity

### 3. Conflict Detection

- Detects username conflicts across all sub-account types
- Aborts by default to prevent data loss

### 4. History Tracking

- Every migration attempt is logged
- Includes success/failure status
- Complete audit trail

### 5. Preview Mode

- Test migration without making changes
- See exactly what will happen
- Review conflicts before deciding

## What Gets Migrated

### ✅ Migrated Collections

| Collection | Migration Method | Notes |
|------------|------------------|-------|
| **Clients** | Parent reference updated | Merged if conflicts |
| **Pilots** | Parent reference updated | Merged if conflicts |
| **Vehicles** | Parent reference updated | Merged if conflicts |
| **Jobs** | `byPuid` reference updated | All related data follows |
| **Applications** | Via job relationship | No direct update needed |
| **App Files** | Via application relationship | No direct update needed |
| **App Details** | Via app file relationship | No direct update needed |
| **Job Logs** | Via job relationship | No direct update needed |
| **Job Assigns** | Via job relationship | Updated if account merge |
| **Products** | `byPuid` reference updated | - |
| **Crops** | `byPuid` reference updated | - |
| **Costing Items** | `byPuid` reference updated | - |
| **Invoices** | Customer reference updated | Optional via `--skip-invoices` |

### ❌ Not Migrated

- Stripe subscriptions (requires manual handling)
- Bill periods
- Payment logs
- User authentication tokens
- Session data

## Troubleshooting

### Issue: Transaction Timeout

**Symptom**: Migration fails with transaction timeout error

**Solution**: The script uses enhanced transaction handling with 60-second timeout. For very large migrations:
- Ensure database connection is stable
- Consider migrating in smaller batches (fewer source customers at once)

### Issue: Conflicts Detected

**Symptom**: Migration aborted with conflict list

**Solutions**:
1. **Recommended**: Manually resolve conflicts by renaming accounts in source or destination
2. Use `--merge-conflicts` flag (USE WITH CAUTION)
3. Migrate source customers without conflicting accounts first

### Issue: Validation Failed

**Symptom**: "Source or target customer not found"

**Solutions**:
- Verify customer IDs are correct
- Ensure customers are not already deleted
- Check database connection

### Issue: Permission Denied

**Symptom**: MongoDB permission errors

**Solutions**:
- Ensure database user has write permissions
- Check that user can create transactions
- Verify user has access to all collections

## Best Practices

### 1. Always Preview First

```bash
# ALWAYS run preview before execution
node scripts/migrateCustomerData.js --preview --sources ... --destination ...
```

### 2. Backup Database

Before running any migration, ensure you have a recent database backup.

### 3. Test on Non-Production First

Test the migration process on a development or staging environment first.

### 4. Review Conflicts Carefully

If conflicts are detected, review them carefully. Consider manually resolving rather than using `--merge-conflicts`.

### 5. Monitor History File

Regularly review `migration_history.json` to track all migrations.

### 6. Plan Stripe Subscriptions

Customer subscriptions cannot be automatically migrated and require manual intervention through Stripe.

### 7. Use Custom Environment Files

For production migrations, use the `--env` flag to specify the production environment file:

```bash
# Production migration
node scripts/migrateCustomerData.js \
  --env ../environment_prod.env \
  --sources acme_aviation \
  --destination consolidated_ag \
  --preview

# Development/testing (default)
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag \
  --preview
```

## Rollback Migrations

If you need to undo a migration, use the rollback script:

```bash
# Rollback the last migration (development environment)
node scripts/rollbackMigration.js

# Rollback with production environment
node scripts/rollbackMigration.js --env ../environment_prod.env
```

### What Rollback Does

- ✅ Restores all migrated entities (`byPuid` references)
- ✅ Restores deleted products/crops (if entity reuse was used)
- ✅ Restores invoice references
- ✅ Restores sub-account parent references
- ✅ Restores source customer from admin back to customer
- ✅ Reactivates deactivated source customers

### Rollback Limitations

- ⚠️ Can only rollback the **last** completed migration
- ⚠️ Must be run against the same database as the migration
- ⚠️ Cannot rollback if database has been manually modified since migration
- ⚠️ Check entities exist before restoring (handles already-rolled-back state)

## Database Schema Impact

### Updated Fields

- `User.parent` - Updated for sub-accounts
- `Job.byPuid` - Updated to destination customer
- `Product.byPuid` - Updated to destination customer
- `Crop.byPuid` - Updated to destination customer
- `CostingItem.byPuid` - Updated to destination customer
- `Invoice.customer` - Updated to destination customer
- `Job.client/operator/vehicle` - Updated if account merged

### Marked for Deletion

- `Customer.markedDelete` - Source customers (unless converted to admin)
- `Customer.username` - Suffixed with timestamp
- `User.markedDelete` - Merged sub-accounts
- `User.username` - Suffixed with timestamp

## Support and Debugging

Enable debug output:

```bash
DEBUG=agm:migrate-customer-data node scripts/migrateCustomerData.js ...
```

For detailed transaction debugging:

```bash
DEBUG=agm:migrate-customer-data,agm:mongo_enhanced node scripts/migrateCustomerData.js ...
```

## Related Files

- `scripts/migrateCustomerData.js` - Main migration script
- `scripts/rollbackMigration.js` - Rollback script for undoing migrations
- `helpers/mongo_enhanced.js` - Transaction utilities
- `helpers/constants.js` - User types and constants
- `model/job.js` - Job model with `byPuid` reference
- `migration_history.json` - Migration history log (tracks all migrations and rollbacks)

## License

Internal use only - AgMission Customer Data Management