agmission/Development/server/scripts/MIGRATION_QUICK_REFERENCE.md

Customer Data Migration - Quick Reference

Quick Start

1. Preview Migration (Always Do This First!)

# Using usernames (recommended - more readable)
node scripts/migrateCustomerData.js \
  --preview \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag

# Using IDs (also supported)
node scripts/migrateCustomerData.js \
  --preview \
  --sources <SOURCE_ID1>,<SOURCE_ID2> \
  --destination <DEST_ID>

# Mixed: usernames and IDs work together
node scripts/migrateCustomerData.js \
  --preview \
  --sources acme_aviation,507f1f77bcf86cd799439012 \
  --destination consolidated_ag

2. Execute After Reviewing Preview

# Using usernames
node scripts/migrateCustomerData.js \
  --sources acme_aviation,regional_ag \
  --destination consolidated_ag

# Using IDs
node scripts/migrateCustomerData.js \
  --sources <SOURCE_ID1>,<SOURCE_ID2> \
  --destination <DEST_ID>

Common Scenarios

Scenario 1: Simple Single Customer Migration

Use Case: Merge one customer into another

# Step 1: Preview (using usernames - easier to read)
node scripts/migrateCustomerData.js \
  --preview \
  --sources acme_aviation \
  --destination consolidated_ag

# Step 2: Execute
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag

# Alternative: Using IDs
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea

Result:

• Source customer deactivated
• All data moved to destination
• Aborts if username conflicts detected

---

Scenario 2: Consolidate Multiple Regional Accounts

Use Case: Merge multiple regional accounts into corporate account

# Using usernames (recommended)
node scripts/migrateCustomerData.js \
  --sources acme_west,acme_east,acme_south \
  --destination acme_corporate

# Using IDs
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011,507f1f77bcf86cd799439012,507f1f77bcf86cd799439013 \
  --destination 507f191e810c19729de860ea

Result:

• All 3 regional accounts deactivated
• All data consolidated under corporate account
• Regional managers can be converted to admins (see Scenario 4)

---

Scenario 3: Merge with Known Conflicts

Use Case: Known username overlaps, want to merge accounts

# Preview first to see conflicts (using usernames)
node scripts/migrateCustomerData.js \
  --preview \
  --sources acme_aviation \
  --destination consolidated_ag

# Execute with merge if conflicts are acceptable
node scripts/migrateCustomerData.js \
  --sources acme_aviation \
  --destination consolidated_ag \
  --merge-conflicts

Result:

• Conflicting accounts merged (old accounts marked deleted)
• Job/data references updated to existing accounts
• Old account data preserved in history

⚠️ WARNING: Review conflicts carefully before merging!

---

Scenario 4: Convert Customers to Admin Accounts

Use Case: Keep source customer as admin user under destination

# Using usernames
node scripts/migrateCustomerData.js \
  --sources acme_west,acme_east \
  --destination acme_corporate \
  --convert-to-admin

# Using IDs
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011,507f1f77bcf86cd799439012 \
  --destination 507f191e810c19729de860ea \
  --convert-to-admin

Result:

• Source customers become admin (kind='2') accounts under destination
• All sub-accounts and data moved to destination
• Original customer accounts deactivated but admin access preserved

Perfect For:

• Regional managers becoming corporate admins
• Acquired companies maintaining some autonomy
• Franchise consolidation

---

Scenario 5: Data Only Migration (Skip Invoices)

Use Case: Migrate operational data but keep invoices separate

node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea \
  --skip-invoices

Result:

• All data migrated except invoice customer references
• Useful for financial separation requirements
• Invoices remain linked to original customer

---

Scenario 6: Custom History File Location

Use Case: Store migration logs in specific location

node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea \
  --output-file /var/log/agmission/migrations/customer_migration_2025.json

Result:

• Migration history saved to custom location
• Useful for compliance/audit requirements

---

Decision Tree

Do you want to migrate customer data?
│
├─ Single source customer?
│  ├─ Yes → Use Scenario 1
│  └─ No → Continue
│
├─ Multiple source customers?
│  ├─ Yes → Use Scenario 2
│  └─ No → Continue
│
├─ Known username conflicts?
│  ├─ Yes → Use Scenario 3 (with --merge-conflicts)
│  └─ No → Continue
│
├─ Keep source customers as admins?
│  ├─ Yes → Use Scenario 4 (with --convert-to-admin)
│  └─ No → Continue
│
├─ Need to preserve invoice separation?
│  ├─ Yes → Use Scenario 5 (with --skip-invoices)
│  └─ No → Use Scenario 1 or 2
│
└─ Always preview first with --preview!

Conflict Resolution Strategies

Strategy 1: Manual Resolution (RECOMMENDED)

Before migration:

1. Run preview to identify conflicts
2. Manually rename conflicting accounts in source OR destination
3. Run migration without conflicts

# Example: Rename client in source database
db.users.updateOne(
  { _id: ObjectId("507f1f77bcf86cd799439015") },
  { $set: { username: "john_farmer_regional" } }
)

# Then run migration
node scripts/migrateCustomerData.js --sources ... --destination ...

Strategy 2: Automatic Merge

During migration:

• Use --merge-conflicts flag
• Old accounts marked as deleted
• References updated to existing accounts

node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea \
  --merge-conflicts

Strategy 3: Selective Migration

Migrate in phases:

1. Migrate non-conflicting customers first
2. Resolve conflicts for remaining customers
3. Migrate remaining customers

# Phase 1: No conflicts
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439011 \
  --destination 507f191e810c19729de860ea

# Phase 2: After resolving conflicts
node scripts/migrateCustomerData.js \
  --sources 507f1f77bcf86cd799439012 \
  --destination 507f191e810c19729de860ea

Verification Steps

After Migration

1. Check Migration History

   cat migration_history.json | jq '.[-1]'
   

2. Verify Source Customer Deactivated

   db.users.findOne({ 
     _id: ObjectId("507f1f77bcf86cd799439011"),
     kind: "1"
   })
   // Should have: markedDelete: true, active: false
   

3. Verify Job Migration

   db.jobs.countDocuments({ 
     byPuid: ObjectId("507f191e810c19729de860ea") 
   })
   // Should equal total jobs from source + existing destination jobs
   

4. Verify Sub-Account Migration

   db.users.countDocuments({ 
     parent: ObjectId("507f191e810c19729de860ea"),
     kind: { $in: ["3", "5", "9"] }
   })
   // Should equal expected count
   

5. Check for Errors in History

   cat migration_history.json | jq '.[-1].stats.errors'
   

Troubleshooting Common Issues

Issue: "Source customer(s) not found"

# Verify customer exists
mongo
> use agmission
> db.users.findOne({ _id: ObjectId("YOUR_ID"), kind: "1" })

Issue: "Migration aborted due to conflicts"

# View conflicts in detail
node scripts/migrateCustomerData.js --preview --sources ... --destination ...
# Then use Strategy 1 or 2 above

Issue: Transaction timeout

# Reduce number of source customers per migration
# Instead of migrating 10 at once, do 2-3 at a time

Issue: Need to rollback

# MongoDB transactions auto-rollback on error
# If completed but need to reverse:
# 1. Restore from backup
# OR
# 2. Manually reverse using migration_history.json details

Environment Variables

# Enable detailed debugging
DEBUG=agm:migrate-customer-data,agm:mongo_enhanced \
  node scripts/migrateCustomerData.js ...

# Custom MongoDB connection
MONGODB_URI=mongodb://user:pass@host:port/dbname \
  node scripts/migrateCustomerData.js ...

Data Validation Queries

Check Job Distribution

// Count jobs per customer
db.jobs.aggregate([
  { $match: { markedDelete: { $ne: true } } },
  { $group: { _id: "$byPuid", count: { $sum: 1 } } },
  { $sort: { count: -1 } }
])

Check Sub-Account Distribution

// Count sub-accounts per parent
db.users.aggregate([
  { $match: { kind: { $in: ["3", "5", "9"] }, markedDelete: { $ne: true } } },
  { $group: { _id: "$parent", count: { $sum: 1 } } },
  { $sort: { count: -1 } }
])

Find Orphaned Data

// Jobs without valid customer
db.jobs.aggregate([
  { $match: { markedDelete: { $ne: true } } },
  {
    $lookup: {
      from: "users",
      localField: "byPuid",
      foreignField: "_id",
      as: "customer"
    }
  },
  { $match: { customer: { $size: 0 } } },
  { $count: "orphaned_jobs" }
])

Checklist

Before Migration:

• Database backup completed
• Source and destination customer IDs confirmed
• Preview completed and reviewed
• Conflicts identified and strategy decided
• Stakeholders notified
• Maintenance window scheduled (if applicable)

After Migration:

• Migration history file reviewed
• No errors in migration log
• Job counts verified
• Sub-account counts verified
• Invoice references checked (if not skipped)
• Source customer deactivated (if not converted to admin)
• Destination customer has all expected data
• Users notified of changes

Support

For issues or questions:

1. Check migration_history.json for details
2. Enable debug logging: DEBUG=agm:*
3. Review this guide and main README
4. Contact database administrator

Related Commands

# View migration history
cat migration_history.json | jq '.'

# View latest migration
cat migration_history.json | jq '.[-1]'

# View all failed migrations
cat migration_history.json | jq '.[] | select(.status == "failed")'

# View all completed migrations
cat migration_history.json | jq '.[] | select(.status == "completed")'

# Count migrations by status
cat migration_history.json | jq 'group_by(.status) | map({status: .[0].status, count: length})'