How to Migrate Data Into a New System Without Losing Anything
Data migration is where custom system projects most often overrun. Auditing, cleaning, mapping, dry runs, cutover, and rollback — done properly.
How to Migrate Data Into a New System Without Losing Anything
Data migration is quietly where custom system projects most often overrun — not because moving data is technically hard, but because the existing data is rarely as clean or complete as anyone assumed when the project was scoped. Here's how to do it properly.
Audit the source data first
Before writing any migration logic, look at what you actually have: duplicate records, inconsistent formatting, missing fields, data entered by five different people over ten years in five different conventions. This audit should happen before the migration is estimated, not during it — it's usually the single biggest driver of how long migration actually takes.
Clean before you map
Fixing obvious data quality issues in the source system, or in an intermediate export, is far cheaper than building migration logic clever enough to handle every inconsistency automatically. Some cleanup genuinely needs a human to look at ambiguous records — budget time for that, not just engineering time.
Map fields deliberately
Document exactly how every source field maps to the new system's structure, including what happens to fields that don't have an obvious home. "We'll figure it out during migration" is how data quietly goes missing — a field nobody explicitly mapped often just doesn't make it across.
Run dry runs
Migrate a full copy of the data into a test environment before touching production, and have someone who knows the data well check the results against the source. Dry runs are where you find the edge case that breaks the migration script on record 40,000 — far better to find it there than during the real cutover.
Plan the cutover
Decide exactly when the switch happens, how long the old and new systems run in parallel (if at all), and who verifies the migrated data before the old system is switched off. A cutover done at a quiet time with a clear go/no-go checkpoint is far less risky than one done under deadline pressure.
Have a rollback plan
If something goes wrong during or immediately after cutover, know in advance how you'd revert to the old system without losing anything entered in the meantime. Hoping you won't need this is not a plan.
Where the time actually goes
| Stage | Typical share of migration effort |
|---|---|
| Audit and cleaning | Usually the largest share — often 40%+ on older or messier datasets |
| Mapping and migration logic | Moderate — depends on how different the two data structures are |
| Dry runs and verification | Should not be skipped to save time — this is where errors get caught cheaply |
| Cutover and rollback planning | Small in effort, high in consequence if skipped |
Be direct with yourself about this before scoping a new system: existing data quality, not the new system's complexity, is usually the largest variable in how long the whole project actually takes.