The first version of any importer writes directly. You point it at a file, it creates the records, and the run either succeeds or throws. For a while this is fine, because you are watching it, and because the input is the file you just looked at.
It stops being fine on the day the input is slightly wrong in a way nobody noticed. The damage is rarely the bad record itself, which is usually easy to spot and delete. The damage is that from then on you do not know which of the good looking records to trust, and checking them by hand costs more than the automation ever saved.
Preview is the default, not a flag
We make the safe mode the one you get by doing nothing. When writing is the default and previewing is the option you remember to pass, the first hurried afternoon is the afternoon it writes.
The preview prints the complete set of records it would create, each one with the rule or the source document that produced it, and then stops. Not a count, not a sample: the actual list, in full, because the errors you are looking for are the ones that look reasonable in aggregate. Committing is a separate command against the same input, run by a person who has read the list.
Starting from the last record, not from a date
The other common way to ruin a dataset is telling the importer where to begin. A human picks a date, remembers it slightly wrong, and the batch either duplicates a week or skips one.
Ours asks the destination instead. Before a run it queries the latest record already stored for each account, each stream, each customer, whatever the natural partition is, and starts from there. The system of record already knows where it got to.
Reading back what was written
Once records are written, the importer reads them back. It confirms each record exists as stored, and that any derived total the destination maintains still agrees with what was expected. A write path that trusted its own request would report success while quietly doing nothing.
Timeouts get their own handling, because a timeout is an unknown rather than a failure: the record may well have been created. Retrying an unknown is how one payment becomes two, so on a timeout the importer stops, looks at the destination, and lets a person decide what happened.
Saying what would be taken back
Each committed batch is journaled: the identifiers, in order, written as they are confirmed. A rollback then works like everything else here, printing the exact records it would remove and waiting for approval before removing them.
Knowing you can undo the last batch changes how the whole system feels to use. It is the difference between running an import and committing to one.
What this design gives up
This design is slow, and that is the trade.
- A person in the loop. Someone reads the preview. That is a real cost and it caps how much you can process in a day.
- Two passes over the data. Preview and commit both do the work of building the records.
- Extra state. The journal is another file to keep correct, including when a run fails halfway.
Those costs buy very little at high volume, where the right answers are idempotency keys, deduplication, and reconciliation running after the fact. This shape is for the other case: low volume, high consequence, and a dataset a human would have to repair by hand.
Three questions before it writes
Before our automation changes a record we could not reconstruct, we want it to answer three questions. What exactly will it change, where does it start, and what would it take to undo this?
Until it can, it stays in preview.