The first version of any importer writes directly. You point it at a file, it creates the records, and the run either succeeds or throws. For a while this is fine, because you are watching it, and because the input is the file you just looked at.
It stops being fine on the day the input is slightly wrong in a way nobody noticed. The damage is rarely the bad record itself, which is usually easy to spot and delete. The damage is that from then on you do not know which of the good looking records to trust, and checking them by hand costs more than the automation ever saved.
Preview has to be the default, not a flag
The safe mode has to be the one you get by doing nothing. If writing is the default and previewing is the option you remember to pass, then the first hurried afternoon is the afternoon it writes.
A preview should print the complete set of records it would create, each one with the rule or the source document that produced it, and then stop. Not a count, not a sample: the actual list, in full, because the errors you are looking for are the ones that look reasonable in aggregate. Committing is then a separate command against the same input, run by a person who has read the list.
Start from the record, not from a date
The other common way to ruin a dataset is telling the importer where to begin. A human picks a date, remembers it slightly wrong, and the batch either duplicates a week or skips one.
Ask the destination instead. Before a run, query the latest record already stored for each account, each stream, each customer, whatever the natural partition is, and start from there. The system of record already knows where it got to, and it is the only participant that cannot be mistaken about it.
A successful response is not proof
Once records are written, read them back and check them. Confirm the record exists as stored, and confirm any derived total the destination maintains still agrees with what you expected. A write path that trusts its own request is a write path that reports success while quietly doing nothing.
Timeouts deserve their own rule, because a timeout is not a failure. It is an unknown, and the record may well have been created. Retrying an unknown is precisely how one payment becomes two payments, so the correct response is to stop, look at the destination, and let a person decide what actually happened.
Be able to say what you would take back
Journal each committed batch: the identifiers, in order, written as they are confirmed. A rollback then works like everything else here, printing the exact records it would remove and waiting for approval before removing them.
Knowing you can undo the last batch changes how the whole system feels to use. It is the difference between running an import and committing to one.
What it costs
This design is slow on purpose, and it is not free.
- A person in the loop. Someone has to read the preview. That is a real cost and it caps how much you can process in a day.
- Two passes over the data. Preview and commit both do the work of building the records.
- Extra state. The journal is another file that has to stay correct, including when a run fails halfway.
Those costs buy very little at high volume, where the right answers are idempotency keys, deduplication, and reconciliation running after the fact. This shape is for the other case: low volume, high consequence, and a dataset a human would have to repair by hand.
The general shape
Before an automation is allowed to change a record you could not reconstruct, it should be able to answer three questions. What exactly will you change, how do you know where to start, and what would it take to undo this?
A program that cannot answer all three is not finished. It is a draft that happens to have write access.