Every company that moves money has a ledger, and every ledger starts as a table. Ours was a table for six years. It was a good table: append-only, indexed, boring. Then it became the thing that every other service in the company read from, wrote to, and quietly depended on in ways nobody had written down.
In March of last year we decided to replace it. This is an account of how that went — what we built, what we refused to touch, and the two weeks in August when we very nearly stopped.
Why a rewrite, and why now
Rewrites have a bad reputation, and they've earned it. The honest reason we did this one is that the old ledger had stopped being able to answer the questions we were being asked. Finance wanted balances as of any past instant. Support wanted to see why a number was what it was. Our largest customers wanted to reconcile against their own books without waiting for a nightly batch.
The old design could do none of that cheaply. Balances were derived by summing rows, which was fast when there were millions of rows and had become slow with billions. We had patched around it with caches, and the caches had become the real source of truth for anything customer-facing. When a cache and the table disagreed, we trusted whichever one a human had last looked at.

The rule: no one should be able to tell
Before we wrote any code, we wrote one sentence and pinned it to the top of the project document: a customer must not be able to tell that the ledger changed. Same API shapes, same response times within a margin, same numbers to the cent.
That sentence did more work than any architecture diagram. It ruled out the tempting clean-slate redesign of our public API. It ruled out changing how we round. It turned a hundred arguments about taste into one question — would a customer notice? — that usually had an obvious answer.
What we kept
- The external data model: accounts, entries, transfers and holds, with identical identifiers.
- Integer minor units everywhere. No floats, no decimals, no exceptions, ever.
- Entries are immutable. Corrections are new entries that reference the old one.
- The public webhook payloads, down to field ordering.
What we changed
- Balances are now maintained incrementally per account, with periodic checkpoints, instead of summed on read.
- Every entry carries a causal timestamp so we can answer as of queries without scanning.
- Writes are partitioned by account rather than by time, which removed the hottest lock in the system.
The goal of a migration is not to arrive. It is to arrive without anyone having noticed you left.
Shadowing before believing
For eleven months the new ledger did not serve a single customer request. Instead, every write to the old system was replayed into the new one, and every read was executed against both. A comparator service diffed the results and recorded any disagreement with enough context to reproduce it.
In the first week the comparator reported about two hundred thousand mismatches a day. This was not alarming; it was the point. Most were our own bugs. A surprising number were the old system's bugs, long-standing behaviors that we had to decide to either reproduce faithfully or fix openly.
// Simplified shape of the comparator's report
type Mismatch = {
operation: "balance" | "entries" | "transfer";
accountId: string;
old: unknown;
next: unknown;
// How we classified it after triage
verdict: "new-bug" | "old-bug-reproduced" | "old-bug-fixed" | "timing";
};The old-bug-fixed category is where we spent the most time arguing. Our rule held: if fixing a bug would change a number a customer had already seen, we reproduced the bug and filed a ticket to retire it later, in public, with notice.
The two weeks we nearly stopped
In August the mismatch rate stopped falling. It plateaued at around forty a day and no amount of triage moved it. Every individual case had an explanation. Taken together, they had none.
The cause, when we found it, was embarrassingly small. The old system recorded timestamps in the database's local clock; the new one used the application's. Under load, the two drifted by a few milliseconds, which was enough to reorder entries that arrived in the same instant. Neither order was wrong. They were just different, and our comparator, correctly, said so.

We resolved it by making ordering explicit: a monotonic sequence number assigned at the account partition, independent of any clock. It was the first genuinely good design idea of the project, and it came from a bug, not a whiteboard.
Cutover, in three boring steps
The cutover itself took less time than the meeting to approve it. We did it account by account, in rings, starting with our own internal accounts and widening only after each ring had been stable for a full billing cycle.
- Read from new, write to both. The new ledger serves reads; the old one is kept warm as a fallback.
- Write to new, mirror to old. The new ledger becomes the system of record. The mirror exists purely so we can roll back.
- Retire the mirror. After sixty days without a rollback, we turned the old writes off and archived the table.
At no point during the three steps did a customer-facing metric move outside its normal band. The migration completed on a Tuesday. Nobody outside the company noticed, which was exactly the brief.
What we'd do again, and what we wouldn't
We would write the one-sentence rule again. We would build the comparator first again. We would, without hesitation, keep the migration boring on purpose.
We would not wait eleven months before talking to finance. They were the ledger's most demanding customer, and their questions shaped the checkpoint design more than any engineer's did. If we'd asked in month one, we would have saved about six weeks.



