Also published on Medium on 2026-10-24.
The first time I built a backend that moved real money, I learned a rule I still apply to almost everything I build: never trust a single write.
When you’re building a to-do app, a failed or duplicated request is an annoyance. When you’re building payments, wallets or transaction processing, the same bug means someone is charged twice, a balance goes negative or money disappears from a report. The engineering is mostly the same. The consequences are not.
Things that “never happen” happen every day
Before working on financial systems, I thought of edge cases as rare. Afterwards, I thought of them as scheduled. At enough volume, every one of these is routine:
- The same request arrives twice. A mobile client retries after a timeout. A user taps the button again. A load balancer replays a request.
- A provider times out, but the payment went through. From your side, the call failed. On their side, the money moved.
- Two requests try to spend the same balance. Each one checks the balance, sees enough money and proceeds. Together, they overspend.
- A webhook arrives late, out of order or more than once.
- A process crashes halfway through a multi-step operation.
If your design assumes any of these won’t happen, it will be wrong regularly.
Idempotency keys on everything that moves money
The first defence is idempotency. Every request that changes money gets a unique key from the client. The server stores the key with the result. If the same key arrives again, the server returns the original result instead of doing the work twice.
It sounds simple, and the concept is. The details matter:
- The key and the operation need to be recorded atomically, so a crash can’t leave one without the other
- A second request with the same key but different parameters should be rejected, not silently accepted
- Keys need a sensible retention period
- Every layer that can retry, including your own background jobs, needs to respect them
Locks belong in the database
The classic overspend bug looks like this in application code: read the balance, check it’s enough, then write the new balance. Two concurrent requests can both pass the check before either writes.
Application-level checks don’t solve this. The database has to. That means row-level locks, conditional updates that only succeed if the balance is still sufficient, or serialisable transactions for the critical section. Choose based on your database and load, but make sure the guarantee lives where the data lives.
Append to a ledger, don’t overwrite a balance
One of the most valuable design decisions in financial systems is to stop treating a balance as the source of truth.
Instead, record every movement as an immutable entry in a ledger: what happened, when, how much, why and as a result of which request. The balance is derived from the ledger, or cached from it, but never edited directly.
This gives you things an overwritten balance never can:
- A complete history for every account
- The ability to answer “where did this money go?” with evidence
- Corrections that are visible, made through new entries rather than silent edits
- A foundation for audits and reconciliation
Assume something went wrong and go looking
Even with idempotency, locking and a ledger, you still need reconciliation. Scheduled jobs compare your records with your payment providers, banks and internal systems, and flag anything that doesn’t match.
The mindset matters. Reconciliation isn’t a formality. It’s a process that assumes something has gone wrong somewhere and goes looking for it. The goal is to find discrepancies before customers or finance teams do.
Visibility is a feature
The final lesson is that observability in financial systems isn’t optional. For every transaction, you want to be able to answer:
- Who initiated it, and through which request?
- What state is it in right now?
- What external calls were made, and what did they return?
- If it failed, where did it stop?
If your team can’t answer “where did this money go?” in a few minutes, that’s a gap worth prioritising.
The bigger principle
You can’t prevent every failure in a distributed system. Networks fail, providers fail, processes crash and users double-tap. The goal is to make every failure recoverable and visible.
I learned that building transaction systems, but it now shapes how I design everything else, including AI systems. When something can go wrong, make sure you’ll know, and make sure you can fix it.