Isaac Emmanuel

2026.10.30

Idempotency must survive process death

An in-memory flag can stop a duplicate inside one process. Real idempotency has to survive retries, replicas, restarts, and the moment after a side effect succeeds.

I used to think of idempotency as a duplicate-request problem.

A client sends the same request twice, so the server keeps a key and returns the first result. That is part of it, but production systems create duplicates in less polite ways.

A worker finishes the external call and dies before acknowledging the job. A load balancer sends a retry to another replica. A scheduler overlaps during a deployment. A network timeout hides a successful response. Two users press the button at nearly the same time because the interface appears stuck.

Idempotency is not complete until it survives the death of the process that first handled the work.

Memory is not the authority

An in-memory set of processed keys can prevent a duplicate inside one running instance. It loses its knowledge on restart and knows nothing about the other replicas.

The durable decision belongs in shared storage.

A basic relational design might include:

sql
create table operation_requests (
  owner_id uuid not null,
  idempotency_key text not null,
  request_hash text not null,
  status text not null,
  response jsonb,
  created_at timestamptz not null default now(),
  primary key (owner_id, idempotency_key)
);

The composite key matters. An idempotency key supplied by one customer should not collide with another customer's operation.

The request hash matters too. Reusing the same key with a different payload should be rejected. Otherwise the server may return a successful response for an operation the caller did not actually request.

The first write has to win atomically

The unsafe pattern is:

text
check whether key exists
  -> no
  -> perform work
  -> insert key

Two replicas can both observe "no" before either inserts.

The safer pattern lets the database decide the winner through a unique constraint or an atomic insert. The loser reads the existing operation and responds according to its state.

text
insert operation key
  -> inserted: this request owns the work
  -> conflict: inspect the existing operation

The application should not recreate database concurrency control with hopeful timing.

In progress is a real state

If another request finds the key, the result may not exist yet. I find it useful to model at least these states:

  • IN_PROGRESS
  • SUCCEEDED
  • FAILED_RETRYABLE
  • FAILED_FINAL
  • UNKNOWN

UNKNOWN is uncomfortable but honest. It represents the case where a side effect may have occurred and the service lost the evidence needed to decide safely.

Automatically retrying an unknown payment, message, or publication can create a duplicate. The correct response may be reconciliation rather than another attempt.

The external side effect is the hard part

Database-only work can often be wrapped in one transaction. External providers do not share that transaction.

Suppose the system records a payment request, calls the provider, and receives success. It crashes before updating the local row. On restart, the database says IN_PROGRESS while the provider has already acted.

There are several ways to reduce this ambiguity:

  • Send a stable provider idempotency key when the provider supports one.
  • Persist the intent before calling the provider.
  • Store provider references as soon as they are available.
  • Reconcile unknown attempts by querying the provider.
  • Do not treat a client timeout as proof of failure.

The provider's idempotency behaviour is part of our design, not an implementation detail hidden inside an SDK.

Schedulers need durable singleton semantics

Scheduled work creates its own duplicate problem. Running one scheduler process is not proof that one will always run. Deployments overlap, containers restart, and a second replica may be started for availability.

For a job that must happen once per time window, I prefer a durable tick key:

text
job name + intended UTC window -> unique execution record

Every replica may attempt to claim the tick. Only one insert succeeds. The record tells us which windows ran, failed, or never appeared.

This is stronger than a process-level mutex because the truth survives the process.

I test failure, not only repetition

Calling the same endpoint twice in a unit test is useful, but it misses the situations that usually hurt.

In a local multi-replica experiment, I want to see what happens when:

  • Requests are distributed across several application pods.
  • The same key reaches different pods concurrently.
  • A pod disappears while requests are active.
  • A retry lands on a pod with no local memory of the first attempt.
  • The scheduler has multiple eligible replicas.

The evidence I care about is not merely that every HTTP response was successful. I want to inspect the durable state and prove that one logical action produced one durable result.

Exactly once is usually assembled

Networks do not hand us exactly-once execution. We assemble the useful approximation from several parts:

  • At-least-once delivery.
  • Stable operation identity.
  • Atomic ownership of that identity.
  • Idempotent state transitions.
  • Provider-supported deduplication where available.
  • Durable outcomes and reconciliation.

The phrase "we added an idempotency key" hides all of this.

The key is only the name of the decision. The system is idempotent when every replica, retry, and restart reaches the same durable answer about what happened.