Build for Failure

There is a moment in every payment engineer's career that changes how they build software forever. Yours will look something like mine. A partner bank's API goes down, not with clean errors, but with hanging connections and occasional successes, during your highest-traffic hour. Retries pile up. Some customers are charged twice. Some aren't charged at all. And the worst part? For forty minutes, you cannot say with confidence which was which.

After that night, you stop asking "how do I make this work?" and start asking the only question that matters in financial infrastructure. "What happens when this fails?"

Failure is the normal case

In a payment system of any real scale, failure is not an edge case, it is weather. Third-party processors time out. Banks return undocumented error codes. Networks partition. Webhooks arrive late, duplicated, or out of order. A system that handles only the success path is not mostly done; it is mostly untested against reality.

The discipline is to enumerate failure the way you enumerate features. For every external call, five questions.

1. What if it never responds?
2. What if it responds after we gave up?
3. What if it succeeded, but we never learned?
4. What if we retry? Can that cause harm?
5. Who resolves it if the answer stays unknown?

Question three is the one that separates payments from most other domains. In a CRUD app, an ambiguous write is an annoyance. In a payment system, it's money in an unknown state, the one condition a financial system must never tolerate silently.

Idempotency, the load-bearing wall

Everything else in reliable payments rests on one property. An operation retried is an operation performed once. Without idempotency, every retry is a gamble, every timeout is a potential double-charge, and every incident response is paralyzed by fear of making things worse.

async function executeTransfer(cmd: TransferCommand): Promise<TransferResult> {
  // One key per *intent*, minted when the user acts,
  // not per attempt. This is the whole trick.
  const existing = await store.findByIdempotencyKey(cmd.idempotencyKey); // [!code highlight]
  if (existing) return existing.result; // replay, not re-execution
 
  const transfer = await store.create({
    key: cmd.idempotencyKey,
    state: "pending",
    amount: cmd.amount,
  });
 
  const outcome = await provider.send(transfer, {
    reference: cmd.idempotencyKey, // the provider dedupes on this too
  });
 
  return store.complete(transfer.id, outcome);
}

The subtlety most teams miss is that the idempotency key must be minted at the moment of intent, when the user taps "Send", and carried through every layer, including to the provider. A key generated per attempt protects nothing; it just gives each duplicate a fresh disguise.

Once operations are idempotent, retries stop being dangerous, and you can be aggressive about recovering from transient failure, because the worst a retry can do is nothing.

Timeouts are answers you choose in advance

A timeout is not an error. It is your system telling you that the deadline arrived and the truth did not. What you do next must be a product decision made calmly at a desk, not improvised in an incident channel.

For every dependency, decide in advance.

• Fail closed. Decline the transaction. Right for fraud checks on high-risk flows.
• Fail open. Proceed with a fallback. Right for a recommendation or limit-check that has a safe default.
• Fail pending. Accept the intent, queue it, and resolve asynchronously. Right for transfers, where users prefer "processing" to "try again later."

The third option is underused. Users tolerate honest pending states far better than they tolerate errors. A payment that says "we're on it" and completes in two minutes feels more trustworthy than one that fails fast and makes the user retry. Pending is a promise with a deadline; an error is a promise refused.

Reconciliation, the system that checks the system

Idempotency and timeout policy handle failures you can see. Reconciliation handles the ones you can't, the webhook that never arrived, the settlement file with one extra row, the ledger entry that drifted from the provider's record.

Reconciliation is not a batch job. It is the immune system of a financial platform, always running, always comparing what we believe against what the world reports.

The mechanics are unglamorous. Pull provider reports, compare against internal state, and force every difference into a workflow with an owner and a deadline. The principle behind them is anything but.

Every unit of money is in a known state,
or in a queue that will make it known,
with a human accountable for the queue.
 
There is no third category.

Teams that treat reconciliation as an afterthought discover their discrepancies from customers. Teams that treat it as core infrastructure discover their discrepancies from dashboards, usually before the customer notices anything at all. That difference is, roughly, the difference between a fintech people trust and one they used to use.

Blast radius, or failing small

Finally, reliable systems fail small. A degraded FX-rate service should slow down currency conversion, not take card authorizations down with it. This is bulkheading, the art of isolating components so failure in one cannot flood the others.

In practice this means separate queues and worker pools per flow, per-dependency circuit breakers, load-shedding rules that protect the money paths first, and kill switches that let you disable a single corridor or feature without touching the rest. When something breaks at 2 a.m., and it will, the on-call engineer should be reaching for a labeled switch, not improvising surgery.

The mindset

None of these techniques is exotic. Idempotency keys, timeout budgets, reconciliation loops, bulkheads. These are old ideas, documented everywhere. What separates reliable payment platforms from fragile ones is not knowledge. It's the mindset that failure is the primary design input.

• In design reviews, the failure paths get more scrutiny than the success path.
• "What happens when this fails?" is asked about every arrow on the diagram.
• Ambiguity is treated as the enemy, worse than failure itself.

Build for failure, and failure becomes an operational event, one that is detected, bounded, resolved, and reviewed. Build for success only, and failure becomes an existential one.

The systems people trust are not the ones that never fail. They are the ones that fail honestly, recover quickly, and never, ever lose track of the money.