On the Anatomy of a SagaHow to Keep an Order from Getting Lost

An order begins its long journey the moment a user clicks “Buy.” It has to pass through authentication, inventory locking, risk checks, a payment gateway, and logistics allocation before finally landing in a row of some table you have never heard of. If any stop along the way breaks down, the order may disappear somewhere in between — neither completed nor back where it started.

Over the past ten years, our industry has packaged this problem as distributed transactions, then repackaged distributed transactions as the Saga pattern, and finally admitted that Sagas cannot fix everything either, adding event sourcing. But the underlying question has not changed — how do we know exactly how far an order has gone, and how do we bring it back from whatever strange intermediate state it has reached?

Why Not Wrap It All in One Big Transaction?

The most straightforward thought is to put every step inside a single database transaction. That made sense in the days of one database and one service. But when inventory lives in service A, payments in service B, and logistics in service C, you cannot make a single transaction span three independent process boundaries — even with the XA protocol, because under heavy traffic XA either slows down or falls over.

In a distributed system, there is no such thing as an “atomic operation,” only an operation that “looks atomic.”

That sounds like an admission of defeat. In fact, it is one of the most costly lessons our industry has agreed on. Once you accept it, every architectural choice that follows asks the same question — how atomic can I make it look?

How Sagas Come to the Rescue

A Saga essentially breaks one long transaction into a sequence of short ones, each with a corresponding “compensating action” — if step 5 fails, execute the compensating actions in reverse: 4, 3, 2, and 1. It sounds simple. In practice, two things make it difficult:

First, compensation is not simply the inverse operation. If you charge a user 100 yuan, a refund is not simply “adding 100 back,” because exchange rate fluctuations, processing fees, or risk-related freezes may be involved. If you reserve inventory, releasing it is not simply “adding the stock back” either, because other people placing orders in the meantime may have seen your “phantom inventory.”

Second, the Saga state machine itself must be persisted. Otherwise, after a process restarts, you will not know which step it reached. This is why we use the event stream (event log) as the system’s “source of truth” — state can be lost; events cannot.

// A minimal Saga implementation, in pseudocode
async function placeOrder(input) {
  const ev = await events.append("ORDER_RECEIVED", input);
  try {
    await events.append("INVENTORY_RESERVED", await reserveInventory(ev));
    await events.append("PAYMENT_AUTHORIZED",  await authorizePayment(ev));
    await events.append("ORDER_COMMITTED",     await commit(ev));
  } catch (err) {
    await compensate(ev, err);   // Walk back through the steps; each one is idempotent
    throw err;
  }
}

The Debate over cron as a Safety Net

A common approach is to use a Saga for the main flow and cron as a safety net — scan for “stuck” orders every minute and see whether they can move forward. This works up to a certain scale, but it has two hidden costs:

One is response latency. If cron runs once a minute, users may have to wait 60 seconds in the worst case before they see the status move forward. The other is competing writes. The main flow advances the order, then cron tries to advance it again. The state machine must be idempotent, or you get a strange “advanced twice” bug — users see their money charged twice and then refunded once, an experience even worse than a stuck order.

So the more reliable approach is this: the main flow must complete inline; cron only checks the facts and does not acquire locks on its own. We learned this through repeated stumbles in the unified order system at GGBOYS — at first we treated cron as a lifeline, then discovered that what really helps is giving the main flow its own ability to “retry + advance.” cron only enters the picture once the main flow is completely dead, and even then it only observes; it does not mutate.

Keeping a Record of What We Learned

After every incident in the order system, I archive screenshots of the state machine at the time, the logs, and the reasoning behind our decisions in the team’s PITFALLS documents. These documents are for me two years from now, when I have forgotten the details, and for colleagues who have just joined, rather than for today — they record “why we cannot do it this way,” rather than “this is how it should be done.”

I increasingly believe that a team’s real asset is its incident archive rather than its codebase. Code gets rewritten. Lessons do not.

Translation log: translated from the Chinese original by Codex on 2026-11-03, 17:53–17:55 (US Eastern). Main thread GPT-6 Astra (gpt-6-astra), reasoning effort ultra; it spawned 1 sub-thread: 1 on GPT-6 Astra (effort ultra). Together they read 407,973 tokens (384,128 of them from cache) and wrote 7,188 (1,327 of them reasoning).