On Scheduled Jobs as Production CodeCron Is Not a Fallback · It Is a Second Production Line

An order was stuck for nearly a day after a rental was returned.

The item had long since been returned, but the system simply would not acknowledge it. When I looked into it, I found five problems piled on top of one another, rather than a single bug: the platform had blocked an upstream endpoint at the time, taking out the main path; the fallback path happened to hit a page that was not returning the data it needed either; the other party's client was not running, so that side of the transaction was never connected; the item's internal identifier changed with every trade, so comparing it against an old identifier would always be wrong; and finally, no path checked the most conclusive fact of all—had the item actually left the other party's possession?

Fixing any one of those five would have kept the order from getting stuck. But all five failed together, and there it stayed. That incident changed how I think about scheduled jobs. A scheduled job is not a fallback. It is a second production line—usually dressed as a “fallback,” but holding the same write permissions as the main flow.

The Illusion of a Fallback

It is natural to want a fallback. Real systems always have rough edges: a user loses their connection, a payment callback is slow, a webhook is retrying, a database connection falters, a worker gets restarted. Expecting the synchronous flow to resolve every intermediate state in a single pass is unrealistic.

So a row of scheduled jobs grows up inside the system. Every few minutes, they scan for stuck orders and move them forward, release expired reservations, and finish compensation work left incomplete. There is nothing wrong with the jobs themselves. The trouble starts when they are treated as “unimportant fallbacks”—so they lack the boundary checks used in the main flow, their logs have less detail, no one has tested them in a real environment, and their time windows are arbitrary. Yet they modify orders, inventory, and wallets. An incident tends to look something like this:

Main flow:     The user has just paid; the state has not been fully written back
Scheduled job: Finds “removed / expired / pending too long”
Action:        Clears inventory or cancels the order
Result:        Both flows think they are right

The question is not which one is smarter. The problem is that two actors in the system can modify the same assets, and they do not share the same safety checks.

A Timestamp Is an Intermediate Signal, Not a Fact

One kind of incident has stayed with me. The logic for deciding “has the item been moved away?” was simple: if a certain “removed” timestamp field had a value more than a few minutes old, treat the removal as established and move the rest of the process forward.

Then, after an upstream state rollback released a lock, the inventory snapshot drifted, and several orders were mistakenly marked complete a dozen or so hours before they should have ended. That populated field looked like solid proof. In fact, it was only an intermediate signal within one flow—when it was written, payment, inventory, notifications, and the next full snapshot were all still on their way. The job stamped its approval ahead of them.

Thinking a little further: using a snapshot from just one side to judge something that spans services is unsound to begin with. Snapshots drift, and that drift can even correct itself—the next full snapshot resets the state, but decisions already made from the wrong state do not roll back with it. What truly cannot be undone is the event itself: once a trade is complete, it is complete, and once that record is written, it does not change. Inferring “did this happen?” from a state field and relying directly on an irrevocable event offer two different levels of reliability. The eventual fix here was to move from “checking the state” to “recognizing the event that both sides had confirmed.”

So now, whenever I see a scheduled job triggering writes based on signals such as “removed,” “expired,” or “too old,” I first ask: is this an irrevocable fact, or a transitional state still propagating through the system?

A Grace Period Acknowledges That Reality Fluctuates

Grace periods are often criticized for not acting in real time. Why not release a reservation the moment it expires? Why not clean up the moment the signal appears?

Because “the moment” is mostly an illusion in a distributed system. The browser may have only just sent its request, the payment service may have only just received a callback, a worker may have only just begun retrying, or the main table's transaction may have committed while an auxiliary index has yet to catch up. The time you read in one flow does not mean the whole system has settled.

There is another, firmer fallback: elapsed time. Some platforms impose hard rules of their own: certain assets simply cannot be transferred during a specified period. Any “completed” signal that appears during that period must therefore be false—this is a hard constraint imposed by the outside world, rather than caution in business logic, and it can be used directly as a hard-coded filter. A grace period gives the fluctuations of reality some room; it is not procrastination. I would rather have a system that cleans up a dozen or so minutes later but does not mistakenly delete a user's assets.

Different Entrances, the Same Brakes

It is fine for the main flow and scheduled jobs to run at different times. Different safety boundaries can be disastrous. One design came close to going live and made that lesson hard to forget.

We were designing a compensation process that crossed services. The first version seemed straightforward: detect an anomaly, insert a “pending” record, and let a reconciliation job take over a few minutes later to move it to the next step. The problem was that actively moving things forward had never been that job's responsibility. It was designed only to verify that actions already taken reconciled correctly, and to handle compensation after timeouts. It checked; it did not initiate the crucial action across services. So the record the system inserted would wait forever for someone to perform that step. It was bound to be marked as failed, and the resources that should have been restored by the rollback would never come back.

What unnerved me most was this: the design had passed several rounds of internal and external review, and no one had noticed. Only at the end, when someone actually read the reconciliation job's code from beginning to end, did that hidden assumption come to light. The conclusion was simple, and costly: the initiator must carry out the crucial action from end to end. It cannot “make a record first and wait for the scheduled job to pick it up.” The reconciliation job should keep its original role. No one should give it the work of actively advancing the process, or it will compete with the active trigger to do the same thing.

Since then, I have put every scheduled job that writes state through these questions:

What state does it write?
Does the main flow write that state too?
What checks does the main flow make before writing?
Does the scheduled job reuse those same checks?
If it makes one wrong judgment, what happens to the user?
Could it start with a dry run, audit-only mode, or a grace period?

Whenever the answer is “it only does a little cleanup,” I automatically translate that as: it has the power to damage live state; it just does not usually look that way.

Fail-safe Means Limits on Actions, Not try/catch

Having try/catch and retries in the code does not mean it is fail-safe. The point of fail-safe is not “do not crash when something goes wrong.” It is “however things go wrong, never cross a certain limit on what you are allowed to do.”

A cleanup job may fail, skip work, raise an alert, or hand things to a queue for human review. But it must not delete anything while the state of a user's assets is uncertain. Payment reconciliation may be slow, rerun, or flag an anomaly. But it must not treat a paid order as an orphan and dispose of it on the strength of an intermediate signal.

Put plainly, the fallback needs a fallback of its own. The first layer is the scheduled job: it sweeps up whatever the main flow leaves behind. The second layer requires the job itself to stop when it is uncertain—without enough evidence, do not advance to an irreversible state; raise an alert and hand it to a person. If the stuck order at the beginning had had this second layer, it would not have lingered for nearly a day. It would have reached a person as soon as the first problem appeared.

Three Questions to Ask of a Scheduled Job

Does it have write access? Read-only monitoring and jobs that write state are different things. If it can write, review it to the same standard as a production flow.

Does it share an object with the main flow? If it can modify orders, inventory, wallets, or notifications, it will compete with the main flow over whose interpretation of the same thing prevails.

Can it stop when it is uncertain? A fallback with no way to stop is overconfident, not reliable.

These three questions matter far more than “how often does it run?” Frequency only determines how soon an incident arrives. Boundaries determine whether it happens at all.

A Second You

This is really another side of the same issue as the article on keeping orders from getting lost (№ 006) and the one on password rotation (№ 011)—there are no “atomic operations” in a distributed system; the little atomicity a database transaction gives you stops holding once you cross a service boundary. A scheduled job is not a quiet script tucked away in a corner of the bilge. It is a second you, with write access, able to touch users' assets, sharing the same data.

The order that was stuck for nearly a day was eventually recovered by hand. The cost was not large, but that stretch of time kept reminding me of one sentence: the background has no special privileges, only fewer witnesses. Whenever I hear “it is only a fallback” now, I automatically translate it as “it can cause trouble; there is just no one watching.”

Translation log: translated from the Chinese original by Codex on 2026-11-03, 17:46–17:49 (US Eastern). Main thread GPT-6 Astra (gpt-6-astra), reasoning effort ultra; it spawned 1 sub-thread: 1 on GPT-6 Astra (effort ultra). Together they read 348,846 tokens (316,160 of them from cache) and wrote 6,479 (898 of them reasoning).