On the Discipline of Catching CarefullyExceptions Are Not Failures · They Are Signals of Severity

One Sunday in late April 2026, I stared at the monitoring dashboard for 24 hours.

It began with an unremarkable change to how we lift bans. We deployed it on Friday, and nothing seemed wrong. When I looked back at the metrics on Monday morning, one number was off: there should have been more than 80 “account unbanned” notifications, but there were 0. Not a single user had received one.

The strange thing was that the service reported no errors. The logs were full of INFO-level “success” messages, and nothing looked wrong in the database either. On the surface, everything seemed normal.

Only after tracing the whole chain did I find it: when we added a new notification type, we forgot to update the database CHECK constraint to match. The INSERT raised an IntegrityError, but a try/except in the code logged it with logger.error, then quietly swallowed it. The main flow went on to commit. The user's ban really had been lifted in the DB, but the notification never went out.

24 hours, more than 80 users, and not one received a message. They had to refresh the page themselves and discover, “Oh, I can log in again?”

I thought about it for a while afterward. The immediate cause was, of course, the missed CHECK constraint update. That was a separate PITFALL. But the deeper cause was the presence of that try/except. It should not have been there.

Exceptions Are Not Failures; They Are Signals of Severity

A lot of try/except code looks like this:

try:
    do_something()
except Exception as e:
    logger.error(f"failed: {e}")
    # continue

It looks like “defensive programming”: I won't let any exception take down the process; I'll catch and log them all. It looks “resilient.”

But I now think this is one of the most dangerous patterns in code. It treats every exception as the same thing: “Something failed. Log it and carry on.” Exceptions are not all the same thing.

Exceptions are signals at different severity levels:

· Business-level exceptions: invalid user input, a third-party API returning 400, a quota exceeded. These are expected setbacks. Catching them and giving the user a helpful message is the right thing to do.

· System-level exceptions: a dropped database connection, a Redis timeout, a full disk. These are infrastructure problems. You can catch and retry or fail fast, but you need an alert.

· Invariant violations: IntegrityError, NOT NULL violations, CHECK constraint failures, foreign key failures. These mean the code's assumptions do not match the reality in the database. They do not mean “something failed.” They mean “the code is wrong.”

There is only one acceptable way to handle the third kind: logger.critical + raise. Let it crash. Let monitoring pick it up. Let me get paged in the middle of the night.

Because if you catch an invariant violation and carry on, you are telling the system: “I know this step didn't succeed, but I'll pretend it did, then use that false state for everything that follows.” Those 24 hours of silent failure were a consequence of that thinking.

Why Database Invariant Violations Are Especially Dangerous

Of the three kinds of exceptions, the third is the easiest to handle badly and the easiest to swallow.

There are three reasons:

First, it looks like an ordinary failure. To the caller, an IntegrityError is just an Exception raised by SQLAlchemy / asyncpg, much like a connection timeout. A generic except can catch it along with everything else.

Second, it usually happens in the middle of a transaction. You have already completed three steps when the fourth raises an invariant violation. If you catch it, the first three steps are still in the transaction, and you go on to commit. You have committed a transaction in an “intermediate state.” That state belongs to no state machine defined by the business logic. It is simply a hole in the data.

Third, it leaves the work “logically half-finished”. The process is still running, the endpoint hasn't returned a 500, and the user hasn't received an error. But the job is not done. This kind of unfinished work is the hardest to detect, because no signal tells you anything went wrong.

In this incident, our try block looked roughly like this:

# lift_expired_bans.py
async with session.begin():
    user.ban_status = 'lifted'
    user.ban_lifted_at = now()

    try:
        await session.execute(insert(Notification).values(
            user_id=user.id,
            type='account_unbanned',  # ← new value not allowed by CHECK
            ...
        ))
    except Exception as e:
        logger.error(f"failed to insert notification: {e}")
        # IntegrityError gets swallowed here

    # commit still happens — ban_lifted_at is written, notification is not

Whoever read this code at the time probably thought: “A notification failure shouldn't affect the main flow. The user's ban still needs to be lifted.” The instinct was right, but the implementation was wrong: one except caught every kind of exception, including invariant violations.

How to Write try/except Without Swallowing Invariant Violations

The right approach is to catch specific exceptions and handle each kind separately:

from sqlalchemy.exc import IntegrityError

async with session.begin():
    user.ban_status = 'lifted'
    user.ban_lifted_at = now()

    try:
        await session.execute(insert(Notification).values(...))
    except IntegrityError:
        # Invariant violation: my assumed schema does not match the actual DB
        # This is a critical bug; it must not be swallowed
        logger.critical(
            "BUG: notification CHECK violation on lift_expired_bans, "
            "type='account_unbanned' missing from DB enum?"
        )
        raise  # ← roll back the transaction, notify monitoring, get me paged
    except (httpx.TimeoutException, redis.ConnectionError) as e:
        # System-level exception: an infrastructure problem
        # Fall back to a retry queue for the notification
        logger.warning(f"notification delivery degraded: {e}")
        await enqueue_for_retry(...)

A few things to notice:

· Catch specific exceptions. Write explicit exception classes, not except Exception. This forces you to think through how each kind should be handled.

· logger.critical, not logger.error. Most teams' alert thresholds start at critical. Errors only turn up in grep; they don't page anyone.

· raise, not continue. An invariant violation must roll back the transaction. A half-finished state is worse than “nothing was done.”

· Explain clearly in the comments why this kind of exception must cause a crash. Six months later, when another engineer (or an AI) wants to “helpfully” add a catch-all except, they can read the explanation.

Build Exception Classification into CI

After the incident, I did three things.

First, I added a CI rule: every code path that INSERTs into a table with a CHECK constraint must have a corresponding enum test. The rule automatically diffs “the enum values the code assumes” against “the enum values the DB actually allows.” Any mismatch fails the build.

The implementation is a pytest test:

# tests/test_notification_types_align.py

CODE_TYPES = {  # all type strings found by grepping the code
    'rental_ended', 'account_unbanned', 'order_completed', ...
}

def test_notification_types_match_db_check():
    db_types = await fetch_check_constraint_values('notifications', 'type')
    missing = CODE_TYPES - db_types
    assert not missing, (
        f"code uses these types but DB CHECK forbids: {missing}"
    )

Second, I marked every except Exception in the library for review, using a simple system of grep and comment annotations. It is not a ban; it requires an explicit explanation of why the catch is so broad here.

Third, I added a feedback memory entry to CLAUDE.md:

Rule:  Invariant violations (IntegrityError / NOT NULL / CHECK / FK violation)
       must never be swallowed. Always use logger.critical + raise.

Why:   On 2026-04-27, a try/except swallowed an IntegrityError. The main flow
       committed, but notifications never reached the DB. For 24h, 80+ users
       got no notification and found out only by refreshing the page themselves.

How:   Applies to all SQLAlchemy / asyncpg / psycopg paths.
       Does not apply to business ValidationError / third-party API 4xx
       (those are valid “business failures”; catch them and show a user message).

With this memory entry in place, any agent changing similar code next time will be reminded of the rule. The incident record is always more important than the code itself.

Exception Handling Is Design, Not a Fix

This incident made me rethink something: most of the time, we treat exception handling as a “fix.” Something goes wrong, we wrap it in a try/except, and we're done.

But good exception handling is design. Before writing the code, you should already have decided:

· How many kinds of exceptions will this function face?

· What does each kind mean? (A business setback / an infrastructure hiccup / a wrong assumption in the code)

· Who should see each kind? (The user / the on-call engineer / me in the middle of the night)

· How severe should each kind be? (info / warning / error / critical)

· What state should the system be in afterward? (rollback / partial commit + queued retry / fail-fast)

Only after those five questions are answered can try/except take a sensible shape. Without answers, it is just a reflex: “Put a try around it, in case something goes wrong.”

A reflex is a reaction, not a design. Design means deciding in advance what happens to each kind of exception.

Closing Thoughts: Silent Failure Is the Most Expensive Failure

If a system breaks and returns a 500, that is a cheap failure. You know immediately, fix it immediately, and review the incident within an hour.

If a system breaks and stays silent for 24 hours, that is an expensive failure. You don't know anything is wrong, and neither do the users. By the time you discover it, 80 events have piled up. The review requires 24 hours of logs, and even after the fix, you still have rollback and compensating work to do.

The difference between those two failures is often a single line inside a try/except.

So the next time you write a try/except, think it through: What kind of exception am I catching? Could catching it silence an alarm that ought to sound?

If it would, don't catch it. Let it crash.

A crash is the best alert.

Translation log: translated from the Chinese original by Codex on 2026-11-03, 17:49–17:51 (US Eastern). Main thread GPT-6 Astra (gpt-6-astra), reasoning effort ultra; it spawned 1 sub-thread: 1 on GPT-6 Astra (effort ultra). Together they read 325,544 tokens (297,344 of them from cache) and wrote 7,063 (976 of them reasoning).