On the Coverage of a RotationA password is not a value · It is a set of contracts

On Tuesday afternoon, April 15, 2026, I did a routine password rotation. It took five minutes, and I thought, "That's done." Half an hour later, Discord alerts started arriving one after another.

The first: a container in data-services failed its health check. "auth failed." I assumed it was an isolated hiccup and restarted it. The second: all of Umami's website analytics went down. "Cannot connect to database." The third: Infisical's own admin panel started returning 401s. The fourth: I SSHed into VM103 to take a look with docker logs and found an old container started with docker-run stuck in a reconnect loop too.

The reason was simple, and discouraging: what I had changed was not "a" password, but a contract — and there were 28 signatories on the other side of that contract. I had notified only one.

A password is not configuration; it is a subscription relationship

This used to be my mental model of a password:

A password is a string. It lives in .env (or a secrets manager), and the application reads it at startup to connect to the database. Changing the password means changing the string + restarting the application. Done.

That model works for a monolithic application. One .env, one process, a one-to-one relationship.

But for a system that has grown even a few branches, it is way off the mark. Because the reality looks like this:

· The main backend's password: in infra/.env, injected by docker-compose at startup

· The individual docker-compose configurations for the 25 data-services containers: each reads from data-services/.env

· Umami analytics: runs in another stack, with its own .env

· Infisical: keeps its own copy because it uses the same database for its audit log

· An old script started with docker-run in 2024: the password is hardcoded in the startup command, absent from every .env

· lib/db_config.py uses getenv, but its default value also contains a hardcoded copy for the development environment

· My own ~/.pgpass

· DBeaver's connection settings

These are 8 independent "consumers." Each has cached a copy of the password. Changing the password = signing a new contract with each of these 8 parties.

That time, I only changed infra/.env + restarted the backend. The other 7 parties were still using the old password. The moment it became invalid, all 7 went down.

Finding every consumer is the real problem in rotation

After the incident, I realized that the hard part of password rotation was never how to change it. The change takes 1 line of SQL.

The hard part is how to find every consumer. This is a distributed auditing problem.

I made a list of all the places a password might be hiding in the system:

· Explicit configuration: .env / .env.production / the environment section in docker-compose.yml / Kubernetes secrets / Infisical / Vault — these are easy to find; grep for keywords and list the files

· Implicit configuration: -e DB_PASSWORD=xxx in a docker-run startup command / Environment= in a systemd unit / strings embedded in cron jobs — digging these out takes ps aux | grep + systemctl show + crontab -l

· Hardcoded values in code: string literals / dict defaults / argparse defaults / test fixtures — these require grepping the entire repository

· Client caches: the SDK gets a copy at startup and keeps using it until the process restarts. If the SDK does not use lazy reload, changing .env does nothing — the process has to restart

· Copies in people's heads: my ~/.pgpass / a teammate's DBeaver / a temporary SSH tunnel session — these can only be addressed by telling people

Five categories, each requiring an audit. Miss one, and the rotation is incomplete.

The evolution of rotate-passwords.sh

My rotate-passwords.sh started out like this:

#!/bin/bash
NEW_PW=$(openssl rand -hex 32)
psql -c "ALTER USER ggboys WITH PASSWORD '$NEW_PW'"
sed -i "s/DB_PASSWORD=.*/DB_PASSWORD=$NEW_PW/" /infra/.env
docker-compose restart backend
echo "done"

Four lines. It looked very professional. This was the version that caused the incident.

I rewrote it after the incident. The current version is more than 200 lines long and does the following:

1. Generate a new password, but do not change the database yet.

2. Scan the entire filesystem for .env / .env.* / docker-compose*.yml / any configuration file containing the keyword DB_PASSWORD. List them for me to confirm.

3. Scan ps aux for embedded DB_PASSWORD= strings in all docker-run / systemd / cron tasks.

4. Scan git repositories for hardcoded passwords (within limits, mostly legacy remnants).

5. List every "location to update" for confirmation. I review the list and check that nothing is missing.

6. Update the database password + all configuration files + Infisical at the same time + restart all affected containers/services.

7. Run a connection test — connect to the database with the new password to confirm that the rotation succeeded.

8. Monitor health checks for all affected services for 5 minutes. Roll back immediately if any of them turns red.

This process turns "changing a password" from an action into an atomic event. Either all consumers switch together, or none of them do.

A rotation is a saga

By this point, you have probably noticed — password rotation has the same shape as a saga:

· Multiple independent services/systems need to be updated

· A failure in any one of them requires compensation (roll everything back)

· The intermediate state is unsafe (half on the new password and half on the old = 7 services down)

· An orchestrator has to actively move the process forward; a cron job relying on "it'll be consistent eventually" cannot be the fallback

This is why rotate-passwords.sh looks more and more like a saga engine as I work on it. Every step needs an explicit forward + compensating action. It cannot allow a state of "oh, I changed this one and forgot that one."

At a deeper level — a password is not configuration; it is a distributed contract. In essence, it is no different from "an order spanning multiple services." The contract concerns authentication between services, rather than payments between users.

Three failures + three better approaches

Failure 1: after changing the password, only the main backend was restarted. The secondary services were still using the old password and went down 30 minutes later.

Failure 2: sed was used to change .env, but someone forgot that environment variables passed inline when starting docker-compose take precedence. The .env file changed, but the container still read the old inline value.

Failure 3: .env was changed, but the container was not restarted (the SDK caches the value within the process). The problem only appeared "suddenly" when the container later restarted on its own. By then, several days had passed, and no one remembered the connection to the password rotation.

Better approach 1: use a secrets manager (Infisical / Vault) for dynamic retrieval, with an SDK that supports lazy reload, so changing the password does not require restarting the process. This is the ultimate solution, but the integration cost is high.

Better approach 2: bring all password consumers onto one library, such as lib/db_config.py. Changing the password then only requires changing the secrets source; all consumers pick it up automatically through the library. This is the middle option, and the one we use now.

Better approach 3: build a "consumer discovery + update all + verify all" pipeline into rotate-passwords.sh. This is the fallback option, suited to systems that have yet to consolidate around a secrets manager.

Closing — check the blast radius before any operation

The lesson I took from this incident went beyond "be careful with password rotation." It was a more general discipline:

Any operation that looks "local" may be global in a distributed system.

· Changing a schema column? — Check which services read the table / which ORM models have that field / which caches hold that row.

· Deleting an API endpoint? — Check which clients (web / app / third-party plugins) call it.

· Shutting down a container? — Check who is still writing to its mounted volume / who is polling its health.

· Changing a password? — Check the 8 categories of consumers above.

Before every "operation," ask how large the blast radius is. If you cannot answer, the operation is not ready to be carried out.

This, sagas (№ 006), and exception classification (№ 010) are different sides of the same thing — in a distributed system, there are no "atomic operations," only sagas that need an orchestrator to move them forward. The atomicity a database transaction gives you ends at the boundary between services.

We have always thought we were doing "engineering," but most of our time is actually spent on contract management — who has agreed to what with whom, and how a change in one party's state should be communicated to everyone involved. Password rotation is merely the smallest of these contracts.

Translation log: translated from the Chinese original by Codex on 2026-11-03, 17:48–17:51 (US Eastern). Main thread GPT-6 Astra (gpt-6-astra), reasoning effort ultra; it spawned 1 sub-thread: 1 on GPT-6 Astra (effort ultra). Together they read 321,830 tokens (298,112 of them from cache) and wrote 6,044 (659 of them reasoning).