There is a specific kind of message that only arrives when you own code that moves money. It comes at a bad hour. It is short. Someone, somewhere, tried to do something that matters to them, and your code said no. Or worse, said yes twice.
I live alone in KL, so when the phone lights up at that hour, there is no one to turn to and say “can you believe this”. It is just me, the notification, and the slow cold understanding that the cause has my name on it. The first time it happened, I understood my job differently by morning.
Before the message
Before you have received the message, engineering is mostly a craft problem. Clean code, nice abstractions, tests that pass. You optimize for the review, for the merge, for the satisfying green checkmark. Failure is theoretical. It lives in a unit test called shouldHandleError.
You ship on Friday because the code looked right and the tests agreed. Confidence comes from the work you did, which sounds healthy until you notice what it means: your confidence has no input from reality yet.
Two cold sweats, documented
The message does not always come from other people. Sometimes you send it to yourself. Two real ones from my own infrastructure, both from this year, both fixed, both instructive.
The secrets in the layers. While hardening this website’s Docker build, I went looking for how the environment file got into the image. The answer was worse than expected: a plain COPY . . had been quietly baking it into an image layer, and anything in a layer is recoverable by anyone who can pull the image, with a single docker history or docker save. The build was green. The site worked. Nothing anywhere hinted that credentials were sitting in an archive format whose whole purpose is to remember things. The fix was BuildKit secret mounts, which exist for exactly this: the file is present for one build step and never becomes a layer. But the lesson that stayed was about the silence. The dangerous state produced no error, no warning, no symptom. Some classes of bug you only find by going looking, on a quiet afternoon, before the bad hour finds them for you.
The version I could not name. During one deploy debugging session I realised I could not answer a basic question with certainty: which commit is the running container actually built from? The site said nothing. The container said nothing. I was reasoning about production behavior from what I believed I had shipped. Belief is not an input you want in an incident. So the deploy now bakes the git commit into the image at build time, the admin dashboard displays exactly what is serving, and the deploy script hard-fails if the running container’s commit does not match what was just built. A broken identity chain cannot ship silently anymore. The 2am question “what is actually running” now has a lookup, not a guess.
What it builds in you
You stop trusting a green pipeline as proof of anything except that the pipeline is green. You start asking what happens when this fails, not if. And you learn the deepest lesson of money-path code: silence is not success. A payment flow that throws an error is having a good day. The bad day is the one where everything returns fine and a number is quietly wrong somewhere, compounding, until a reconciliation job or an angry customer finds it. The scariest issues I have dealt with never crashed anything.
The habits compound. Deploys that keep the old version running when the new one fails to start. Migrations written so running them twice hurts nothing. Alarms on the absence of activity, not just the presence of errors, because the quietest graph is sometimes the loudest warning. And a personal rule that has never once been wrong: if a fix feels clever, sleep on it. Clever is what you call a thing before it wakes you up.
You can usually tell, working with an engineer, whether they have received the message yet. It is not about skill or seniority. It shows up in small places: the extra question in the design review, the rollback plan nobody asked for, the refusal to say “done” before touching the thing in production with their own hands.
I would not wish the message on anyone. I also would not trade what it taught me. Some lessons only arrive at 2am, addressed personally to you.