A practical, repeatable framework for finding a real root cause instead of stopping at the first plausible explanation — the Five Whys, contributing-vs-root-cause, and writing a blame-free postmortem.
The most common failure in incident analysis isn't a lack of effort — it's stopping at the first explanation that sounds plausible ("the server ran out of memory") without asking why it ran out of memory, and why that condition wasn't caught sooner. A fix aimed at a contributing factor instead of the actual root cause tends to produce the same incident again later, in a slightly different shape.
The difference between a root cause and a contributing factor, the "Five Whys" technique for getting past the first plausible answer, and how to write up a finding without turning it into blame.
A contributing factor made an incident worse, more likely, or harder to detect — but removing it alone wouldn't have prevented the incident entirely. A root cause is the underlying condition that, if it had been different, the incident would not have happened at all. Real incidents almost always have several contributing factors and one (sometimes two) genuine root causes. Treating a contributing factor as if it were the root cause is how the same category of incident quietly recurs months later.
Starting from the observed symptom, ask "why" repeatedly, each time about the answer just given, until you reach something you can actually act on:
Five is a guideline, not a rule — stop when you reach a cause that is genuinely actionable ("update the configuration and add a check that would have caught this class of mismatch"), not necessarily on exactly the fifth question.
"Someone forgot to update the config" is rarely a useful stopping point — it identifies a person, not a fixable system condition, and offers no way to prevent a recurrence. Push one level further: why was it possible for a config mismatch to go unnoticed until production broke? That question usually leads somewhere genuinely fixable (a missing validation check, no automated environment-parity test, a deployment process with no built-in review step).
A finding should describe the sequence of events and the systemic conditions that allowed them, written so that naming the specific person involved wouldn't change the recommended fix at all. Avoid "X forgot to Y" phrasing entirely — write "the process allowed Y to be skipped without a check" instead. This isn't about protecting feelings for its own sake; a blame-free writeup is what actually gets people to report near-misses honestly next time, which is where most future incidents get caught early.
Once you've identified a genuine root cause, the practical next question is often "how do we prevent this from being possible again" — for deployment-related root causes specifically, see Writing Idempotent Deployment Scripts. For the investigation phase that comes before root-cause analysis, see Isolating "Is It the Network or the App?".
Part of: Support Engineers, System Administrators, Technical Leads