A concrete, step-by-step playbook for rolling back a bad production deployment: confirming the problem, choosing revert vs. redeploy, executing safely, and verifying the rollback actually worked.
Confirm you actually need a rollback, not a smaller fix. A rollback is appropriate when: the current production version has a confirmed, user-impacting defect, and a fix isn't ready to ship immediately. It is not the first response to every alert — check whether the issue is actually caused by the most recent deployment (compare the timing of the deploy against the timing of the first error) before assuming a rollback is the right move; rolling back a deployment that wasn't actually the cause wastes time and can mask the real issue.
Have ready before you start: the last known-good commit hash or deployment version, and confirmation of who else (if anyone) needs to know a rollback is happening.
Confirm the last known-good state.
git log --oneline -10
Identify the commit hash immediately before the suspect deployment.
Choose your rollback method based on what's already merged:
main yet (still on a feature branch), simply don't merge it — no rollback needed, this is the easy case.git revert over resetting history — it creates a new commit undoing the change, preserving full history and triggering the normal deployment pipeline cleanly (see step 3).Revert the problematic commit(s).
git switch main
git pull --ff-only origin main
git revert <bad-commit-hash>
If the bad change was a merge commit, use git revert -m 1 <merge-commit-hash> instead (see the Git guide's section on reverting merges).
Push the revert to trigger the normal deployment pipeline.
git push origin main
This is deliberately the same deployment path as any other change — a rollback via CI/CD should not require a special manual deployment procedure; if it does, that's worth fixing separately once the immediate incident is resolved.
Watch the deployment complete through your normal pipeline visibility (Cloud Build logs, Cloud Run revision list, or equivalent), the same way you'd watch any other deployment.
git rev-parse HEAD and git rev-parse origin/main match, exactly as in the standard post-push verification habit — a rollback is not "done" until this is confirmed, the same as any other deployment.If reverting introduces a new problem (rare, but possible if other work depended on the reverted change), the same process applies in reverse: git revert the revert commit. This is why using revert rather than a hard history rewrite matters here — every step remains cleanly undoable.
Once the immediate incident is resolved, the underlying defect still needs a real fix — Root Cause Analysis: A Simple Framework covers how to make sure the eventual re-deployment doesn't reintroduce the same problem.
Part of: System Administrators, DevOps Beginners
A structured, practical course in Git and GitHub, taught through the actual workflow used to build and maintain a production Node.js site — not a shallow command dump.
CI/CD explained without the buzzwords — what continuous integration and continuous deployment actually automate, and what a real pipeline looks like end to end.