There is a persistent myth in tech that the best way to handle a production incident is to throw everyone into a conference bridge, yell about dashboards for two hours, and hope someone figures it out. The "war room" approach sounds decisive. It feels like action. But more often than not, it is the opposite of effective incident response. It is chaos wearing the costume of urgency.
I have managed incidents across cloud infrastructure teams for years, most recently at Citrix where our DaaS platform supports hundreds of thousands of virtual desktops for enterprise customers. When something breaks at that scale, the pressure is real. People's entire workdays - their applications, their files, their ability to do their jobs - depend on the service staying up. That pressure makes it tempting to pull the war room lever. But I have learned, sometimes the hard way, that structure beats adrenaline every single time.
The War Room Problem
War rooms fail for predictable reasons. You get fifteen people on a call. Half of them do not know why they are there. Three people are talking over each other about different hypotheses. Someone is sharing their screen but nobody can see the relevant data because the dashboard is zoomed to the wrong time range. Meanwhile, the person who actually knows what is going on is quietly debugging in a terminal, unable to focus because someone keeps asking them for a status update every ninety seconds.
The fundamental issue is that war rooms confuse presence with contribution. Having more people in the room does not make the incident resolve faster. It usually makes it resolve slower, because the people doing the real work are constantly interrupted by the people who need to feel involved. Information overload sets in quickly. Nobody is sure who is making decisions, who is communicating to stakeholders, or who is actually investigating the root cause. You end up with duplicated effort, conflicting instructions, and a resolution time that balloons well past what it should have been.
Structure Over Spectacle
What works better is a structured incident response model with clearly defined roles. At minimum, you need three: an incident commander, a communications lead, and one or more technical leads. The incident commander owns the process. They do not need to be the most senior engineer in the room - they need to be the person who can keep things organized, make sure the right questions are being asked, and drive toward decisions. The communications lead handles all outbound updates: status pages, executive notifications, customer-facing messaging. The technical leads own the investigation and remediation work.
This separation of concerns is critical. When the same person is trying to debug a failing service while also fielding Slack messages from a VP asking "what is the customer impact," both tasks suffer. The debugging gets interrupted and the VP gets a half-baked answer. Split those responsibilities and suddenly both get done well.
At Citrix, our DaaS incidents follow this model closely. When a control plane issue threatens desktop availability for a large customer segment, we cannot afford confusion. A hundred thousand virtual desktops going dark means a hundred thousand people staring at a loading screen instead of doing their work. The incident commander establishes a timeline, assigns investigation tracks, and gates all communication through the comms lead. Technical leads work their respective areas - networking, broker services, hypervisor layer - and report findings back through a single channel rather than cross-talking.
Where the TPM Fits
As a TPM, my role during an incident is not to fix the code. I am not going to be the one tracing a memory leak in a microservice or rolling back a bad deployment. My job is to make sure the right people are talking to each other, that status is flowing in the right direction, and that decisions actually get made instead of debated endlessly.
That sounds simple, but in practice it is one of the hardest parts of incident response. Engineers under stress tend to go heads-down. They stop communicating. They pursue their own theory without checking whether someone else already ruled it out. The TPM's job is to maintain the connective tissue - asking "has anyone looked at the recent config change in region X?" or "do we have confirmation that the rollback actually took effect?" These are not technical contributions in the traditional sense, but they routinely shave thirty minutes or more off resolution times.
I also track the decision log during incidents. Who decided to fail over to the secondary region, and at what time? Who approved the customer notification? When did we first see the anomaly in monitoring? This log is not bureaucracy for its own sake. It is the raw material for the post-incident review, and it saves hours of reconstruction after the fact.
Post-Incident Reviews Are Where the Real Work Happens
Here is an opinion that I will stand behind: the post-incident review matters more than the incident itself. Resolving the immediate problem is necessary, obviously. But if you do not learn from it - really learn, not just write a document that nobody reads - you are going to have the same incident again in six months.
Good post-incident reviews are blameless, thorough, and action-oriented. "The on-call engineer should have caught it sooner" is not an action item. "Add a monitor that alerts when broker connection latency exceeds 500ms for three consecutive minutes" is an action item. The difference matters enormously. One creates a culture where people hide mistakes. The other creates a culture where every failure makes the system stronger.
I push hard for follow-through on post-incident action items. It is easy to write them up in the heat of the moment and then let them languish in a backlog. The TPM's role here is accountability - tracking those items through sprint planning, making sure they get prioritized against feature work, and closing the loop weeks later to verify they actually shipped. Without that follow-through, the review is just theater.
Automation Is the Long Game
The best incident response is the one that never requires a human to wake up at 3 AM. That is the long game, and it is where automation comes in. Documented runbooks are the starting point - if your on-call engineer has to improvise every response from scratch, you have already lost. Runbooks capture institutional knowledge and turn it into repeatable procedures that anyone on the rotation can execute.
But runbooks are just step one. The real leverage comes from automating those runbooks entirely. Auto-scaling responses that kick in when traffic patterns shift. PagerDuty routing rules that page the right team based on the alerting service rather than blasting everyone. Automated rollback triggers when error rates cross a threshold. Each of these reduces the human coordination overhead during an incident, which means fewer people need to be involved, which means less chaos.
At Citrix, we have invested heavily in this kind of automation for our cloud infrastructure. Auto-remediation handles a meaningful percentage of issues that used to require manual intervention. That does not mean we never have incidents - we absolutely do. But the incidents that reach a human are the genuinely hard ones, the novel failures that automated systems have not seen before. And for those, structured response with clear roles beats a war room full of panicking engineers every time.
Incident management is not about heroics. It is about preparation, structure, and the discipline to learn from every failure. Get those right, and the war room becomes unnecessary.