In March 2026, a thermal event at an AWS data center in Virginia caused power loss that took out a chunk of US-East-1. EC2 instances went dark. EBS volumes became unavailable. For a lot of teams, that meant their production workloads simply stopped.
We felt it. Not catastrophically - I'll explain why in a minute - but enough to trigger our incident process and spend a stressful few hours validating that our DaaS sessions were healthy. It was a useful reminder that the cloud is, at the end of the day, someone else's building full of servers. And buildings have problems.
This was predicted
Forrester put out a prediction earlier this year that we'd see at least two major multi-day hyperscaler outages in 2026. Their reasoning was straightforward: the cloud providers are pouring massive resources into AI infrastructure buildouts, and legacy systems - the stuff that runs most enterprise workloads - are getting less investment and attention as a result. More change, more complexity, same or fewer people watching the legacy infrastructure. That's a recipe for incidents.
The March outage fit this pattern. And I'd be surprised if it's the last one this year.
Why DaaS outages hit different
When a backend API goes down, you get errors, maybe some queued requests, but users can usually do something else while you fix it. When virtual desktops go down, people literally cannot work. They're staring at a connection error where their entire workspace used to be. No email, no files, no apps. Just gone.
For Citrix DaaS customers, an outage in the region hosting their desktop sessions means thousands of employees are idle. That's a direct, calculable cost per minute. It's also a political problem - VPs start calling, the CEO wants to know why IT can't keep the lights on. The tolerance for downtime in DaaS is close to zero because the impact is so visible and so immediate.
This is what makes resilience planning for desktop workloads different from most cloud services. You can't fall back to a degraded mode. You're either serving desktops or you're not.
The real lesson isn't "go multi-cloud"
Every time there's a big cloud outage, the hot takes start flying. "This is why you need multi-cloud." "You should never be single-region." And yeah, sure. But that's not the actual lesson.
The actual lesson is simpler and more uncomfortable: most organizations have resilience plans that have never been tested. They have a DR document on Confluence that someone wrote two years ago. It describes a failover process that references services that have been renamed, scripts that no longer exist, and runbooks that assume an architecture from three migrations ago.
I've seen this across multiple teams. The plan exists on paper. Nobody has ever run it end to end. And when the outage happens, people scramble, and the plan turns out to be more of a rough suggestion than an executable procedure.
What actually works
Active-active, not just failover. The traditional DR model is active-passive: your workloads run in one region, and you have a standby in another that you switch to when things break. The problem is that the standby is, by definition, not handling real traffic. So you never really know if it works until you need it. Active-active - where both regions are serving real users all the time - means your "failover" is just routing more traffic to something that's already running and proven. It's more expensive, yes. But it eliminates the scariest part of DR, which is "will this thing actually boot up when I need it?"
For our DaaS infrastructure at Citrix, we've been pushing toward active-active across regions for exactly this reason. When US-East-1 had its moment in March, our sessions in US-West-2 kept serving without anyone noticing. The users who were pinned to East had a bad time, but the blast radius was contained. We're still working on making session placement smarter so fewer users are single-region, but the architecture is right.
Regular game days. Netflix popularized this with Chaos Monkey years ago, and most companies nodded along and then never did it. Actually killing infrastructure on purpose - in production, during business hours - is terrifying. It's also the only way to know if your systems handle failure correctly. We run game days quarterly. We pick a failure scenario, announce it (barely) in advance, and execute it. The first one was rough. The second one was better. By the fourth, the team handled it like a drill because it was a drill.
The key is making game days a scheduled, expected thing. Not a surprise. Not a punishment. A practice, like a fire drill. If your team groans when you bring it up, that's fine. They'll stop groaning when the real outage hits and they've done this before.
Treat your DR plan like code. Version it. Put it in a repo, not a wiki. Review changes. Test it. Run it through CI if you can - at minimum, validate that the scripts it references still exist and the endpoints it mentions still resolve. A DR plan that drifts out of date is worse than no plan at all, because it gives you false confidence.
I maintain our resilience runbooks as markdown files in a repo with the infrastructure code. Every PI, we review them as part of planning. If the architecture changed, the runbooks change. If a runbook references a manual step, we try to automate it. The goal is that anyone on the team can pick up the runbook during an incident and execute it without prior knowledge.
The TPM's role in all of this
Resilience work has a coordination problem. It crosses team boundaries - infra, platform, networking, application teams all have a piece of it. Nobody owns it end to end. Which means, left to its own, it doesn't happen. Each team assumes someone else is handling it, or it gets deprioritized in favor of feature work because there's no immediate customer asking for DR testing.
This is where I spend a meaningful chunk of my time as a TPM. Getting DR testing into PI planning as a first-class commitment, not a stretch goal. Running the cross-team coordination for game days. Making sure the resilience runbooks stay current. Tracking the gap between what our architecture supports and what our procedures actually exercise.
It's not glamorous work. Nobody gives you a promotion for running a successful game day. But when US-East-1 goes down and your DaaS sessions keep running, that's the payoff. The March outage was a stress test we didn't choose, but because we'd been doing the unglamorous work, we passed it. Barely, in some areas. But we passed it. And that's the whole point.