Google Cloud europe-west2 Incident Report: 6 Lessons for CABs
No affiliate links appear in this article. Every incident fact comes from Google’s own incident report and product documentation, linked inline and in Sources. This is our follow-up to the day-one timeline and zonal design analysis, written before Google published its root cause. That piece covers the live status updates, how each zonal and regional configuration is documented to behave, and what to do while a provider incident is in progress; this one covers what the report found.
Table of contents
- Why this matters
- What did the incident report find?
- Why was the maintenance posture the real risk?
- Why did failover fail for some customers?
- How long did each service take to recover?
- Which failure modes should your CAB check for?
- What is Google changing, and what isn’t it promising?
- What to do about it
- FAQ
- Sources
Why this matters
Google’s incident report for europe-west2, revised on November 4, is a change-management story on both sides of the shared responsibility line.
On Google’s side, a facility was put into a reduced-redundancy state for repairs, and a hidden configuration flaw turned one external cable fault into a full zone power loss. On the customer side, failover plans that looked fine on paper ran into capacity limits, slow failure detection and control-plane dependencies that crossed zone boundaries.
Google recommends deploying across multiple zones and multiple regions, and our day-one analysis compares how common zonal and regional configurations are documented to behave in a zone loss. The report shows where those documented behaviors held on October 20 and where they didn’t. For the cloud and SaaS reliability work most change boards own, the report gives six checks worth adding to review templates.
What did the incident report find?
The facility behind europe-west2-b normally runs a 2N power design with two utility feeds, a standby utility connection, about five minutes of UPS battery and a generator bank run by a dual-redundant programmable logic controller (PLC). Google gives this sequence:
- After problems found in a quarterly generator load test on October 14, feed B’s breaker was opened for repairs. The site ran in what Google calls a “temporary single-feed (1N) bridged maintenance posture” on feed A.
- At 06:53 BST on October 20, an underground cable fault outside the facility took out feed A.
- The PLC’s undervoltage sensors were wired upstream of the open feed B breaker. Feed B was still live on that side, so both PLCs saw “a healthy utility source” and never sent the generators a start signal.
- The UPS carried the load until the batteries ran out. Impact started at 07:00 BST.
- Staff moved the site to the standby utility source. Power to the data halls was back at 08:20 BST.
Then came the slow part. Storage devices with unclean shutdowns had to recover before Google would restore service, and responders “chose not to restore services that would otherwise make this worse.” Platform services and APIs were fully available at 12:11 BST, and the report puts total main impact at 6 hours 8 minutes.
Why was the maintenance posture the real risk?
The cable fault was outside Google’s control. The exposure that turned it into an outage was a planned state: a facility deliberately running with one feed while a known defect was repaired. That is a change, and it is the kind most change processes underweight because nothing is being deployed.
The PLC flaw only mattered once feed B was isolated, because the logic checked line-side voltage rather than voltage and breaker position. Redundancy-reducing windows are when latent assumptions get tested for real.
Google’s own fix makes the point. Among its data-center action items, under “Strengthen change management,” it froze planned maintenance at sites with similar topologies and added “elevated approvals and reviews for data center configuration changes which are outside their standard designs,” marked complete on October 30.
Few enterprises run data centers, but most run this pattern often:
- Draining one node of a two-node cluster for patching.
- Taking a replica offline for a storage migration.
- Running a firewall pair on one unit during a firmware upgrade.
- Pausing a disaster-recovery link to reseed it.
Each one leaves the next unrelated failure with no backstop. Our deployment rings guide covers blast radius for things you push; this incident is about time spent with less redundancy than your risk assessment assumes.
Why did failover fail for some customers?
This is the sentence most architects will remember. For Compute Engine, Google writes that zonal resources “require customers to provision cross-zone failover and recovery mechanisms,” and that “during this incident, however, some of these failover attempts were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism.”
Google’s disaster recovery architecture guide has said this for years. Automatic failover “is subject to availability of resources in active zones in the same region,” and teams using zonal resources must do their own “designing, building, and testing failover and recovery.” The report shows what that clause looks like at scale. Google says the “sudden influx of redirected query executions, high-availability database failovers, and compute workloads saturated available compute slots” in zones a and c, producing admission queues and intermittent 500 and 503 errors across the region.
There are two remedies Google documents, and both cost money. Compute Engine reservations give “a high level of assurance of capacity,” but you pay the on-demand rate for reserved resources “regardless of whether or not its resources are being used.” For GKE, Google’s regional clusters page suggests overprovisioning: a three-zone cluster at 150% can route all traffic to the surviving zones if one zone is lost. Notably, Google says GKE engineers “adjusted scheduler allocations and capacity limits across unimpacted infrastructure to prioritize active customer reservations.” Reserved capacity got priority.
How long did each service take to recover?
We built the chart below from the times and durations in Google’s report. Power to the data halls was out for 80 minutes. The managed services that are designed to survive a zone loss took far longer to fully recover.
Google gives Apigee X as a duration and Looker as an end time only, so both bars start at the power loss. The Cloud SQL bar shows the tail: Google says a majority of affected HA instances recovered by 09:00 BST, with “a small number” not fully back until 19:00 BST.
The lesson for recovery objectives is plain. If your RTO assumes a zonal event lasts as long as the physical fault, it is too short. Storage integrity checks, controlled traffic re-admission and backlog draining ran for hours after the power came back.
Which failure modes should your CAB check for?
The report’s per-product sections describe distinct failure modes. We grouped them into a table, with the customer-side check each one suggests.
| Failure mode | What Google reported | Customer-side check |
|---|---|---|
| Capacity contention | Some Compute Engine failovers failed on capacity; GKE scale-ups in zones a and c “impacted by available capacity” | Reservations or overprovisioning sized for N-1 zones, reviewed at each capacity change |
| Slow failure detection | AlloyDB regional primaries had “up to a 95-minute delay” starting failover due to delayed notifications | Your own zonal health alarms; don’t rely only on provider-triggered failover |
| Storage-layer stalls | Cloud SQL HA instances on failed Regional Persistent Disk waited for disk recovery; about 4% of AlloyDB regional instances needed manual recovery | Test failover with the disk layer impaired, not just the VM |
| Replica placement drift | Apigee X had two of three datastore replicas in the failed zone after “a scheduling misconfiguration,” so it lost quorum | Periodic placement audit: replicas per zone, not just replica count |
| Cross-zone control-plane dependency | Regional control planes timed out calling metadata in the offline zone; BigQuery had an autoscaling dependency in a data plane job | Map which control-plane calls your failover needs, and whether they work with one zone gone |
| Bulk node loss | Apigee Edge’s node replacement “didn’t handle multiple nodes going down simultaneously”; some VMs booted an older binary when OS repository calls timed out | Pin boot-time dependencies; test multi-node loss, not single-node |
The placement-drift row deserves special attention because it applies to almost every Kubernetes estate. The Kubernetes topology spread documentation is explicit: “There’s no guarantee that the constraints remain satisfied when Pods are removed,” and scaling down can leave an imbalanced distribution. A design that is spread across three zones on day one can quietly collapse into two. Placement needs to be checked as an operating control, not assumed from the original design review.
The storage rows matter if you plan around documented failover times. Cloud SQL’s HA page says to expect about sixty seconds of unavailability, and the Regional Persistent Disk guide describes force-attach in “O(seconds).” Those held for many instances, but not for disks caught in what Google called “sequencing atypical of previous zonal outages.”
What is Google changing, and what isn’t it promising?
Google’s action list is specific on the facility side:
- The utility is repairing feed A, estimated November 13.
- Both feeds return and full generator load transfer tests follow, estimated November 17.
- A comprehensive PLC code review was due November 6.
- PLC configurations at sites with similar topologies are being validated, estimated November 15.
- The emergency procedure is now practiced twice per shift.
On the platform side, Google says it is expanding “regular disaster-recovery testing for full cluster power-loss scenarios,” adding detection signals for Regional Persistent Disk, and having Apigee X enforce three replicas in three different zones.
What the report doesn’t do is promise spare regional capacity for zonal failover. The capacity commitment remains the documented one: reservations are what Google prioritized, and the DR guide still says failover depends on “availability of resources in active zones.” The report also says Google “did not identify any impact of the outage outside of the europe-west2 region,” which is the strongest argument in it for multi-region designs on critical paths. We covered how to test those in cross-region failover testing.
If your change records cite this report, keep a dated copy; our guide to retaining provider incident evidence explains why.
What to do about it
- Treat redundancy-reducing maintenance as a risk-rated change. Any change that leaves a service at N rather than N+1 (one feed, one node, one firewall, a paused replica) should name the window, the next failure it can’t absorb, and who can restore redundancy fast. Score it with the change risk score tool rather than treating it as routine.
- Size failover capacity for a lost zone, and pay for it where it matters. For Tier 0 workloads, use reservations or overprovisioning sized for N-1 zones, and record the cost decision in the change record.
- Add a placement audit to the quarterly control set. Count replicas per zone for databases, quorum systems and stateful sets. A design reviewed once can drift with every scale event.
- Test failover with the storage layer impaired. Game days that kill a VM but leave its disk healthy don’t reproduce October 20. Include a case where the regional disk or database failover stalls and someone has to recover by hand.
- Set RTOs that include recovery tail, not just the fault. Use this report’s 80-minute fault and 5-to-12-hour recovery as a reference point when you challenge an RTO that assumes a zone is back when its power is.
- Watch provider status alongside your own alarms. AlloyDB failovers started up to 95 minutes late. Your own zonal health checks plus our cloud status heatmap give the CAB a second signal. Put provider incidents on the standing CAB agenda, and re-check Google’s open actions after November 17.
FAQ
What caused the Google Cloud europe-west2 outage on October 20, 2026?
An external underground cable fault cut the facility’s only active utility feed while the second feed was isolated for repairs. A PLC misconfiguration read live voltage upstream of the open breaker as a healthy source, so generators never started. UPS batteries ran down and the zone lost power at 07:00 BST.
How long did the europe-west2 outage last?
Google’s report gives 6 hours 8 minutes of main impact, from 07:00 to 13:08 BST. Data hall power was back at 08:20 BST, but storage recovery and controlled re-admission took hours. Some Cloud SQL HA instances weren’t fully recovered until 19:00 BST. The incident page’s first-day figure of 7 hours 52 minutes, which appears in early coverage, has been superseded.
Why did multi-zone workloads fail during a single-zone outage?
Google names two causes: demand surges on the surviving zones a and c, and control-plane dependencies across zone boundaries. Some Compute Engine failovers failed on capacity, and Apigee X lost quorum because two of three replicas sat in the failed zone.
Does Google guarantee capacity for zonal failover?
No. Google’s disaster recovery guide says automatic failover is subject to resource availability in the remaining zones. Compute Engine reservations provide a high level of capacity assurance, but you pay for reserved resources whether you use them or not.
Was this a change-management failure?
Partly. The trigger was external, but the outage happened during a planned single-feed maintenance posture, and Google’s remediation includes freezing similar maintenance and adding elevated approvals for non-standard data center configuration changes.
What should a CAB do after reading the report?
Rate redundancy-reducing maintenance as a real risk, size failover capacity for a lost zone, audit replica placement each quarter, test failover with storage impaired, and set RTOs that include the recovery tail.
Sources
- Google Cloud Service Health: europe-west2 incident report (revised November 4, 2026)
- Google Cloud: Regions and zones
- Google Cloud: Architecting disaster recovery for cloud infrastructure outages
- Google Cloud: About Compute Engine reservations
- Google Cloud: GKE regional clusters
- Google Cloud: Build HA services using synchronously replicated disks
- Google Cloud: About high availability (Cloud SQL for MySQL)
- Kubernetes: Pod Topology Spread Constraints
Published November 11, 2026. Last updated November 13, 2026.