Avoiding disaster in the face of cloud chaos

The recent disruption in services illustrates the core issue: even major cloud regions can fail, and organisations that depend on one region, one cluster, or one database endpoint can lose customer-facing capability very quickly. On 20 October 2026, Google Cloud reported elevated errors and intermittent network failures affecting multiple services in its London europe-west2 region, with reports of impact on banks, retailers and other large enterprises.

What the outage exposes

While the headlines will focus on the provider, more useful questions could be asked about the customer side:

• Single points of failure: organisations that depend on one region, one cluster or one database endpoint can lose customer-facing capability very quickly.
• Failover that was never built: Google’s guidance to affected customers was to move workloads to another zone or region where possible. That is only an option if the second environment, the replicated data and the traffic routing already exist. It can’t be improvised in the middle of an incident.
• Unclear priorities: when everything is down at once, teams without agreed priorities struggle to decide what to restore first, and what can wait.

None of these are really technology failures. They are planning failures, and they are decided long before an outage, when a cloud environment is first designed and set up.

Resilience starts with planning, not technology

Building a resilient cloud environment isn’t about buying more infrastructure. It’s about a sequence of decisions made in the right order: what the business can’t afford to lose, how quickly it must recover, and how much it is prepared to invest to protect it. Only then do architecture and platform choices follow.

That is why the first step is to start with business impact, not technology.

Start with business impact, not technology

Before choosing a recovery model, businesses need to define two targets for every critical application:

• RTO (Recovery Time Objective): how quickly the service must be usable again.
• RPO (Recovery Point Objective): how much data loss is acceptable, measured in seconds, minutes or hours.

For example, a payments platform may require an RTO of minutes and an RPO near zero, while an internal reporting system may tolerate hours of downtime and a daily backup restore. These targets should determine the architecture—not the other way round.

When looking at business impact and the suitable response or posture, decision making teams should classify workloads into tiers:

Avoid the single-region trap

Availability zones (isolated data-centre locations within a cloud region, with independent power, networking and infrastructure) improve resilience against some local failures, but they do not remove regional risk. Google’s own guidance during the recent incident was for customers to move applications to another zone or region where possible.

At minimum, organisations should:

• Deploy production workloads across multiple availability zones.
• Use separate Kubernetes node pools and topology spread constraints so pods are not concentrated on one zone or node.
• Keep container images, Helm charts, Terraform/OpenTofu code, secrets and configuration in a second region or separate cloud account.
• Replicate object storage, databases and message queues across regions.
• Use global DNS, health checks and traffic steering to redirect users when a region degrades.

For Kubernetes, a practical baseline is:

• primary cluster in Region A
• standby cluster in Region B
• identical deployments through GitOps
• database replication continuously running to the secondary.

On failover, the database replica would be promoted, the standby cluster scaled up, and traffic shifted through DNS or a global load balancer.

Choose the right recovery model

On the rare occasions where businesses need to invoke their disaster recovery plans, the chosen recovery process needs to meet the needs of their systems and their users. Recovery models can either run hot, run warm or run cold.

Run hot: active-active

Best for: payments, core commerce, customer-facing platforms where downtime directly costs revenue or trust.

Both regions serve live traffic. A global load balancer distributes requests, and the data layer is either multi-region/active-active or has a clearly designated writable primary with synchronously replicated copies.

Benefits:

• Near-immediate failover.
• Capacity is already running in the secondary region.
• Regular production traffic validates the secondary environment.

Trade-offs:

• Highest cost and operational complexity.
• Multi-master databases can create conflict-resolution and consistency challenges.
• Requires mature observability, automated failover and regular failover testing.

Active-active is generally appropriate only where the data model and operational maturity support it; for many enterprises, a controlled active-passive design is more realistic.

Run warm: standby with continuous replication

Best for: most Tier 1 and Tier 2 enterprise applications.

The primary region handles traffic. A secondary region runs a reduced but functional copy of the application (perhaps 20–50% of normal capacity) with database replicas, configuration and container images already deployed.

During an outage:

1. Confirm the incident and declare failover according to a pre-agreed decision matrix.
2. Stop writes to the impaired primary, where safe and possible.
3. Promote the database replica in the secondary region.
4. Scale Kubernetes deployments, node pools and autoscaling capacity.
5. Update DNS, load-balancer targets or service endpoints.
6. Run smoke tests: login, search, add-to-basket, checkout, payment authorisation and key integrations.
7. Communicate status to customers and internal stakeholders.

A warm standby can typically achieve a much lower RTO than rebuilding from backup, especially when database replication lag is monitored and kept within agreed limits.

Run cold: rebuild from code and backups

Best for: lower-tier systems, or as a last-resort recovery path for critical systems.

A cold site has no continuously running application capacity. Recovery depends on infrastructure-as-code, container registries, configuration management, database backups and documented runbooks.

The recovery sequence should be:

1. Provision network, compute and Kubernetes infrastructure in the recovery region or on-premises environment.
2. Restore secrets, certificates, DNS records and external integration configuration.
3. Deploy applications from GitOps repositories or immutable images.
4. Restore the latest consistent database backup.
5. Replay transaction logs or change-data-capture streams where available.
6. Validate data integrity and reconcile financial or order transactions.
7. Gradually reopen traffic, starting with read-only or limited-functionality modes if appropriate.

Cold recovery is cheaper, but it is slow and error-prone unless it is rehearsed. Kubernetes teams should maintain etcd snapshots, workload backups using tools such as Velero, and cross-region copies of backups; monthly restore drills are a sensible minimum.

Minimum viable service: degrade deliberately

A common mistake is treating every feature as essential. During a regional outage, it is often better to run a minimum viable application than to remain fully offline.

Examples:

• Allow customers to browse and place orders, but defer non-essential recommendations, personalisation or loyalty features.
• Accept orders and queue fulfilment events rather than requiring every downstream integration to be available.
• Enable read-only access to account information while writes are restricted.
• Serve cached product and content pages from a CDN or edge platform.
• Use a lightweight status page and customer communication channel that does not depend on the impaired cloud region.

This “reduced functionality mode” should be designed in advance, with feature flags, circuit breakers, queues and graceful degradation paths. The goal is to preserve the most valuable customer journeys rather than attempt a like-for-like recovery of every service.

Kubernetes resilience: more than one node

A single Kubernetes node is not a resilience strategy; it is a temporary survival tactic. If an organisation must run a minimal footprint, it should still separate control-plane availability from workload availability and ensure the node can be recreated automatically.

A more robust minimum configuration would include:

• At least three control-plane nodes across availability zones, or a managed Kubernetes control plane with regional redundancy.
• Multiple worker nodes across zones for production workloads.
• Pod anti-affinity and topology spread constraints.
• Persistent volumes replicated or backed up outside the failed zone.
• External database services, or database operators with tested primary/replica failover.
• Health probes, PodDisruptionBudgets, horizontal pod autoscaling and cluster autoscaling.
• GitOps-based redeployment so the entire cluster can be recreated from source control.

For critical systems, a single-node Kubernetes deployment may be acceptable only as a short-term degraded mode (for example, hosting a small order-intake service) provided its data is externally replicated and its loss does not prevent recovery. Multi-region Kubernetes patterns generally rely on separate clusters, global routing and a data layer that can be promoted or operated across regions.

Database migration and failover

The database is usually the hardest part of recovery. Applications can often be redeployed quickly; data consistency cannot be improvised.

Practical approaches include:

Cross-region read replicas: continuously replicate from primary to secondary; promote the replica during failover.

Managed global databases: use provider-native capabilities where RPO requirements are tight but understand their regional dependencies.

Streaming replication: for self-managed PostgreSQL, MySQL or similar databases, maintain a continuously synchronised standby.

Change data capture: stream changes to a secondary platform or data store when a full active-active database is not practical.

Object-storage backups: retain encrypted, versioned, cross-region backups, but do not treat them as the primary failover mechanism for Tier 1 systems.

Replication lag must be monitored continuously. A standby that is hours behind is not a credible failover target for a transactional system. Failover should be tested with controlled switchovers under production-like conditions, not merely documented.

On-premises DR: useful, but not automatic

On-premises recovery can reduce dependence on a single cloud region, but it introduces its own risks: capacity, hardware lead times, network connectivity, patching, security and operational skills.

It is most effective when used for a narrowly defined critical subset – for example, order capture, authentication, or a read-only catalogue – rather than attempting to recreate the entire cloud estate.

A pragmatic hybrid pattern is:

• Cloud-primary for normal operations.
• On-premises or secondary-cloud “break-glass” capability for the few services needed to trade, serve customers or maintain safety-critical operations.
• Regularly tested connectivity, DNS and certificate paths.
• Independent identity and access management, so a cloud identity outage does not lock teams out of recovery tooling.

What to do immediately after an outage

1. Declare an incident and establish a single incident commander and communication channel.
2. Assess customer impact by business journey, not by individual microservice.
3. Check dependencies: DNS, identity, APIs, payment gateways, message queues, CDN, monitoring and third-party SaaS.
4. Decide whether to fail over using pre-agreed criteria; avoid improvised architecture changes during an active incident.
5. Shift traffic only after data promotion and smoke tests.
6. Communicate early and honestly, including which functions are unavailable and what customers should do.
7. Capture evidence for the post-incident review: timelines, decisions, failed dependencies and gaps in runbooks.

Board-level actions

For leadership, the question is not “Can we survive any cloud outage?” but “Which customer promises must survive, and what are we prepared to pay to protect them?”

Practical commitments include:

• Fund multi-region or hybrid resilience for Tier 1 services.
• Set explicit RTO and RPO targets per service.
• Require quarterly failover tests, including database promotion and DNS cutover.
• Mandate infrastructure-as-code and GitOps for recoveryability.
• Review concentration risk across cloud, SaaS, DNS, identity and network providers.
• Treat resilience as a product feature, with investment prioritised alongside new functionality.

The recent Google Cloud disruption shows that regional cloud failures can rapidly affect banks, retailers and consumer services. Organisations that have already defined their critical journeys, replicated data, maintained recoverable infrastructure and rehearsed failover will be far better placed than those attempting to design resilience in the middle of an incident.

How DeeperThanBlue helps

Resilience is built before an incident, not during one. Our cloud team supports organisations across the full lifecycle:

• Multi-cloud expertise: we’re a trusted partner of Google Cloud, AWS, Microsoft Azure and IBM Cloud, and design hybrid and multi-cloud architectures
• Readiness assessments and secure architecture design: we review your infrastructure, dependencies and risks, then design environments with resilience and compliance built in
• Kubernetes depth: we’re one of only 195 CNCF-approved Kubernetes Certified Service Providers globally, with Certified Kubernetes Administrators on the team
• Infrastructure as code: Terraform and Vault consultancy so your estate can be rebuilt from source control
• Proactive, 24/7 support: managed services and monitoring with a dedicated Customer Success Team, backed by a Service Desk that runs 24/7 for clients who need it

The question for leadership isn’t “can we survive any outage?” but “which customer promises must survive, and what are we prepared to invest to protect them?”

• Share