Technical Program Manager – System Design

Global resource scaling in AWS: How would you design a multi-region deployment architecture to route and fail over high-throughput traffic for an AWS foundational service during a regional availability-zone outage?

A regional outage should end in an automated re-route, not an incident bridge. Because AWS's failure hierarchy is Region → Availability Zone, spreading compute across AZs only survives a single-AZ failure — genuine resilience against a whole-region outage requires a second, independent region, kept continuously in sync and reachable through health-check-driven global traffic steering.

  1. 1.Assess & classify the failure domains — separate single-AZ failures from whole-region failures, pin down the RTO/RPO targets the business actually requires, and baseline peak throughput and every downstream dependent.
  2. 2.Design the multi-region topology — default to active-passive: Region A serves 100% of production traffic while Region B runs a fully-provisioned, continuously-replicated standby, chosen far enough away to sit on an independent power grid and network backbone.
  3. 3.Build the global traffic-steering layer — configure an Amazon Route 53 Failover routing policy with Region A as primary, an Elastic Load Balancer distributing traffic within each region across its Availability Zones, and DNS TTLs tuned to bound the failover window against the RTO target.
  4. 4.Automate health-based failover detection — probe both regions with Route 53 Health Checks from multiple AWS edge locations, require a sustained run of failures before Route 53 acts, and page on-call via CloudWatch Alarms and SNS in parallel with the automated re-route.
  5. 5.Validate failover through regular game-day testing — simulate regional outages and measure actual failover time against the RTO target, confirm Region B's Aurora Global Database replica matches Region A's primary within the RPO target, and treat every game day as a test of the written runbook.
  6. 6.Execute failback & post-incident stabilization — resync Region A from Region B before sending any traffic back, shift traffic back gradually via Route 53 weighted routing, and feed the real failover/failback timings back into the runbook and game-day script.

This holds up for a high-throughput foundational service specifically because the decision of where traffic goes is driven entirely by an automated, independently-verified health signal — never by a human under pressure mid-outage; the human's job is validating the failover and deciding when it's safe to fail back, not making the call itself.