Technical Program Manager – System Design

Global resource scaling: How would you design a multi-region deployment architecture to route and fail over high-throughput traffic for an OCI foundational service during a regional availability-zone outage?

A regional outage should end in an automated re-route, not an incident bridge. Because OCI's real failure hierarchy is Region → Availability Domain → Fault Domain, spreading compute across ADs and FDs only survives a partial failure — genuine resilience against a whole-region outage requires a second, independent region, kept continuously in sync and reachable through health-check-driven global traffic steering.

  1. 1.Assess & classify the failure domains — separate single-AD/FD failures from whole-region failures, pin down the RTO/RPO targets the business actually requires, and baseline peak throughput and every downstream dependent.
  2. 2.Design the multi-region topology — default to active-passive: Region A serves 100% of production traffic while Region B runs a fully-provisioned, continuously-replicated standby, chosen far enough away to sit on an independent power grid and network backbone.
  3. 3.Build the global traffic-steering layer — configure an OCI Traffic Management Steering Failover policy at the DNS layer, with an OCI Load Balancer distributing traffic within each region and DNS TTLs tuned to bound the failover window against the RTO target.
  4. 4.Automate health-based failover detection — probe both regions with OCI Health Checks from multiple external vantage points, require a sustained run of failures before steering acts, and page on-call via OCI Notifications in parallel with the automated re-route.
  5. 5.Validate failover through regular game-day testing — simulate regional outages and measure actual failover time against the RTO target, confirm Region B's data matches Region A's within the RPO target, and treat every game day as a test of the written runbook.
  6. 6.Execute failback & post-incident stabilization — resync Region A from Region B before sending any traffic back, shift traffic back gradually via weighted steering, and feed the real failover/failback timings back into the runbook and game-day script.

This holds up for a high-throughput foundational service specifically because the decision of where traffic goes is driven entirely by an automated, independently-verified health signal — never by a human under pressure mid-outage; the human's job is validating the failover and deciding when it's safe to fail back, not making the call itself.