Technical Program Manager – System Design

Capacity planning under ambiguity: Design an end-to-end resource forecasting and allocation system for GPU clusters that must handle sudden enterprise spikes in AI model training demand.

GPU capacity can't be provisioned instantly, so the honest answer to "how much will we need" is always a distribution, never a point estimate. The design isn't "scale reactively once the queue backs up" — it's "forecast a range of demand, pre-position capacity against that range, and let a priority-based admission queue absorb the difference between the forecast and what actually happens."

  1. 1.Assess & classify the demand signal — separate steady-state, forecastable production training from sudden enterprise-driven spikes, classify workloads into priority tiers, and audit how much lead time bringing new GPU capacity online actually requires.
  2. 2.Build the forecasting pipeline — feed historical utilization, queue depth, and known business signals (committed contracts, sales pipeline) into a scheduled time-series model (OCI Data Science) that outputs a rolling P50/P90 confidence range, not a single number, published with its uncertainty visible.
  3. 3.Design the tiered capacity model — size OCI Capacity Reservations to the forecast's P50 baseline as a hard guarantee, and an on-demand burst pool to cover the P50–P90 gap as best-effort capacity, so predictable demand never competes with a spike for the same pool.
  4. 4.Build priority-based admission & allocation — route every incoming job through a priority queue that allocates against reserved capacity first and burst second, with explicit preemption/backfill rules and visible queue position and wait-time instead of silent job failure.
  5. 5.Automate scale-out & scale-in — trigger OCI Autoscaling and Resource Manager (Terraform) to provision burst capacity on queue-depth or forecast signals, set thresholds early enough to cover the provisioning lead time from Step 1, and scale in only after confirming in-flight jobs have checkpointed safely.
  6. 6.Monitor, validate & feed back into forecasting — continuously compare forecast P50/P90 against actual demand and feed the error back into the model, track queue wait time by priority tier as a leading indicator, and alert before the burst pool's own capacity ceiling is reached.

This holds up under genuinely ambiguous, spike-prone enterprise demand specifically because no single number is ever trusted to be right — the forecast is a range, the capacity model has two tiers with different guarantees, and the admission queue enforces priority explicitly instead of assuming the forecast and the capacity behind it will always match reality.