Technical Program Manager – System Design

A pipeline ingesting millions of metrics per second can never let a slow downstream consumer become the producer's problem. The design isn't "write directly to wherever the data ends up" — it's "land everything in a durable, replayable buffer first, and let every downstream consumer read from that buffer at its own pace." The core mechanism is OCI Streaming as a partitioned buffer with a real retention window, so a struggling consumer causes measurable lag, never lost data.

1. 1.Assess & classify the telemetry workload — baseline sustained and peak metrics/logs-per-second and label cardinality, separate telemetry into loss-tolerance tiers (zero-tolerance security/audit logs vs. sample-able debug logs), and distinguish real-time consumers from near-real-time ones.
2. 2.Design the ingestion & buffering layer — lightweight collector agents (Fluent Bit or the OCI Unified Monitoring Agent) batch and compress telemetry at the source, landing everything in OCI Streaming, partitioned by data center and service, with retention sized to survive a realistic downstream outage.
3. 3.Build backpressure-aware stream processing — run consumer groups sized to partitions, autoscale on per-partition consumer lag rather than CPU, and gracefully degrade by shedding the lowest-tier telemetry first if sustained load outpaces even a fully-scaled consumer group.
4. 4.Route & fan out via Service Connector Hub — consume from Streaming once and fan out independently to OCI Logging Analytics (search/dashboards), OCI Monitoring (metrics/alarms), and Object Storage (durable archive), each with its own retry policy so one slow destination never blocks the others.
5. 5.Prevent data loss via dead-letter and retry patterns — route any record that exhausts its retries to a dedicated dead-letter Object Storage bucket instead of discarding it, treat dead-letter growth as its own alarm, and build a defined reprocessing job to replay it once the destination issue is fixed.
6. 6.Monitor the pipeline itself & alert on backpressure — alarm on consumer lag with enough runway to intervene before Streaming's retention window ages data out, size partitions against sustained growth trends rather than historical peaks, and run end-to-end synthetic canary checks on a fixed interval.

This holds up at millions of metrics per second across multiple data centers specifically because no component ever writes directly to a destination it doesn't control the pace of — every write goes through a buffer with real retention, every consumer scales independently against measurable lag, and every failure has a durable landing spot instead of a silent drop.