Technical Program Manager – System Design

A pipeline ingesting millions of metrics per second can never let a slow downstream consumer become the producer's problem. The design isn't "write directly to wherever the data ends up" — it's "land everything in a durable, replayable buffer first, and let every downstream consumer read from that buffer at its own pace." The core mechanism is Amazon Kinesis Data Streams as a sharded buffer with a real retention window, so a struggling consumer causes measurable lag, never lost data.

1. 1.Assess & classify the telemetry workload — baseline sustained and peak metrics/logs-per-second and label cardinality, separate telemetry into loss-tolerance tiers (zero-tolerance security/audit logs vs. sample-able debug logs), and distinguish real-time consumers from near-real-time ones.
2. 2.Design the ingestion & buffering layer — lightweight collector agents (Fluent Bit or the AWS Distro for OpenTelemetry Collector) batch and compress telemetry at the source, landing everything in Amazon Kinesis Data Streams, sharded by data center and service, with retention configurable up to 365 days.
3. 3.Build backpressure-aware stream processing — run Kinesis Client Library (KCL) consumers or Lambda with a Kinesis event source mapping sized to shard count, autoscale on IteratorAge rather than CPU, and gracefully degrade by shedding the lowest-tier telemetry first if sustained load outpaces even a fully-scaled consumer group.
4. 4.Route & fan out via Kinesis Data Firehose — consume from the stream once and deliver independently to Amazon OpenSearch Service (search/dashboards), Amazon CloudWatch (metrics/alarms), and Amazon S3 (durable archive), each with its own retry duration so one slow destination never blocks the others.
5. 5.Prevent data loss via dead-letter and retry patterns — route any record that exhausts its retries to a dedicated dead-letter S3 bucket instead of discarding it, treat dead-letter growth as its own alarm, and build a defined reprocessing job to replay it once the destination issue is fixed.
6. 6.Monitor the pipeline itself & alert on backpressure — alarm on IteratorAge with enough runway to intervene before the stream's retention window ages data out, size shard counts against sustained growth trends rather than historical peaks, and run end-to-end synthetic canary checks on a fixed interval.

This holds up at millions of metrics per second across multiple data centers specifically because no component ever writes directly to a destination it doesn't control the pace of — every write goes through a buffer with real retention, every consumer scales independently against measurable lag, and every failure has a durable landing spot instead of a silent drop.