Design Metrics Monitoring

Every server in a company reports numbers every few seconds: CPU, memory, requests served, errors, latency. Engineers look at them on dashboards, and rules watch them around the clock and page someone when error rates climb. When production is on fire, this is the system everyone opens at once, so it must be at its best exactly when everything else is at its worst.

Think of an in-house version of a hosted product such as Datadog. Interviewers ask it because it looks like "store numbers, draw graphs" and hides a write-heavy store that must compress well, a cardinality problem that can take it down, and a design rule most candidates miss: the monitor must not fail along with what it monitors. The observability lesson covers what to measure and how to alert well; this one builds the machine. The steps follow A Framework for Any Design.

#Clarify the problem

• Functional: collect metrics with labels (service, host, status) from every server; query them for dashboards by time range and labels, with sums, rates and percentiles; evaluate alert rules and notify on-call.
• Non-functional: ingest 5 million samples a second; a sample is queryable within 30 seconds; a 1-hour dashboard panel loads in about 1 second; alerts fire within a minute once a rule's condition has held for its set duration, and keep working during partial outages; raw data for 7 days, rollups for 2 years.
• Out of scope: logs, traces, anomaly detection, the agent's install and upgrade tooling.

#Estimate the scale

• Series: 100,000 × 500 = 50 million active time series.
• Samples: 50 million ÷ 10 s = 5 million samples a second, nearly constant all day. At about 20 bytes per sample on the wire, that is 100 MB/s into the system.
• Raw storage: a sample is a timestamp and a value, 16 bytes uncompressed. 5 × 10⁶ × 86,400 = 432 billion samples a day, 6.9 TB a day uncompressed. Time-series compression (next section) typically gets a sample down to one or two bytes; assume 2: 864 GB a day.
• Queries: in an incident, say 2,000 dashboards open with 20 panels refreshing every 30 seconds: 2,000 × 20 ÷ 30 ≈ 1,300 queries a second. 100,000 alert rules every 30 seconds add a steady 3,300 a second.

#Define the API

POST /v1/write                     from agents, every 10 s, compressed
  body: [ { "labels": {"__name__": "http_requests_total", "service": "checkout",
                       "host": "web-17", "status": "500"},
            "samples": [[1763114700, 48213]] }, ... ]
  204 No Content
  429 Too Many Requests   tenant over its sample-rate limit (agent backs off)
 
GET /v1/query_range?q=sum by (status) (rate(http_requests_total{service="checkout"}[1m]))
                   &start=...&end=...&step=60s
  200 { "series": [ { "labels": {"status": "500"}, "points": [[t, v], ...] } ] }
  422 Unprocessable   query would touch more than 100,000 series
 
Alert rule
  expr:  error ratio for checkout > 5%
  for:   5m                 the condition must hold this long before firing
  notify: team-checkout

The decision worth arguing is step. A panel 1,000 pixels wide cannot show more than about 1,000 points, so the query service picks the coarsest data that still fills it. Asking for a week at a 10-second step returns 60,480 points per series that the browser throws away. Making step part of the API lets the server pick a rollup tier and refuse queries that would touch too many series.

#Model the data

series      id = hash(metric name + sorted labels)
            labels  {__name__="http_requests_total", service="checkout", host="web-17", status="500"}
 
chunk       series_id, start_time, end_time, compressed bytes   (120 samples, 20 minutes)
            only the open chunk stays in memory; full ones go to local disk
 
index       "service=checkout" -> [series ids, sorted]
            "status=500"       -> [series ids, sorted]
            query {service="checkout", status="500"}  =  intersect the two lists
 
blocks      every 2 hours: chunks + index for that window, written as immutable
            files to object storage; compaction merges blocks and builds rollups

The access pattern is unusual. Writes are append-only, always at "now", and arrive for every series at once. Reads ask for a range of time for a set of series, and data is almost never updated. A general database would spend its effort on things this workload does not need. A time-series database (such as Prometheus, M3 or VictoriaMetrics) keeps each series' open chunk in memory and the rest of the last two hours on local disk, then writes immutable, compressed blocks that are cheap to keep in object storage.

#High-level design

#Deep dives

#Push or pull

Something has to move numbers from 100,000 hosts to the system. Either the system pulls (scrapes an endpoint on each target) or the targets push.

The agent buffers a few minutes of samples on local disk when ingest is slow or returns 429, and backs off with jitter so 100,000 agents do not retry in lockstep. Every agent also sends an up series; an alert on that series going absent is how you tell a dead host from a quiet one.

#Storing time series

Compression works because neighbouring samples look alike. Timestamps arrive every 10 seconds: 1000, 1010, 1020, 1031. Store the first, then the difference of differences: 10 − 10 = 0, 11 − 10 = 1. Mostly zeros, which take a single bit. Values change slowly, so store each one XOR-ed with the previous one: an unchanged value becomes a single 0 bit, and a small change leaves only a few meaningful bits. That is how 16 bytes becomes one or two, and why samples are compressed per series in time order.

Downsampling and retention tiers. Old data is looked at in coarse views, so compaction rewrites it: raw for 7 days, 1-minute rollups for 30 days, 1-hour rollups for 2 years. Each rollup point keeps min, max, sum and count, not just an average, so a rollup can still answer "the highest CPU in that hour" and averages over any longer span stay correct (total sum ÷ total count). The tiers cut storage from about 630 TB to 30 TB. Percentiles cannot be rebuilt from those four values, so latency should be recorded as a histogram (counts per bucket), whose buckets roll up by adding.

The cardinality problem. A series is a unique combination of label values, so labels multiply. http_requests_total with 200 services × 50 endpoints × 10 status codes is 100,000 series, fine. Add user_id and every active user becomes new series: 1 million users each hitting 5 endpoint and status combinations is 5 million series, 5 GB of ingester memory at 1 KB each, from one line of code. Memory, index size and query cost all scale with series, so this is the usual way such systems fall over.

• Limit per tenant: cap active series per team and refuse new ones beyond it, as in the tracer. Existing series keep working.
• Make it visible: show each team its series count and the labels with the most values.
• Send unbounded values elsewhere: user ids, request ids and URLs with ids belong in logs and traces, not labels.

#The query path

A dashboard query such as "error rate by status for checkout over the last hour" runs in four steps:

1. Find series: intersect the index's sorted id lists for __name__=http_requests_total and service=checkout. Refuse with 422 if more than 100,000 series match, before reading any data.
2. Pick a tier: use the coarsest tier that is still finer than the step. A 7-day panel 1,000 points wide has a step of about 10 minutes, so over 1,000 series it reads 10 million 1-minute points instead of 60 million raw ones.
3. Fan out: series are sharded across ingesters by id, so the query service asks every shard holding matches for recent data, and object storage for older blocks, in parallel.
4. Aggregate early: each shard computes partial sums by status and returns a few series, not thousands; the query service adds them up.

During an incident hundreds of engineers open the same dashboards. Align each query's start and end to multiples of the step and cache results by (query, step, time range): the older part of the range never changes, so a refresh only computes the newest few points.

#Alerts that survive the outage

Evaluate each rule on a fixed schedule through the same query path, and require the condition to hold for a duration before firing, which filters out brief spikes. Run two evaluator replicas so one can die, and deduplicate in the notifier by rule and labels, so redundancy never means double pages. The observability lesson covers which conditions deserve a page.

The harder problem is shared fate. If monitoring runs on the same cluster, network and service discovery as production, the outage that needs an alert can take out the alerting too. Then the system is silent at the worst moment, and silence reads as "all fine".

• Separate failure domain: run monitoring on its own machines and network path, with no dependency on the services it watches (not their database, not their auth service, not their deploy system).
• No data is a signal: a rule whose series vanish fires a "no data" alert instead of quietly resolving.
• Watch the watcher: a small second monitoring stack in another region checks that the main one is scraping, storing and evaluating.
• Dead man's switch: an alert that is always firing sends a heartbeat to the outside paging provider every minute. If heartbeats stop, the provider pages, as in the last tracer step.

#Bottlenecks and failures

At 10x, 50 million samples a second over 500 million series, ingest and the log scale out by adding nodes, and ingesters need 500 GB of memory for active series before replication. The index and query fan-out are next: split tenants onto separate shard groups so one team's huge query cannot slow everyone, and add more rollup tiers. Storage grows to about 300 TB, cheap in object storage.

#What a strong answer sounds like

#Follow-up questions

#How do you compute p99 latency across 1,000 servers?

Not by averaging each server's p99, which is meaningless. Each server records a histogram of request counts per latency bucket; add the buckets across servers, then read the 99th percentile from the summed histogram.

#Can you lose samples?

A few seconds' worth, yes, if an agent's disk buffer fills. That is acceptable for metrics; the guarantee to protect is that alerts keep firing.

#How do you support many teams fairly?

Per-tenant limits on samples per second, active series and query cost, separate shard groups for the largest tenants, and showing each team its usage.

#How would you detect anomalies, not just thresholds?

Compare the current value with the same hour in previous weeks, or with a moving baseline, as an extra rule type. It runs on the same evaluator and query path.

#What about very short-lived jobs?

A batch job that runs for 3 seconds can finish between collections. It pushes its final values to the local agent before exiting, and the agent sends them on with its next batch.