Observability
Know when your system is broken and why: metrics, logs and traces, percentiles instead of averages, SLOs, error budgets and alerts that matter.
On this page
A checkout API's dashboard shows an average latency of 120 ms, comfortably green. Meanwhile support tickets pile up: "checkout hangs". Look at ten real requests and the problem is plain: nine finish in 20 to 43 ms, and one takes 950 ms. The average hides the one request that users complain about.
It gets worse with fan-out. If a page makes 20 backend calls and each has a 1-in-100 chance of being slow, the chance that at least one is slow is 1 − 0.9920 ≈ 18%. Nearly one page in five is slow, while each backend's average looks fine.
The idea
Observability is being able to answer, from the outside, two questions: is it broken for users right now? and where and why? Three kinds of data do this, each good at a different part:
- Metrics: numbers over time, such as requests a second, error rate, latency percentiles, CPU. Cheap to keep for months, ideal for dashboards and alerts. They tell you that something changed.
- Logs: one record per event, with detail: "payment 9f2 timed out after 800 ms, attempt 2". They tell you why, for one request. Write them as structured fields (JSON), not free text, so you can search them.
- Traces: the path of one request across services. Each service records a span (name, start, duration) and passes a trace id on to the next call in a header. They tell you where the time went.
| Option | Good at | Bad at | Pick it when |
|---|---|---|---|
| Metrics Usual default | Cheap at any traffic; fast to query; the basis for alerts and SLOs. | Aggregated: no detail on a single request. Labels with many values (user id) explode the cost. | Always. Alert on them. |
| Logs | Full detail for one event; answer questions you did not plan for. | Expensive at volume; noisy unless structured. Sample or drop debug logs at high traffic. | Explaining a failure once a metric or trace points at it. |
| Traces | Show which service and which call made a request slow. | Every service must pass the trace id on. Usually sampled, so a rare request may be missing. | Several services on one request path. |
For each service, start with three metrics, often called RED: Rate (requests a second), Errors (failed requests a second), Duration (latency percentiles). For machines and queues, add saturation: how full the CPU, memory, connection pool or backlog is.
Percentiles, not averages
Sort the ten requests: 20, 22, 24, 25, 26, 28, 30, 32, 43, 950 ms. Their sum is 1,200, so the average is 120 ms, a number no request actually took. The median (p50) is 27 ms: what a typical user sees. The slowest is 950 ms. With thousands of requests, the p99 is the latency that 99% of requests beat, which is what your unluckiest regular users feel every day.
Track p50 for the typical case and p99 (sometimes p99.9) for the tail. One trap: you cannot average percentiles. The p99 of two servers is not the mean of their two p99s. Collect latency as histograms (counts per latency bucket), add the histograms, then compute the percentile.
From alert to cause
Step through an incident: an alert fires, a dashboard narrows it down, a trace finds the slow call, and a log explains it.
- checkout p99
- 2.1 s
- SLO
- 99.9% under 1 s
1An alert pages the on-call engineer: checkout has used 2% of its monthly error budget in the last hour. It is about users, not CPU.
- rate
- normal
- errors
- normal
- checkout p99
- 2.1 s
2The RED dashboard: rate is normal, errors are normal, but checkout's p99 jumped from 250 ms to 2.1 s. So it is slowness, in checkout or below it.
- trace
- 4bf9
- gateway
- 2,050 ms
- checkout
- 2,040 ms
3Open one slow trace. The gateway span took 2,050 ms; inside it, the checkout span took 2,040 ms.
- trace
- 4bf9
- inventory
- 15 ms
4Checkout's child spans: the inventory query took 15 ms. Not the problem.
- trace
- 4bf9
- inventory
- 15 ms
- payment
- 1,980 ms
5The payment call took 1,980 ms. That is where the time went.
- attempt 1
- 800 ms, timeout
- attempt 2
- 800 ms, timeout
- attempt 3
- 300 ms, ok
6Search logs for trace 4bf9: two payment attempts (same idempotency key) timed out at 800 ms each, the third succeeded in 300 ms, with 80 ms of backoff. 800 + 800 + 300 + 80 = 1,980.
- checkout p99
- back to 280 ms
7Cause found in minutes: the payment provider is slow. Mitigate (switch to a backup provider if you have one), tell the provider, and add payment-call latency to the dashboard.
SLOs and alerts
A service level indicator (SLI) is a measured ratio of good events: "share of checkout requests that succeed in under 1 second". A service level objective (SLO) is the target for it: "99.9% over 30 days". What is left, 0.1%, is the error budget: the amount of failure you are allowed. While budget remains, ship features; when it runs low, slow down and fix reliability.
Alert on symptoms users feel (the SLO burning), not on causes that may not matter (CPU at 80%). A useful rule: page a human if the budget is burning 14.4 times faster than sustainable over the last hour, because at that rate a 30-day budget is gone in 30 ÷ 14.4 ≈ 2 days, and one hour of it uses 14.4 ÷ 720 = 2% of the budget. Slower burns open a ticket for working hours. Every page should need a human to act; if people learn to ignore pages, they will ignore the real one.
How many requests may fail the SLI in a 30-day month without breaking the SLO?
- 500 checkout requests a second, steady
- SLO: 99.9% of requests are good over 30 days
- 30 days = 2,592,000 seconds
Show the working
- Requests a month: 500 × 2,592,000 ≈ 1.3 billion
- Budget: 0.1% of that = 0.001 × 1.296 × 109 ≈ 1.3 million bad requests
Estimate 1.3 million (anything from 400 thousand to 4 million is a good answer)
A 30-minute full outage at 500 a second is 500 × 1,800 = 900,000 requests: about 70% of the month's budget in one incident.
In the interview
Expect "How would you know it is broken?" near the end of any design. A weak answer lists tools: "I'd add Prometheus and Grafana." A strong answer starts from the user: name the one or two actions that matter, define an SLI and SLO for each, alert on the SLO, and say how you would find the cause.
Your notification system is live. How do you know it is working?
The user-facing promise is "a notification arrives within 10 seconds". So my SLI is the share of notifications delivered within 10 seconds of the event, with a 99.9% SLO over 30 days, and I page when the budget burns fast. Under that, each service has rate, errors and p99 latency, and the queue has consumer lag, which usually moves first.
The SLO alert fires. What then?
Check the dashboards to see which stage's lag or latency moved, open a few slow traces to find the span, then read that service's logs for those trace ids. I would also run a synthetic check that sends a test notification every minute, so silence is caught even when there is no traffic.
Follow-ups you should expect:
- "Why p99 and not the average?" The average hides the slow tail, and with fan-out most pages hit someone's tail.
- "Logs are too expensive. What do you cut?" Sample successful requests, keep all errors, drop debug level in production, and lean on metrics.
- "How do you catch a total outage with no traffic?" Synthetic checks from outside, and an alert when traffic drops to near zero.
Common mistakes
A page makes 10 backend calls, each slow 1% of the time, independently. About what share of pages has at least one slow call?
Which alert should wake someone at 3 a.m.?
A request is slow and passes through five services. Which signal shows fastest which service is responsible?
What to practise next
Observability closes this module. Put it to work in a full design with Design a Notification System, and review how to discuss trade-offs in Talking Through Trade-offs.