Metrics

Logging gave you the detailed event record — the what. But you cannot alert on detail, and you cannot dashboard millions of individual lines. To answer “is the system healthy right now, and how does that compare to an hour ago?” you need the aggregate pillar: metrics. Metrics are cheap, always-on numbers over time, and they are what page you at 3am.

The problem they solve is summarisation. A log line per request tells you everything about one request and nothing, at a glance, about the population of requests. A metric throws away the per-request detail on purpose and keeps a running number — “requests per second,” “p99 latency,” “error rate” — so you can watch the health of the whole service in one number that updates continuously. You trade detail for the ability to see the forest.

The time-series model

A metric is a time series: a stream of (timestamp, value) points, identified by a name and a set of labels (key–value pairs). The labels are what make a metric sliceable.

   http_requests_total{service="payments", method="POST", status="500"}
   ▲                    ▲                                                ▲
   metric name          labels (dimensions you can filter/group by)      one time series

   t=14:00:00  → 4012
   t=14:00:15  → 4090     each scrape appends a (timestamp, value) point
   t=14:00:30  → 4171

Every unique combination of name + labels is a distinct time series, stored independently. That single fact — one series per label combination — is the source of both the power (slice by any label) and the danger (cardinality explosion) we’ll hit below.

Four metric types

Prometheus, the de-facto standard, defines four types. Knowing which to use is most of the skill.

Type	What it models	Rule of thumb	Example
Counter	A value that only ever goes up (resets to 0 on restart)	Count of things that happened	http_requests_total, errors_total
Gauge	A value that goes up and down	A current measurement	memory_bytes, queue_depth, temperature
Histogram	Distribution of values into buckets	”How many requests fell under 100ms? under 500ms?”	request_duration_seconds
Summary	Like a histogram but computes quantiles client-side	Pre-computed percentiles	request_duration_seconds{quantile="0.99"}

The non-obvious one is counter vs gauge. A counter only goes up because you don’t care about its absolute value — you care about its rate of change. http_requests_total climbing forever is fine; what you graph is rate(http_requests_total[5m]) — requests per second. A gauge is the opposite: its current value is the point (memory in use right now).

Histograms matter because averages lie. An average latency of 50ms can hide that 1% of requests take 5 seconds — and that 1% is your angriest users. A histogram preserves the distribution, so you can ask for the 99th percentile (p99): the latency that 99% of requests beat. Always alert on percentiles, never on averages.

Histogram vs summary: where the percentile is computed

A histogram ships raw bucket counts and lets the server (PromQL) compute quantiles — which means you can aggregate across many instances correctly. A summary computes the quantile inside the application before shipping — cheaper to query but you cannot meaningfully average a p99 across instances. Prefer histograms unless you have a specific reason; aggregatability usually wins.

Prometheus and the pull model

Most monitoring systems are push: the app sends metrics to a collector. Prometheus is pull: a central Prometheus server scrapes an HTTP endpoint (/metrics) that each target exposes, on a fixed interval.

   PUSH model                        PULL model (Prometheus)
   ──────────                        ───────────────────────
   app ──► sends metrics ──► server  Prometheus ──► GET /metrics ──► app
   app decides when                  server decides when (every 15s)
   server can be overwhelmed         server controls load; sees "target down" for free

The pull model has properties that matter operationally:

• The scraper controls the rate, so a misbehaving app can’t flood your metrics backend.
• Liveness comes free: if a scrape fails, Prometheus knows that target is down — that’s a signal, not a gap. A push system can’t distinguish “app is healthy but quiet” from “app is dead.”
• Service discovery is built in: Prometheus asks the Kubernetes API for the current set of Pods and scrapes them as they come and go — which is essential when scaling and scheduling churns your replicas constantly.

A scrape target just exposes plaintext on /metrics:

# HELP http_requests_total Total HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="POST",status="200"} 48123
http_requests_total{method="POST",status="500"} 37

PromQL: asking questions of time series

PromQL is the query language. You rarely graph a raw counter; you derive rates, ratios, and percentiles from it. A few load-bearing examples:

# Request rate (per second), averaged over 5 minutes — PromQL
rate(http_requests_total[5m])

# Error ratio: fraction of requests that are 5xx
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# p99 latency from a histogram
histogram_quantile(0.99, sum by (le) (rate(request_duration_seconds_bucket[5m])))

That middle query — the error ratio — is the literal definition of an SLI you’ll meet in SLOs & Error Budgets. PromQL is how a Golden Signal becomes a number you can alert on.

The cardinality explosion

Now the danger the time-series model set up. Because every label combination is a separate stored series, a high-cardinality label multiplies your series count catastrophically.

   http_requests_total{method, status}            ~ 5 methods × 6 statuses   =   30 series   ✅
   http_requests_total{method, status, user_id}   × 1,000,000 users          = 30,000,000   💥

Adding user_id didn’t add a label — it added a series per user, forever. This is the cardinality explosion, and it is the number-one way teams take down their own Prometheus: memory blows up, scrapes slow, queries time out.

Metrics are for low-cardinality dimensions only

The rule is blunt: never put unbounded values (user_id, request_id, email, full URLs) in metric labels. Those are high-cardinality detail — they belong in logs and traces, which are built to carry them. Keep metric labels to bounded, low- cardinality dimensions: method, status code, region, service. When in doubt, count how many distinct values the label can take; if it’s “lots,” it’s not a metric label.

What to measure: Golden Signals, RED, and USE

Metrics are only useful if you measure the right numbers. Three well-known frameworks tell you which, and they overlap heavily.

The Four Golden Signals (from Google’s SRE practice) — measure these for any user-facing service:

Signal	Question	Typical metric
Latency	How long do requests take?	p50/p99 of request_duration_seconds
Traffic	How much demand?	rate(http_requests_total[5m])
Errors	What fraction fail?	5xx ratio
Saturation	How full is the system?	CPU, memory, queue depth vs capacity

Two narrower methods you’ll hear constantly:

• RED (for request-driven services): Rate, Errors, Duration. It’s the Golden Signals minus saturation — the three you watch per endpoint.
• USE (for resources: CPUs, disks, queues): Utilization, Saturation, Errors. Use it to find which resource is the bottleneck.

The mental split: RED watches the service from the outside (what users feel); USE watches the resources from the inside (why it’s struggling). Latency spiking (RED) plus CPU pinned at 100% (USE) tells you the what and the why in two glances.

Alerting basics

An alert is a PromQL expression that, when true for some duration, pages a human. The art is alerting on symptoms users feel, not internal causes:

# Prometheus alerting rule (routed via Alertmanager): page if the 5xx error ratio exceeds 5% for 5 minutes
- alert: HighErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m]))
      / sum(rate(http_requests_total[5m])) > 0.05
  for: 5m
  labels:    { severity: page }
  annotations:
    summary: "Error rate above 5% on {{ $labels.service }}"

Two principles make alerts survivable: the for: clause prevents flapping on a brief blip, and you should alert on user-facing symptoms (errors, latency) rather than causes (high CPU). High CPU that nobody feels is not worth waking someone for; high CPU plus rising latency is. We’ll make this rigorous with burn-rate alerting in SLOs & Error Budgets.

By the numbers — what a cardinality explosion costs in RAM

Prometheus keeps each active series in memory. A common capacity-planning rule of thumb is very roughly 1–3 KB of RAM per active series (it drifts with version and label sizes — treat it as an order-of-magnitude guide, mid-2020s). Now price the user_id mistake from above:

   30 series (method × status)        × ~2 KB  ≈  60 KB        (trivial)
   30,000,000 series (× 1M user_ids)  × ~2 KB  ≈  ~60 GB RAM   (Prometheus is dead)

Adding one unbounded label didn’t make the metric “a bit bigger” — it turned a 60 KB metric into one no reasonable server can hold in memory. That is why the rule is blunt and absolute: unbounded values never go in metric labels.

The thread: from “is it up?” to a number you can trust

The manual, error-prone step metrics remove is eyeballing: a human watching a terminal, refreshing a page, “feeling” whether the site is slow. That doesn’t scale past one service and it doesn’t work while you sleep. Metrics replace the human watcher with a tireless one — a query that’s always evaluating, an alert that fires on a real symptom — much like the reconciliation loop replaced the human who watched and fixed. Production gets safer because problems are caught by threshold, in seconds, on the signal users actually feel — not minutes later when someone happens to glance at a graph.

Metrics tell you that a service is unhealthy. To find where in a distributed call graph the problem lives, you need the next pillar: Tracing.

The architect’s lens

Five questions for the aggregate pillar:

• Why does it exist? Because you cannot alert on or dashboard millions of individual log lines — to answer “is the system healthy right now, and how does that compare to an hour ago?” you need cheap, always-on numbers over time.
• What problem does it solve? Summarisation and alerting: a time series sliceable by labels, queried with PromQL, watched against the Four Golden Signals — so a real symptom pages a human in seconds instead of a person eyeballing a terminal.
• What are the trade-offs? Every name+label combination is a separately stored series, so one high-cardinality label (user_id) triggers a cardinality explosion — ~30 series becomes 30 million, ~60 GB of RAM, a dead Prometheus — which is why metric labels must stay bounded.
• When should I avoid it? For per-request, high-cardinality detail — that belongs in logs and traces, never in metric labels.
• What breaks if I remove it? Detection reverts to a human watching a graph: problems are caught minutes late by whoever happens to glance over, and nothing pages you while you sleep.

Check your understanding

1. What is a time series, and why does “every unique combination of name + labels is a separate series” explain both the power and the danger of metrics?
2. Explain counter vs gauge using rate of change. Why do you graph rate(http_requests_total[5m]) rather than the counter itself?
3. Why do averages lie about latency, and what does a histogram give you instead? When would you choose a histogram over a summary?
4. Describe the pull model. Give two concrete operational advantages it has over push (one about load, one about detecting a dead target).
5. A teammate adds a user_id label to http_requests_total to “debug per-user.” What happens, why, and where should that dimension actually live?

Show answers

1. A time series is a stream of (timestamp, value) points identified by a metric name plus a set of labels. Because each distinct name+label combination is stored as its own series, you can slice by any label (power) — but a label with many possible values multiplies the series count explosively (danger).
2. A counter only increases (you care about its rate of change, not its absolute value); a gauge goes up and down (its current value is the point). You graph rate(...) because a counter climbing forever is meaningless on its own — the useful quantity is requests per second, which is its slope.
3. An average hides the tail: 1% of requests at 5 seconds barely move a 50ms mean, yet that 1% is your angriest users. A histogram preserves the distribution in buckets so you can compute percentiles like p99. Prefer a histogram over a summary when you need to aggregate quantiles across many instances — summaries compute the quantile client-side and can’t be meaningfully averaged.
4. In the pull model, the Prometheus server scrapes each target’s /metrics endpoint on a schedule. Advantages: (a) the scraper controls the rate, so a misbehaving app can’t flood the backend; (b) a failed scrape is the “target down” signal — a push system can’t tell “healthy but quiet” from “dead.”
5. You get a cardinality explosion — a separate time series per user (millions), which blows up memory and crashes/slows Prometheus. It happens because each label combination is its own series. user_id is high-cardinality detail and belongs in logs and traces, not metric labels.