The Observability Stack

We have a signed image ready to deploy. But deploying something you can’t see is flying blind — and the next page’s canary literally depends on real signals to decide whether to promote or roll back. So before we deploy, we make Snip observable. This page instruments it across all three pillars — metrics, logs, traces — and wires the metrics into Prometheus so they’re queryable.

The manual step being removed here is guessing: SSHing into a box, tailing a log, and forming a hunch about whether the service is healthy. In its place: numeric signals you can query, alert on, and — crucially — let an automated rollout make decisions on.

The four golden signals

You could emit a thousand metrics. Start with the four that matter for almost any request-serving service — the golden signals (Metrics covers the theory):

Signal	Question it answers	Snip’s metric
Latency	How long do requests take?	http_request_duration_seconds (histogram)
Traffic	How much demand is there?	http_requests_total (counter)
Errors	What fraction are failing?	http_requests_total{status=~"5.."}
Saturation	How full is the service?	db_pool_in_use, cache_hit_ratio

Snip exposes these on a /metrics endpoint in Prometheus text format. Latency is a histogram, not an average, on purpose: an average hides the slow tail, and it’s the tail (p95, p99) that users actually feel. With a histogram you can ask “what’s the 99th-percentile latency?” — which an average can never answer.

   GET /metrics
   ─────────────────────────────────────────────────────────
   # HELP http_requests_total Total HTTP requests.
   # TYPE http_requests_total counter
   http_requests_total{method="GET",route="/:code",status="302"} 184213
   http_requests_total{method="GET",route="/:code",status="404"} 1192
   http_requests_total{method="POST",route="/shorten",status="201"} 5043

   # TYPE http_request_duration_seconds histogram
   http_request_duration_seconds_bucket{route="/:code",le="0.005"} 170431
   http_request_duration_seconds_bucket{route="/:code",le="0.01"}  182004
   http_request_duration_seconds_bucket{route="/:code",le="+Inf"}  185405

   # TYPE cache_hit_ratio gauge
   cache_hit_ratio 0.94

Labels are the dimensions you’ll slice by

Labels (route, status, method) turn one metric into many you can filter and group. But labels multiply: every distinct combination is a separate time series. Keep label values bounded — route is the template /:code, never the actual short code, or you’d create a new series per URL and melt Prometheus. High-cardinality labels are the classic way to take down a metrics system.

Structured logs and traces — the other two pillars

Metrics tell you that something is wrong (error rate up). Logs and traces tell you what and where.

Structured logs are JSON, not prose, so they’re queryable. Every log line carries a trace_id so a log can be tied back to the exact request that produced it:

{"ts":"2026-06-24T18:03:21Z","level":"error","msg":"db query failed",
 "route":"/shorten","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","err":"timeout","duration_ms":312}

Traces (Tracing) follow one request across service boundaries. A single GET /:code becomes a tree of timed spans — and the trace immediately shows where the time went:

   GET /:code                         [■■■■■■■■■■] 12ms   trace 4bf92f35…
   ├─ check Redis cache               [■■]         2ms   (hit → skip Postgres)
   └─ (on miss) query Postgres        [ — not hit this request — ]
   └─ write 302 redirect              [■]          1ms

Snip emits traces with OpenTelemetry (OTel) — the vendor-neutral standard — so the same instrumentation can ship to Jaeger, Tempo, or any backend without rewriting the app. The three pillars share IDs (trace_id in the log, the same trace in the tracer), so you can pivot: a metric alert fires → find an example trace → read its logs. That pivot is the whole point of having all three.

Deploying Snip with the probes and the metrics port

Here is Snip’s Kubernetes Deployment. It pulls the SHA-tagged image, runs as non-root, gets its config from a ConfigMap/Secret (config & secrets), exposes the metrics port, and wires the /healthz probes from the containerize page:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: snip
  namespace: snip
  labels: { app: snip }
spec:
  replicas: 3
  selector:
    matchLabels: { app: snip }
  template:
    metadata:
      labels: { app: snip }
    spec:
      securityContext:
        runAsNonRoot: true        # match the non-root image; refuse to run as root
      containers:
        - name: snip
          image: ghcr.io/acme/snip:a1b2c3d4e5f6   # the immutable SHA tag from CI
          ports:
            - { name: http,    containerPort: 8080 }
            - { name: metrics, containerPort: 8080 }   # /metrics on the same port
          envFrom:
            - configMapRef: { name: snip-config }   # PORT, LOG_LEVEL, REDIS_URL
            - secretRef:    { name: snip-secrets }   # DATABASE_URL with password
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
          resources:
            requests: { cpu: "100m", memory: "64Mi" }
            limits:   { memory: "128Mi" }

Telling Prometheus to scrape Snip

Prometheus needs to discover the /metrics endpoint. With the Prometheus Operator, you declare that with a ServiceMonitor — a custom resource that says “scrape every pod behind Services matching these labels.” This is the reconciliation pattern again: Prometheus’s config is reconciled from the ServiceMonitors in the cluster, so adding monitoring to a new service is a declarative manifest, never a hand-edit of Prometheus config.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: snip
  namespace: snip
  labels: { release: prometheus }   # so the Prometheus instance selects it
spec:
  selector:
    matchLabels: { app: snip }      # match the Snip Service
  endpoints:
    - port: metrics                 # the named port above
      path: /metrics
      interval: 15s

An alert rule and a Grafana panel

Metrics you never look at are dead weight. Two outputs make them useful: an alert (tell a human when something’s wrong) and a dashboard (let a human see what’s happening). Both are just PromQL.

The alert: fire if Snip’s 5xx error rate exceeds 5% for 5 minutes. (The for: 5m avoids paging on a single transient blip — a real signal, not noise.)

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: snip-slo
  namespace: snip
  labels: { release: prometheus }
spec:
  groups:
    - name: snip.rules
      rules:
        - alert: SnipHighErrorRate
          expr: |
            sum(rate(http_requests_total{app="snip",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{app="snip"}[5m]))
            > 0.05
          for: 5m
          labels: { severity: page }
          annotations:
            summary: "Snip 5xx error rate above 5% for 5 minutes"

The Grafana panel: p99 latency over time, the number users actually feel.

# PromQL
histogram_quantile(
  0.99,
  sum(rate(http_request_duration_seconds_bucket{app="snip"}[5m])) by (le)
)

These exact queries drive the canary

Hold onto that error-rate and latency PromQL. On the next page, Argo Rollouts runs these same queries against the canary’s slice of traffic and uses the result to decide — automatically — whether to promote the new version or roll it back. The observability you build here isn’t just for humans to look at; it’s the input to an automated deployment decision.

The thread

The manual step this page removes is guessing whether production is healthy — logging into a server, tailing a file, and forming a hunch. In its place are queryable, numeric signals: the golden signals as metrics, structured logs joined to traces by trace_id, and a declarative ServiceMonitor that wires it all into Prometheus without hand-editing config. Production gets safer in two compounding ways: a human gets paged on a real threshold before a small problem becomes an outage, and — the bigger win — the deploy machinery itself can now read these signals and make safety decisions no human is fast enough to make. The cost is the instrumentation effort and the discipline to keep label cardinality bounded.

With Snip both deployable and observable, we can finally ship it the safe way — gradually, watching these exact signals, ready to undo itself: Canary Deploy.

Check your understanding

1. Name the four golden signals and Snip’s metric for each. Why is latency stored as a histogram rather than an average?
2. All three pillars carry a shared trace_id. Walk through how you’d use metrics, traces, and logs in sequence to debug a spike in errors.
3. What does the ServiceMonitor do, and why is declaring monitoring this way better than editing Prometheus’s config file directly?
4. The alert has for: 5m. What problem would you have if you removed it, and what problem if you set it to for: 1h?
5. Why does this page insist the metrics aren’t only for humans? What consumes the exact same PromQL on the next page, and to what end?

Show answers

1. Latency (http_request_duration_seconds histogram), traffic (http_requests_total), errors (http_requests_total{status=~"5.."}), saturation (db_pool_in_use, cache_hit_ratio). Latency is a histogram because an average hides the slow tail; only a histogram lets you ask for p95/p99 — the latency users actually feel.
2. A metric alert tells you that errors spiked (rate of 5xx up). You find an example trace for a failing request to see where the time/failure was (e.g. the Postgres span). You read that request’s logs, joined by trace_id, to see what exactly failed (e.g. a query timeout). Metrics → traces → logs: that-then-where-then-what.
3. The ServiceMonitor declaratively tells the Prometheus Operator to scrape pods matching given labels at a given path/interval; Prometheus’s config is reconciled from it. That’s better than hand-editing config because adding monitoring becomes a reviewable manifest in git, can’t drift, and doesn’t require touching the Prometheus instance directly.
4. Without for, a single transient blip (one bad scrape) pages a human — noise that erodes trust in alerts. With for: 1h, a genuine outage burns an hour before anyone is told — far too slow. for: 5m balances: ignore blips, page on sustained problems.
5. Because the canary’s automated analysis (next page) runs the exact same error-rate and latency PromQL against the new version’s traffic slice and uses the result to auto-promote or auto-rollback. The metrics are the input to a deployment decision, not just a dashboard.