Logging

The three pillars page placed logging as the pillar that answers “what exactly happened?” — maximum detail per event, at the cost of being expensive and hard to aggregate at scale. This page is about making logs actually useful: not the print() statements you scattered while debugging on your laptop, but a disciplined event stream you can ship off the machine, query across a whole fleet, and trust during an incident.

The problem logging solves is old and concrete. A process on one box writes to a file; you read it with tail -f. That works for one process on one machine. It collapses the moment you have fifty replicas across twenty nodes, where the Pod that logged the error was rescheduled and deleted ten minutes ago. The log line you need is gone unless you got it off the box first. Everything below is in service of that: get logs off the machine, in a shape you can query.

Unstructured vs structured logs

The single highest-leverage change you can make to logging is to stop writing prose and start writing data. Compare:

   UNSTRUCTURED (a human sentence)
   2026-06-24T14:02:11Z ERROR user 8842 failed checkout: card declined after 1240ms

   STRUCTURED (JSON: machine-queryable fields)

{
  "timestamp": "2026-06-24T14:02:11Z",
  "level": "error",
  "msg": "checkout failed",
  "user_id": "8842",
  "reason": "card_declined",
  "duration_ms": 1240,
  "service": "payments",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

Both carry the same information. But the unstructured line is a string — to find “all checkouts that took over 1000ms,” a machine has to parse English with a regex, and that regex breaks the moment someone rewords the message. The structured line is fields. The query is duration_ms > 1000 AND level = "error", and it keeps working no matter how the human-readable msg changes.

Structured logging is the cheapest observability upgrade you can make

Switching to structured (usually JSON) logs costs almost nothing — one logging-library change — and it is the difference between logs you grep and logs you query. Every field becomes a filterable, aggregatable dimension. If you do one thing after reading this page, do this.

Log levels

Levels are a volume knob: they let you emit detail cheaply in development and suppress it in production, and they let you filter to severity during an incident. The conventional ladder, most to least severe:

Level	Meaning	Use it for
ERROR	Something failed and needs attention	Failed requests, exceptions, exhausted resources
WARN	Suspicious but handled	Retries, fallbacks, deprecated-path hits
INFO	Normal, noteworthy events	Service started, request completed, config loaded
DEBUG	Detailed internal state	Variable values, branch decisions — off in prod
TRACE	Firehose	Every step; almost never on in prod

You run production at INFO (or WARN) and can dial up to DEBUG for a specific service when investigating. The discipline: a level is a promise about volume and severity, not a mood. ERROR should mean “a human may need to look,” or your error alerts become noise everyone ignores.

The aggregation pipeline

A log line is worthless if it dies with the Pod. The job of a logging pipeline is to get every line off every machine and into one queryable place. The canonical shape has four stages:

   APP            AGENT                STORE                QUERY
   ───            ─────                ─────                ─────
   write JSON ──► collect & ship ──►  index & retain  ──►  search / dashboard
   to stdout      (Fluent Bit,         (Loki, Elastic,      (Grafana, Kibana)
                   Vector, Fluentd)     OpenSearch)
       │              │                     │                    │
   one line per   tails container       stores with          "level=error AND
   event          logs on each node,    labels/index,         service=payments
                  adds metadata,        enforces retention    since 14:00"
                  forwards

1. App writes structured logs to stdout/stderr. In a container, you do not manage log files — you write to standard output and let the platform capture it. This is the “logs are a stream, not a file” principle.
2. Agent — a lightweight collector like Fluent Bit, Vector, or Fluentd — runs on every node (usually as a Kubernetes DaemonSet), tails every container’s output, enriches each line with metadata (pod name, namespace, node), and forwards it.
3. Store indexes and retains the logs. Loki indexes only labels and keeps the log body compressed (cheaper); the ELK / Elastic stack (Elasticsearch) and OpenSearch index the full text (more powerful queries, more expensive). Retention policy lives here.
4. Query is the UI — Grafana for Loki, Kibana for Elasticsearch — where you search and build dashboards.

Why “write to stdout” is the right default in containers

Writing to a file inside a container means the logs vanish when the container does, and you’d have to mount volumes and rotate files yourself. Writing to stdout makes the platform responsible for capture — the same separation of concerns that makes containers portable. The app emits a stream; the node’s agent owns shipping it.

Correlation IDs: stitching a request together

Here is the problem structured logs alone don’t solve. One user request touches the gateway, then orders, then payments, then inventory — four services, four sets of log lines, interleaved with thousands of other requests’ logs in the same store. How do you pull out the lines for this one request?

You attach a correlation ID (often the same as the trace_id from tracing): a unique identifier generated when the request enters the system and propagated, unchanged, through every downstream call’s headers. Every service includes it in every log line.

   request arrives ─► gateway generates  trace_id=4bf92f...
        │
        ├─ gateway   logs {trace_id: 4bf92f..., msg: "received"}
        ├─ orders    logs {trace_id: 4bf92f..., msg: "reserved stock"}
        ├─ payments  logs {trace_id: 4bf92f..., msg: "card declined"}   ◄── the error
        └─ inventory logs {trace_id: 4bf92f..., msg: "rollback"}

Now the query trace_id = "4bf92f..." returns the entire journey of one request across all four services, in order — the difference between “somewhere in the system, checkouts are failing” and “here is exactly this checkout, hop by hop.” Without correlation IDs, distributed logs are a pile of unrelated sentences. With them, they’re a story.

Cardinality and cost — applied to logs

Logs are the high-detail pillar, so the temptation is to log everything, and the bill arrives fast. Three levers control log cost, and you tune all three deliberately:

• Volume — how much you emit. A DEBUG line in a hot loop running millions of times a second can dwarf everything else. Log events, not iterations.
• Retention — how long you keep it. Thirty days of full logs costs far more than seven; most teams keep recent logs hot and archive or drop older ones.
• Indexing — what you make searchable. Indexing every field is what makes Elasticsearch fast and expensive; Loki’s cheaper model indexes only a few labels and scans the rest.

The cardinality lesson from the three pillars reappears here as a gift, not a curse: logs are designed to carry high-cardinality fields like user_id and trace_id. Put that detail here, where it’s cheap to store and the cost is governed by volume and retention — not in metrics, where each distinct value spawns a new time series.

What you must never log

This is the part that becomes a security incident if you get it wrong, so it gets a hard warning.

Logs are a copy of your data — treat them like one

Logs are aggregated, shipped across the network, stored for weeks, and read by many engineers. Anything you log is now in all of those places. Never log:

• Secrets — passwords, API keys, tokens, private keys, session cookies. A logged token is a leaked token. (See Secrets Management.)
• PII / sensitive data — full credit-card numbers, government IDs, health data, raw personal data. Logging it can violate PCI-DSS, GDPR, or HIPAA and turn your log store into regulated data you now have to protect and audit.
• Full request/response bodies by default — they’re the most common way secrets and PII leak in by accident.

Redact at the source ("card": "****4242"), and prefer logging references (a user_id) over the sensitive value itself.

A logged secret is worse than no log at all: it’s a vulnerability that’s been replicated to every node, every backup, and every engineer’s query history. The fix is cultural and automated — code review for log statements, plus redaction filters in the agent as a backstop.

By the numbers — why “log everything” gets expensive fast

The three cost levers aren’t abstract — multiply them out for one moderately busy service:

   1,000 requests/sec  ×  ~1 KB per JSON line   =  1 MB/sec
   1 MB/sec  ×  86,400 sec/day                  ≈  86 GB/day
   86 GB/day  ×  30 days retention              ≈  2.6 TB/month

That’s one service at a single log line per request. Add a DEBUG line in a hot loop, or full request/response bodies, and the same service can emit ten times that — which is why volume and retention are knobs you set deliberately, not defaults you ignore.

The thread: from tailing files to querying a fleet

The manual, error-prone step logging-done-right removes is SSHing into individual boxes and tailing files by hand — a process that doesn’t even work once a request crosses many short-lived Pods, because the box you’d SSH into may be gone. The pipeline gets every line off every machine into one place before the Pod dies; structured fields and correlation IDs turn that pile into something you query. Production gets safer two ways: you can reconstruct exactly what happened during an incident without touching the (possibly broken) hosts, and you stop the far more dangerous habit of restarting things just to reproduce a log line.

Next, the aggregate pillar — numbers over time, and the alerts that wake you up: Metrics.

The architect’s lens

Five questions for logging done right:

• Why does it exist? Because tail -f on one box collapses the moment a request crosses fifty replicas on twenty nodes and the Pod that logged the error was rescheduled and deleted — you need the line off the machine before it dies.
• What problem does it solve? The forensic what: structured JSON makes every field queryable (duration_ms > 1000), the agent → store → query pipeline lifts every line off every node, and a correlation/trace_id stitches one request’s journey across services.
• What are the trade-offs? Logs are the high-detail, high-cost pillar — one busy service at one line/request is ~2.6 TB/month — so you tune three deliberate levers (volume, retention, indexing); and a logged secret or PII line is a breach replicated to every node, backup, and query history.
• When should I avoid it? You don’t avoid logging, but resist “log everything at DEBUG in a hot loop” and resist pushing high-cardinality detail into metrics — log events, not iterations.
• What breaks if I remove it? You lose the what entirely — debugging reverts to SSHing into (possibly broken) boxes to tail files, which doesn’t even work once the Pod is gone.

Check your understanding

1. Both an unstructured sentence and a JSON object can carry the same information. Mechanically, why is the structured version so much more useful at scale? Give a query that works on one and breaks on the other.
2. Walk the four stages of the aggregation pipeline (app → agent → store → query). Why is “write to stdout, not a file” the right default inside a container?
3. What problem do correlation IDs solve that structured logging alone does not? Show what the query looks like with and without them.
4. Logs are the high-cardinality pillar. Why is that a feature here, and what three levers do you tune to keep log cost under control?
5. Name three things you must never log and explain why a logged secret is worse than having no log at all.

Show answers

1. The unstructured line is a string, so a machine must parse English with a regex that breaks when the wording changes; the structured line is typed fields, so every field is directly filterable and aggregatable. duration_ms > 1000 AND level = "error" works on the JSON but requires fragile regex parsing on the sentence.
2. App writes structured logs to stdout; agent (Fluent Bit/Vector/Fluentd, one per node) tails container output, enriches with metadata, and forwards; store (Loki/Elasticsearch) indexes and retains; query (Grafana/Kibana) searches. Stdout is right in containers because the platform then owns capture — files inside a container vanish with it, and you’d have to mount volumes and rotate logs yourself.
3. They let you pull one request’s lines out of a store full of thousands of interleaved requests across many services. Without them you can only ask “are checkouts failing somewhere?”; with them, the single query trace_id = "4bf92f..." returns that exact request’s whole journey, hop by hop, in order.
4. It’s a feature because high-cardinality fields like user_id and trace_id belong in logs (and traces), where storage is cheap and cost is governed by volume/retention rather than by spawning a new metric series per value. The three levers: volume (log events, not loop iterations), retention (how long you keep them), and indexing (what you make searchable).
5. Never log secrets (passwords, API keys, tokens), PII/sensitive data (full card numbers, IDs, health data), or full request/response bodies by default. A logged secret is worse than no log because it’s been replicated across every node, backup, and engineer’s query history — a leaked token is a live vulnerability, whereas a missing log is merely inconvenient.