Cluster Networking Internals

Services & Networking gave you the user-facing story: a Pod talks to http://api, DNS resolves it to a stable ClusterIP, and kube-proxy rewrites that virtual IP to a healthy backend Pod. It deliberately waved at the parts underneath — “the CNI is what lets Pod IPs route at all,” “kube-proxy programs the kernel.” This page is those parts. We’ll establish the network model Kubernetes mandates, see how a CNI plugin delivers it, then open up the three ways kube-proxy can turn a Service into kernel rules (iptables, IPVS, and the eBPF dataplane that replaces kube-proxy entirely), how CoreDNS implements the DNS spec, and how NetworkPolicy is enforced not by Kubernetes but by the CNI.

The thread: this machinery removes the manual job of hand-wiring routes, editing load-balancer and /etc/hosts files, and maintaining firewall ACLs by IP every time a Pod moves — the work that, done by hand, makes one forgotten entry into an outage or an open door.

The network model: routable Pod IPs, no pod-to-pod NAT

Kubernetes doesn’t ship a network. It ships three rules every network implementation must satisfy, and then lets a plugin satisfy them however it likes:

1. Every Pod gets its own cluster-unique IP. Not a shared host IP with port-mapping — a real IP.
2. Every Pod can reach every other Pod directly, across nodes, without NAT. The IP a Pod sees as its own is the same IP other Pods use to reach it.
3. Agents on a node (kubelet, kube-proxy) can reach all Pods on that node.

   THE FLAT MODEL — no NAT between Pods
   ┌───────── node A (10.0.0.5) ────────┐   ┌───────── node B (10.0.0.6) ───┐
   │  pod-w  10.244.1.7                  │   │  pod-api  10.244.2.4           │
   └───────────────┬─────────────────────┘   └──────────────┬────────────────┘
                   │   pod-w → 10.244.2.4 directly           │
                   └──────────── routed by the CNI ──────────┘
   The source IP that pod-api sees IS 10.244.1.7 — not node A's 10.0.0.5.

That “no NAT, identity preserved” rule is the load-bearing one. It means NetworkPolicy can match on real Pod IPs, service meshes can identify peers, and you never reason about a translation layer between two Pods. Contrast Docker’s default bridge, where containers behind one host hide behind that host’s IP and need port-mapping to be reachable — Kubernetes outlaws exactly that for Pod-to-Pod traffic.

Three IP ranges, three jobs

Don’t confuse the Pod CIDR (e.g. 10.244.0.0/16 — real, routable, one slice per node), the Service CIDR (e.g. 10.96.0.0/12 — virtual ClusterIPs that no interface owns; they exist only as kube-proxy rules), and the node IPs (the machines’ real addresses). A packet to a ClusterIP is always rewritten to a Pod IP before it leaves; ClusterIPs are never the source or final destination on the wire.

CNI: how a Pod gets its IP

When the kubelet creates a Pod sandbox (the pause container from Kubelet & CRI), it has a network namespace with no connectivity. It then invokes the configured CNI plugin — a binary in /opt/cni/bin, configured by JSON in /etc/cni/net.d/ — with the namespace and the verb ADD. The plugin’s contract is narrow and that’s the point:

   kubelet → CNI ADD(netns, podName)
        plugin must:
          • create a veth pair: one end in the Pod netns (eth0), one on the host
          • assign an IP from this node's slice of the Pod CIDR (IPAM)
          • set up routes so the IP is reachable from other nodes
        plugin returns: { ip: 10.244.1.7, routes, dns }
   ...later, on Pod teardown → CNI DEL(netns) frees the IP

How the plugin makes the IP routable across nodes is where plugins differ — and the choice is a real trade-off:

CNI plugin	Cross-node mechanism	Trade-off
Flannel (VXLAN)	Encapsulate Pod packets in UDP, tunnel node→node	Simple, works anywhere; encap overhead
Calico (BGP)	Advertise Pod routes via BGP; native routing, no encap	Fast, no overhead; needs an L3 fabric that allows it
Cilium (eBPF)	eBPF programs on the datapath; tunnel or native routing	Highest performance + policy/observability; newer, complex

Overlay plugins (VXLAN) wrap each Pod packet so the underlying network only ever sees node-to-node traffic — they work on any substrate but pay an encapsulation tax. Native-routing plugins (Calico BGP) make the underlying network itself aware of Pod routes — faster, but they need a network that will carry those routes. Same CNI contract above; very different packet on the wire.

kube-proxy: turning a Service into kernel rules

A ClusterIP is virtual, so something must translate “packet to 10.96.0.10:80” into “packet to a healthy Pod.” That something is kube-proxy, watching Services and EndpointSlices and programming the node’s kernel. Services & Networking made the key point: kube-proxy is not in the data path as a byte-forwarding process — it configures the kernel and gets out of the way. There are three implementations, and they scale very differently.

iptables mode (the long-time default)

For each Service, kube-proxy writes a chain of iptables rules; a packet to the ClusterIP jumps to that chain, which uses a statistical match to pick one backend and DNATs the destination to that Pod IP.

   -A KUBE-SERVICES -d 10.96.0.10/32 -p tcp --dport 80 -j KUBE-SVC-API
   -A KUBE-SVC-API  -m statistic --mode random --probability 0.3333 -j KUBE-SEP-1
   -A KUBE-SVC-API  -m statistic --mode random --probability 0.5000 -j KUBE-SEP-2
   -A KUBE-SVC-API                                                  -j KUBE-SEP-3
   -A KUBE-SEP-1 -j DNAT --to-destination 10.244.2.4:8080

It works and it’s battle-tested, but it’s a linear, sequential ruleset: matching is O(n) in the number of Services × endpoints, and every Service change rewrites a large rule set. On clusters with tens of thousands of Services, rule sync time and per-packet matching both become a problem.

IPVS mode

IPVS (IP Virtual Server) is the kernel’s purpose-built L4 load balancer. kube-proxy in IPVS mode keeps Service→endpoint mappings in hash tables, so lookup is O(1) regardless of Service count, and it offers real LB algorithms (rr, lc, wrr, …) instead of iptables’ statistical hack. It’s the answer when iptables’ linear scaling hurts.

eBPF / Cilium dataplane (replacing kube-proxy)

The newest model removes kube-proxy entirely. Cilium (and similar eBPF dataplanes) attaches eBPF programs at the socket and network-driver layers; Service translation, load balancing, and policy happen in those programs using eBPF maps as the lookup tables — often before a packet ever reaches the iptables/netfilter machinery. Run Cilium in kube-proxy-replacement mode and there is no kube-proxy DaemonSet at all. eBPF is the subject of its own page; the relevant point here is where it sits — in the kernel’s fast path, programmable without forwarding packets through a userspace proxy.

   iptables : O(n) linear chains    | mature, ubiquitous, slow to sync at scale
   IPVS     : O(1) hash + real LB    | better scale, still uses netfilter for the rest
   eBPF     : programmable fast path | best scale + built-in policy/observability; replaces kube-proxy

The choice is a scaling story, not a feature war

For a small cluster, iptables mode is completely fine and the simplest to reason about. IPVS earns its keep when the number of Services grows large enough that iptables sync time and O(n) matching bite. eBPF/Cilium goes furthest — folding Services, NetworkPolicy, and deep observability into one kernel-level dataplane — at the cost of a newer, more complex stack to operate. Pick by the problem you actually have.

CoreDNS: the cluster DNS spec made real

Half of service discovery is the stable name. The cluster runs CoreDNS (as an ordinary Deployment + Service), and the kubelet writes every Pod’s /etc/resolv.conf to point at CoreDNS’s ClusterIP with a search list that makes short names work. The Kubernetes DNS specification defines exactly what must resolve:

   A/AAAA records:
     <svc>.<ns>.svc.cluster.local            → the Service's ClusterIP
     <pod-ip-dashed>.<ns>.pod.cluster.local  → a Pod's IP   (10-244-2-4.ns.pod...)
   SRV records:
     _<port>._<proto>.<svc>.<ns>.svc...      → port + target for named ports
   Headless Service (clusterIP: None):
     <svc>.<ns>.svc.cluster.local            → A records for EVERY ready Pod IP

The search domains in resolv.conf (<ns>.svc.cluster.local, svc.cluster.local, cluster.local) are why curl http://api works from inside namespace prod: the resolver tries api.prod.svc.cluster.local first. CoreDNS’s kubernetes plugin watches the API server’s Services and EndpointSlices and answers from that live data — so a name resolves to the current ClusterIP the moment the Service exists, no zone files edited by hand.

ndots:5 is the classic latency trap

The default resolv.conf carries options ndots:5. Any lookup with fewer than five dots is tried against each search domain first. So resolving an external name like api.github.com (3 dots) fires several failed cluster-suffixed queries before the real one — multiplied across every request, that’s real latency and CoreDNS load. The fix is a fully-qualified name with a trailing dot (api.github.com.) or a tuned dnsConfig on the Pod.

NetworkPolicy: written by you, enforced by the CNI

By default the flat model is allow-all: any Pod can reach any Pod. A NetworkPolicy is how you add deny rules — but here’s the subtlety that surprises people: Kubernetes does not enforce NetworkPolicy. The API object is just declared intent. The CNI plugin is what enforces it, which is why NetworkPolicy silently does nothing on a CNI that doesn’t implement it (plain Flannel, for one).

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-allow-from-web, namespace: prod }
spec:
  podSelector: { matchLabels: { app: api } }   # applies TO these Pods
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: { matchLabels: { app: web } }   # only web Pods may connect
      ports:
        - { protocol: TCP, port: 8080 }

The crucial semantic: the moment any NetworkPolicy selects a Pod, that Pod flips from allow-all to default-deny for the covered direction — only explicitly listed traffic is then permitted. So a policy that “allows web→api” also blocks everything else to api, which is usually what you want but bites if you forget you applied it. The CNI translates the selector-based rule into dataplane rules: Calico into iptables/eBPF, Cilium into eBPF programs keyed on pod identity rather than raw IP (so the policy keeps holding as Pods churn and recycle IPs). This is the cluster-level face of identity, RBAC & network policy from the security part — RBAC governs who can call the API; NetworkPolicy governs which Pods can reach which Pods.

By the numbers — what ndots:5 costs one external lookup

The default Pod resolver carries options ndots:5 and a three-entry search list. Resolve an external name with fewer than five dots — say api.github.com (three dots) — and the stub resolver tries every search domain before the name as written, doubling each attempt for A and AAAA records:

   query 1  api.github.com.<ns>.svc.cluster.local   (A + AAAA)  → NXDOMAIN ×2
   query 2  api.github.com.svc.cluster.local        (A + AAAA)  → NXDOMAIN ×2
   query 3  api.github.com.cluster.local            (A + AAAA)  → NXDOMAIN ×2
   query 4  api.github.com.                         (A + AAAA)  → the real answer
   → up to 8 DNS packets, 6 of them guaranteed-doomed, for ONE name

Multiply that by every outbound request in a busy service and it’s real CoreDNS load and tail latency. A trailing dot (api.github.com.) makes the name fully qualified, skipping the search list and cutting eight packets to two — the cheapest DNS win in the cluster.

The thread, made concrete

The manual steps this whole stack removes are the oldest, most outage-prone jobs in networking. Without the flat CNI model, connecting a new service means a human allocating an IP, adding routes on every box, and praying nothing overlaps. Without kube-proxy, every Pod restart means re-pointing a load balancer by IP. Without CoreDNS, it means editing /etc/hosts or a DNS zone on each change — the very thing that, forgotten once, sends traffic into a black hole. Without NetworkPolicy enforced by the CNI, isolating a sensitive service means hand-maintaining firewall ACLs keyed on IPs that change hourly. Kubernetes turns each into declarative data reconciled continuously: Pod IPs are assigned and routed automatically, Service membership and DNS follow healthy Pods on their own, and reachability rules are expressed by label and enforced in the kernel. The cost is a layered dataplane (CNI + kube-proxy/eBPF + CoreDNS + policy) where a problem can hide in any layer — which is why connectivity debugging gets its own section in Debugging Kubernetes, and why the eBPF approach, which folds several of these layers into one programmable place, is worth the deep dive.

With placement (scheduler), execution (kubelet), and connectivity (this page) covered, the remaining internals govern what’s allowed in and how the cluster protects itself — admission control and security, the next pages in this Part.

The architect’s lens

Step back from the four layers and answer the five questions an architect asks of the cluster dataplane:

• Why does it exist? Because hand-wiring routes, editing load-balancer and /etc/hosts files, and maintaining firewall ACLs by IP every time a Pod moves is the most outage-prone job in networking. The flat CNI model, kube-proxy, CoreDNS, and NetworkPolicy turn each into declarative data reconciled continuously.
• What problem does it solve? It guarantees three rules — every Pod gets a unique routable IP, every Pod reaches every other without NAT (source identity preserved), node agents reach local Pods — and that no-NAT rule is what lets NetworkPolicy match real Pod IPs, meshes identify peers, and a virtual ClusterIP transparently DNAT to a healthy backend.
• What are the trade-offs? It’s a layered dataplane (CNI + kube-proxy/eBPF + CoreDNS + policy) where a fault can hide in any layer; the CNI choice is a scaling story — Flannel VXLAN’s encap tax vs Calico BGP needing an L3 fabric, and kube-proxy iptables’ O(n) chains vs IPVS’ O(1) hash vs eBPF replacing kube-proxy entirely; and ndots:5 fires up to 8 doomed DNS packets for one external name.
• When should I avoid a piece? iptables mode is fine — and simplest to reason about — for a small cluster; reach for IPVS or eBPF only when Service count makes sync time and O(n) matching bite. And don’t expect a NetworkPolicy to do anything on a CNI that doesn’t enforce it (plain Flannel silently ignores it).
• What breaks if I remove it? Without the flat CNI a human allocates IPs and routes per box; without kube-proxy every Pod restart re-points a load balancer by IP; without CoreDNS you edit a zone file on every change; without NetworkPolicy enforced by the CNI, isolating a sensitive service means hand-maintained ACLs keyed on IPs that churn hourly.

Check your understanding

1. State the Kubernetes network model’s core rules. Why is “no NAT between Pods, source IP preserved” the load-bearing one — what does it make possible?
2. What is the CNI plugin’s job when the kubelet creates a Pod sandbox, and how do an overlay plugin (Flannel VXLAN) and a native-routing plugin (Calico BGP) differ in making the Pod IP routable across nodes?
3. kube-proxy “isn’t in the data path.” Explain what iptables mode actually does to a packet aimed at a ClusterIP, and why IPVS mode scales better as the number of Services grows.
4. Why does curl http://api resolve from inside a namespace, and what is the ndots:5 latency trap?
5. A teammate applies a NetworkPolicy allowing web → api and suddenly a monitoring Pod can no longer scrape api. Explain why, and identify what component actually enforces the policy (and why it might do nothing on some clusters).

Show answers

1. Every Pod gets its own cluster-unique IP; every Pod can reach every other Pod directly across nodes without NAT; node agents can reach local Pods. The no-NAT rule is load-bearing because identity is preserved — the source IP a peer sees is the Pod’s real IP — which is what lets NetworkPolicy match on Pod IPs, service meshes identify peers, and you reason about Pod-to-Pod traffic with no translation layer.
2. On sandbox creation the CNI plugin creates a veth pair into the Pod’s netns, assigns an IP from the node’s slice of the Pod CIDR (IPAM), and sets up routes so the IP is reachable cross-node. Flannel (VXLAN) encapsulates Pod packets in UDP tunnels between nodes (works anywhere, encap overhead); Calico (BGP) advertises Pod routes so the underlying L3 network routes them natively (faster, no encap, but needs a fabric that carries the routes).
3. iptables mode writes per-Service chains; a packet to the ClusterIP jumps to the chain, a statistical match selects a backend, and the destination is DNATed to a real Pod IP — all in the kernel, kube-proxy only writes the rules. It’s a linear O(n) ruleset, so matching and rule-sync degrade with many Services. IPVS stores Service→endpoint mappings in hash tables (O(1) lookup) with real LB algorithms, so it scales far better.
4. The kubelet writes each Pod’s resolv.conf with CoreDNS as nameserver and a search list (<ns>.svc.cluster.local, …), so api is tried as api.prod.svc.cluster.local and resolves to the Service’s ClusterIP. The ndots:5 trap: names with fewer than 5 dots are tried against every search domain first, so an external name like api.github.com fires several failed cluster-suffixed queries before succeeding — extra latency and CoreDNS load; fix with a trailing-dot FQDN or tuned dnsConfig.
5. Selecting a Pod with any NetworkPolicy flips it from allow-all to default-deny for that direction, so only explicitly listed sources (here, web) are allowed — monitoring is now blocked because it isn’t listed. The policy is enforced by the CNI plugin (Calico/Cilium translating it into iptables/eBPF), not by Kubernetes itself — so on a CNI that doesn’t implement NetworkPolicy (e.g. plain Flannel) the object is accepted but does nothing.