The Scheduler in Depth

Scaling & Scheduling drew the scheduler as a two-phase filter-then-score loop: drop the nodes a Pod can’t fit on, rank what’s left, bind to the winner. That picture is true but it’s a silhouette. This page fills it in. The real kube-scheduler is not a hard-coded algorithm — it’s a small framework with a fixed set of extension points, and every piece of placement logic (resource fit, affinity, taints, topology spread, even preemption) is a plugin registered at one of those points. Once you see the extension points, every scheduling feature stops being a special case and becomes “a plugin at point X.”

The thread for this page: the scheduler removes the manual job of a human staring at a fleet and choosing which box has room and which constraints it satisfies — a job that, done by hand across hundreds of nodes and a dozen overlapping rules, is both slow and quietly wrong. Encoding those rules as plugin data makes placement reproducible and auditable instead of tribal.

Scheduling is constraint satisfaction, one Pod at a time

Strip away the Kubernetes vocabulary and the scheduler is solving a classic problem: given a set of items (Pods) with requirements (CPU, memory, “must be near X,” “must not be near Y,” “must spread across zones”) and a set of bins (nodes) with capacity and labels, assign each item to a bin so every hard requirement holds, optimising soft preferences. That’s bin-packing with constraints — NP-hard in the general case.

Kubernetes makes it tractable with two deliberate simplifications:

• One Pod at a time. The scheduler does not solve for the whole queue globally. It pops one Pod off a priority queue, places it, commits, and moves on. This is greedy, not optimal — but it’s fast and predictable, and the reconciliation loop cleans up the rest over time.
• Hard constraints filter, soft constraints score. A hard requirement (a Pod needs 4Gi and the node has 2Gi free) eliminates a node outright. A soft preference (prefer spreading across zones) only nudges the ranking. Splitting the problem this way is what lets a single Pod be placed in milliseconds instead of seconds.

Why greedy is good enough

A globally optimal packer would have to reconsider every placement whenever a new Pod arrives — and the cluster changes constantly anyway. The scheduler bets that fast and mostly-right, continuously beats slow and optimal, occasionally. When greedy paints itself into a corner (a high-priority Pod won’t fit), preemption — covered below — is the escape hatch.

The scheduling framework: extension points

Modern kube-scheduler runs each Pod through a scheduling cycle (which must pick a node) followed by a binding cycle (which writes the decision). Each cycle is a sequence of named extension points, and plugins register at the points they care about. This is the architecture that replaced the old monolithic “predicates and priorities.”

   Pop Pod from scheduling queue
        │
   ┌────────────── SCHEDULING CYCLE (synchronous, one Pod) ──────────────┐
   │  PreFilter   compute Pod-wide state once (e.g. total requests)      │
   │  Filter      per node: can this Pod run here?  → feasible node set  │
   │  PostFilter  ran only if Filter found ZERO nodes → try preemption   │
   │  PreScore    precompute state for scoring                           │
   │  Score       per (surviving) node: a number; higher is better       │
   │  NormalizeScore  rescale each plugin's scores to 0–100              │
   │  Reserve     tentatively claim the node's resources (in-memory)     │
   │  Permit      allow / deny / WAIT (e.g. gang scheduling)             │
   └─────────────────────────────────┬──────────────────────────────────┘
   ┌──────────────────── BINDING CYCLE (may be async) ───────────────────┐
   │  PreBind     do work before binding (e.g. attach/provision a volume) │
   │  Bind        write the chosen node into Pod.spec.nodeName via API    │
   │  PostBind    cleanup / notify                                        │
   └─────────────────────────────────────────────────────────────────────┘

The split between cycles matters: the scheduling cycle is serial (only one Pod is being decided at a time, so resource accounting stays consistent), while the binding cycle can run concurrently because the node is already chosen and Reserve has already accounted for the resources. PreBind is where the slow, I/O-heavy work lives — notably waiting for a CSI volume to be provisioned and attachable (see Storage Internals (CSI)) — so it must not block the next Pod’s placement.

Extension point	Plugin returns	Used for
PreFilter	ok / error + shared state	One-time per-Pod precompute; reject impossible Pods early
Filter	pass / fail per node	Hard constraints: resource fit, taints, node affinity, volume limits
PostFilter	a victim set or “no luck”	Preemption — only when Filter yields zero nodes
Score	an integer per node	Soft preferences: spread, affinity weight, least/most allocated
Reserve / Unreserve	—	Tentatively book resources; undo on later failure
Permit	allow / deny / wait	Gang scheduling, external approval
Bind	—	Persist nodeName (default plugin just PATCHes the Pod)

Built-in plugins you’ll meet by name: NodeResourcesFit (Filter + Score), NodeAffinity, InterPodAffinity, TaintToleration, PodTopologySpread, NodePorts, VolumeBinding, NodeResourcesBalancedAllocation, and ImageLocality. You can re-weight scoring plugins or disable filters per scheduler profile in the scheduler’s config — multiple profiles can run inside one kube-scheduler process, and a Pod selects one via spec.schedulerName.

Filter plugins: the hard constraints

A node survives Filter only if every Filter plugin passes it. The important ones:

• NodeResourcesFit — does the node have enough allocatable CPU/memory/ephemeral-storage to honour the Pod’s requests? (Requests, not limits — limits never affect placement; see Resource Management & Node Pressure.)
• NodeAffinity — does the node’s labels satisfy the Pod’s required nodeAffinity (requiredDuringSchedulingIgnoredDuringExecution)? nodeSelector is the blunt subset of this.
• TaintToleration — for every node taint with effect NoSchedule/NoExecute, does the Pod carry a matching toleration? No toleration → node filtered out.
• InterPodAffinity — does the node satisfy required pod (anti-)affinity given which Pods are already running where?
• NodePorts, VolumeBinding, PodTopologySpread — port conflicts, can the Pod’s volumes bind on this node (topology), and do required spread constraints hold?

nodeAffinity, podAffinity, anti-affinity

These are the expressive successors to nodeSelector. nodeAffinity attracts a Pod to nodes by their labels. podAffinity/podAntiAffinity attract or repel a Pod relative to other Pods — the constraint is evaluated over a topology domain named by topologyKey (a node label like kubernetes.io/hostname or topology.kubernetes.io/zone).

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:   # HARD → a Filter
        nodeSelectorTerms:
          - matchExpressions:
              - { key: topology.kubernetes.io/zone, operator: In, values: ["us-east-1a","us-east-1b"] }
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:   # HARD: never two replicas on one host
        - labelSelector: { matchLabels: { app: web } }
          topologyKey: kubernetes.io/hostname
      preferredDuringSchedulingIgnoredDuringExecution:  # SOFT → a Score
        - weight: 100
          podAffinityTerm:
            labelSelector: { matchLabels: { app: web } }
            topologyKey: topology.kubernetes.io/zone

Read the field names literally: requiredDuringScheduling… is a hard constraint enforced at Filter; preferredDuringScheduling… is a soft preference applied at Score. The …IgnoredDuringExecution half means the rule is checked when placing, not re-checked while the Pod runs — a Pod is never evicted because a label changed under it (the NoExecute taint effect is the exception that does act during execution).

podAntiAffinity is O(pods × nodes) — handle with care

Inter-pod affinity must, for each candidate node, look at the Pods already scheduled across the whole topology. On large clusters required anti-affinity with a wide topologyKey is expensive and can make Pods unschedulable in subtle ways. For the common “spread my replicas” case, prefer topologySpreadConstraints (below) — it was built precisely to be cheaper and more predictable than anti-affinity.

Taints and tolerations: repulsion lives on the node

Scaling & Scheduling gave the mnemonic — attraction lives on the Pod, repulsion on the node. The depth detail is the three effects, because they hook different extension points:

Taint effect	Enforced at	Meaning
NoSchedule	Filter (scheduling time)	Don’t place new Pods here unless they tolerate it
PreferNoSchedule	Score (a penalty)	Avoid if possible, but allowed
NoExecute	the node controller, at runtime	Evict already-running Pods that don’t tolerate it

NoExecute is the one that acts during execution: when a node goes NotReady, the node controller taints it node.kubernetes.io/not-ready:NoExecute, and Pods are evicted after their tolerationSeconds expires (default 300s, set by the DefaultTolerationSeconds admission plugin). That is the mechanism behind “Pods drain off a dead node after five minutes.”

Score plugins: ranking the survivors

Every node that survives Filter goes to Score. Each Score plugin returns 0–100 (after NormalizeScore), the scheduler multiplies by the plugin’s configured weight, sums across plugins, and binds to the highest total. Key scorers:

• PodTopologySpread — rewards nodes that keep the Pod’s label set evenly distributed across topology domains.
• InterPodAffinity — applies the preferred affinity/anti-affinity weights.
• NodeResourcesBalancedAllocation — prefers nodes where CPU and memory utilisation end up balanced, avoiding a node that’s 90% CPU / 10% memory.
• NodeResourcesFit with a scoring strategy: LeastAllocated (spread, the default) vs MostAllocated (bin-pack tightly, good for autoscaling cost) vs RequestedToCapacityRatio.
• ImageLocality — prefers nodes that already have the Pod’s image cached, shaving pull time.

topologySpreadConstraints

This is the modern, first-class way to spread Pods. maxSkew bounds how unbalanced the distribution across domains may get; whenUnsatisfiable decides whether an unmet constraint is a hard filter (DoNotSchedule) or a soft score (ScheduleAnyway).

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule          # hard: zones must stay within skew 1
      labelSelector: { matchLabels: { app: web } }
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway         # soft: prefer host spread
      labelSelector: { matchLabels: { app: web } }

With maxSkew: 1 across zones, if zone A has 3 matching Pods, zones B and C must each have at least 2 before another lands in A. It expresses “keep my replicas evenly spread so one zone failure can’t take me down” far more cheaply than required anti-affinity, and it composes — host-level and zone-level constraints together.

Priority and preemption: making room for what matters

Filtering can fail. Sometimes no node passes — every machine is full. Greedy placement would simply leave the Pod Pending. Priority plus preemption is the escape hatch, and it runs at the PostFilter extension point (only when Filter returned zero feasible nodes).

A PriorityClass assigns Pods an integer priority. When a high-priority Pod can’t be scheduled, the preemption plugin asks: is there a node where, if I evicted some lower-priority Pods, this Pod would fit? If yes, it picks the node requiring the fewest, lowest-priority victims, deletes them (respecting their graceful termination and, on a best-effort basis, their PodDisruptionBudgets), and the pending Pod is scheduled in the freed space.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: high-priority }
value: 1000000
preemptionPolicy: PreemptLowerPriority    # or Never: jump the queue but don't evict anyone
globalDefault: false
description: "Latency-critical services that may evict batch workloads."

   payments Pod (priority 1,000,000) is Pending — no node fits
        │  Filter → 0 feasible nodes
        ▼  PostFilter / preemption plugin runs:
   node-7 holds 2 batch Pods (priority 100) using just enough CPU
        │  evict the 2 batch Pods (graceful, honor PDB if possible)
        ▼
   payments Pod schedules onto node-7   (batch Pods re-queue, wait for room)

Priority is also queue order

Priority does double duty. Beyond preemption, it sets the order Pods come off the scheduling queue — higher-priority Pods are considered first. So a flood of low-priority batch jobs can’t starve a latency-critical service of scheduling attention, even before any eviction happens. Two built-in classes, system-cluster-critical and system-node-critical, sit far above the highest value a user-defined class may set (one billion), protecting control-plane and node-critical add-ons from ever being preempted.

By the numbers — the scheduler doesn’t even look at every node

One-Pod-at-a-time would still be slow if every placement examined all 5,000 nodes. So on large clusters the scheduler stops the feasibility search early — once it has found “enough” feasible nodes — and scores only those. The default cutoff shrinks as the cluster grows:

   feasible nodes to find ≈ max( numNodes × (50 − numNodes/125)% , 100 )   (floor: 5%)
   100 nodes   → the floor of 100 dominates → it checks ~all of them
   5,000 nodes → 50 − 5000/125 = 10%        → stop after ~500 feasible nodes, score those

The trade-off is explicit: the scheduler accepts a good node found quickly over the globally best node found slowly. Spreading quality can suffer slightly on huge clusters (it may not survey every candidate), which is exactly why percentageOfNodesToScore is a tunable — raise it when even placement matters more than scheduling throughput.

The thread, made concrete

The manual step the scheduler removes is a human deciding placement: reading free capacity off a dashboard, remembering that the GPU nodes are reserved, recalling that the two database replicas must never share a host or a zone, and SSHing in to start the process on the box they picked. Every one of those rules now lives as declarative data — requests, tolerations, affinity, spread constraints, priority — evaluated identically every time by Filter and Score plugins. The safety win is that placement policy becomes reviewable and reproducible: it’s in the manifest, it’s in code review, and it can’t be forgotten at 2 a.m. The cost is real complexity — these knobs interact, and an over-constrained Pod that won’t schedule is its own failure mode (a Pending Pod whose events read 0/40 nodes are available — see Debugging Kubernetes).

Once the scheduler has chosen a node and written spec.nodeName, its job is done. Everything after — pulling the image, setting up the sandbox, wiring the network, running the container — belongs to the agent on that node. That’s the Kubelet & Container Runtime, next.

The architect’s lens

Step back from the extension points and answer the five questions behind how placement is decided:

• Why does it exist? Because placement by hand — reading free capacity off a dashboard, remembering the GPU nodes are reserved, recalling two DB replicas must never share a host or zone — is slow and quietly wrong across hundreds of nodes. The scheduler encodes those rules as plugin data evaluated identically every time.
• What problem does it solve? It solves constraint satisfaction one Pod at a time: hard constraints filter (resource fit on requests, taints, required affinity, topology spread), soft preferences score (spread, LeastAllocated, image locality), and Reserve → Bind writes spec.nodeName — making placement policy reviewable and reproducible in the manifest.
• What are the trade-offs? It’s greedy, not globally optimal (it bets fast-and-mostly-right continuously beats slow-and-optimal occasionally); on huge clusters it scores only the first percentageOfNodesToScore feasible nodes (~10% at 5,000), trading spread quality for throughput; and required podAntiAffinity is O(pods × nodes).
• When should I avoid a knob? Prefer topologySpreadConstraints over wide required anti-affinity for the common “spread my replicas” case; don’t over-constrain a Pod into permanent Pending (0/40 nodes are available); and reserve a high PriorityClass + preemption for workloads whose eviction of others is genuinely justified.
• What breaks if I remove it? A human picks the box again and placement becomes tribal instead of in-manifest; without preemption a high-priority Pod that fits nowhere just stays Pending; and without priority-driven queue order a flood of batch jobs starves a latency-critical service of scheduling attention — long before any eviction.

Check your understanding

1. Why does the scheduler place Pods one at a time and greedily, rather than solving for the whole pending queue optimally? What feature compensates when greedy placement gets stuck?
2. A plugin needs to reject a Pod that can never fit anywhere, do per-node feasibility checks, and rank the feasible nodes. Which three extension points map to those three jobs?
3. Distinguish requiredDuringSchedulingIgnoredDuringExecution from preferredDuringSchedulingIgnoredDuringExecution — which extension point does each hook, and what does the IgnoredDuringExecution half promise?
4. The three taint effects (NoSchedule, PreferNoSchedule, NoExecute) are enforced in different places. Name where each acts, and which one can evict a Pod that is already running.
5. Walk through what happens when a Pod with PriorityClass value 1,000,000 finds zero feasible nodes. Which extension point runs, how are victims chosen, and what role does priority play before any eviction?

Show answers

1. Global optimal placement is NP-hard and the cluster changes constantly, so the scheduler bets that fast-and-mostly-right, continuously beats slow-and-optimal occasionally: it pops one Pod off a priority queue, places it greedily, commits, and moves on, letting the reconciliation loop clean up over time. When greedy gets stuck (a high-priority Pod fits nowhere), preemption at PostFilter frees room.
2. Reject-the-impossible-early is PreFilter; per-node feasibility (can it run here?) is Filter; ranking the survivors is Score.
3. required… is a hard constraint enforced at the Filter extension point — fail it and the node is dropped. preferred… is a soft preference applied at Score — it only nudges the ranking. IgnoredDuringExecution means the rule is checked only when placing the Pod, not re-checked while it runs, so a Pod is never evicted because a label changed under it.
4. NoSchedule acts at Filter (scheduling time — blocks new Pods). PreferNoSchedule acts at Score (a soft penalty). NoExecute acts at runtime via the node controller and is the one that evicts already-running Pods lacking the toleration (after tolerationSeconds).
5. Filter returns 0 feasible nodes, so the PostFilter preemption plugin runs: it looks for a node where evicting some lower-priority Pods would make room and picks the node needing the fewest, lowest-priority victims, deleting them gracefully (honoring PDBs best-effort). Before any eviction, priority also sets scheduling-queue order, so the high-priority Pod is considered first and can’t be starved of scheduling attention by low-priority jobs.