How Kubernetes decides where to schedule a pod

The Problem Statement

Interviewer: "You run kubectl apply to deploy a pod. Kubernetes needs to pick a node to run it on. Walk me through how the scheduler decides which node gets the pod. What happens if no node fits?"

This question tests three things: your understanding of resource-aware scheduling in a distributed cluster, your knowledge of the constraint system (affinity, taints, topology spread) that lets operators control placement, and whether you can reason about failure modes like preemption and unschedulable pods.

Most candidates say "the scheduler picks the node with the most resources." Strong candidates walk through the full scheduling cycle: filtering nodes that meet hard constraints, scoring survivors on soft preferences, handling ties, and explaining what happens when no node passes the filter (priority preemption or the pod stays pending).

Clarifying the Scenario

You: "Before I walk through the scheduling flow, I want to make sure I scope this correctly."

You: "Are we talking about the default kube-scheduler, or should I also cover custom scheduler extensions and the scheduling framework plugins?"

Interviewer: "Focus on the default scheduler. Mention the plugin framework briefly."

You: "Got it. And should I assume a typical production cluster with resource requests, affinity rules, and taints already configured? Or a bare cluster with no constraints?"

Interviewer: "Production cluster. I want to see you reason about the full constraint set."

You: "Perfect. I will structure my answer around the scheduling cycle: how the scheduler watches for unscheduled pods, filters nodes by hard constraints, scores the survivors by soft preferences, binds the winning node, and handles the case where no node qualifies."

My Approach

I break this into five parts:

1. The scheduling cycle: How the scheduler picks up unscheduled pods and processes them one at a time through filter and score phases
2. Filtering (predicates): The hard constraints that eliminate nodes. A node either passes or fails. No partial credit.
3. Scoring (priorities): The soft preferences that rank surviving nodes. Higher score means better fit, but no node gets eliminated.
4. Binding and preemption: How the scheduler commits the decision, and what happens when zero nodes pass the filter phase
5. Advanced constraints: Topology spread, pod anti-affinity, and how operators use these to achieve high availability

The mental model I use: think of the scheduler as a two-stage funnel, similar to a feed ranking system. The filter phase is a hard pass/fail gate (like database WHERE clauses). The score phase is a weighted ranking (like ORDER BY with multiple columns). The best node wins.

For your interview: stating this five-part breakdown upfront signals structured thinking. I would say "Let me walk through the scheduling cycle in five parts" and list them. Interviewers can then choose which part to drill into, rather than watching you ramble through an unstructured answer.

The Architecture

Here is how the full cycle works:

The kube-scheduler watches the API Server for pods that have no spec.nodeName set. When it finds one, it places the pod into a priority queue sorted by PriorityClass (higher priority pods get scheduled first). The scheduler dequeues the highest-priority pod and starts the scheduling cycle.

The Filter phase evaluates every node in the cluster against hard constraints. Does the node have enough CPU and memory for the pod's resource requests? Does the node match the pod's node selector or node affinity? Is the node tainted with a taint the pod does not tolerate? If a node fails any filter, it is eliminated. There is no partial credit.

The Score phase evaluates all surviving nodes against soft preferences. Each scoring plugin returns a score from 0 to 100 for each node. The scores are weighted and summed. The node with the highest total score wins.

The scheduler then reserves the node (an optimistic lock to prevent double-booking), patches pod.spec.nodeName via the API Server, and the kubelet on that node takes over: it pulls the container image, creates the containers, and reports status back.

If zero nodes pass the filter phase, the scheduler checks whether any lower-priority pods on any node could be evicted (preempted) to make room. If so, it evicts the victims and retries. If preemption is not possible, the pod stays in Pending state with a FailedScheduling event.

I like to emphasize that the scheduler is not a "one-shot" decision. It continuously retries pending pods whenever cluster state changes (a new node appears, a pod terminates, resources are freed). The scheduling queue is not static. It reprocesses pods as the cluster evolves.

The Filter-Score Pipeline

This is the core of the scheduler. I want to walk through exactly which plugins run in each phase and how they interact.

Before diving into the plugins, here is a practical note on debugging. When a pod is stuck in Pending, the first thing I do is run kubectl describe pod <name>. The Events section shows exactly which filter failed and why:

Events:
  Type     Reason            Message
  ----     ------            -------
  Warning  FailedScheduling  0/10 nodes are available:
           3 Insufficient cpu,
           4 node(s) had untolerated taint {dedicated: ml},
           3 node(s) didn't match Pod's node affinity/selector

This tells you exactly which filters eliminated which nodes. In this example: 3 nodes lacked CPU, 4 had a taint the pod did not tolerate, and 3 did not match the affinity rule. Understanding the filter plugins lets you read this output instantly and know exactly what to fix.

The filter phase runs these plugins (in order):

1. NodeResourcesFit: Does the node have enough allocatable CPU, memory, and ephemeral storage for the pod's requests? This checks requests, not limits. A node with 4 CPU allocatable and 3.5 CPU already requested only has 0.5 CPU remaining. If the new pod requests 1 CPU, this node fails.

2. NodeName: If the pod specifies spec.nodeName, only that exact node passes. All others fail.

3. NodeAffinity: Evaluates requiredDuringSchedulingIgnoredDuringExecution rules. If a pod requires node-type=gpu, only nodes with that label pass.

The filter plugins run in a defined order, and the scheduler uses early termination: if a node fails the first filter, it skips the remaining filters for that node. This is important for large clusters where evaluating 6 plugins across 5,000 nodes would be expensive.

Here is a concrete example that ties several filters together. Suppose you deploy a pod with:

• resources.requests.cpu: 2
• nodeAffinity requiring label tier=compute
• Toleration for taint environment=staging:NoSchedule
• podAntiAffinity against pods with label app=my-app

The scheduler evaluates every node. Node-1 has 4 CPU free but lacks the tier=compute label, so it fails NodeAffinity. Node-2 has the label and the taint but only 1 CPU free, so it fails NodeResourcesFit. Node-3 has the label, 3 CPU free, tolerates the taint, but already runs a pod with app=my-app, so it fails InterPodAffinity. Node-4 passes all filters. Only Node-4 enters the scoring phase.

4. TaintToleration: If a node has a taint (like dedicated=ml-training:NoSchedule), the pod must have a matching toleration or the node fails.

5. PodTopologySpread: If the pod has topology spread constraints with whenUnsatisfiable: DoNotSchedule, nodes that would violate the spread constraint fail.

6. InterPodAffinity: If the pod has requiredDuringSchedulingIgnoredDuringExecution anti-affinity rules (like "do not co-locate with pods labeled app=redis"), nodes already running those pods fail.

A concrete scoring example: suppose 3 nodes pass the filter. Node-A scores 80 on ResourcesBalanced, 60 on NodeAffinity, and 90 on TopologySpread. Node-B scores 70, 90, and 50 respectively. Node-C scores 90, 40, and 70. With weights of 1, 2, and 2:

• Node-A: 80×1 + 60×2 + 90×2 = 80 + 120 + 180 = 380
• Node-B: 70×1 + 90×2 + 50×2 = 70 + 180 + 100 = 350
• Node-C: 90×1 + 40×2 + 70×2 = 90 + 80 + 140 = 310

Node-A wins with 380. Even though Node-B had the best affinity score and Node-C had the best resource balance, the weighted combination favors Node-A. This illustrates why understanding plugin weights matters for debugging scheduling decisions.

The score phase runs on every node that survived the filters. Each scoring plugin produces a value from 0-100. The scheduler normalizes these scores and applies configurable weights. The default weights give extra influence to topology spread and affinity (weight: 2) over resource balance (weight: 1).

For your interview: the critical insight is that filters are boolean (pass/fail) and scores are numeric (0-100). A pod might match 50 nodes on all filters, but the scores determine which of those 50 is "best." This is why you can have a pod that is schedulable (passes filters) but placed on a suboptimal node (low score).

Resource Bin Packing vs Spreading

When multiple nodes pass the filter phase, the scheduler must decide between two competing strategies: bin packing (fill nodes to capacity before using new ones) and spreading (distribute pods evenly across nodes). The default behavior uses a balanced approach, but understanding both extremes is important.

Bin packing packs pods tightly onto the fewest nodes possible. This is cost-efficient because empty nodes can be deallocated (especially in cloud environments with cluster autoscaler). The NodeResourcesFit scoring plugin can be configured with MostAllocated strategy to prefer nodes with the most resources already consumed.

Spreading distributes pods evenly across nodes. This provides better fault isolation (if a node dies, fewer pods are affected) and more headroom for burst traffic. The default LeastAllocated strategy prefers nodes with more free resources.

In practice, most production clusters use spreading for application pods (fault tolerance) and bin packing for batch jobs (cost efficiency). You can configure different scoring weights per scheduler profile, or even run multiple schedulers.

Priority Preemption and Pod Eviction

When no node passes the filter phase, the pod is technically unschedulable. But if the pod has a higher PriorityClass than pods already running on some nodes, the scheduler can evict (preempt) lower-priority pods to make room.

The priority system is one of the most powerful and most dangerous features in Kubernetes. Without it, a burst of low-priority batch jobs can consume all cluster resources, leaving critical production pods pending. With it, the scheduler can make explicit tradeoffs, but misconfigured priorities can also trigger cascading evictions.

I always recommend designing priority classes before they are needed in a real incident. Retrofitting priorities during an outage is stressful and error-prone.

Here is how preemption works:

1. The scheduler identifies all nodes where removing one or more lower-priority pods would make the node feasible for the pending pod.
2. It picks the node that requires evicting the fewest pods and the lowest-priority victims.
3. It deletes the victim pods (they get a graceful termination period).
4. Once the victims are gone and resources are freed, the pending pod retries scheduling and lands on that node.

This is how Kubernetes handles resource contention in production. Critical services (like your payment API) get PriorityClass 1000000. Background jobs (like log processing) get PriorityClass 100. When the cluster is full, the payment API pod preempts a log processing pod, not the other way around.

There are important guardrails around preemption:

• PodDisruptionBudget (PDB): If a Deployment has a PDB saying "min 2 replicas must be available," the scheduler will not preempt a pod if doing so violates the PDB. This protects critical services from cascading preemption.

• Graceful termination: Victims get a graceful shutdown period (default 30 seconds). They are not instantly killed. This matters for pods that need to drain connections or flush data.

• No cross-priority preemption: A pod with PriorityClass 500 can only preempt pods with PriorityClass < 500. Same-priority pods cannot preempt each other.

The Tricky Parts

1. Requests vs limits confusion at scheduling time: The scheduler only sees requests, not limits. A node with 4 CPU that has pods requesting 3.8 CPU appears full to the scheduler even if actual CPU usage is low. This is why over-provisioning requests wastes cluster capacity, and why right-sizing requests with tools like Vertical Pod Autoscaler matters.

2. Topology spread vs affinity conflicts: You can set both topologySpreadConstraints (spread pods across zones) and podAffinity (co-locate with database pods in the same zone). These constraints can conflict. Hard topology spread constraints can make the pod unschedulable if satisfying both is impossible. Prefer soft affinity and whenUnsatisfiable: ScheduleAnyway unless you truly need hard guarantees.

3. Scheduler latency in large clusters: In a 5,000-node cluster, the filter phase still evaluates a large set of nodes against multiple plugins. Even with early termination and percentageOfNodesToScore, scheduler latency can become visible when hundreds of pods enter the queue together.

4. Preemption cascading: Pod A preempts Pod B. Pod B's controller creates a replacement. That replacement may preempt Pod C. Without PodDisruptionBudgets, this can ripple across the cluster. Configure PDBs before you introduce aggressive priority classes.

5. DaemonSet special treatment: DaemonSets historically bypassed the scheduler and still behave differently from normal workloads. They tolerate node.kubernetes.io/unschedulable by default and are not common preemption victims, which surprises teams during maintenance operations.

6. Scheduling race conditions: The scheduler makes decisions from a snapshot of cluster state. Another pod can bind to the same node before your pod is bound. Kubernetes handles this with optimistic concurrency and retries, but you still see SchedulingRetry events in high-churn clusters.

7. Node NotReady during scheduling: A node can transition from Ready to NotReady after the scheduler chooses it but before kubelet starts the containers. The bind can still succeed because it is just an API Server update. The pod then sits in ContainerCreating until the node recovers or the pod is recreated elsewhere.

Debugging Scheduling Failures

When a pod is stuck in Pending, the debugging workflow follows a predictable sequence:

First, run kubectl describe pod <name> and read the FailedScheduling event. That event tells you exactly which filter blocked placement: insufficient CPU, affinity mismatch, taint mismatch, topology spread violation, or something else.

Second, inspect candidate nodes with kubectl describe node <name>. Look at allocatable resources, current requested resources, labels, and taints. This tells you whether the scheduler is correctly rejecting the node or whether the pod spec is simply unrealistic.

Third, check topology spread and affinity constraints together. A pod can have enough free CPU on a node and still be rejected because zone distribution or anti-affinity would be violated.

Fourth, confirm whether preemption is possible. If the pod is high priority but still pending, a PodDisruptionBudget may be blocking eviction of lower-priority victims.

What Most People Get Wrong

Follow-up: Autoscaler Interaction

If the interviewer asks "How does this interact with the cluster autoscaler?", I would say:

"The scheduler and autoscaler are separate components with a simple contract. The scheduler tries to place pods on existing nodes. If no node passes the filter phase and preemption is not possible, the pod stays Pending. The autoscaler watches for those Pending pods and provisions more capacity. Once the new node is Ready, the scheduler retries placement. The subtle point is that the new node does not belong to the pod that triggered scale-up; the highest-priority pending pod gets first access to it."

How I Would Communicate This in an Interview

Here is how I would actually say this:

"Kubernetes scheduling is a filter-then-score pipeline.

First, the scheduler filters out every node that cannot run the pod. That includes resource requests, node affinity, taints and tolerations, topology spread, inter-pod affinity, and volume constraints. This is where the most important production detail matters: the scheduler uses requests, not limits, for placement.

Then it scores the feasible nodes using plugins like resource balance, topology spread, and affinity. The highest-scoring feasible node wins. That means a pod can be schedulable on many nodes, but the score decides which one is best according to the cluster's policy.

If no node fits, the scheduler can try preemption for a higher-priority pod. If that still does not produce a feasible placement, the pod stays Pending and the cluster autoscaler may add capacity. The scheduler itself never creates nodes; it only chooses among the nodes that already exist.

When debugging, I start with kubectl describe pod and read the FailedScheduling events before I inspect nodes manually. That usually tells me whether the real issue is resource requests, constraints, or preemption blocking."

Interview Cheat Sheet

• "How does the scheduler pick a node?" -> Two-phase pipeline: filter (hard constraints, pass/fail) then score (soft preferences, 0-100 per plugin). Highest total score wins.
• "What are the main filter plugins?" -> NodeResourcesFit (CPU/memory), NodeAffinity (label matching), TaintToleration (taint/toleration pairs), TopologySpread (zone constraints), InterPodAffinity (co-location rules)
• "What is the difference between requests and limits?" -> Requests are what the scheduler uses for placement decisions. Limits are what kubelet enforces at runtime (CPU throttling, OOM kill). The scheduler never sees limits.
• "What are taints and tolerations?" -> Taints are on nodes ("I am special"). Tolerations are on pods ("I accept that specialty"). A node tainted gpu=true:NoSchedule only accepts pods that tolerate that taint. Used to reserve nodes for specific workloads.
• "What is topology spread?" -> Constraints that distribute pods across failure domains (zones, nodes). maxSkew: 1 with topologyKey: topology.kubernetes.io/zone means pods must be evenly spread across zones, with at most 1 pod difference between any two zones.
• "What happens when no node fits?" -> Scheduler checks for preemption. If the pending pod has higher priority than existing pods, it evicts victims. Otherwise, the pod stays Pending and the cluster autoscaler may provision a new node.
• "What is PodDisruptionBudget?" -> A guardrail that sets minAvailable or maxUnavailable for a set of pods. The scheduler will not preempt a pod if it violates the PDB. Prevents cascading evictions.
• "How does bin packing vs spreading work?" -> Configurable via NodeResourcesFit strategy: MostAllocated (bin pack) or LeastAllocated (spread). Spread for stateless services (fault tolerance), bin pack for batch (cost efficiency).
• "Can you run multiple schedulers?" -> Yes. Each scheduler watches for pods that specify its name in spec.schedulerName. Common pattern: one scheduler for interactive workloads, another for batch with different scoring weights.
• "How does the scheduler scale to large clusters?" -> percentageOfNodesToScore limits how many feasible nodes are scored (default: 50% or 100, whichever is larger). Filter phase uses early termination. Combined, these keep scheduling latency under 100ms per pod.

Test Your Understanding

Quick Recap

1. The kube-scheduler processes one pod at a time through a two-phase cycle: filter (hard constraints, pass/fail) and score (soft preferences, 0-100 per plugin per node). It continuously watches for pods with no spec.nodeName set.

2. The filter phase eliminates nodes that do not meet resource requests, node affinity rules, taint tolerations, topology spread constraints, or inter-pod affinity rules. If a node fails any single filter, it is eliminated with no partial credit.

3. The score phase ranks surviving nodes using weighted plugin scores for resource balance, affinity preference, taint counts, and topology spread evenness. Default weights give extra influence to topology spread and affinity (weight: 2) over resource balance (weight: 1).

4. The scheduler only evaluates resource requests, never limits. Limits are a runtime concern enforced by kubelet (CPU throttling if the pod exceeds its CPU limit, OOM kill if it exceeds its memory limit). A node can be full to the scheduler while running at 20% actual utilization.

5. When zero nodes pass the filter, the scheduler attempts priority preemption: evicting lower-priority pods to make room. PodDisruptionBudgets protect critical services from preemption cascades by setting minimum replica counts.

6. Bin packing (MostAllocated) concentrates pods onto fewer nodes for cost efficiency, enabling cluster autoscaler to remove empty nodes. Spreading (LeastAllocated) distributes pods for fault tolerance and burst headroom. Production clusters typically use spreading for stateless services and bin packing for batch.

7. Topology spread constraints distribute pods across failure domains (zones, nodes) with configurable maxSkew. The whenUnsatisfiable field controls whether violations are hard (DoNotSchedule) or soft (ScheduleAnyway with scoring preference).

8. The scheduler uses optimistic concurrency for binding: if the node state changes between scoring and binding (a race with another scheduling cycle), the bind fails and the pod re-enters the queue.

9. The percentageOfNodesToScore parameter limits how many feasible nodes are scored (default: at least 100 or 50% of cluster). This trades theoretical optimality for scheduling throughput in large clusters.

10. Debugging scheduling failures starts with kubectl describe pod to read FailedScheduling events, which tell you exactly which filter failed and how many nodes were eliminated. Then check node allocatable resources, taints, labels, and topology distribution.

Related Concepts

• Cluster autoscaler: The component that adds or removes nodes based on unschedulable pods and node utilization. Understanding how it interacts with the scheduler is a frequent follow-up. When a pod is pending because no node has capacity, the autoscaler detects this and provisions a new node. The scheduler then retries the pending pod.

• Vertical Pod Autoscaler (VPA): Automatically adjusts pod resource requests based on actual usage history. Since the scheduler uses requests for placement, VPA directly affects scheduling density. A pod requesting 4 CPU but only using 0.5 CPU wastes 3.5 CPU of schedulable capacity. VPA would lower the request to match actual usage, freeing capacity.

• Horizontal Pod Autoscaler (HPA): Scales the number of pod replicas based on CPU utilization, memory, or custom metrics. HPA creates new pods which need scheduling. This interacts with the scheduler and potentially the cluster autoscaler: HPA scales up replicas, scheduler tries to place them, if no nodes fit, autoscaler adds nodes. This three-component chain is the core autoscaling story.

• Resource quotas and limit ranges: Namespace-level controls that cap total resource requests and set default requests/limits for pods. These constrain what pods can request before they reach the scheduler. A resource quota capping a namespace at 10 CPU means the scheduler will never see a pod from that namespace requesting more than 10 CPU in aggregate.

• Node pools and cluster topology: How cloud providers organize nodes into pools (same instance type, same zone) and how this topology maps to Kubernetes labels, taints, and topology keys. Understanding node pools helps you reason about heterogeneous clusters where some nodes have GPUs, more memory, or specialized hardware.

• Custom scheduler extensions: The scheduling framework (since Kubernetes 1.15) lets you write plugins that add custom filter, score, and bind logic. Teams use this for GPU scheduling (match GPU type and count), NUMA-aware placement (pin memory-sensitive workloads to specific NUMA nodes), and gang scheduling (schedule all pods of a distributed training job simultaneously or not at all).