Karpenter in depth¶
Status: AI draft, 2026-11-16. Not yet verified by me. Written from the Karpenter documentation (
karpenter.sh/v1andkarpenter.k8s.aws/v1APIs); nothing was run.
In one line¶
Karpenter launches EC2 instances to fit pending pods, guided by NodePools (what may be launched, limits, disruption rules) and EC2NodeClasses (how: AMI, subnets, security groups, role), and tracks each launch as a NodeClaim. It then disrupts nodes on purpose (consolidation, drift, expiration), rate-limited by disruption budgets and PDBs, and handles Spot interruptions from an SQS queue.
The plain idea¶
Karpenter is a controller that turns "these pods cannot be scheduled" into "here is a machine shaped for them" (Cluster Autoscaler vs Karpenter). You do not list instance types one group at a time; you describe the space it may choose from ("compute or memory families, generation 5 or newer, Spot preferred, these three zones, at most 1000 CPUs in total") and it picks.
Because it chose every node, it also feels free to change its mind later: replace two half-empty nodes with one, swap an On-Demand node for a cheaper one, or retire nodes built from an old AMI. That continuous reshuffling is where the savings come from, and where most of the surprises come from.
How it works¶
The three objects¶
| Object | API | Purpose |
|---|---|---|
| NodePool | karpenter.sh/v1 |
What may be launched: requirements, taints, labels, expireAfter, limits, weight, disruption settings |
| EC2NodeClass | karpenter.k8s.aws/v1 |
How to launch on AWS: AMI selection (for example the al2023@latest alias), subnet and security group selectors (often by a discovery tag), IAM role, kubelet settings, block devices |
| NodeClaim | karpenter.sh/v1 |
One per node Karpenter launched: its chosen type, zone, capacity type and status |
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: general
spec:
template:
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
requirements:
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values: ["4"]
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
- key: topology.kubernetes.io/zone
operator: In
values: ["ap-south-1a", "ap-south-1b", "ap-south-1c"]
expireAfter: 720h
limits:
cpu: "1000"
memory: 1000Gi
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 1m
budgets:
- nodes: "10%"
From a pending pod to a node¶
sequenceDiagram
participant S as Scheduler
participant K as Karpenter
participant E as EC2
S->>S: no node fits,<br/>pod marked unschedulable
K->>K: batch pending pods, match<br/>NodePools by requirements and taints
K->>K: compute instance types that fit,<br/>within limits, by price
K->>E: CreateFleet with<br/>many candidate types
E-->>K: instance launched
K->>K: create NodeClaim,<br/>wait for the node to register
S->>S: binds pods to the new node
- Karpenter respects the pods' requests, nodeSelector, node affinity, tolerations, topology spread and volume zones when choosing (Scheduling).
- Several NodePools may match a pod. Make them mutually exclusive (taints, labels), or give them a weight: the highest-weight NodePool that can be used wins, falling back when its limits are reached.
- Limits cap a NodePool's total CPU and memory; once reached, it launches nothing more.
Disruption¶
Karpenter removes or replaces nodes in two ways:
| Kind | Methods | Respects NodePool budgets | Respects PDBs and do-not-disrupt |
|---|---|---|---|
| Graceful (voluntary) | Drift, consolidation | Yes | Yes; replacement launched first when needed |
| Forceful | Expiration, Spot interruption, node deletion | No | PDBs only, and only until the termination grace period |
- Node auto repair, when enabled, follows its own rules: it replaces unhealthy nodes without waiting for budgets or
do-not-disrupt, and stops if too many nodes in a pool are unhealthy at once. - Consolidation:
WhenEmptyremoves only nodes with no workload pods;WhenEmptyOrUnderutilizedalso removes or replaces underused nodes with cheaper ones;Balancedconsiders the same nodes but only acts when the saving is worth the disruption.consolidateAfterwaits after pod changes before acting. - Drift: a node whose NodePool or EC2NodeClass no longer matches it (new AMI, changed requirements) is replaced. This is how node upgrades happen (EKS upgrades).
- Expiration:
expireAfter(for example 720h) retires nodes older than that, whatever their state; changing it drifts existing nodes. - Budgets:
spec.disruption.budgetscaps how many nodes may be voluntarily disrupted at once; if none is set, the default is one budget ofnodes: 10%. Budgets can have schedules, for example no disruption during business hours. - Termination: Karpenter taints the node
karpenter.sh/disrupted:NoSchedule, evicts pods respecting PDBs (Disruptions and PDBs), then terminates the instance. Its finalizer on the node makes this happen in order (Finalizers and deletion).
Opting out¶
- On a pod:
karpenter.sh/do-not-disrupt: "true"blocks voluntary disruption of its node while it runs; a duration such as"30m"protects it only for that long after start. Use it for jobs that cannot be restarted halfway (Jobs and CronJobs). - On a node: the same annotation blocks voluntary disruption of that node.
- Neither stops forceful methods such as expiration or Spot interruption.
Spot interruptions¶
EC2 warns two minutes before reclaiming a Spot instance. Karpenter receives these warnings (and rebalance recommendations, scheduled maintenance and instance state changes) from an SQS queue fed by EventBridge rules, which you create when installing it. On a warning it cordons and drains the node and launches a replacement right away. Without the queue, the node simply disappears. Diversify: allowing many instance types and zones lowers the chance of interruptions and of InsufficientInstanceCapacity.
A concrete example¶
A cluster with one general NodePool (above) and one batch NodePool.
batchallows only large sizes, carries a taintworkload=batch:NoSchedule, and hasconsolidationPolicy: WhenEmptyso running jobs are never consolidated away.- Batch Job pods tolerate the taint and set
karpenter.sh/do-not-disrupt: "true". generalusesWhenEmptyOrUnderutilizedwith a budget ofnodes: "10%", plus a scheduled budget ofnodes: "0"during business hours to avoid churn at peak.- EC2NodeClass
defaultuses theal2023@latestAMI alias; when AWS publishes a new AMI, nodes drift and are replaced gradually within the budget. - Karpenter itself runs on a two-node managed node group (EKS architecture and node options).
Why not just...?¶
- ...allow only one instance type, to keep things predictable? Karpenter then cannot fall back when that type has no capacity in a zone, and Spot interruptions rise. Allow families and sizes, and constrain with requirements.
- ...turn consolidation off to stop churn? Nodes then stay half-empty and cost grows. Keep it, and make workloads disruption-friendly, or use budgets and
WhenEmptywhere needed. - ...set
do-not-disrupton everything? Then drift and consolidation never happen, AMIs never update by drift, and only forceful expiration remains. Reserve it for work that cannot be interrupted.
Commands to know¶
These were not run for this page.
| Command | What it does |
|---|---|
kubectl get nodepools |
Pools, their usage against limits, readiness. |
kubectl get nodeclaims -o wide |
Every node Karpenter launched: type, zone, capacity type, age. |
kubectl describe nodeclaim <name> |
Launch errors and disruption conditions (drifted, expired). |
kubectl get ec2nodeclasses |
Node classes and whether their AMIs, subnets and security groups resolved. |
kubectl logs -n <karpenter-namespace> deploy/karpenter |
Why it launched, did not launch, or disrupted a node. |
kubectl get events -A --field-selector source=karpenter |
Karpenter's events on pods and nodes. |
kubectl annotate pod <pod> karpenter.sh/do-not-disrupt=true |
Protect one running pod's node from voluntary disruption. |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
| Pods Pending, log says no instance type met the scheduling requirements | Requirements | Pod requests and constraints against NodePool requirements | Pod too large for allowed sizes, wrong arch, zone or taint mismatch |
| NodePool launches nothing | Limits or class | kubectl get nodepools, EC2NodeClass status |
Limits reached; AMI, subnet or security group selectors match nothing |
InsufficientInstanceCapacity in logs |
EC2 capacity | Allowed types and zones | Too few instance types or zones allowed |
| Nodes replaced every few minutes | Consolidation | Disruption events | Aggressive policy with churning pods; tighten budgets or consolidateAfter |
| A long job was killed mid-run | Disruption | Pod annotation, node events | Consolidation or drift without do-not-disrupt, or expiration (forceful) |
| Spot nodes vanish without draining | Interruption handling | SQS queue setting, EventBridge rules | Interruption queue not configured |
| Nodes launch but never become Ready | Bootstrap | Node role access, subnets, AMI | Missing access entry for the node role, no route to the API server, wrong AMI |
Interview questions¶
1. What are NodePool, EC2NodeClass and NodeClaim?
Answer
- NodePool: what may be launched (requirements, taints, limits, disruption rules).
- EC2NodeClass: how to launch on AWS (AMI, subnets, security groups, role).
- NodeClaim: one launched node and its chosen type, zone and status.
2. Walk through what happens when a pod is Pending in a Karpenter cluster.
Answer
- The scheduler marks it unschedulable; Karpenter batches pending pods.
- It matches NodePools by requirements and tolerations, finds instance types that fit within limits, and calls EC2 with many candidates.
- A NodeClaim tracks the launch; the node registers and the scheduler binds the pods.
3. Explain consolidation, drift and expiration.
Answer
- Consolidation: remove empty or underused nodes, or replace with cheaper ones.
- Drift: replace nodes that no longer match their NodePool or EC2NodeClass, such as after an AMI change.
- Expiration: replace nodes older than
expireAfter, forcefully.
4. How do you stop Karpenter from disrupting a running batch job?
Answer
karpenter.sh/do-not-disrupt: "true"on the pod, or a duration.- A NodePool with
WhenEmptyconsolidation for batch nodes. - It still cannot stop forceful events such as Spot interruption or expiration, so long jobs should checkpoint.
5. How does Karpenter handle Spot interruptions?
Answer
- EventBridge rules send interruption warnings to an SQS queue that Karpenter reads.
- On a two-minute warning it cordons and drains the node and launches a replacement.
- Diversifying instance types and zones reduces interruptions.
6. Karpenter keeps logging that no instance type meets the pod's requirements. What do you check?
Answer
- The pod: requests, nodeSelector, affinity, tolerations, volume zones.
- The NodePools: requirements, taints, limits; whether any allowed type is big enough in an allowed zone.
- The EC2NodeClass status and capacity errors in the logs.
7. What protects production from Karpenter replacing too many nodes at once?
Answer
- NodePool disruption budgets (10% of nodes by default), optionally scheduled.
- PodDisruptionBudgets on the workloads, honoured during voluntary disruption.
- Replicas spread across nodes and zones, and graceful shutdown in the apps.
Try it¶
Without AWS:
- Write a NodePool for GPU jobs that allows only GPU instance categories, taints its nodes, and never consolidates non-empty nodes. Write the pod spec that would land there.
- For the example cluster, list what happens step by step when AWS publishes a new AL2023 AMI.
- Explain to a colleague, in three sentences, why a pod with
do-not-disruptcan still be stopped.
Zoom out¶
Karpenter is the reconcile loop applied to capacity: desired (pending pods) versus actual (nodes), corrected continuously (Architecture and the reconcile loop). Its value is cost and speed; its price is frequent, controlled disruption, which only well-built workloads absorb (Probes and graceful shutdown). EKS Auto Mode runs it for you (EKS architecture and node options).