Architecture and the reconcile loop¶
Status: AI draft, 2026-11-16. Not yet verified by me. Commands and output were run on a local kind cluster (Kubernetes 1.37); addresses are rewritten to example ranges.
In one line¶
Kubernetes stores the desired state of everything in etcd behind the API server, and controllers keep comparing it with the actual state and acting to close the gap. The control plane (API server, etcd, scheduler, controller manager) decides; on each node the kubelet, the container runtime, kube-proxy and the CNI plugin carry it out.
The plain idea¶
With plain servers, you give orders: "start this process", "copy that file". If the process dies at 3 a.m., nothing restarts it unless you wrote something to.
Kubernetes works the other way round. You write down what you want ("three copies of sample-app, version 1.0.2, each with 256 MiB"), and Kubernetes keeps comparing that wish with reality. A copy crashed? Reality is two, the wish is three: start one. A node vanished? Its pods are gone: recreate them elsewhere. That endless compare-and-fix cycle is the reconcile loop, and almost every part of Kubernetes is a small program running one.
How it works¶
The parts¶
flowchart TD
U["kubectl, CI,<br/>GitOps"] --> A["kube-apiserver"]
A <--> E["etcd"]
A <--> S["scheduler"]
A <--> C["controller<br/>manager"]
A <--> K["kubelet,<br/>each node"]
K --> R["containerd"]
Every arrow goes through the API server: components watch it and write back to it. kube-proxy and the CNI plugin also run on each node (table below).
Control plane
| Component | Job |
|---|---|
| kube-apiserver | The front door. Authenticates and authorises every request (RBAC and service accounts), runs admission checks (CRDs, operators and admission webhooks), validates objects, and stores them in etcd. Every other component talks only to it. |
| etcd | A consistent, replicated key-value store holding every object. The cluster's only memory (etcd, backups and cluster upgrades). |
| kube-scheduler | Watches for pods with no node, picks the best node by filtering and scoring, and writes the choice back (Scheduling). |
| kube-controller-manager | Runs the built-in controllers: Deployment, ReplicaSet, StatefulSet, Job, Node, EndpointSlice, garbage collector and more. |
| cloud-controller-manager | Talks to the cloud: creates load balancers for Services, labels nodes with zones, removes nodes whose instances are gone. |
Each node
| Component | Job |
|---|---|
| kubelet | Watches for pods assigned to its node, asks the runtime to start them, runs probes, mounts volumes, reports status (Probes and graceful shutdown). |
| Container runtime | containerd or CRI-O, through the CRI (Container runtimes). |
| kube-proxy | Programs iptables, IPVS or nftables rules so Service addresses reach pods (Services and DNS). Some network plugins replace it. |
| CNI plugin | Gives each pod a network interface and address (Pod networking and NetworkPolicy). |
On a managed service such as EKS, the provider runs the control plane for you; you see only the API endpoint and your nodes (EKS architecture and node options).
Objects: spec and status¶
Every object has a spec (what you want) and a status (what is true now, written by controllers). You write the spec; you never write the status. kubectl get and describe show both. Objects also carry metadata: name, namespace, labels, annotations, and owner references.
What happens after kubectl apply of a Deployment¶
sequenceDiagram
participant K as kubectl
participant A as API server
participant C as Controllers
K->>A: apply Deployment
A->>A: auth, admit,<br/>store in etcd
C->>A: Deployment controller:<br/>create ReplicaSet
C->>A: ReplicaSet controller:<br/>create 3 Pods
C->>A: Scheduler:<br/>bind Pods to nodes
C->>A: kubelet: start,<br/>report Ready
A-->>K: rolled out
- The API server checks who you are, whether you may do this, runs admission (defaults, policies), and stores the Deployment.
- The Deployment controller sees a Deployment with no matching ReplicaSet and creates one.
- The ReplicaSet controller sees zero pods for a ReplicaSet that wants three, and creates three pod objects with no node.
- The scheduler sees unscheduled pods, picks a node for each, and records it.
- The kubelet on that node sees pods assigned to it, pulls images, starts containers through the runtime, runs probes, and reports status.
No component calls another directly. Each one watches the API server for the objects it cares about and writes its results back. That is why the system keeps working when any single component restarts: when it comes back, it reads the current state and continues.
A concrete example¶
A node in a three-node cluster loses power.
- Its kubelet stops reporting. After a grace period the node controller marks the node
NotReadyand taints it. - Pods on the node are eventually marked for eviction (by default after about five minutes of the node being unreachable, through a toleration Kubernetes adds to pods).
- Each pod's ReplicaSet now counts fewer ready replicas than desired and creates replacements.
- The scheduler places the new pods on the two healthy nodes, if they have room.
- Nobody ran a command. The desired state (three replicas) never changed; the controllers closed the gap.
If the pods had been created bare, with no controller, step 3 would never happen: nothing owns them, so nothing replaces them (Pods).
Why not just...?¶
- ...run containers with a script that starts them on servers? It works until a server dies, a container crashes, or you need to change the version on fifty machines at once. The reconcile loop handles all three with the same mechanism.
- ...let components talk to each other directly? Every pair would need to know about every other. With everything going through the API server, each component only watches objects, and new controllers (yours included) plug in the same way.
- ...edit the running state by hand (
kubectl edit,kubectl scale) in production? It works, until the next deploy or GitOps sync puts back what is written in Git (GitOps principles). Change the declared state, not the live one.
Commands to know¶
The node and the control plane, which on kind runs as pods on the same node:
$ kubectl get nodes -o wide
NAME STATUS ROLES VERSION INTERNAL-IP CONTAINER-RUNTIME
notes-lab-control-plane Ready control-plane v1.37.0 172.18.0.2 containerd://...
$ kubectl get pods -n kube-system -o custom-columns=NAME:.metadata.name,STATUS:.status.phase
NAME STATUS
coredns-559f6c778d-dz9l7 Running
coredns-559f6c778d-p79mw Running
etcd-notes-lab-control-plane Running
kindnet-jzwmm Running
kube-apiserver-notes-lab-control-plane Running
kube-controller-manager-notes-lab-control-plane Running
kube-proxy-z9rd4 Running
kube-scheduler-notes-lab-control-plane Running
Columns trimmed. kindnet is kind's CNI plugin. On managed services you do not see the control-plane pods.
The chain of owners the controllers created:
$ kubectl apply -f deploy.yaml
deployment.apps/sample-app created
$ kubectl get pod <a sample-app pod> -n demo -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}'
ReplicaSet/sample-app-dcc66c4d7
$ kubectl get rs sample-app-dcc66c4d7 -n demo -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}'
Deployment/sample-app
| Command | What it shows |
|---|---|
kubectl get <kind> <name> -o yaml |
The full object: spec, status, metadata. |
kubectl get events -n <ns> --sort-by=.lastTimestamp |
What controllers and the kubelet did, in order. |
kubectl get --raw /readyz?verbose |
The API server's own health checks. |
kubectl api-resources |
Every object type the API server knows, including custom ones. |
kubectl get <kind> -w |
Watch changes live, as the controllers do. |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
kubectl times out or connection refused |
API server | Endpoint, network path, credentials | API server down, wrong context, network blocked |
Pods created but stay Pending |
Scheduler | kubectl describe pod events |
No node fits (Scheduling) |
Pods scheduled but stay ContainerCreating |
kubelet, runtime, CNI | Pod events; kubelet and runtime logs on the node | Image pull, volume mount or network setup failing |
| Deployment exists but no pods appear | Controllers | kubectl get rs; events |
Controller manager unhealthy, or admission rejecting pods (quota, policy) |
Node NotReady |
kubelet | kubectl describe node conditions |
kubelet stopped, runtime down, disk or memory pressure, network partition |
| Manual changes keep reverting | Reconciliation | Who owns the object? | A controller or GitOps tool restoring the declared state |
Interview questions¶
1. Name the control-plane components and what each does.
Answer
- kube-apiserver: the only entry point; authenticates, authorises, admits, validates and stores objects.
- etcd: the consistent key-value store holding all cluster state.
- kube-scheduler: assigns pods to nodes.
- kube-controller-manager: runs controllers that reconcile desired and actual state; cloud-controller-manager talks to the cloud.
2. What runs on every worker node?
Answer
- The kubelet, which runs the pods assigned to the node and reports status.
- A container runtime such as containerd, used through the CRI.
- kube-proxy (or a replacement) for Service routing, and the CNI plugin for pod networking.
3. What is a reconcile loop?
Answer
- A controller repeatedly compares the desired state (spec) with the actual state (status and the world).
- It takes the smallest action to close the difference, then checks again.
- It makes the system self-healing and lets components restart safely.
4. Walk me through what happens when you apply a Deployment.
Answer
- The API server authenticates, authorises, admits and stores it in etcd.
- The Deployment controller creates a ReplicaSet; the ReplicaSet controller creates pods.
- The scheduler binds each pod to a node; that node's kubelet starts the containers and reports Ready.
5. A node goes down. What does Kubernetes do, and what does it not do?
Answer
- The node controller marks it NotReady; after a timeout its pods are evicted.
- Controllers (ReplicaSets, StatefulSets) create replacements, which the scheduler places on healthy nodes.
- Bare pods with no controller are not replaced; and volumes tied to that node or zone may block StatefulSet pods.
6. Why does every component talk only to the API server?
Answer
- One place for authentication, authorisation, validation and auditing.
- Components stay decoupled: each watches objects and writes results, so they can restart or be replaced independently.
- Custom controllers and operators plug in the same way.
Try it¶
Step 1. Create a kind cluster, or use an existing one, and look at the control plane.
kind create cluster --name arch-demo
kubectl get pods -n kube-system
Step 2. Create a Deployment, then delete one of its pods. Predict what happens before you run the last command.
kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl rollout status deployment/web
kubectl delete pod $(kubectl get pods -l app=web -o jsonpath='{.items[0].metadata.name}')
kubectl get pods -l app=web
Step 3. Read the events to see which controller did what, then delete the cluster.
kubectl get events --sort-by=.lastTimestamp | tail -10
kind delete cluster --name arch-demo
Zoom out¶
Every Kubernetes topic is a variation on this page: an object type with a spec, and a controller that reconciles it, whether it is a Deployment, a Service, a Karpenter NodePool or an Argo CD Application (CRDs, operators and admission webhooks). The same desired-state idea drives GitOps one level up (GitOps principles). The principle to carry: change the declared state and let controllers do the work, and when something is wrong, ask which controller should have acted and why it did not.