Sidecars and traffic flow¶
Status: AI draft, 2026-11-16. Not yet verified by me. Written from the Istio documentation; Istio was not installed for these notes.
In one line¶
Istio injects an Envoy sidecar into each pod through a mutating webhook, redirects the pod's traffic into it with iptables rules (from an init container or the CNI agent), and reads each port's protocol from its name or appProtocol. Most surprises are protocol detection and start-up order.
The plain idea¶
The application keeps calling http://payments:8080 as if nothing changed. Underneath, the pod's network namespace has been rewired so every packet leaving or entering the app goes through the local Envoy first (What a container is, Firewalls and connection tracking). That Envoy, configured by istiod, adds mTLS, routing, retries and telemetry, then hands the traffic to the Envoy in the destination pod, which passes it to the destination app.
How it works¶
Injection¶
- Label a namespace
istio-injection=enabled(oristio.io/rev=<revision>when running control plane revisions); a pod can opt out with the labelsidecar.istio.io/inject: "false". - When a pod is created, Istio's mutating admission webhook adds the
istio-proxycontainer (and, without the CNI agent, theistio-initinit container) to the pod spec (CRDs, operators and admission webhooks). - Existing pods are only changed when they are recreated: restart Deployments after labelling a namespace or upgrading Istio.
Redirecting traffic into the proxy¶
| Method | How | Trade-off |
|---|---|---|
istio-init init container (default) |
Writes iptables rules in the pod's network namespace before the app starts | Pods need the NET_ADMIN and NET_RAW capabilities, which strict Pod Security levels forbid (Pod security) |
| Istio CNI node agent | A DaemonSet configures the redirection from the node as pods start | No elevated capabilities in application pods |
The rules send outbound traffic to Envoy's outbound port (15001) and inbound traffic to its inbound port (15006). Other ports used by the sidecar include 15000 (Envoy admin, pod-local), 15020 (merged Prometheus metrics), 15021 (health checks) and 15090 (Envoy's own Prometheus endpoint); applications must not use them.
The path of one request¶
flowchart TD
A["App A calls http://payments:8080"] --> E1["iptables in pod A redirect<br/>to local Envoy (outbound 15001)"]
E1 --> R["Envoy A: route, pick an endpoint,<br/>retries, timeout, start mTLS"]
R -->|"mTLS over the pod network"| E2["iptables in pod B redirect<br/>to Envoy B (inbound 15006)"]
E2 --> P["Envoy B: verify identity, apply<br/>authorization policy, record metrics"]
P --> B["App B on localhost:8080"]
Both proxies log the request and emit metrics, so one request appears twice in access logs (outbound in A, inbound in B), which is useful for telling which side failed (Mesh troubleshooting).
Protocol detection¶
Envoy needs to know whether a port carries HTTP, HTTP/2, gRPC or plain TCP to apply L7 features. Istio decides from the Service definition:
- the port name:
name: <protocol>[-<suffix>], such ashttp-web,grpc-api,tcp-db; - or the
appProtocolfield (takes precedence when both are set); - otherwise it tries automatic detection, which does not work for "server-first" protocols where the server speaks first (many databases).
A database port detected wrongly, or an HTTP port treated as TCP (no L7 retries, routing or metrics), are classic causes of confusing behaviour (Services and DNS).
Startup and shutdown ordering¶
- Startup: if the app starts before its proxy is ready, its first outbound calls fail. Setting
holdApplicationUntilProxyStarts: true(globally or per pod) puts the proxy first and blocks the other containers until it is ready. - Shutdown: if the proxy exits before the app finishes in-flight requests, those requests fail; configure drain durations and graceful shutdown so the proxy outlives the app's work (Probes and graceful shutdown).
- Jobs: a sidecar that never exits keeps a Job's pod running after the work finishes; Kubernetes native sidecar containers (init containers with
restartPolicy: Always) solve this where used (Jobs and CronJobs).
Resource overhead¶
Each sidecar requests CPU and memory and uses more with traffic and configuration size; across hundreds of pods this is significant (Requests, limits and QoS). Limiting which services each proxy needs to know about (the Sidecar resource's egress hosts) shrinks proxy memory in large meshes. Ambient mode avoids per-pod proxies entirely (Why a mesh, and Istio architecture).
A concrete example¶
A team enables injection for team-a:
kubectl label namespace team-a istio-injection=enabled, then a rolling restart of each Deployment; pods now show2/2containers.- The Istio CNI agent is installed so pods need no
NET_ADMIN, keeping the namespace at Pod Securitybaseline. - Service ports are renamed
http-webandtcp-postgres, so HTTP gets L7 features and the database connection is treated as TCP. holdApplicationUntilProxyStartsis set mesh-wide after an app crashed on startup trying to reach its config service.- Sidecar requests are tuned from observed usage.
Why not just...?¶
- ...label the namespace and wait? Injection happens only at pod creation. Restart workloads, and verify they show the extra container.
- ...leave port names generic? Protocol detection may guess wrong, and server-first protocols fail detection. Name ports or set
appProtocol. - ...run sidecars with tiny resource requests to save money? A CPU-starved proxy adds latency to every request in the pod. Size proxies from real usage.
Commands to know¶
These were not run for this page.
| Command | What it does |
|---|---|
kubectl get namespace -L istio-injection,istio.io/rev |
Which namespaces inject sidecars. |
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].name}' |
Is istio-proxy present? |
istioctl x check-inject -n team-a <pod> |
Why a pod was or was not injected. |
kubectl rollout restart deployment -n team-a <name> |
Recreate pods so they get (or lose) the sidecar. |
kubectl logs <pod> -c istio-proxy |
The sidecar's access and error logs. |
kubectl exec <pod> -c istio-proxy -- pilot-agent request GET stats |
Envoy statistics from inside the pod. |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
| Pod has no sidecar | Injection | Namespace label, pod label, webhook | Label missing or wrong revision, pod created before labelling, opted out |
| Pod rejected with capability errors | Redirection | Pod Security level | istio-init needs NET_ADMIN/NET_RAW; use the Istio CNI agent |
| App fails on startup, works after restart | Startup order | App logs, timing | App started before the proxy; use holdApplicationUntilProxyStarts |
| Errors during every rollout | Shutdown order | Termination timing | Proxy exits before the app drains |
| Database connections fail or hang | Protocol | Service port name, appProtocol |
Server-first protocol not declared as TCP |
| Job pods never complete | Lifecycle | Container status | Sidecar keeps running after the main container exits |
Interview questions¶
1. How does a sidecar get into a pod?
Answer
- The namespace (or pod) is labelled for injection.
- When the pod is created, Istio's mutating webhook adds the proxy container (and an init container unless the CNI agent is used).
- Existing pods need recreating.
2. How is traffic redirected into Envoy?
Answer
- iptables rules in the pod's network namespace send outbound traffic to Envoy's outbound port and inbound to its inbound port.
- Set by the
istio-initinit container (needsNET_ADMIN) or by the Istio CNI node agent. - The app is unaware and still calls normal Service names.
3. Why do Service port names matter in Istio?
Answer
- They (or
appProtocol) tell Istio the protocol, which decides whether L7 features apply. - Wrong or missing values lead to wrong detection; server-first protocols cannot be detected automatically.
- Name them like
http-web,grpc-api,tcp-db.
4. What startup and shutdown problems do sidecars cause, and how are they solved?
Answer
- Startup: the app runs before the proxy is ready; fix with
holdApplicationUntilProxyStarts. - Shutdown: the proxy stops before in-flight requests finish; fix with drain settings and graceful shutdown.
- Jobs never finishing; native sidecar containers or ambient mode avoid it.
5. Describe the path of a request between two meshed pods.
Answer
- App A to local Envoy (redirected), which routes, retries and starts mTLS to pod B.
- Envoy B terminates mTLS, checks authorization, and forwards to app B on localhost.
- Both proxies log and emit metrics for the request.
6. How would you reduce sidecar overhead in a large mesh?
Answer
- Right-size proxy requests from usage.
- Limit each proxy's configuration scope with the
Sidecarresource. - Consider ambient mode for namespaces that only need mTLS and L4.
Try it¶
Without installing Istio:
- For a Service with ports 80 (web), 5432 (PostgreSQL) and 9090 (gRPC), write port names that make Istio's protocol selection correct.
- On a kind cluster, look at a pod's network namespace with
kubectl debugandip addr; then sketch where the iptables redirection would sit (Network tools on Linux). - Write the pod annotation that keeps the app from starting before its proxy.
Zoom out¶
Sidecars are the Kubernetes pod pattern applied to networking (Pods); routing and security on top of them are on Traffic management and mTLS and authorization, and debugging them on Mesh troubleshooting.