Mesh troubleshooting¶
Status: AI draft, 2026-11-16. Not yet verified by me. Written from the Istio and Envoy documentation; Istio was not installed for these notes, so the commands have no sample output.
In one line¶
Debug a mesh from the outside in: istioctl analyze for configuration mistakes, istioctl proxy-status to see whether proxies have their configuration (SYNCED or STALE), istioctl proxy-config to read the listeners, routes, clusters and endpoints a proxy actually uses, and Envoy access logs, whose response flags (NR, UH, UF, URX, UT, DC) say whether the mesh or the application failed.
The plain idea¶
With a mesh, every request passes through two proxies that each make decisions: routing, retries, mTLS, authorization. When a request fails, the first question is who said no? The application, the caller's proxy, the destination's proxy, or the network between them. The mesh gives you good tools to answer that, as long as you look at what the proxies actually have, not at the YAML you think you applied.
How it works¶
The order¶
flowchart TD
A["1. istioctl analyze:<br/>configuration mistakes"] --> B["2. istioctl proxy-status:<br/>is every proxy SYNCED?"]
B --> C["3. Access logs on both sides:<br/>status code and response flags"]
C --> D["4. istioctl proxy-config: routes, clusters,<br/>endpoints, secrets the proxy really has"]
D --> E["5. Application logs: if the proxies passed<br/>the request, the app answered"]
istioctl analyze¶
Checks Istio and Kubernetes resources together and reports problems such as a VirtualService pointing at a subset no DestinationRule defines, a Gateway with no matching selector, ports without protocol names, or namespaces missing injection labels. Run it before applying changes and in CI.
proxy-status¶
Lists every proxy with the state of each configuration type (CDS clusters, LDS listeners, EDS endpoints, RDS routes) and which istiod it talks to:
| State | Meaning |
|---|---|
SYNCED |
The proxy acknowledged the latest configuration istiod sent |
NOT SENT |
istiod has nothing to send for that type (often normal) |
STALE |
istiod sent an update the proxy has not acknowledged: a networking problem between them, or istiod overloaded |
proxy-config¶
| Command | Shows | Typical question |
|---|---|---|
istioctl proxy-config listeners <pod> |
Ports the proxy listens on | Is there a listener for this port at all? |
istioctl proxy-config routes <pod> |
Routing tables | Does my VirtualService route exist and match? |
istioctl proxy-config clusters <pod> |
Destinations and their policies (TLS mode, load balancing) | Is mTLS configured as expected for this destination? |
istioctl proxy-config endpoints <pod> |
Endpoint IPs and their health | Are there healthy endpoints behind this cluster? |
istioctl proxy-config secret <pod> |
Certificates the proxy holds | Is the workload certificate valid and not expired? |
These map to the Envoy concepts on Traffic management.
Access logs and response flags¶
Envoy's access log line shows the status code followed by response flags, which explain mesh-generated failures. Common ones:
| Flag | Meaning | Usual cause |
|---|---|---|
NR |
No route configured | VirtualService or DestinationRule mismatch, wrong host name |
UH |
No healthy upstream hosts | Subset selects no pods, all endpoints failing health or ejected by outlier detection |
UF |
Upstream connection failure | Often an mTLS mismatch (plaintext vs STRICT), or the app not listening |
URX |
Upstream retry limit (or connect attempt limit) exceeded | The destination kept failing through all retries |
UT |
Upstream request timeout | Route timeout shorter than the backend's response time |
UO |
Upstream overflow | Connection pool or circuit breaker limits in the DestinationRule |
DC |
Downstream connection terminated | The caller hung up (its own timeout, client cancelled) |
A 503 with a flag was produced by a proxy; a 503 with no flag in the destination's inbound log usually came from the application itself. Comparing the caller's outbound line with the destination's inbound line tells you which side failed (Sidecars and traffic flow, HTTP).
The usual causes of 503¶
| Cause | Signs |
|---|---|
| No healthy upstream | UH; proxy-config endpoints shows none or all unhealthy; subset labels do not match pods |
| mTLS mismatch | UF or URX, connection resets; a STRICT destination receiving plaintext, or a DestinationRule TLS mode disagreeing with the destination (mTLS and authorization) |
| Port naming or protocol | HTTP features missing, or a server-first protocol broken by detection; ports not named or no appProtocol |
| Routing | NR; VirtualService host or gateway binding wrong |
| Overflow | UO; connection pool limits too low for real traffic |
| Authorization | 403 with RBAC: access denied, not 503 |
A concrete example¶
After a deploy, frontend calls to reviews return 503:
istioctl analyze -n shop: a warning that VirtualServicereviewsrefers to subsetv3, which the DestinationRule does not define.- The frontend proxy's access log shows
503 UHfor the outbound call, andreviewspods show no inbound line: the mesh rejected it before it left. istioctl proxy-config endpoints deploy/frontend -n shopshows no endpoints for thev3cluster.- The new pods are labelled
version: 3instead ofversion: v3. Fixing the label restores traffic; the analyze check is added to CI.
Why not just...?¶
- ...restart the proxies or istiod first? It may hide the evidence (STALE state, logs) and usually does not fix a configuration error. Look first, then change.
- ...read the YAML to see what should happen? The proxy may not have it (STALE), or another resource may override it. Read the proxy's actual configuration with
proxy-config. - ...blame the mesh for every 503? Check the flags: a 503 the application returned passes through the proxies unchanged.
Commands to know¶
These were not run for this page.
| Command | What it does |
|---|---|
istioctl analyze -A |
Check all namespaces for configuration problems. |
istioctl proxy-status |
Sync state of every proxy. |
istioctl proxy-config routes deploy/frontend -n shop -o json |
Full routing configuration of one proxy. |
istioctl proxy-config endpoints deploy/frontend -n shop |
Endpoints and health per cluster. |
kubectl logs deploy/frontend -n shop -c istio-proxy --tail=50 |
Recent access log lines with response flags. |
istioctl x describe pod <pod> |
Policies and routes that apply to a pod. |
istioctl proxy-config log <pod> --level debug |
Raise Envoy's log level temporarily. |
How it breaks¶
| Symptom | First check | Likely cause |
|---|---|---|
503 UH |
proxy-config endpoints |
Subset or Service selects no ready pods |
503 UF after enabling STRICT |
Caller has a sidecar? DestinationRule TLS mode | Plaintext caller or conflicting TLS settings |
404 NR |
Routes, VirtualService host | No matching route |
| Config change ignored | proxy-status |
Proxy STALE, or resource selects nothing (analyze) |
504 UT |
Route timeout vs backend latency | Timeout too short, or slow backend |
403 RBAC: access denied |
AuthorizationPolicy | No ALLOW matches, or a DENY matches |
Interview questions¶
1. How do you start debugging a 503 in an Istio mesh?
Answer
- Read the access logs on both sides: status code and response flags.
- A flag means a proxy produced the error (UH, UF, NR, UO, UT); no flag in the destination's log points to the app.
- Then
proxy-config endpointsandrouteson the caller, andistioctl analyze.
2. What does istioctl proxy-status tell you?
Answer
- Whether each proxy has acknowledged the latest configuration for clusters, listeners, endpoints and routes.
- STALE means an update was sent but not acknowledged.
- Use it when a change seems to have no effect.
3. What do the response flags UH, UF and NR mean?
Answer
- UH: no healthy upstream hosts.
- UF: upstream connection failure, often mTLS mismatch.
- NR: no route configured for the request.
4. How do you tell an application error from a mesh error?
Answer
- Compare the caller's outbound log line with the destination's inbound line.
- Mesh errors carry response flags or never reach the destination; app errors show a normal inbound request returning an error code.
- Application logs confirm.
5. What is istioctl proxy-config used for?
Answer
- Reading a proxy's actual listeners, routes, clusters, endpoints and secrets.
- It shows what Envoy will do, not what the YAML intended.
- It is the main tool for routing, TLS and endpoint problems.
6. A VirtualService weight change has no effect. What do you check?
Answer
istioctl analyzefor errors;proxy-statusfor STALE proxies.proxy-config routeson a caller: is the new weight there?- Whether callers' traffic passes through the mesh at all, and the VirtualService host matches what callers use.
Try it¶
Without installing Istio:
- For each response flag in the table, write the first command you would run.
- Write two access log lines (caller outbound and destination inbound) for a request that fails because the application returned 500, and two for a request rejected with
UH. - Draft a one-page mesh runbook for your on-call rotation using the order diagram above.
Zoom out¶
Mesh troubleshooting follows the same layered approach as the Kubernetes and network playbooks: find which layer said no, then read that layer's real state (Troubleshooting playbook, The debugging walk).