High availability and failure domains¶
Status: AI draft, 2026-11-16. Not yet verified by me. Written from the AWS Well-Architected reliability pillar, the AWS Builders' Library and the Kubernetes documentation; the availability arithmetic is worked out on this page.
In one line¶
Design for failure at each level (process, node, Availability Zone, region): run redundant copies across failure domains so one loss does not stop the service, keep stateful tiers replicated with quorum, fail fast with timeouts, retries with backoff and jitter and circuit breakers, and remember that dependencies in series multiply their availability while redundant copies in parallel multiply their unavailability.
The plain idea¶
Everything fails eventually: a process crashes, a node disappears, a whole data centre loses power, occasionally an entire region has trouble. High availability means asking, for each of those, "what happens to users?", and arranging things so the answer is "nothing much".
The tools are simple to name: redundancy (more than one of everything that matters), spread (copies in different failure domains, not side by side), detection (health checks that notice failure and move traffic), and degradation (when something is broken, fail quickly and partially rather than slowly and completely).
How it works¶
Failure domains¶
| Domain | Fails when | Protection |
|---|---|---|
| Process or container | Crash, OOM, deadlock | Restart policy, liveness probes, multiple replicas (Probes and graceful shutdown) |
| Node or instance | Hardware fault, kernel panic, Spot interruption | Replicas on different nodes, PDBs, autoscaling (Disruptions and PDBs) |
| Availability Zone | Power, network or facility problem | Replicas across zones, multi-AZ databases, zone-aware load balancing |
| Region | Rare large-scale event | Multi-region designs, DR plans (Backup and disaster recovery) |
Removing single points of failure¶
flowchart TD
U["Users"] --> LB["Load balancer: nodes in every zone"]
LB --> A1["App pod, zone a"]
LB --> A2["App pod, zone b"]
LB --> A3["App pod, zone c"]
A1 --> DB["Database primary in zone a,<br/>synchronous standby in zone b"]
A2 --> DB
A3 --> DB
Walk every request path and ask "if this one thing disappears, what stops?" Typical hidden single points: one NAT gateway for all zones, one CoreDNS replica, a single-zone EBS volume, a single database instance, one person who knows the runbook (NAT, endpoints and PrivateLink, EBS and EFS CSI drivers).
Stateless vs stateful tiers¶
- Stateless services (web and API tiers) are easy: run several copies anywhere, any copy can serve any request.
- Stateful tiers (databases, queues, caches) need replication and a way to agree on the current primary. Consensus systems need a quorum, a majority of members, which is why they run 3 or 5 members across 3 zones (etcd, backups and cluster upgrades).
Health checks and failover¶
Load balancers send traffic only to healthy targets; Kubernetes Services only to ready pods; databases fail over to a standby. Health checks must test the thing users need (can it serve a request?) without failing just because a shared dependency is slow, or every instance is removed at once (ALB and NLB).
Graceful degradation¶
| Technique | Protects against |
|---|---|
| Timeouts on every remote call | Hanging forever on a slow dependency, exhausting threads and connections |
| Retries with exponential backoff and jitter | Transient errors, without synchronised retry storms |
| Retry budgets and idempotency | Retries multiplying load, and duplicate side effects |
| Circuit breakers | Hammering a dependency that is already failing; fail fast instead |
| Fallbacks | Serving cached or reduced results when a non-critical dependency is down |
| Load shedding | Overload: reject some requests quickly so the rest succeed |
AWS's Well-Architected guidance (REL05-BP03) is to retry with exponential backoff, add jitter, and limit the number of retries; retries without these turn a small outage into a large one.
Availability arithmetic¶
- In series (each one required): multiply availabilities.
A = A1 × A2 × A3. - In parallel (any one is enough): multiply the unavailabilities.
A = 1 − (1 − A1) × (1 − A2).
Worked example (illustrative numbers):
| Component | Availability |
|---|---|
| Load balancer | 99.99% |
| One app instance | 99.5% |
| Two app instances in parallel | 1 − 0.005² = 99.9975% |
| Database (multi-AZ) | 99.95% |
- With one app instance: 0.9999 × 0.995 × 0.9995 ≈ 99.44%, about 242 minutes of downtime in 30 days.
- With two app instances: 0.9999 × 0.999975 × 0.9995 ≈ 99.94%, about 27 minutes in 30 days.
- The weakest serial component dominates: no amount of app redundancy makes this system better than its database.
Real failures are not independent (both instances share a zone, a deploy, a configuration), so treat these numbers as an upper bound.
Multi-AZ on AWS and Kubernetes¶
- Subnets, NAT gateways, load balancers and node groups in every zone the service uses (VPC basics).
- Pods spread with topology spread constraints on
topology.kubernetes.io/zone, plus PDBs (Scheduling). - Multi-AZ databases (RDS Multi-AZ, Aurora replicas) (RDS and Aurora).
- Enough spare capacity that losing one zone's worth of pods still serves peak load.
A concrete example¶
sample-app after an Availability Zone outage review:
- Found: all three replicas had been scheduled into one zone after a node replacement; the single NAT gateway lived in that zone.
- Fix: a topology spread constraint with
maxSkew: 1on zone andwhenUnsatisfiable: DoNotSchedule; minimum replicas raised to 3; a NAT gateway per zone with per-zone route tables. - Calls to the payment API got a 2-second timeout, 2 retries with jittered backoff, and a circuit breaker; the checkout page now shows "payment temporarily unavailable" instead of hanging.
- A game day drained one zone in staging to prove it.
Why not just...?¶
- ...run everything in one zone to save inter-AZ data transfer? One zone failure then takes everything down. Pay for redundancy where the service needs it, and design to limit cross-zone chatter.
- ...retry immediately and forever? Synchronised retries multiply load on a struggling dependency and turn a blip into an outage. Back off, add jitter, cap retries.
- ...go multi-region for everything? It multiplies cost and complexity (data replication, failover, testing). Most services need solid multi-AZ first; choose multi-region based on the business need.
Commands to know¶
| Command | What it does |
|---|---|
kubectl get pods -o wide -l app=sample-app |
Which nodes the replicas run on. |
kubectl get nodes -L topology.kubernetes.io/zone |
Zone of each node. |
kubectl get pdb -A |
Disruption budgets and allowed disruptions. |
aws ec2 describe-nat-gateways --query 'NatGateways[].{id:NatGatewayId,subnet:SubnetId}' |
Is there one NAT gateway per zone? |
aws rds describe-db-instances --query 'DBInstances[].{id:DBInstanceIdentifier,multiAZ:MultiAZ}' |
Which databases are Multi-AZ. |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
| Zone outage took the whole service down | Spread | Pod and node placement | Replicas in one zone; single-zone dependency (NAT, volume, database) |
| Small dependency outage became a full outage | Degradation | Timeouts, retries | No timeouts; retry storms; no circuit breaker |
| Health checks removed every instance at once | Health checks | What the check tests | Check depends on a shared dependency |
| Failover took far longer than expected | Stateful tier | Failover tests | Never rehearsed; DNS caching; connection pools pinned to the old primary |
| Redundant copies failed together | Correlation | Shared causes | Same deploy, config or zone |
Interview questions¶
1. What failure domains do you design for, and how?
Answer
- Process (restarts, replicas), node (spread across nodes, PDBs), zone (multi-AZ everything), region (DR or multi-region).
- Remove single points of failure along every request path.
- Keep spare capacity to absorb the loss of one domain.
2. Compute the availability of two 99.9% components in series and in parallel.
Answer
- Series: 0.999 × 0.999 = 99.8%.
- Parallel: 1 − 0.001² = 99.9999%.
- Real failures are correlated, so parallel figures are optimistic.
3. Why add jitter to retries?
Answer
- Without it, many clients retry at the same moments, creating synchronised load spikes.
- Jitter spreads retries out so the dependency can recover.
- Combine with exponential backoff, caps and retry budgets.
4. How do you make a Kubernetes workload survive the loss of a zone?
Answer
- At least one replica per zone with topology spread constraints, plus a PDB.
- Nodes available in every zone (node groups or Karpenter allowing all zones).
- Zone-independent dependencies: NAT per zone, multi-AZ databases, no single-zone volumes for critical state.
5. What is a circuit breaker?
Answer
- A client-side switch that stops calling a failing dependency after errors cross a threshold.
- Calls fail fast (or use a fallback) while open; it tests again after a pause.
- Prevents wasting resources and piling load on something already failing.
6. Your service is 99.99% available but depends on a 99.9% database. What can you promise?
Answer
- At most about 99.89% (0.9999 × 0.999) for paths that need the database.
- Improve the database (multi-AZ, failover), or reduce dependence (caching, degraded modes).
- Set SLOs to what the chain can deliver (SLOs and error budgets).
Try it¶
- Draw your service's request path and mark every component that exists only once.
- Compute the availability of: load balancer 99.99%, three app instances at 99% each in parallel, cache 99.9% (required), database 99.95%.
- Write a topology spread constraint that keeps
sample-apppods balanced across zones.
Zoom out¶
High availability is the structural side of reliability; SLOs say how much is enough, backups and DR cover the failures redundancy cannot, and deployment strategies stop changes from being the most common cause of outages (SLOs and error budgets, Backup and disaster recovery, Deployment strategies).