Question index¶
Status: generated. Built by
tools/build-question-index.pyfrom every page's interview questions. Do not edit by hand.
In one line¶
Every interview question in these notes, 1024 of them across 165 pages, grouped by section. Answer from memory, then open the page and compare with the hidden answer.
How to drill with this page¶
- Pick one section and set a timer.
- Answer each question aloud or on paper before opening anything.
- Open the page, expand the answer, and score yourself: 2 for every point, 1 for some, 0 for none or wrong.
- A section passes at 70%. Anything scored 0 goes on your list for the next loop.
Start here¶
- Describe everything that happens when a user opens
https://shop.example.comserved from EKS. - Where does TLS usually terminate in this design, and what are the implications?
- The site returns
503from the load balancer after a deploy. Walk through your diagnosis. - How does traffic reach a pod with no public address?
- Some users see an old version of the site after a DNS change. Which hop, and why?
- A request fails only for large file uploads. Which hops would you suspect?
- A user says "the website is down". Walk me through how you diagnose it.
- What is the difference between "connection refused" and "connection timed out", and what does each tell you to check?
- Why should you run checks from the failing component rather than from your laptop?
pingto a server fails but the website works. What do you conclude?- A health check passes but users get errors. How can that be?
- You get a
504from the load balancer. Where in the walk are you, and what next?
Linux¶
- What happens between typing
ls -land seeing its output? - Why is
cda shell builtin and not a program in/usr/bin? - What is an inode, and what does a directory actually contain?
- What is the difference between a hard link and a symbolic link?
- A script works when you run it, but fails in cron with
command not found. Why? rm -rf $DIR/*deleted far more than intended. What went wrong?- Two versions of
terraformare installed and the wrong one runs. How do you find out why?
- Explain the output
-rwxr-x--- 1 app deploy deploy.sh. Who can do what? - How does Linux decide which permission set applies to a process?
- What do
r,wandxmean on a directory? - What are umask, setuid and the sticky bit?
- A colleague fixed "permission denied" with
chmod -R 777. What do you say? - You added
cito thedockergroup, butdocker psstill says permission denied. Why? - An application running as
appcannot read/opt/sample-app/config/app.yaml, yet the file is mode644. What do you check?
- What is the difference between a program and a process?
- What is the difference between SIGTERM and SIGKILL?
- A container exits with code 137. What does that tell you, and what do you check?
- What is a zombie process, and how do you get rid of one?
- What do exit codes 126, 127 and 130 mean?
- Why does
docker stoptake 10 seconds for some containers, and how do you fix it? - A server has load average 30 but CPU is mostly idle. What is going on?
Streams, pipes and redirection
- What are stdin, stdout and stderr, and why does stderr exist?
- What is the difference between
cmd > file 2>&1andcmd 2>&1 > file? - What does a pipe do, and does the second command wait for the first to finish?
- What is
set -o pipefailfor? - A cron job's log shows nothing on nights it failed. What do you change?
- A colleague runs
sort data.txt > data.txtand the file is now empty. Why?
Text tools: grep, sed, awk, find
- How do you find the 5 IP addresses that made the most requests in an access log?
- Why does
uniqneed sorted input? - When would you use
awkinstead ofcut? - How do you show 3 lines of context around every "Exception" in a log, case-insensitively?
- How do you replace a value in a config file safely with
sed? - A disk is filling with old logs. How do you find and delete logs older than 14 days under
/var/log/appsafely? - Using only the access log, how would you work out whether a spike of 502s came from one client or from everyone?
- What is systemd, and what is a unit?
- What is the difference between
After=andRequires=? - You edited a unit file and restarted the service, but nothing changed. Why?
- A service keeps restarting. Walk me through your investigation.
- How do you change a setting of a packaged service without your change being lost on upgrade?
- What happens when you run
systemctl stopon a service? - When would you choose supervisord over systemd?
- Where do logs go on a modern Linux server?
- How do you see only errors from one service in the last hour?
- What is the difference between
copytruncateand rotating with a reopen signal? - You are paged at 3 a.m. for a failing service. How do you read its logs efficiently?
- After logrotate ran, the application's log file is empty but the application is clearly busy. Why?
- Why are structured logs better than free-text logs?
- Walk through what it takes to use a new disk on a Linux server.
- What is the difference between
dfanddu? - A disk shows 100% in
df, butdufinds only half of that. What is happening and how do you fix it? - "No space left on device", but
df -hshows 40% free. What do you check? - You enlarged a cloud volume from 50 GB to 100 GB, but the server still shows 50 GB. Why?
- What is the safe way to clear a huge log file that a running service is writing to?
- Why should
/etc/fstabuse UUIDs, and what doesnofaildo?
free -hshows 200 MiB free on a 16 GiB server. Is that a problem?- What is the difference between VSZ and RSS?
- What does the OOM killer do, and how does it choose?
- A container was OOM-killed, but the host had 8 GiB free. How?
- How would you confirm that a process was OOM-killed on a plain Linux server?
- A Java service in a 2 GiB container sets
-Xmx2gand keeps getting OOM-killed. What is wrong? - Should a production server use swap?
- What does load average mean on Linux, and how do you judge whether 8 is high?
- What is the USE method?
- In
vmstat, what dor,b,waandsttell you? - A server's load is 20 on 4 CPUs, but CPU is 80% idle. What is happening and what do you run?
- A containerised service has high latency but its CPU use never reaches its limit. What could it be?
- When would you use
strace, and what is the risk? - Users say a web app is slow. Walk me through your first five minutes on the server.
- What happens when you run
ssh user@host? - What are
known_hostsandauthorized_keys, and on which side does each live? - Why is
ProxyJumpsafer than agent forwarding? - You get
Permission denied (publickey)with a key you know is correct. What do you check? - SSH warns that the remote host identification has changed. What do you do?
- How do you reach a database in a private subnet from your laptop?
- How would you harden SSH on a public server?
- What is the difference between
apt updateandapt upgrade? - How do you find which package installed a given file, and what files a package installed?
- How do you stop one package from being upgraded?
- You patched OpenSSL on a server, but a scanner still reports the vulnerability. Why?
- How do you add a third-party repository safely?
- Should production servers patch themselves automatically?
- What does
set -euo pipefaildo, line by line? - Why should you quote variables, and what goes wrong if you do not?
- What is
trapused for in scripts? - How do you read a file line by line correctly in bash?
- A script runs fine by hand but fails in cron. What do you check?
- How do you debug a bash script that does the wrong thing?
- Write the outline of a script that deploys only if a health check passes, and rolls back otherwise.
- Explain
30 2 * * 1-5. - Why does a script that works in your shell fail under cron?
- How do you stop a cron job from overlapping with itself?
- What does a systemd timer give you that cron does not?
- Where would you look to see whether a cron job ran last night?
- A daily cleanup must run at 23:00 India time, but the server uses UTC. What do you write?
- How do you list the names of all pods that are not Running, using
kubectlandjq? - What do
-r,-cand--argdo in jq? - How do you give a default when a field might be missing?
- Why use yq instead of sed to change an image tag in a manifest?
- A script runs
jq ".items[] | select(.name == \"$NAME\")"and breaks for some values. Why, and what is the fix? - How would you see which resources a Terraform plan will destroy, from a script?
Networking¶
- What happens at L2, L3, L4 and L7? Give one example protocol and one device for each.
- What is encapsulation?
- What is the difference between a MAC address and an IP address?
- What does ARP do, and does a host ARP for a destination on another subnet?
- What is the difference between an L4 and an L7 load balancer?
- A service cannot reach a database. How do you use the layers to narrow it down?
- Small API responses work through a VPN, but large downloads hang. What layer and what cause?
- How many addresses are in a
/24, a/20and a/28? - What does
/16mean in10.0.0.0/16? - What are the private IPv4 ranges, and why do they need NAT to reach the internet?
- Split
10.0.0.0/16into 6 equal subnets. What prefix and what ranges? - Does
10.0.1.0/24overlap with10.0.0.0/20? How do you tell quickly? - Two teams want to peer their VPCs, but both used
10.0.0.0/16. What are the options? - A Kubernetes cluster on AWS stops scheduling pods with IP allocation errors. How is that a subnetting problem?
- What is a default route, and when is it used?
- Explain longest prefix match with an example.
- What is NAT and why do private networks need it?
- What is the difference between SNAT and DNAT?
- Instances in a private subnet suddenly cannot reach the internet. What do you check?
- How does traceroute work?
- A request reaches the server, but the client times out waiting for the reply. What could cause that at the routing level?
- Walk me through the TCP three-way handshake.
- What is the difference between TCP and UDP? Give a use for each.
- What is the difference between "connection refused" and "connection timed out"?
- What is TIME-WAIT, and should thousands of them worry you?
- What is an ephemeral port?
- A service cannot connect to a database on port 5432.
nctimes out. Walk through your next steps. - Connections from an application pool fail with resets after the system has been quiet for a while. Why?
- What does DNS do, and what does a user see when it fails?
- What is the difference between a recursive resolver and an authoritative server?
- What is the difference between an A record and a CNAME, and why can the top of a zone not be a CNAME?
- What is a TTL, and what does "DNS propagation" really mean?
- Walk me through what
dig +trace www.example.comdoes. - You changed an A record ten minutes ago. Some users still reach the old server. Why, and how do you confirm it?
curlsays "Could not resolve host" on one server, but the same name resolves on your laptop. What do you check, in order?digreturns the correct address, but the application still connects to the wrong one. What could explain it?
- What is the difference between a
401and a403? - What do
502,503and504each mean, and who usually sends them? - What does idempotent mean, and why does it matter for retries?
- What is the
Hostheader for? - What does HTTP/2 change compared with HTTP/1.1?
- After moving TLS termination to a load balancer, the site loops between redirects. Why?
- Users report slowness. How would you use
curlto find which part of a request is slow?
- What does TLS give you?
- Explain the certificate chain of trust.
- What is SNI and why is it needed?
- A site works in Chrome but
curland a Java client fail with a certificate error. What is the likely cause? - What does "TLS termination at the load balancer" mean, and what are the trade-offs?
- Why does forward secrecy matter?
- How do you stop certificate expiry from causing outages?
- What is the difference between an L4 and an L7 load balancer?
- How do health checks work, and how can a bad one cause an outage?
- What do
502,503and504from a load balancer tell you? - What is connection draining, and why does it matter during deploys?
- How does the application learn the real client address behind a proxy, and what is the risk?
- Users get occasional
502s under normal load, with no deploys. What would you check? - When would you choose sticky sessions, and what is the better alternative?
Firewalls and connection tracking
- What is the difference between a stateful and a stateless firewall?
- What is the difference between DROP and REJECT?
- Explain connection tracking and the ESTABLISHED state.
- Which iptables chains does a packet pass through when it is routed through a host, for example a container's traffic?
- You allowed port 443 inbound on a host, but HTTPS connections still time out. What do you check?
- How do you change firewall rules on a remote server without locking yourself out?
- Which tool do you use to see which ports a server is listening on, and which program owns each?
- What is the difference between
ping,tracerouteandmtr? - How do you test whether a TCP port is reachable without the application's client?
- How would you capture only the traffic to and from port 5432 on all interfaces, and save it?
- A container image has no networking tools. How do you debug its network?
tracerouteto a host shows* * *from hop 5 onwards, but the website loads. What does that mean?
- In
curl -voutput, what do lines starting with*,>and<mean? - Walk me through the stages
curl -v https://example.comshows. - The transcript stops at
Trying 203.0.113.10:443...and eventually times out. What does that tell you? - How do you test a new server's certificate before DNS points at it?
curl -vshowsSSL certificate verify okand then< HTTP/2 502. Where is the problem?- What does
ALPN: server accepted h2mean?
- How does Nginx decide which
serverandlocationhandle a request? - What is the difference between
proxy_pass http://app;andproxy_pass http://app/;? - A user gets a
502from Nginx. Where do you look and what might you find? - What causes a
504from Nginx, and how do you fix it properly? - What is the difference between
nginx -s reloadand a restart? - Why should a reverse proxy set
HostandX-Forwarded-Forheaders?
Git¶
- Does Git store diffs or snapshots? Explain.
- What are blobs, trees and commits?
- What is a branch, technically? What is HEAD?
- What is the staging area for?
- Why does rewriting history cause trouble for collaborators?
- A large file was committed by mistake and deleted in the next commit. Is the repository smaller now?
- What is the difference between a fast-forward merge and a merge commit?
- Merge or rebase: what is the difference and when would you use each?
- How do you resolve a merge conflict?
- What does
git stashdo, and when would you use it? - What is cherry-pick, and what is a typical use?
- What is the difference between a lightweight and an annotated tag?
- What is the difference between
git reset --soft,--mixedand--hard? - What is the difference between
git resetandgit revert? - You ran
git reset --hardand lost commits. How do you get them back? - A bad commit was merged to
mainand deployed. How do you undo it? - How would you find which of the last 200 commits introduced a bug?
- What can Git not recover?
Remotes, pull requests and access
- What is the difference between
git fetchandgit pull? - Why is a push rejected, and how do you fix it safely?
- What does
--force-with-leaseprotect against? - What would you put in branch protection for
main? - How should a deployment server or CI job authenticate to Git?
- Are Git hooks a good way to enforce that no secrets are committed?
Branching and release strategies
- What is trunk-based development?
- Compare GitFlow and trunk-based development.
- What is a feature flag, and what problem does it solve?
- How do you ship an urgent fix to production under trunk-based development, and under GitFlow?
- What does semantic versioning mean by major, minor and patch?
- Which strategy would you pick for a team deploying a web API many times a day, and what would you need in place?
AWS identity and network¶
- What is the difference between a region and an Availability Zone?
- How does the AWS CLI decide which credentials to use?
- Why use many AWS accounts instead of one?
- What should you do with the root user of an account?
- Your script works on your laptop but runs against the wrong account in CI. What do you check?
- What is the difference between
--queryand--filtersin the CLI?
Organizations, SCPs and multi-account
- What is a service control policy, and does it grant permissions?
- Who is affected by SCPs, and who is not?
- How do SCPs combine across OUs?
- Design an account structure for a company with two products, each with dev and prod.
- How do you restrict an organisation to two regions without breaking global services?
- An administrator in a member account gets
AccessDeniedfor an action their policy allows. How do you investigate?
IAM policies and how they are evaluated
- Explain how AWS evaluates whether a request is allowed.
- What is the difference between an identity-based and a resource-based policy?
- What is a permissions boundary, and when would you use one?
- What does an explicit deny in an SCP mean for an account administrator?
- A role can list a bucket but not read objects in it. Why?
- How do you give a role in account A read access to an S3 bucket in account B?
- Why is
"Effect": "Allow"withNotActionrisky?
- What is the difference between an IAM user and an IAM role?
- What is a trust policy, and how does it differ from a permissions policy?
- What does
iam:PassRoleprotect against? - How do temporary credentials from STS differ from access keys?
- What is the confused deputy problem, and how does an external ID help?
- An application on EC2 cannot get AWS credentials. What do you check?
- What is federation in AWS, and why is it preferred over IAM users?
- How does IAM Identity Center grant access to many accounts?
- What is the difference between SAML and OIDC federation?
- Why must a trust policy for a CI OIDC provider check the
subclaim? - An engineer leaves. How do you make sure their AWS access is gone?
- A pipeline can assume its AWS role from feature branches, but it should only work from
main. What is wrong and how do you fix it?
Least privilege and AccessDenied
- What is least privilege, and how do you apply it in practice on AWS?
- Walk me through debugging an
AccessDeniederror. - Name some IAM permissions that allow privilege escalation.
- How would you let developers create IAM roles for their applications without letting them escalate?
- A role was tightened using 30 days of CloudTrail data and broke a month later. What went wrong?
- An application can read an encrypted S3 object but not write one. Why?
- What makes a subnet public or private?
- How many usable addresses does a
/24subnet have in AWS, and why? - Draw a production VPC for a web application.
- What is the difference between an internet gateway and a NAT gateway?
- An instance in a private subnet cannot download packages. Walk through the checks.
- Why plan VPC ranges across the whole organisation?
NAT, endpoints and PrivateLink
- Why do private subnets need a NAT gateway, and where does it live?
- What is the difference between a gateway endpoint and an interface endpoint?
- A NAT gateway was deleted by mistake. What do workloads see, and why?
- How would you reduce a high NAT gateway bill?
- What is PrivateLink, and how is it different from VPC peering?
- How do you make sure an S3 bucket is only used from inside your VPC?
Peering, Transit Gateway and VPN
- What does "VPC peering is not transitive" mean?
- When would you choose Transit Gateway over peering?
- Explain association and propagation in Transit Gateway route tables.
- What are the options for connecting an office to AWS, and how do they differ?
- Two VPCs are peered and routes are added, but connections still time out. What do you check?
- Two teams' VPCs overlap and they need one API between them. What do you do?
- What is the difference between a security group and a network ACL?
- Why do NACLs need rules for ephemeral ports?
- Why reference security groups instead of IP ranges?
- How would you secure a three-tier application with security groups?
- A client gets timeouts. You can see its requests in the server's access log. What is the likely cause?
- How do you find security groups exposed to the internet?
- When would you choose an ALB over an NLB, and the reverse?
- What are listeners, rules and target groups?
- An ALB returns
503. What does it mean and what do you check? - How do you avoid dropping requests during a deploy behind an ALB?
- Why might targets behind an ALB return intermittent
502s? - What does Global Accelerator add on top of a regional load balancer?
- What is an alias record, and how does it differ from a CNAME?
- What is the difference between a public and a private hosted zone?
- Name Route 53 routing policies and a use for each.
- How does ACM validate and renew certificates?
- Why does a CloudFront distribution not see a certificate you created in
ap-south-1? - How would you migrate a domain to a new load balancer with a safe rollback?
- How does CloudFront decide whether to serve from cache?
- How do you keep an S3 bucket private while serving it through CloudFront?
- Why might a CDN make a site slower or more expensive?
- What is a WAF, and how do you roll out new rules safely?
- What does a rate-based rule protect against?
- What is the difference between Shield Standard and Shield Advanced?
- An EC2 instance in a private subnet cannot reach the internet. Walk through your checks.
- What do VPC Flow Logs record, and what do they not record?
- Flow Logs show the request accepted at the server but the reply rejected. What is the cause?
- What does Reachability Analyzer do?
- Why does a timeout point to different causes than "connection refused" inside a VPC?
- Two VPCs are peered, but an application in one cannot reach a database in the other. List the places to check.
AWS services¶
- What is the difference between stopping and terminating an instance?
- Why use IMDSv2, and what does the hop limit do?
- How do you get shell access without SSH keys?
- When would you choose Graviton instances?
- What do system and instance status check failures mean?
- A T-family instance becomes slow every afternoon. Why?
- What is the difference between an AMI and a launch template?
- EC2 vs load balancer health checks in an Auto Scaling group?
- Which scaling policy would you choose and why?
- How do you roll out a new AMI to a running group without downtime?
- Golden images or bootstrapping?
- An Auto Scaling group keeps replacing instances every few minutes. What do you check?
- What is EBS and how is it different from instance store?
- gp3 vs io2 vs st1?
- How do you grow a volume without downtime?
- How do snapshots work, and how are they used for DR?
- Why can a pod's volume not follow it to another zone?
- A database volume is slow. How do you investigate?
- When would you choose EFS over EBS?
- What is a mount target and what does it need?
- What is an EFS access point for?
- An EFS mount hangs from a new node group. What do you check?
- Elastic, Provisioned or Bursting throughput?
- How do you reduce EFS storage cost?
- What consistency does S3 provide today?
- A user deleted an important file from a versioned bucket. How do you recover it?
- How do you cut the S3 bill for a bucket that keeps growing?
- How would you let browsers upload large files without passing them through your servers?
- What does replication need, and what does it not do by default?
- Why serve a static site through CloudFront instead of the S3 website endpoint?
- What is a presigned URL, and what limits it?
- Your Lambda, triggered by S3 uploads, is processing some files twice. Why, and what do you do?
- Walk through how S3 decides whether a request is allowed.
- A role in another account gets
AccessDeniedreading your bucket. What do you check? - SSE-S3 or SSE-KMS: which and why?
- How do you make sure a bucket is never reached over plain HTTP?
- How would you protect backups from deletion by ransomware or a compromised admin?
- How do you find public or shared buckets across an account?
- Governance mode versus compliance mode in Object Lock?
- What is the difference between CloudTrail data events and S3 server access logs?
- Multi-AZ versus read replica: what is each for?
- Someone dropped a table at 14:05. How do you recover?
- Explain Aurora's architecture and why failover is fast.
- Lambda functions keep hitting "too many connections" on RDS. What do you do?
- How does an app survive a database failover?
- You changed
shared_buffersand nothing happened. Why? - How would you do a major version upgrade with little downtime?
- Queue, topic or event bus: when do you use which?
- What is the visibility timeout, and what goes wrong if it is too short?
- How do you make a consumer idempotent?
- Explain the SNS to SQS fan-out pattern and why it is common.
- Messages are landing in the DLQ. Walk me through it.
- Standard or FIFO queue?
- An EventBridge rule is not triggering its Lambda. What do you check?
KMS, Secrets Manager and Parameter Store
- Explain envelope encryption.
- How do key policies and IAM policies interact?
- Secrets Manager or Parameter Store?
- What happens when a KMS key is rotated, and when it is deleted?
- How does a rotated database password reach running applications without an outage?
- A team wants to share an encrypted S3 bucket with another account. What must change on the KMS side?
- How do you keep secrets out of logs and repositories?
- What uniquely identifies a CloudWatch metric?
- Why can't you see memory usage for an EC2 instance by default, and how do you fix it?
- Explain alarm evaluation: period, evaluation periods, datapoints to alarm and missing data.
- Your CloudWatch bill doubled. Where do you look?
- How do you get application logs from CloudWatch into OpenSearch or S3?
- What is the embedded metric format and when is it useful?
- How would you alert on errors that only appear in logs?
- Someone deleted an S3 bucket last night. How do you find out who?
- Management versus data events?
- How do you protect CloudTrail logs from tampering?
- How do you query months of CloudTrail logs efficiently?
- What does AWS Config give you that CloudTrail does not?
- You get
AccessDeniedin an app. How can CloudTrail help? - What would you alarm on from CloudTrail?
GuardDuty, Security Hub and Inspector
- What is the difference between GuardDuty, Inspector and Security Hub?
- GuardDuty reports an EC2 instance communicating with a known malicious IP. What do you do?
- How do you prioritise 500 critical CVE findings?
- When is it acceptable to suppress a finding?
- How do you run these across 50 accounts?
- Why might a CSPM control show failed even though the resource is fine?
- Why use Session Manager instead of SSH?
- An instance does not appear in Systems Manager. How do you debug it?
- How do you patch 500 instances safely?
- How would you reach a private RDS database from your laptop without a bastion?
- What are concurrency and error thresholds in Run Command for?
- What is in an SSM document, and give examples.
- Pods on EKS show
ImagePullBackOffwith an ECR image. What do you check? - How does CI push to ECR without long-lived keys?
- Why turn on tag immutability, and what is the catch?
- The ECR bill keeps growing. What do you do?
- Why use a pull-through cache for public base images?
Serverless and Fargate¶
- Explain a Lambda cold start and how to reduce it.
- What are reserved and provisioned concurrency?
- Compare synchronous, asynchronous and event source mapping invocations.
- How do you deploy a new Lambda version safely?
- Why might more memory make a function cheaper?
- A Lambda behind SQS keeps reprocessing the same messages and the queue grows. What do you check?
Lambda networking and permissions
- What is the difference between a Lambda execution role and its resource-based policy?
- A Lambda in a VPC times out calling S3. Why, and how do you fix it?
- Does putting a Lambda in a public subnet give it internet access?
- How should a Lambda function get a database password?
- How do you let only one S3 bucket invoke a function?
- When should a Lambda function be attached to a VPC?
- REST API or HTTP API?
- What authorizer options are there?
- A client gets "Missing Authentication Token". What is wrong?
- A Lambda-backed API returns 502. What do you check?
- How do you handle an operation that takes two minutes?
- How do you protect an API from one noisy client?
- Explain cluster, task definition, task and service.
- What is the difference between the task execution role and the task role?
- What does the awsvpc network mode change?
- Fargate or EC2 launch type?
- How do ECS rolling deployments and the circuit breaker work?
- A new task revision never becomes healthy. How do you debug it?
- What is the difference between a user pool and an identity pool?
- Which tokens does a user pool issue, and what is each for?
- How do you protect an API with Cognito?
- How do you give admins extra rights?
- Why use managed login instead of building your own sign-in form?
- When would you use Step Functions instead of chaining Lambda functions?
- Standard or Express workflows?
- How do Retry and Catch work?
- How do you wait for a human approval in a workflow?
- What does the Map state do?
Data stores¶
- The app reports "too many clients already". What do you do now and later?
- New tables created by migrations are not readable by the app. Why?
pg_dumpversus physical backups?- Queries are hanging. How do you find the cause?
- What does
sslmode=preferreally give you? - Session versus transaction pooling in PgBouncer?
- How would you upgrade a major PostgreSQL version with little downtime?
- Explain cache-aside and its failure modes.
- What is a cache stampede and how do you prevent it?
- Which eviction policy for a pure cache, and which for sessions?
- RDB versus AOF?
- Redis or Valkey versus Memcached?
- How does Redis Cluster split data, and what breaks?
- After a database update, users still see the old price. Why?
Containers¶
- What is a container, technically?
- How is a container different from a virtual machine?
- What are namespaces and cgroups, and which does what?
- Why can a container ignore
docker stopand get killed after 10 seconds? - What is copy-on-write in the container filesystem?
- A container was OOM-killed although the host had plenty of free memory. Why?
- Is running as root inside a container safe?
- What is an image layer, and why does instruction order matter?
- What is the difference between
CMDandENTRYPOINT? - What is a multi-stage build, and why use one?
- Why should you not use the
latesttag in production? - How can a secret end up in an image even after you delete it?
- A container fails with
exec format error. What happened? - How would you make a Dockerfile more secure?
Running and debugging containers
- A container keeps exiting right after it starts. How do you debug it?
- What is the difference between a volume and a bind mount?
- Why can two containers on the default bridge network not reach each other by name?
- The app works inside the container but the published port refuses connections. Why?
- How do you debug the network of a container whose image has no shell?
- What do exit codes 125, 126, 127 and 137 mean for a container?
- What is the difference between an image tag and an image digest?
- How do you make sure every node runs exactly the same image?
- What does an image scanner check, and what are its limits?
- Your base image is distroless, yet the scan shows critical vulnerabilities. How is that possible?
- Why rebuild images on a schedule, not only on code changes?
- What should a CI pipeline do with image security?
- What is the difference between Docker, containerd and runc?
- What is the CRI?
- Kubernetes removed dockershim. Do images built with Docker still run?
- How do you debug containers on a Kubernetes node that has no Docker?
- What is a containerd shim for?
- What is the pause container in a pod?
- What is Docker Compose for?
- Does
depends_onmake a service wait until its dependency is ready? - How do services in Compose find each other?
- What is the difference between
docker compose downanddown -v? - Why not run production on Compose?
Kubernetes¶
Architecture and the reconcile loop
- Name the control-plane components and what each does.
- What runs on every worker node?
- What is a reconcile loop?
- Walk me through what happens when you apply a Deployment.
- A node goes down. What does Kubernetes do, and what does it not do?
- Why does every component talk only to the API server?
- A pod is in
CrashLoopBackOff. Which kubectl commands do you run, in order? - What is a kubeconfig context?
- What is the difference between
kubectl applyandkubectl create? - How do you preview a change before applying it?
- How do you debug a container that has no shell?
- Why should you be careful with
kubectl editandkubectl scalein production?
- What is a pod, and why does Kubernetes not schedule containers directly?
- What do containers in the same pod share?
- What is the difference between an init container and a sidecar container?
- What does
CrashLoopBackOffmean? - Why should you not create pods directly in production?
- A pod shows
1/2ready. How do you find out which container is the problem?
- How does a Deployment perform a rolling update?
- What do
maxSurgeandmaxUnavailablecontrol? - What is the relationship between a Deployment, ReplicaSets and pods?
- You deployed a bad image. What happens, and how do you recover?
- Why is a readiness probe important for rolling updates?
- When would you use the
Recreatestrategy? - You changed a ConfigMap, but the pods still use the old values. Why, and what do you do?
- When do you use a StatefulSet instead of a Deployment?
- What is a headless Service, and why does a StatefulSet need one?
- What happens to a StatefulSet's volumes when you scale down or delete a pod?
- Does a StatefulSet replicate data or handle failover?
- What is a DaemonSet, and what typically runs as one?
- A StatefulSet's first pod is stuck
Pending. What do you check?
- When do you use a Job instead of a Deployment?
- What do
completions,parallelismandbackoffLimitdo? - What are the CronJob concurrency policies?
- Why must CronJob work be idempotent?
- A CronJob scheduled at 02:30 ran at 08:00. Why?
- How do you test a CronJob without waiting for its schedule?
- A Job with a mesh sidecar never completes. Why, and how do you fix it?
Labels, selectors and namespaces
- What is the difference between labels and annotations?
- How does a Service know which pods to send traffic to?
- Are namespaces a security boundary?
- What do ResourceQuota and LimitRange do?
- A Deployment shows
3/5and no pods are failing. What do you check? - Why is an empty selector dangerous?
- What is the difference between a ConfigMap and a Secret?
- Is a Kubernetes Secret secure? What makes it more secure?
- You updated a ConfigMap, but the application still uses the old value. Why?
- What does
CreateContainerConfigErrormean? - Why might someone who cannot
get secretsstill read one? - What are immutable ConfigMaps for?
- What is the difference between liveness, readiness and startup probes?
- Why should a liveness probe not check the database?
- What happens, in order, when a pod is deleted?
- Users see errors during every rolling update. What do you check?
- A slow-starting app keeps restarting with no errors in its logs. Why?
- Why add a
preStopsleep if the app already handles SIGTERM?
- What is the difference between requests and limits?
- What happens when a container exceeds its CPU limit, and its memory limit?
- What are the QoS classes and how are they assigned?
- A pod is
PendingwithInsufficient memory, but the nodes' memory usage is low. Why? - Should you set CPU limits?
- How do you choose requests and limits for a new service?
- How does the scheduler choose a node for a pod?
- What is the difference between a taint and node affinity?
- A pod is
Pending. How do you find out why? - How do you spread replicas across availability zones?
- What do the
NoSchedule,PreferNoScheduleandNoExecutetaint effects do? - What is preemption?
- What is a Kubernetes Service, and why do we need it?
- Explain ClusterIP, NodePort, LoadBalancer and headless Services.
- What is the difference between
port,targetPortandnodePort? - A Service returns "connection refused". Walk through your checks.
- How does DNS work for Services, and what is
ndots:5? - Why might load not be balanced evenly across pods behind a Service?
- What is the difference between an Ingress and an ingress controller?
- Why is Gateway API replacing Ingress?
- Name the main Gateway API objects and who typically owns each.
- What happened to ingress-nginx, and what would you do about a cluster still using it?
- An Ingress returns 503 for every request. How do you debug it?
- How would you route 10% of traffic to a new version?
- Why put TLS termination at the ingress layer, and what are the trade-offs?
Pod networking and NetworkPolicy
- What is the Kubernetes network model?
- What does a CNI plugin do?
- Explain how NetworkPolicy evaluates traffic.
- You applied a default-deny egress policy and the application broke completely. Why?
- A NetworkPolicy seems to do nothing. What do you check?
- How do you isolate namespaces from each other?
- Explain PersistentVolume, PersistentVolumeClaim and StorageClass.
- What are the access modes, and which work with EBS and EFS?
- Why use
WaitForFirstConsumer? - What does the reclaim policy control?
- A pod with a PVC is stuck
ContainerCreatingwithMulti-Attach error. What happened? - How do you grow a volume used by a StatefulSet?
- How does the HPA decide how many replicas to run?
- An HPA shows
<unknown>for its target. What do you check? - Why does a CPU utilisation target depend on requests?
- What are the differences between HPA, VPA and a node autoscaler?
- Why is scale-down slower than scale-up by default?
- A queue worker should scale on queue length, including down to zero. How?
- Traffic spiked, the HPA scaled out, but users still saw errors for three minutes. Why, and what would you change?
- What is a PodDisruptionBudget, and what does it protect against?
- What is the difference between voluntary and involuntary disruptions?
- A
kubectl drainhangs. What do you check? - What is wrong with
minAvailable: 1on a single-replica Deployment? - What do cordon, drain and uncordon do?
- How would you upgrade the nodes of a cluster without downtime?
- What is the difference between a Role and a ClusterRole, and a RoleBinding and a ClusterRoleBinding?
- How does a pod authenticate to the API server?
- How do you check whether a service account can delete pods?
- Why is
create podsa sensitive permission? - What is the difference between
UnauthorizedandForbidden? - Design RBAC for a team that should manage its own namespace but nothing else.
- What are the three Pod Security Standards levels?
- How do you apply Pod Security Admission to a namespace, and what are its modes?
- A Deployment was applied successfully but has zero pods. The namespace enforces restricted. Why?
- What do
runAsNonRoot,allowPrivilegeEscalation: falseandcapabilities.drop: ["ALL"]each protect against? - What is
fsGroupfor? - A vendor's chart needs
privileged: true. What do you do?
- What is a finalizer, and what happens when you delete an object that has one?
- A namespace has been
Terminatingfor an hour. How do you investigate? - Explain background, foreground and orphan cascading deletion.
- What is the risk of removing a finalizer by hand?
- Why might a PVC stay in
Terminating? - How do you replace a StatefulSet's spec field that cannot be changed, without restarting its pods?
- You need to delete all resources labelled
app=old. How do you do it safely?
CRDs, operators and admission webhooks
- What is the difference between a CRD and an operator?
- Describe the order of admission control.
- A webhook's pods crashed and now nobody can create pods. What happened, and how do you recover?
- When would you use a ValidatingAdmissionPolicy instead of a webhook?
- What is the status subresource of a custom resource for?
- How does Istio get a sidecar into your pod?
- You are about to uninstall an operator. What do you check first?
etcd, backups and cluster upgrades
- Why do etcd clusters have an odd number of members?
- What happens to running pods if etcd loses quorum?
- How do you back up and restore etcd?
- What is the correct order for a Kubernetes upgrade?
- How do you find out whether an upgrade will break any manifests?
- On EKS you cannot reach etcd. How do you back up the cluster?
- How long is a Kubernetes minor release supported?
- A pod is
Pending. What do you check? - What is the difference between
ImagePullBackOffandCrashLoopBackOff? - A pod restarts every few minutes with exit code 137. What could it be, and how do you tell?
- Pods are Running and Ready, but the Service times out. Walk through it.
- How do you debug a container whose image has no shell?
kubectl logsshows nothing useful for a crash-looping pod. What next?- A node shows
NotReady. What happens to its pods, and what do you check? - A pod works in namespace
abut calls towebfail after moving it to namespaceb. Why?
EKS¶
EKS architecture and node options
- What does AWS manage in EKS, and what do you manage?
- Compare managed node groups, Fargate and Karpenter.
- What are the endpoint access options, and which would you choose for production?
- How does kubectl authenticate to EKS?
- What is the difference between the cluster IAM role and the node IAM role?
- New nodes in private subnets never join the cluster. What do you check?
- When would you choose EKS Auto Mode?
Cluster access, IRSA and Pod Identity
- How do you give a new engineer read-only access to one namespace on EKS?
- Explain how IRSA works end to end.
- How does EKS Pod Identity differ from IRSA?
- Why should application permissions not go on the node IAM role?
- A pod gets AccessDenied calling S3. Walk through your debugging.
- What is the risk of an IRSA trust policy without a
subcondition? - You are migrating a cluster from
aws-authto access entries. What are the steps and risks?
- How does the VPC CNI give pods IP addresses?
- Why can a node with free CPU and memory not run more pods?
- What does prefix delegation change?
- The cluster's subnets are running out of addresses. What are your options?
- Why is pod traffic to the internet seen with the node's address?
- When would you use security groups for pods?
- What does the AWS Load Balancer Controller do?
- Explain
target-type: ipversusinstance. - An Ingress was applied an hour ago and still has no address. What do you check?
- How do you share one ALB between several Ingresses, and what is the risk?
- Users see 502 errors for a few seconds on every deploy. How do you fix it?
- What is a TargetGroupBinding for?
- When would you use EBS and when EFS on EKS?
- Why does
WaitForFirstConsumermatter for EBS? - A StatefulSet pod is stuck Pending with
volume node affinity conflict. What happened? - An EFS mount times out. What do you check?
- What is an EFS access point and why use one per application?
- How do the CSI drivers get permission to create volumes?
- Compare the External Secrets Operator and the Secrets Store CSI driver.
- A secret was rotated in Secrets Manager. Will running pods see it?
- How do pods authenticate to Secrets Manager on EKS?
- Are Kubernetes Secrets encrypted on EKS?
- Why commit an ExternalSecret instead of a Secret to Git?
- An ExternalSecret shows a sync error after someone added a KMS key to the secret. What happened?
Node autoscaling: Cluster Autoscaler vs Karpenter
- How does Cluster Autoscaler decide to add a node?
- When does Cluster Autoscaler remove a node?
- What is the main difference in how Karpenter adds capacity?
- Pods are Pending but the autoscaler does nothing. How do you investigate?
- Why run the autoscaler on a separate, fixed node group?
- A team complains their pods restart every few hours since moving to Karpenter. What is happening and what do you change?
- What are NodePool, EC2NodeClass and NodeClaim?
- Walk through what happens when a pod is Pending in a Karpenter cluster.
- Explain consolidation, drift and expiration.
- How do you stop Karpenter from disrupting a running batch job?
- How does Karpenter handle Spot interruptions?
- Karpenter keeps logging that no instance type meets the pod's requirements. What do you check?
- What protects production from Karpenter replacing too many nodes at once?
- What is the order of an EKS upgrade?
- How do you find out whether workloads use APIs removed in the next version?
- Can you roll back an EKS upgrade?
- How does a managed node group update work, and what can make it fail?
- How do Karpenter-managed nodes get upgraded?
- When would you choose a blue-green cluster upgrade over in-place?
- Why not stay on an old version with extended support?
Helm¶
- What problem does Helm solve, and what is a chart?
- What is the order of values precedence?
- What is the difference between
includeandtemplate? - How do you debug a chart that fails with a YAML parse error?
- How would you make a value mandatory?
- How do you stop a chart's replica count fighting an HPA?
- What is a Helm release, and where is its state stored?
- How does
helm rollbackwork, and what does it not undo? - A deploy fails with "another operation is in progress". What happened and how do you fix it?
- How would you run a database migration with Helm?
- What does
--rollback-on-failuredo (formerly--atomic)? - How do you see exactly what is deployed and with which values?
Kustomize, and Helm vs Kustomize
- What is the core idea of Kustomize?
- Why do generated ConfigMaps have a hash suffix?
- When would you choose Helm and when Kustomize?
- Why is
labelspreferred overcommonLabels? - A patch in the prod overlay does nothing. How do you debug it?
- How do you change the image tag per environment with Kustomize?
Service mesh¶
Why a mesh, and Istio architecture
- What problems does a service mesh solve?
- Explain Istio's data plane and control plane.
- What is xDS?
- Sidecar mode or ambient mode?
- What does a mesh cost you?
- When would you not use a service mesh?
- How does a sidecar get into a pod?
- How is traffic redirected into Envoy?
- Why do Service port names matter in Istio?
- What startup and shutdown problems do sidecars cause, and how are they solved?
- Describe the path of a request between two meshed pods.
- How would you reduce sidecar overhead in a large mesh?
- What do VirtualService and DestinationRule each do?
- How do you run a canary with Istio?
- What is outlier detection?
- What are Istio's default retries and timeouts?
- How do Istio resources map to Envoy configuration?
- A canary at 10% receives no traffic. What do you check?
- How does Istio identify workloads?
- PERMISSIVE vs STRICT?
- How are AuthorizationPolicies evaluated?
- Does RequestAuthentication require a token?
- What does mTLS not protect you from?
- After switching a namespace to STRICT, a CronJob starts failing. Why?
- How do you start debugging a 503 in an Istio mesh?
- What does
istioctl proxy-statustell you? - What do the response flags UH, UF and NR mean?
- How do you tell an application error from a mesh error?
- What is
istioctl proxy-configused for? - A VirtualService weight change has no effect. What do you check?
Infrastructure as code¶
- What happens during
terraform init,planandapply? - What is the difference between a resource and a data source?
- How does Terraform decide the order of operations?
- Why commit
.terraform.lock.hcl? - Why save a plan in CI instead of running
applydirectly? - A colleague changed a security group in the console. What happens on the next plan?
- Why does Terraform need state?
- How do you set up remote state on AWS today?
- What sensitive data can end up in state, and how do you protect it?
- A pipeline was killed during apply and now the state is locked. What do you do?
- How do you rename a resource without destroying it?
- How would you split state for a company with several environments and teams?
- What is the difference between
countandfor_each? - When would you use
create_before_destroy? - What does
ignore_changesdo and what is the risk? - What does
cidrsubnet("10.0.0.0/16", 8, 3)return? - Why might
for_eachfail with "known only after apply"? - Does
prevent_destroyprotect a resource if someone deletes its block?
- What is a Terraform module, and what is the root module?
- How do you version modules, and why?
- Directories per environment or workspaces? Defend a choice.
- How do you move resources into a module without recreating them?
- What makes a module well designed?
- How would you roll out a risky change to a shared VPC module across ten accounts?
- What do
~,-/+and+/-mean in a plan? - What does
# forces replacementtell you, and why does it matter? - Why apply a saved plan file in CI?
- When is
-targetacceptable? - How would you automatically block plans that delete databases?
- A plan for a tags-only change wants to replace three NAT gateways. What do you do?
- How do you bring an existing resource under Terraform?
- You renamed a resource and the plan wants to destroy and recreate it. Fix it.
- How do you stop managing a resource without deleting it?
- How do you detect drift automatically?
- Someone changed a security group in the console during an incident. What do you do?
- What are the risks of
-generate-config-out?
- Describe a CI/CD pipeline for Terraform.
- Why apply a saved plan, and what if it is stale?
- What do TFLint, Trivy and Checkov each add over
terraform validate? - What is
terraform testand when would you usecommand = plan? - How do you make sure only reviewed code can change production?
- A plan in CI shows changes nobody made in this pull request. What do you do?
- What problem does Terragrunt solve?
- How do
include,remote_stateandgeneratework together? - What are
dependencyblocks andmock_outputsfor? - How do you run Terragrunt across many units, and what changed recently?
- When would you not use Terragrunt?
- How do you promote a module change through environments with Terragrunt?
CloudFormation and CDK vs Terraform
- What is a CloudFormation change set, and how does it compare to a Terraform plan?
- What happens when a CloudFormation update fails, compared with Terraform?
- What does CDK add on top of CloudFormation?
- When would you choose Terraform over CloudFormation?
- How do you deploy the same infrastructure to many AWS accounts with each tool?
Ansible and configuration management
- What is the difference between provisioning and configuration management?
- What does "agentless" mean for Ansible?
- What makes an Ansible task idempotent?
- What are handlers for?
- Where does Ansible fit in a Kubernetes and Terraform world?
CI/CD¶
- What is the difference between continuous delivery and continuous deployment?
- What does "build once, promote the same artifact" mean, and why?
- Describe the stages of a typical pipeline.
- How do you keep a pipeline fast?
- What metrics tell you whether delivery is healthy?
- A deploy broke production even though all pipeline checks passed. What do you change?
- Compare rolling, blue-green and canary deployments.
- How do you implement blue-green on Kubernetes?
- What is a feature flag and how does it change deployment risk?
- How do you change a database schema safely with zero-downtime deploys?
- When would you roll forward instead of rolling back?
- A canary at 5% shows the same error rate as the stable version, but users complain after full rollout. What went wrong?
- Explain workflow, job, step, action and runner.
- How do jobs share data?
- What is
GITHUB_TOKENand how do you limit it? - What does
concurrencydo in the Pages workflow? - Why are secrets not available to pull requests from forks?
- A workflow did not run after a push. How do you find out why?
- When would you use a reusable workflow versus a composite action?
- How do you require approval before deploying to production?
- Why pin actions to a commit SHA, and what does GitHub now recommend?
- What is the risk of
pull_request_target? - What are the risks of self-hosted runners?
- A workflow fails only sometimes. How do you debug it?
- Explain the controller and agent architecture.
- Declarative vs scripted pipelines?
- What is a shared library and why use one?
- How do you use secrets safely in a Jenkinsfile?
- How do you run Jenkins agents on Kubernetes?
- Builds queue for a long time. What do you check?
- Describe the structure of
bitbucket-pipelines.yml. - What is the difference between caches and artifacts?
- How do deployments and variables work together?
- How do you deploy to AWS from Bitbucket without stored keys?
- What is a pipe?
- Why use OIDC instead of access keys in CI?
- Walk through the OIDC flow from a GitHub Actions job to AWS.
- Why must the trust policy pin
sub? - A job fails with "Not authorized to perform sts:AssumeRoleWithWebIdentity". Debug it.
- How does Bitbucket's OIDC differ from GitHub's?
- How would you design roles for plan and deploy?
- Why is the CI/CD pipeline an attractive target?
- What is poisoned pipeline execution?
- How do you prevent script injection in GitHub Actions?
- How should secrets be handled in pipelines?
- What are SBOMs, provenance and signing for?
- A popular third-party action was compromised yesterday. What do you do?
- What is the difference between SAST, SCA and DAST?
- What is a SonarQube quality gate?
- How do you decide whether a finding fails the build or only warns?
- A critical CVE is reported in a base image used by 30 services. What do you do?
- How do you keep security scanning from slowing delivery?
- A secret was committed and pushed. What now?
Build tools: Maven, Gradle, npm
- What happens when you run
mvn package? - Why commit the Gradle wrapper?
- What is the difference between
npm installandnpm ci? - How do you make builds faster in CI?
- What is a SNAPSHOT version and why not deploy one to production?
- What are hosted, proxy and group repositories?
- Why put a proxy repository in front of public registries?
- What does "immutable artifact" mean and why does it matter?
- How do you promote an artifact from staging to production?
- How are Helm charts stored in container registries?
- What retention policy would you set for an image repository?
GitOps and Argo CD¶
- What is GitOps?
- Compare push-based CI/CD with pull-based GitOps.
- Why separate the config repository from the application repository?
- What does GitOps not solve by itself?
- An engineer scaled a Deployment by hand during an incident; ten minutes later it went back. Why, and what should they have done?
- How would you structure a config repository for three environments?
Argo CD architecture and Applications
- Describe Argo CD's main components.
- What does an Application define?
- What is the difference between sync status and health status?
- What is an AppProject for?
- How does a commit reach the cluster, and how fast?
- An Application shows
Unknownsync status with a comparison error. What do you check?
- What is the difference between refresh, hard refresh and sync?
- What do
pruneandselfHealdo, and what are their defaults? - How do you run a database migration before the new version starts?
- What are sync waves?
- An app is always OutOfSync on
spec.replicas. Why, and how do you fix it? - How do Helm hooks behave under Argo CD?
App-of-apps and ApplicationSets
- What is the app-of-apps pattern?
- What is an ApplicationSet and what are generators?
- When would you use app-of-apps and when ApplicationSets?
- How can one ApplicationSet change cause an outage, and how do you prevent it?
- How do you deploy an add-on to every production cluster?
- How do you build preview environments per pull request with Argo CD?
Deletion, finalizers and safe cleanup
- What happens when you delete an Argo CD Application?
- How do you protect a PVC from being deleted with its Application?
- How do you move resources from one Application to another without downtime?
- How can a single change delete many applications in an app-of-apps or ApplicationSet setup?
- An Application has been "deleting" for an hour. What do you check?
- What does
Prune=confirmdo?
Promotion, rollback and troubleshooting
- How do you deploy a new version with GitOps?
- How do you promote from staging to production?
- How do you roll back, and why not with
argocd app rollback? - An application is OutOfSync. Walk through your triage.
- Production is Degraded right after a deploy. What do you do?
- Someone changed a Deployment by hand and it keeps reverting. Explain.
Observability¶
- What are metrics, logs and traces each best for?
- Explain RED and USE.
- What are the four golden signals?
- What is cardinality and why does it matter?
- What is the difference between monitoring and observability?
- What would you monitor for a new web API before launch?
- How does Prometheus collect metrics, and why pull?
- Explain counter, gauge, histogram and summary.
- What is a time series in Prometheus?
- What causes high cardinality and how do you fix it?
- When should you use the Pushgateway?
- What are kube-state-metrics and node_exporter for?
- What is the difference between an instant vector and a range vector?
- Explain
rate,irateandincrease. - Why must you take
rate()beforesum()? - Write a query for the error ratio of a service.
- How do you compute p99 latency across all pods?
- What are recording rules and how should they be named?
- What does kube-prometheus-stack install?
- What is a ServiceMonitor, and how does Prometheus find it?
- You created a ServiceMonitor but the target never appears. What do you check?
- ServiceMonitor or PodMonitor?
- How do you add an alert rule for your application?
- What do Prometheus and Alertmanager each do in alerting?
- What do
forandkeep_firing_fordo? - Explain grouping, inhibition and silences.
- Why alert on symptoms rather than causes?
- What should a good alert notification contain?
- The on-call engineer gets 40 pages a week, mostly ignored. What do you do?
- What are SLI, SLO and SLA?
- What is the error budget for 99.9% over 30 days?
- What is a burn rate and why alert on it?
- Why use multi-window burn-rate alerts?
- How would you choose SLIs for an API?
- The error budget is exhausted halfway through the month. What happens?
- What is a Grafana data source and why does it matter for queries?
- How do variables make dashboards reusable?
- How do you manage dashboards as code?
- Grafana alerting or Alertmanager?
- How would you lay out a service dashboard?
- Where do container logs go in Kubernetes?
- What does
kubectl logsactually read? - How would you collect logs from every pod in a cluster?
- When are logs lost?
- What is the CRI log format?
- Why prefer structured (JSON) logs?
- Describe the Fluent Bit pipeline.
- How does Fluent Bit avoid re-reading or skipping logs after a restart?
- What does the kubernetes filter do?
- Memory vs filesystem buffering?
- Logs went missing during a 10-minute OpenSearch outage. Why, and how do you prevent it?
- How do you handle multiline stack traces?
- Explain index, shard and replica.
- What do green, yellow and red mean?
- What is mapping explosion and how do you prevent it?
- Writes suddenly fail with an index read-only error. Why?
- How do you manage log index lifecycle?
- Fluent Bit reports HTTP 429 from OpenSearch. What do you do?
- What are a trace and a span?
- How does context propagate between services?
- Head vs tail sampling?
- What does the OpenTelemetry Collector do?
- With Istio installed, why might traces still be broken?
Security¶
- What is the difference between authentication and authorisation, and between OAuth and OIDC?
- Walk through the authorization code flow with PKCE.
- What is inside a JWT and how do you validate one?
- When would you use the client credentials flow?
- What is the difference between an ID token and an access token?
- How do CI pipelines and pods use these ideas to reach AWS?
- Where should secrets never be stored?
- Describe the secret lifecycle.
- How do you manage secrets with GitOps?
- Why can rotating a secret break an application, and how do you avoid it?
- A developer pushed an AWS access key to GitHub. What do you do?
- What is better than a well-managed long-lived secret?
Vulnerability and patch management
- What are CVE, CVSS, EPSS and KEV?
- How do you prioritise vulnerability findings?
- How do you patch containers?
- How do you patch a fleet of EC2 instances or EKS nodes safely?
- What metrics show whether vulnerability management works?
- A critical CVE is announced in a base image used everywhere. Walk through your response.
- What does hardening mean, and how do you start?
- What are CIS Benchmarks and how are they used?
- List cloud posture basics you would check in a new AWS account.
- How would you harden an EKS cluster?
- Why prefer preventive controls over detective ones?
- A scanner reports 400 benchmark failures across your accounts. How do you approach it?
Guardrails for AI agents in operations
- Why is an AI agent with shell access a new kind of risk?
- What is the difference between an instruction and a hook as a control?
- How would you set up identities for an operations agent?
- What should happen before an agent makes a change?
- What are the limits of human approval?
- What would you audit for agent activity?
Operations¶
High availability and failure domains
- What failure domains do you design for, and how?
- Compute the availability of two 99.9% components in series and in parallel.
- Why add jitter to retries?
- How do you make a Kubernetes workload survive the loss of a zone?
- What is a circuit breaker?
- Your service is 99.99% available but depends on a 99.9% database. What can you promise?
- What are RPO and RTO?
- Describe the four DR strategies on AWS.
- Why is replication not a backup?
- If you run everything with GitOps, what still needs backing up?
- How do you know your backups work?
- Design backups for a production database with a 15-minute RPO.
- How would you start reducing an AWS bill?
- Compare On-Demand, Spot, Savings Plans and Reserved Instances.
- Why can a Kubernetes cluster be expensive even when nodes look full?
- How do NAT gateways become a large cost, and what do you do?
- How do you keep costs under control without slowing teams?
- What would you not cut to save money?
Incident response and postmortems
- Walk me through how you handle a production incident.
- What does an incident commander do?
- Why mitigate before finding the root cause?
- What makes a postmortem blameless, and why does it matter?
- What goes into a postmortem?
- Tell me about an incident you handled. (How to structure the answer)
- Why do changes cause most incidents, and what reduces the risk?
- What do you do before making a risky production change?
- What does "reversible steps first" mean?
- How much review should a change get?
- A change went wrong halfway through. What now?
- What are rehost, replatform and refactor?
- How would you move a stateful service from VMs to Kubernetes?
- What is the strangler fig pattern?
- How do you migrate a database with minimal downtime?
- What does a good rollback plan for a migration contain?
- When is a migration finished?