Prometheus¶
Status: AI draft, 2026-11-16. Not yet verified by me. The metric output below is real, read from the API server of a local kind cluster (Kubernetes 1.37) with
kubectl get --raw /metrics; no Prometheus server was run.
In one line¶
Prometheus pulls metrics over HTTP from targets it discovers and stores each time series, a metric name plus labels, for PromQL queries and alerts. Metrics are counters, gauges, histograms or summaries, and every new label value is a new series, so watch cardinality.
The plain idea¶
Every service exposes a page, usually /metrics, listing its current numbers in a simple text format. Prometheus visits each page every few seconds (scraping), writes down the numbers with a timestamp, and keeps the history. Because Prometheus does the asking, it knows which targets exist and whether each one answered: a target that stops answering is itself a signal (up == 0).
How it works¶
flowchart TD
SD["Service discovery: Kubernetes API,<br/>EC2, static lists"] --> P["Prometheus: scrape every target<br/>on an interval"]
P -->|"HTTP GET /metrics"| T1["Your app (client library)"]
P -->|"HTTP GET /metrics"| T2["Exporters: node_exporter,<br/>kube-state-metrics, blackbox"]
P --> DB["Local TSDB: series,<br/>retention 15 days by default"]
DB --> Q["PromQL: Grafana, alert rules"]
Q --> AM["Alertmanager"]
Pull model¶
- Prometheus finds targets through service discovery (Kubernetes pods, services and endpoints; EC2 instances; files) and relabelling rules decide what to keep and how to label it (Prometheus on Kubernetes).
- Each scrape also records
up(1 if the scrape worked) and scrape duration. - Why pull: Prometheus controls the rate, knows when a target is down, and you can curl any target to see exactly what it exposes.
A time series¶
A series is a metric name plus labels; each sample is a value at a time. From the kind API server:
$ kubectl get --raw /metrics | grep -E '^# (HELP|TYPE) apiserver_request_total |^apiserver_request_total\{.*resource="pods".*verb="LIST"'
# HELP apiserver_request_total [STABLE] Counter of apiserver requests broken out for each verb, dry run value, group, version, resource, scope, component, and HTTP response code.
# TYPE apiserver_request_total counter
apiserver_request_total{code="200",component="apiserver",dry_run="",group="",resource="pods",scope="cluster",subresource="",verb="LIST",version="v1"} 7
apiserver_request_total{code="200",component="apiserver",dry_run="",group="",resource="pods",scope="namespace",subresource="",verb="LIST",version="v1"} 89
Two series of one metric, differing in the scope label. Each code, verb, resource combination seen gets its own series.
Metric types¶
| Type | Behaviour | Query it with | Example |
|---|---|---|---|
| Counter | Only goes up (resets to 0 on restart) | rate(), increase() (PromQL) |
apiserver_request_total, http_requests_total |
| Gauge | Goes up and down | The value, avg, max |
process_resident_memory_bytes, queue length |
| Histogram | Counts observations into buckets, plus _sum and _count |
histogram_quantile() over rates |
apiserver_request_duration_seconds |
| Summary | Quantiles computed in the client | The quantile series directly; cannot be aggregated across instances | Some client library defaults |
A gauge:
$ kubectl get --raw /metrics | grep -E '^# (HELP|TYPE) process_resident_memory_bytes|^process_resident_memory_bytes '
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.3441792e+08
A histogram (buckets are cumulative: le means "less than or equal"):
$ kubectl get --raw /metrics | grep -E '^apiserver_request_duration_seconds_(bucket|sum|count)\{.*resource="pods".*scope="namespace".*verb="LIST"'
apiserver_request_duration_seconds_bucket{...,le="0.005"} 88
apiserver_request_duration_seconds_bucket{...,le="0.025"} 89
apiserver_request_duration_seconds_bucket{...,le="0.05"} 89
...
apiserver_request_duration_seconds_bucket{...,le="+Inf"} 89
apiserver_request_duration_seconds_sum{...} 0.20507764900000006
apiserver_request_duration_seconds_count{...} 89
(The repeated labels are shortened to ....) 88 of 89 namespaced pod LIST calls finished within 5 ms, all within 25 ms; together they took about 0.2 seconds.
Exporters¶
| Exporter | Exposes |
|---|---|
| node_exporter | Host metrics: CPU, memory, disk, network (Performance triage) |
| kube-state-metrics | The state of Kubernetes objects: Deployment replicas, pod phases, restarts |
| cAdvisor (in the kubelet) | Container CPU, memory, network |
| blackbox_exporter | Probes from outside: HTTP, TCP, DNS, ICMP checks |
| Service-specific | PostgreSQL, Redis, NGINX and many more |
Storage, retention and limits¶
- Local time-series database on disk; retention defaults to 15 days (
--storage.tsdb.retention.time), or set a size limit. - One Prometheus is not built for long-term or global storage; remote write to systems such as Thanos, Cortex, Mimir or a managed service (Amazon Managed Service for Prometheus) handles that.
- Memory grows with the number of active series.
Cardinality¶
The kind API server alone exposes 314 metric families. The single histogram apiserver_request_duration_seconds has 7,800 bucket series on this one-node lab cluster, because every verb, resource and scope combination gets every bucket:
$ kubectl get --raw /metrics | grep -c '^apiserver_request_duration_seconds_bucket'
7800
Add a label with unbounded values (user ID, request ID, full URL path with IDs) to a metric like this, and series multiply until Prometheus runs out of memory. Keep labels bounded; put identifiers in logs and traces (Signals and methods).
The Pushgateway¶
For short batch jobs that finish before a scrape, the Pushgateway accepts pushed metrics and holds them for Prometheus to scrape. The Prometheus documentation recommends it only for service-level batch jobs: it becomes a single point of failure, loses automatic up health, and keeps stale values until deleted. Most services should be scraped directly.
A concrete example¶
Instrumenting sample-app:
- Add the Prometheus client library; expose
/metricson port9090. - A counter
http_requests_total{route, method, status_class}and a histogramhttp_request_duration_seconds{route, method}with buckets around the latency target.routeis the route template (/orders/{id}), never the real path. - A ServiceMonitor tells Prometheus to scrape it (Prometheus on Kubernetes).
- RED dashboards and SLO alerts query these two metrics (PromQL, SLOs and error budgets).
Why not just...?¶
- ...push metrics from every app to a central server? Then the server cannot tell "no traffic" from "app dead", and every app needs to know where to send. Pull with discovery gives health for free.
- ...use a summary for latency? Summary quantiles cannot be combined across pods; a histogram can be aggregated, then turned into a percentile.
- ...keep a year of metrics in Prometheus? Local storage is not designed as a long-term archive. Use remote write to a long-term store.
Commands to know¶
| Command or query | What it does |
|---|---|
kubectl get --raw /metrics |
The API server's metrics in exposition format (as above). |
curl -s http://<pod-ip>:9090/metrics |
What any target exposes. |
up |
Whether each target's last scrape worked. |
count by (__name__) ({__name__=~".+"}) |
Series per metric name: find cardinality problems. |
prometheus_tsdb_head_series |
Active series in Prometheus itself. |
promtool check rules rules.yml |
Validate rule files. |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
| Target missing | Discovery | Targets page, ServiceMonitor selectors | Labels, port name or namespace do not match (Prometheus on Kubernetes) |
up == 0 |
Scrape | Target endpoint | App down, wrong port or path, NetworkPolicy blocking Prometheus |
| Prometheus out of memory | Cardinality | Head series count | An unbounded label |
| Gaps in graphs | Scrape timing | Scrape duration vs interval | Slow /metrics, timeouts |
| Rates jump after a restart | Counters | Resets | Handled by rate(); wrong if using raw differences |
Interview questions¶
1. How does Prometheus collect metrics, and why pull?
Answer
- It discovers targets and scrapes their
/metricsover HTTP on an interval. - Pull gives health (
up) for free, controls load centrally, and anyone can curl a target to debug. - Short-lived jobs are the exception (Pushgateway).
2. Explain counter, gauge, histogram and summary.
Answer
- Counter: only increases; query with
rate. - Gauge: current value that goes up and down.
- Histogram: bucketed counts plus sum and count, aggregatable into percentiles; summary: client-side quantiles that cannot be aggregated.
3. What is a time series in Prometheus?
Answer
- A metric name plus a unique set of label values, with timestamped samples.
- Each new label combination is a new series.
- Hence cardinality matters.
4. What causes high cardinality and how do you fix it?
Answer
- Labels with unbounded values: user IDs, request IDs, raw URL paths, pod names in long-lived metrics.
- Find them by counting series per metric and per label.
- Drop or rewrite labels (route templates), relabel at scrape time, move identifiers to logs and traces.
5. When should you use the Pushgateway?
Answer
- For service-level batch jobs that end before they can be scraped.
- Not for regular services: it hides health, holds stale values, and is a single point of failure.
- Push the job's result once at the end, and delete stale groups.
6. What are kube-state-metrics and node_exporter for?
Answer
- kube-state-metrics: the state of Kubernetes objects (desired vs available replicas, pod phases, restarts).
- node_exporter: host-level CPU, memory, disk and network.
- Together with cAdvisor they cover cluster, node and container views.
Try it¶
On a kind cluster:
kubectl get --raw /metrics | grep '^# TYPE' | awk '{print $4}' | sort | uniq -cand compare with the type counts (152 counters, 105 gauges, 55 histograms, 2 summaries on this lab's 1.37 API server).- Pick one histogram and find its
_countand_sumfor one label set; compute the average duration. - Count the series of
apiserver_request_totaland list which labels create most of them.
Zoom out¶
Prometheus is the metrics backbone of most Kubernetes platforms; PromQL turns its series into answers, the Prometheus Operator wires it into the cluster, and Alertmanager turns rule results into notifications (PromQL, Prometheus on Kubernetes, Alerting and Alertmanager).
Sources¶
- Prometheus documentation: overview
- Prometheus documentation: data model
- Prometheus documentation: metric types
- Prometheus documentation: exposition formats
- Prometheus documentation: exporters
- Prometheus documentation: storage (retention)
- Prometheus documentation: when to use the Pushgateway
- kube-state-metrics