PromQL¶
Status: AI draft, 2026-11-16. Not yet verified by me. Written from the Prometheus documentation; no Prometheus server was run, so the queries have no sample output. Metric names follow the
sample-appexample on Prometheus.
In one line¶
PromQL selects series by name and labels (an instant vector) or over a time window (a range vector, [5m]), turns counters into per-second rates with rate() before aggregating them with sum by (...), computes error ratios by dividing rates, and gets latency percentiles from histogram buckets with histogram_quantile(); recording rules precompute expensive queries.
The plain idea¶
Prometheus stores raw counters that only go up. Nobody wants "12,804,553 requests since the pod started"; they want "how many per second now, how many failed, how slow". PromQL is the small language that turns stored series into those answers, for graphs and for alert rules.
How it works¶
flowchart TD
S["Stored counter series, one per pod:<br/>http_requests_total{pod=...}"] --> R["rate(...[5m]) per series:<br/>per-second, reset-aware"]
R --> A["sum by (route): combine pods,<br/>keep the labels you care about"]
A --> O["Answer: requests per second,<br/>an error ratio, or histogram_quantile"]
Selectors and vectors¶
http_requests_total # every series of the metric
http_requests_total{route="/orders", status_class="5xx"}
http_requests_total{status_class=~"4xx|5xx"} # regex match
http_requests_total{namespace!="kube-system"} # not equal
http_requests_total[5m] # range vector: samples of the last 5 minutes
- An instant vector has one value per series at the evaluation time: what graphs and alerts plot.
- A range vector has all samples in a window for each series: only functions such as
rate()accept it.
Counters: rate, irate, increase¶
| Function | Gives | Use |
|---|---|---|
rate(c[5m]) |
Per-second average increase over the window, handling counter resets and extrapolating to the window edges | Graphs and alerts |
irate(c[5m]) |
Per-second rate from the last two samples in the window | Fast-moving, zoomed-in graphs; not alerts |
increase(c[1h]) |
Total increase over the window (rate multiplied by the window) | "How many in the last hour" |
Rate first, then aggregate. sum(rate(x[5m])) is correct; rate(sum(x)[5m]) cannot see a single pod's counter reset and produces nonsense. The documentation states this rule for rate and irate.
Use a window several times the scrape interval so each window holds enough samples.
Aggregation¶
sum(rate(http_requests_total[5m])) # total requests per second
sum by (route) (rate(http_requests_total[5m])) # per route
sum without (pod, instance) (rate(http_requests_total[5m])) # drop some labels, keep the rest
topk(5, sum by (route) (rate(http_requests_total[5m]))) # five busiest routes
avg by (namespace) (container_memory_working_set_bytes) # gauges can be aggregated directly
Error ratio¶
sum(rate(http_requests_total{status_class="5xx"}[5m]))
/
sum(rate(http_requests_total[5m]))
A value of 0.02 means 2% of requests failed in the last 5 minutes. Divide aggregated rates; both sides must end up with matching labels (here none), or use on() and ignoring() to control matching.
Latency percentiles from histograms¶
histogram_quantile(
0.99,
sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)
rate(..._bucket[5m]): per-second counts for each bucket.sum by (le): combine all pods, keeping the bucket boundary labelle(addrouteto thebylist for per-route percentiles).histogram_quantile(0.99, ...): the value below which 99% fell, interpolated within the bucket. Accuracy depends on bucket boundaries; put buckets around your latency targets.
Average latency instead: sum(rate(..._sum[5m])) / sum(rate(..._count[5m])).
offset and subqueries¶
sum(rate(http_requests_total[5m])) / sum(rate(http_requests_total[5m] offset 1w)) # vs same time last week
max_over_time(sum(rate(http_requests_total[5m]))[1h:1m]) # subquery: peak 5-minute rate in the last hour
Recording rules¶
Expensive queries used by many dashboards or alerts can be computed on a schedule and stored as new series. Name them level:metric:operations:
groups:
- name: sample-app
rules:
- record: route:http_requests:rate5m
expr: sum by (route) (rate(http_requests_total[5m]))
- record: route:http_request_errors:ratio_rate5m
expr: |
sum by (route) (rate(http_requests_total{status_class="5xx"}[5m]))
/
sum by (route) (rate(http_requests_total[5m]))
A concrete example¶
Five queries to know by heart, for sample-app on Kubernetes:
| Question | Query |
|---|---|
| Requests per second | sum(rate(http_requests_total{namespace="team-a"}[5m])) |
| Error ratio | sum(rate(http_requests_total{status_class="5xx"}[5m])) / sum(rate(http_requests_total[5m])) |
| p99 latency | histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) |
| Pods restarting | increase(kube_pod_container_status_restarts_total{namespace="team-a"}[1h]) > 0 (kube-state-metrics) |
| CPU throttling | sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m])) / sum by (pod) (rate(container_cpu_cfs_periods_total[5m])) (cAdvisor) |
Why not just...?¶
- ...graph the raw counter? It only shows a line going up, with drops at every restart. Rates show behaviour.
- ...average the per-pod p99s? Averaging percentiles is mathematically meaningless. Sum the bucket rates across pods, then take the quantile.
- ...use
iratein alerts? It reflects only the last two samples, so one spike or dip fires or clears the alert. Useratewith a sensible window and afor:duration (Alerting and Alertmanager).
Commands to know¶
| Command or query | What it does |
|---|---|
up == 0 |
Targets whose last scrape failed. |
count by (__name__) ({__name__=~".+"}) |
Series per metric, for cardinality. |
absent(up{job="sample-app"}) |
Alert when a job disappears entirely. |
promtool check rules rules.yml |
Validate recording and alerting rules. |
promtool query instant http://prometheus:9090 'up' |
Run a query from the command line. |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
| Negative or huge spikes in a "rate" | Query order | Where sum sits |
Aggregated before rate, so resets were not detected |
many-to-many matching not allowed |
Vector matching | Labels on each side | Divide aggregates with matching labels, or use on(), ignoring() |
| Percentile stuck at a bucket boundary | Histogram buckets | Bucket boundaries | Too few buckets around the interesting range |
| Empty result | Selector | Label values, time range | Wrong label value, or metric not scraped |
| Query slow or times out | Cost | Series touched | High-cardinality metric; use recording rules or narrower selectors |
Interview questions¶
1. What is the difference between an instant vector and a range vector?
Answer
- Instant vector: one sample per series at a point in time.
- Range vector: all samples in a time window per series, written with
[5m]. - Range vectors feed functions like
rate(); graphs and alerts use instant vectors.
2. Explain rate, irate and increase.
Answer
rate: per-second average increase over the window, reset-aware, extrapolated.irate: per-second rate from the last two samples; very responsive, noisy.increase: total increase over the window; equal to rate times the window length.
3. Why must you take rate() before sum()?
Answer
ratedetects counter resets per series.- Summing first hides a single series' reset inside a larger total, giving wrong values.
- The Prometheus docs state the rule explicitly.
4. Write a query for the error ratio of a service.
Answer
sum(rate(http_requests_total{status_class="5xx"}[5m])) / sum(rate(http_requests_total[5m])).- Aggregate both sides to the same labels before dividing.
- Use a recording rule if alerts and dashboards reuse it.
5. How do you compute p99 latency across all pods?
Answer
histogram_quantile(0.99, sum by (le) (rate(..._bucket[5m]))).- Sum bucket rates across pods keeping
le, then take the quantile. - Never average per-pod percentiles.
6. What are recording rules and how should they be named?
Answer
- Queries evaluated on a schedule and stored as new series, to make dashboards and alerts cheap and consistent.
- Named
level:metric:operations, such asroute:http_requests:rate5m. - Validate with
promtool check rules.
Try it¶
Without a Prometheus server, use the kind API server's real metrics (Prometheus):
- From
kubectl get --raw /metrics, takeapiserver_request_duration_seconds_sumand_countfor one label set and compute the average duration by hand. - Write the PromQL that would give the p99 of
apiserver_request_duration_secondsperverb. - Write a recording rule for the API server's 5xx ratio, named correctly.
Zoom out¶
PromQL is how metrics become dashboards, alerts and SLOs (Grafana, Alerting and Alertmanager, SLOs and error budgets). The same few patterns (rate then sum, ratio of rates, quantile of summed buckets) cover most real questions.