Prometheus on Kubernetes¶
Status: AI draft, 2026-11-16. Not yet verified by me. Written from the Prometheus Operator documentation and the kube-prometheus-stack chart; nothing was installed.
In one line¶
The kube-prometheus-stack Helm chart installs the Prometheus Operator, Prometheus, Alertmanager, Grafana, kube-state-metrics, node-exporter and default rules and dashboards; you add targets and rules with ServiceMonitor, PodMonitor and PrometheusRule objects, which Prometheus picks up only when their labels and namespaces match its selectors, the usual reason a target is missing.
How it works¶
flowchart TD
SM["ServiceMonitor: selects Services by label,<br/>endpoints by port name"] --> OP["Prometheus Operator"]
PR["PrometheusRule: alert and<br/>recording rules"] --> OP
P["Prometheus custom resource: which ServiceMonitors<br/>and rules to select, by label and namespace"] --> OP
OP -->|"generates config"| PS["Prometheus server"]
PS -->|"scrapes"| T["Pods behind the Service"]
| Resource | Purpose |
|---|---|
Prometheus |
A Prometheus deployment: replicas, retention, storage, and selectors for monitors and rules |
ServiceMonitor |
Scrape the endpoints behind Services that match a label selector, on a named port and path |
PodMonitor |
Scrape pods directly, without a Service |
PrometheusRule |
Alerting and recording rules (Alerting and Alertmanager, PromQL) |
Alertmanager, AlertmanagerConfig |
Alertmanager deployment and routing |
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: sample-app
namespace: team-a
labels:
release: monitoring # must match the Prometheus serviceMonitorSelector
spec:
selector:
matchLabels:
app: sample-app # must match the Service's labels
endpoints:
- port: metrics # the Service port's name, not its number
path: /metrics
interval: 30s
Selection happens twice: the Prometheus resource selects ServiceMonitors (by their labels, and by namespace through serviceMonitorNamespaceSelector), and each ServiceMonitor selects Services (by the Services' labels, in the namespaces it lists). In kube-prometheus-stack, serviceMonitorSelectorNilUsesHelmValues defaults to true, so Prometheus only selects monitors carrying the chart's release label unless you change the selector.
Why a target is missing:
- ServiceMonitor lacks the label Prometheus selects on (the
releaselabel in the chart's default). - ServiceMonitor in a namespace Prometheus does not watch.
- Its
selectordoes not match the Service's labels. endpoints.portnames a port the Service does not have (by name).- The Service has no endpoints, or a NetworkPolicy blocks Prometheus (Services and DNS).
- Prometheus's service account lacks RBAC to list endpoints in that namespace.
What the stack scrapes out of the box: the API server, kubelets (including cAdvisor container metrics), CoreDNS, kube-state-metrics, node-exporter on every node, and control plane components where reachable (on EKS the managed control plane exposes only some of them) (Prometheus).
Interview questions¶
1. What does kube-prometheus-stack install?
Answer
- Prometheus Operator, Prometheus, Alertmanager, Grafana.
- kube-state-metrics and node-exporter.
- Default ServiceMonitors, alerting and recording rules, and dashboards.
2. What is a ServiceMonitor, and how does Prometheus find it?
Answer
- A custom resource describing how to scrape Services that match a label selector, on a named port.
- The Prometheus resource selects ServiceMonitors by label and namespace; the operator turns them into scrape config.
- Both selections must match.
3. You created a ServiceMonitor but the target never appears. What do you check?
Answer
- The ServiceMonitor's labels against the Prometheus
serviceMonitorSelector(oftenrelease: <name>), and its namespace against the namespace selector. - Its
selectoragainst the Service's labels, andportagainst the Service port's name. - Endpoints exist, NetworkPolicy, and Prometheus's RBAC.
4. ServiceMonitor or PodMonitor?
Answer
- ServiceMonitor when a Service already fronts the pods; most applications.
- PodMonitor for pods without a Service, such as some DaemonSets or jobs.
- Both end up as scrape configuration generated by the operator.
5. How do you add an alert rule for your application?
Answer
- A PrometheusRule object with the rule group, labelled to match the Prometheus
ruleSelector. - Validate the expression first; include
for, severity and a runbook link. - Alertmanager routing decides who is notified.