Signals and methods¶
Status: AI draft, 2026-11-16. Not yet verified by me. Written from the Google SRE book, Brendan Gregg's USE method, the RED method and the OpenTelemetry documentation.
In one line¶
Metrics answer how much and how often, logs what exactly happened, traces where a request spent its time, events what changed. RED (rate, errors, duration) covers services, USE (utilisation, saturation, errors) covers resources. Alert on what users feel.
The plain idea¶
When something is wrong, you ask questions: is it slow, for whom, since when, where? Different kinds of data answer different questions, at different costs:
- Metrics are numbers over time, cheap to store and fast to query: "500 requests per second, 2% errors".
- Logs are records of individual events, rich and expensive: "request 8f2 failed: connection refused to db:5432".
- Traces follow one request across services: "of its 900 ms, 850 ms was in the payment service".
- Events record changes: a deploy, a pod rescheduled, a config change (kubectl fluency).
Monitoring watches for known failure modes with predefined checks. Observability is being able to ask new questions of the system from its outputs, including about failures nobody predicted, which needs rich, connected data.
How it works¶
Which signal answers what¶
| Signal | Answers | Cost | Example tools |
|---|---|---|---|
| Metrics | How much, how often, trends, alerts | Low per series, explodes with cardinality | Prometheus, CloudWatch metrics (Prometheus) |
| Logs | What exactly happened, error details | High volume | Fluent Bit, OpenSearch, CloudWatch Logs (How Kubernetes logging works) |
| Traces | Where time went across services | Medium, usually sampled | OpenTelemetry, Jaeger, X-Ray (Tracing and OpenTelemetry) |
| Events | What changed and when | Low | Kubernetes events, deploy markers, CloudTrail |
flowchart TD
A["Alert from a metric:<br/>error rate above target"] --> D["Dashboard: which service,<br/>which endpoint, since when"]
D --> T["Trace of a failing request:<br/>which hop is slow or failing"]
T --> L["Logs of that service, filtered<br/>by trace ID: the actual error"]
L --> E["Events: what changed just before?<br/>a deploy, a config change"]
The methods¶
| Method | For | Measure |
|---|---|---|
| RED | Request-driven services | Rate of requests, Errors, Duration (latency distribution) |
| USE | Resources: CPU, memory, disks, network, connection pools | Utilisation, Saturation (queued work), Errors (Performance triage) |
| Four golden signals (Google SRE) | User-facing systems | Latency, traffic, errors, saturation |
Use RED on every service endpoint, USE on everything those services depend on, and the golden signals as the summary on top.
Cardinality¶
Each unique combination of metric name and label values is a separate time series. A label with a few values (method="GET") is fine; a label with unbounded values (user_id, request_id, full URLs) creates millions of series, and the metrics system slows down or falls over. High-cardinality details belong in logs and traces, linked by IDs (Prometheus).
Dashboards vs alerts¶
- Alerts wake people up, so they must be about user impact and need action: "checkout error rate above the SLO burn threshold" (Alerting and Alertmanager, SLOs and error budgets).
- Dashboards support investigation: causes, resources, dependencies. They can be detailed because nobody is paged by them (Grafana).
A concrete example¶
What to measure for sample-app, an HTTP API behind an ALB:
| Layer | Measure |
|---|---|
| User-facing (RED) | Requests per second by route and status class; error ratio (5xx divided by all); p50, p95, p99 latency from a histogram |
| Dependencies (RED) | The same for its calls to the database and the payment API |
| Resources (USE) | Pod CPU against requests and throttling, memory against limits, DB connection pool in use and waiting (Requests, limits and QoS) |
| Saturation | Request queue length, HPA at max replicas |
| Events | Deploys and config changes as annotations on dashboards |
| Logs | Structured JSON with route, status, duration and trace ID |
| Alerts | Error budget burn rate, and p99 latency over the SLO, nothing else paging |
Why not just...?¶
- ...log everything and search the logs when needed? Logs are the most expensive signal and slow to aggregate; "error rate over the last week" is a metrics question. Use logs for detail.
- ...put user IDs in metric labels to debug individual users? Cardinality explodes. Put user IDs in logs and traces.
- ...alert on CPU above 80%? High CPU without user impact is not an emergency, and user impact can happen at low CPU. Alert on symptoms, investigate causes on dashboards.
Commands to know¶
| Command or query | What it shows |
|---|---|
kubectl top pods -n team-a |
Current CPU and memory (USE, coarse) (Autoscaling) |
kubectl get events -A --sort-by=.lastTimestamp |
Recent cluster events |
sum by (status_class) (rate(http_requests_total[5m])) |
Traffic by status class (RED rate and errors) (PromQL) |
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) |
p99 latency (RED duration) |
count by (__name__) ({__name__=~".+"}) |
Series per metric name: find cardinality problems |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
| Monitoring is green, users complain | Coverage | What is measured | Measuring resources, not user-facing RED; no synthetic checks |
| Too many alerts, all ignored | Alert design | What pages | Cause-based alerts (CPU, single pod restarts) instead of symptom-based |
| Metrics system slow or out of memory | Cardinality | Series count by metric | Unbounded label values |
| Cannot connect a slow request to its error | Correlation | Trace IDs in logs | Logs without trace or request IDs |
| Averages look fine, some users suffer | Statistics | Percentiles | Averages hide tails; use p95 and p99 |
Interview questions¶
1. What are metrics, logs and traces each best for?
Answer
- Metrics: cheap numbers over time for trends and alerts.
- Logs: detailed records of individual events, for the exact error.
- Traces: one request's path across services, to find where time or errors come from.
2. Explain RED and USE.
Answer
- RED, for services: request rate, errors, duration.
- USE, for resources: utilisation, saturation, errors.
- Together they cover what users feel and what the system is running out of.
3. What are the four golden signals?
Answer
- Latency, traffic, errors and saturation, from the Google SRE book.
- Latency of successful and failed requests should be tracked separately.
- Saturation shows how close the service is to its limits.
4. What is cardinality and why does it matter?
Answer
- The number of unique label combinations, each a separate time series.
- Unbounded labels (user IDs, request IDs, URLs) create huge numbers of series and break the metrics system.
- Keep such details in logs and traces.
5. What is the difference between monitoring and observability?
Answer
- Monitoring checks known conditions with predefined dashboards and alerts.
- Observability lets you ask new questions of the system's outputs to explain unknown failures.
- It depends on rich, connected signals: metrics, structured logs, traces with shared IDs.
6. What would you monitor for a new web API before launch?
Answer
- RED per route, with latency histograms; RED for its dependencies.
- USE for pods and the database; saturation signals like queue length and pool waits.
- An SLO with burn-rate alerts, structured logs with trace IDs, deploy annotations.
Try it¶
- For a service you know, fill in the concrete-example table above.
- List three labels currently on your metrics that could grow without limit.
- Write one alert that pages and three graphs that do not, for the same service.
Zoom out¶
These ideas organise the rest of the observability pages: Prometheus and PromQL for metrics, Fluent Bit and OpenSearch for logs, OpenTelemetry for traces, Alertmanager and SLOs for deciding when to wake someone (Prometheus, Fluent Bit, Tracing and OpenTelemetry, SLOs and error budgets).