CloudWatch¶
Status: AI draft, 2026-11-16. Not yet verified by me. Written from the Amazon CloudWatch and CloudWatch Logs user guides; no AWS calls were made.
In one line¶
CloudWatch is AWS's built-in monitoring: metrics identified by namespace, name and dimensions; alarms that evaluate a metric over periods and act; and Logs, with log groups, retention, metric filters, subscriptions and Logs Insights queries. The usual cost traps are high-cardinality custom metrics and logs that never expire.
The plain idea¶
Every AWS service already reports numbers about itself: CPU, requests, errors, queue depth. CloudWatch is where those numbers land, where you can add your own, and where something watches them and raises a hand when a line crosses a threshold. Logs are the same idea for text. It is the AWS-native answer to Prometheus and a log stack, and many teams use both (Signals and methods).
How it works¶
flowchart TD
S["AWS services<br/>(EC2, ALB, RDS, SQS...)"] -->|"built-in metrics"| M["Metrics<br/>namespace + name + dimensions"]
A["CloudWatch agent / EMF logs"] -->|"custom metrics"| M
A -->|"log events"| LG["Log group<br/>(retention, class)"]
LG -->|"metric filter"| M
LG -->|"subscription filter"| SUB["Lambda, Kinesis, Firehose"]
LG --> Q["Logs Insights queries"]
M --> AL["Alarm<br/>OK / ALARM / INSUFFICIENT_DATA"]
AL --> ACT["SNS, Auto Scaling,<br/>EC2 action, Systems Manager"]
Metrics¶
- A metric is identified by namespace (
AWS/EC2,AWS/ApplicationELB, or your own likeSampleApp), name (CPUUtilization) and dimensions (name/value pairs such asInstanceId=i-1234567890abcdef0). Each unique combination is a separate metric, and custom metrics are billed per metric. - A statistic (Average, Sum, Minimum, Maximum, SampleCount, percentiles such as p99) is computed over a period.
- Retention rolls up with age: data under 60 seconds (high-resolution custom metrics) for 3 hours, 1-minute data for 15 days, 5-minute for 63 days, 1-hour for 455 days (15 months).
- EC2 sends basic metrics every 5 minutes (detailed monitoring: 1 minute, charged). Memory and disk usage are not built-in: the hypervisor cannot see inside the guest. Install the CloudWatch agent to publish
mem_used_percent,disk_used_percentand logs (EC2).
Custom metrics: PutMetricData or EMF¶
PutMetricDataAPI calls publish metrics directly.- The embedded metric format (EMF) writes structured JSON log lines; CloudWatch extracts metrics from them asynchronously. Good for Lambda: no extra API calls, and the log line keeps the detail.
- Cardinality is the cost trap: a dimension like
UserIdorRequestIdcreates one billed metric per value.
Alarms¶
- States: OK, ALARM, INSUFFICIENT_DATA.
- Settings: period, evaluation periods (N) and datapoints to alarm (M): "M out of N" avoids alerting on one spike.
- Missing data can be treated as
missing(the default),notBreaching,breachingorignore. An error-count metric that only reports when errors happen should usenotBreaching; a heartbeat metric that stops reporting should usebreaching. - Composite alarms combine alarms with AND/OR to cut noise. Anomaly detection alarms use a learned band instead of a fixed threshold.
- Actions: SNS notifications, Auto Scaling policies, EC2 stop/reboot/recover, Systems Manager incidents (Alerting).
Logs¶
- Log group (one per app or source: retention, encryption, access) contains log streams (one per source instance or container) of log events.
- Retention defaults to never expire. Set it on every group, or storage grows forever.
- Log classes: Standard (all features) and Essentials (formerly Infrequent Access): cheaper ingestion, a subset of features (for example no subscription filters or Live Tail).
- Metric filters turn matching log lines into metrics (count of
ERROR), which alarms can watch. - Subscription filters stream events in near real time to Lambda, Kinesis Data Streams or Firehose (for example to OpenSearch or S3) (OpenSearch).
- Logs Insights queries log groups with its own query language, or OpenSearch PPL or SQL.
fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as errors by bin(5m)
| sort errors desc
Dashboards¶
Dashboards combine graphs, numbers, alarms and Logs Insights results, across Regions and accounts. Prefer a few dashboards built around user-facing signals over one per resource (Grafana can read CloudWatch too).
A concrete example¶
sample-app on ECS and RDS:
- Built-in metrics: ALB
HTTPCode_Target_5XX_CountandTargetResponseTime, RDSCPUUtilizationandFreeStorageSpace, SQSApproximateAgeOfOldestMessage. - The app writes EMF log lines with
OrdersPlaced(dimensionServiceonly, not per customer). - Log groups
/ecs/sample-appwith 30-day retention; a subscription filter streams to Firehose and on to S3 for a year. - Alarms: p99 latency above 1 second for 3 out of 5 one-minute periods; 5XX rate above 1 percent; DLQ depth above 0. All go to SNS, which pages on-call.
- A composite alarm suppresses the latency alarm while a deploy alarm is active.
- A dashboard per service shows the four golden signals.
Why not just...?¶
- ...use Prometheus and skip CloudWatch? Many AWS service metrics only exist in CloudWatch (ALB, RDS, SQS, Lambda). Prometheus can pull them with an exporter, but CloudWatch stays the source.
- ...alarm on every metric at a fixed threshold? Noise. Alarm on symptoms users feel, use M-of-N, and route causes to dashboards (SLOs).
- ...keep all logs forever in CloudWatch? Costly. Set retention per group and archive to S3 for long-term needs.
- ...add
UserIdas a dimension to see per-user latency? Every user becomes a billed metric. Put the ID in the log line and query it with Logs Insights.
Commands to know¶
These were not run for this page.
| Command | What it does |
|---|---|
aws cloudwatch list-metrics --namespace AWS/EC2 --metric-name CPUUtilization |
Metrics and their dimensions. |
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilization --dimensions Name=InstanceId,Value=i-1234567890abcdef0 --start-time <t1> --end-time <t2> --period 300 --statistics Average Maximum |
Datapoints for one metric. |
aws cloudwatch describe-alarms --state-value ALARM |
Alarms currently firing. |
aws cloudwatch describe-alarm-history --alarm-name <name> |
When an alarm changed state and why. |
aws logs describe-log-groups --query "logGroups[?!retentionInDays].logGroupName" |
Log groups that never expire. |
aws logs put-retention-policy --log-group-name <g> --retention-in-days 30 |
Set retention. |
aws logs tail /ecs/sample-app --follow --since 10m |
Follow a log group live. |
aws logs start-query --log-group-name <g> --start-time <epoch> --end-time <epoch> --query-string '<query>' |
Run a Logs Insights query (then get-query-results). |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
Alarm stuck in INSUFFICIENT_DATA |
Data | Metric datapoints, dimensions | Metric not published, wrong dimensions, or sparse metric with missing treatment |
| Alarm never fires during an outage | Alarm config | Statistic, period, missing data | Average hides spikes; missing data treated as not breaching |
| No memory metric for EC2 | Collection | Agent installed and running | Memory needs the CloudWatch agent |
| CloudWatch bill jumps | Cost | Cost Explorer by usage type | High-cardinality custom metrics, verbose logs, never-expiring log groups, frequent API polling |
| Logs missing from a container | Shipping | Log driver or agent, IAM permissions | Task role or node role lacks logs:PutLogEvents; wrong group name (Fluent Bit) |
| Logs Insights query times out | Query | Time range, log group size | Range too wide; filter early, narrow time |
Interview questions¶
1. What uniquely identifies a CloudWatch metric?
Answer
- Namespace, metric name and the full set of dimensions.
- A different dimension set is a different metric (and a separate charge for custom metrics).
2. Why can't you see memory usage for an EC2 instance by default, and how do you fix it?
Answer
- Built-in EC2 metrics come from the hypervisor, which cannot see inside the guest OS.
- Install the CloudWatch agent with a config that collects memory and disk metrics, with an IAM role that allows publishing.
3. Explain alarm evaluation: period, evaluation periods, datapoints to alarm and missing data.
Answer
- Each period produces one datapoint (a statistic).
- The alarm looks at the last N periods and fires when M breach (M out of N).
- Missing data treatment decides whether gaps count as breaching, not breaching, are ignored or leave the alarm in its state.
4. Your CloudWatch bill doubled. Where do you look?
Answer
- Cost Explorer by usage type: custom metrics count, log ingestion, log storage, API requests.
- High-cardinality dimensions; debug logging left on; log groups with no retention; dashboards or tools polling
GetMetricDataoften.
5. How do you get application logs from CloudWatch into OpenSearch or S3?
Answer
- A subscription filter on the log group to Firehose (to S3 or OpenSearch) or to Lambda or Kinesis for custom processing.
- Or ship from the source with Fluent Bit to both destinations.
6. What is the embedded metric format and when is it useful?
Answer
- Structured JSON log lines that CloudWatch turns into metrics asynchronously.
- No
PutMetricDatacalls in the hot path; good for Lambda and containers; the detailed log line remains for queries.
7. How would you alert on errors that only appear in logs?
Answer
- A metric filter matching the error pattern, producing a count metric.
- An alarm on that metric with
notBreachingfor missing data, M out of N periods.
Try it¶
Without AWS:
- Write a Logs Insights query that shows the top 10 URL paths returning 5XX in the last hour, from JSON logs with fields
pathandstatus. - Design alarms for an SQS worker: which metrics, statistics, periods, M out of N, and missing data treatment.
- Estimate how many custom metrics are created by one metric with dimensions
Service(5 values),Endpoint(40 values) andStatusCode(6 values).
Zoom out¶
CloudWatch is the default observability layer for AWS-managed services; Prometheus, Grafana and OpenSearch often sit beside it for Kubernetes and application detail. The design ideas are the same everywhere: few good alerts on symptoms, cheap high-volume logs with retention, and care with cardinality (Prometheus).
Sources¶
- CloudWatch User Guide: metrics concepts and retention
- CloudWatch User Guide: alarms
- CloudWatch User Guide: missing data in alarms
- CloudWatch User Guide: embedded metric format
- CloudWatch User Guide: metrics collected by the agent
- EC2 User Guide: instance metrics
- CloudWatch Logs User Guide: log groups and streams
- CloudWatch Logs User Guide: log classes
- CloudWatch Logs User Guide: subscriptions
- CloudWatch Logs User Guide: Logs Insights