I have inherited platforms with 140 CloudWatch alarms and a Slack channel nobody unmuted. The fix was not more graphs. It was a smaller set of SLIs that map to user experience, traces that actually connect the BFF to the worker, and logs that carry the same ids the trace does.
Three questions, three signals
- Is the user broken? Availability and latency SLIs at the edge (CloudFront / ALB / API Gateway), sliced by route, not by instance.
- Which hop? A trace id from the browser or BFF through SQS attributes onto the worker. X-Ray or OTel to ADOT; pick one propagator and be dogmatic.
- Us or them? Dependency SLIs: DynamoDB throttles, RDS CPU, PSP latency, IoT Core rule errors. Page on burn rate, not on a single 5xx.
Correlation is a field name
Every log line in a request path should include `trace_id`, `tenant_id` when it exists, and the business id (`order_id`, `device_id`). JSON logs to CloudWatch, not text. Insights queries become trivial; without the fields they become folklore. SQS and EventBridge must forward the trace context in message attributes — if you drop it at the queue, you have two systems, not one.
1. Edge availability SLO (30-day, 99.9%) — burn-rate page
2. Edge p99 latency vs budget per critical route
3. Queue age + DLQ depth per worker
4. Downstream error rate (PSP, RDS, DynamoDB) vs baseline
5. Certificate and secret expiry (days, not hours)
No CPU-on-a-single-task alarms for Fargate APIs
unless that task is a singleton you already regret.Instrument the user journey first. Then the queues. Then the databases. If budget remains, pretty dashboards. Never the other way around.