Every tired on-call engineer has a mental list of alerts they have learned to swipe away. That list is the real state of your alerting, and it is usually longer than the runbook admits.
Noise is a trust problem
An alert is a promise: this is worth your attention right now. Break that promise often enough and the alert stops meaning anything, no matter how urgent its colour. The cost of a noisy alert is not the notification. It is the credibility of every alert next to it.
Fewer, sharper, actionable
Good alerting is ruthless. It fires on conditions a human can act on, it groups related symptoms instead of paging five times for one incident, and it ties to a clear next step. If there is nothing to do, it is a report, not an alert.
- Alert on user-facing impact, not on every underlying wobble.
- Group related signals so one incident is one page.
- If no action follows, move it to a dashboard or a digest.
The measure of an alerting setup is not how much it catches. It is whether the person holding the pager still believes it at 3 a.m. Everything else is detail.
