Triage my alerting setup. I will paste the alerts; you sort them and tell me what to change about each one.
Alerting and monitoring tools: {{e.g. Prometheus + Alertmanager routed to PagerDuty; Datadog monitors posting to Slack}}
The service or system these cover: {{one paragraph: what it does, who depends on it, what "down" means for a user}}
SLOs, if we have them: {{e.g. 99.9% availability on the checkout API over 30 days; or "none defined"}}
Alerts, with how often each fired in the last {{30}} days and what happened when it did: {{paste the list: name, condition and threshold, severity, channel, fire count, and for each one roughly how often someone had to act}}
What on-call complains about most: {{e.g. the disk alert on the batch host fires every night at 02:00 and clears itself; three different alerts fire for the same upstream outage}}
For every alert, give a verdict from exactly one of these four:
- **Page.** Fires for something that consumes a large share of the error budget or blocks users, and a human has to act within minutes. Keep it, and say whether its threshold or duration needs tightening.
- **Ticket.** Real, but tolerates hours. Move it out of the paging channel into a queue with an owner.
- **Rewrite.** The concern is valid, but the alert as written fires on a cause rather than on user impact (raw CPU, memory, queue length with no latency effect), or it flaps, or it duplicates another alert. Say what it should measure instead.
- **Delete.** Nobody has acted on it, or it announces something the dashboard already shows. Say so plainly.
Then:
**Duplicates and cascades.** Group alerts that fire together for the same underlying failure. For each group, name the one alert that should page and how the rest get grouped, deduplicated on a shared key, or inhibited while the parent is active, using the mechanism my tool has.
**Predictable alerts.** Anything that fires on a schedule, during a known job, or in a maintenance window: propose the window, the silence rule, or the threshold that reflects normal load at that hour, so a human stops acknowledging it every time.
**Where an SLO alert would replace several threshold alerts.** If I gave you SLOs, propose burn-rate alerts with a long window and a short confirmation window, with the numbers, and list the cause-based alerts they retire. If I have no SLOs, list the two or three user-facing indicators you would define them on.
**Before and after.** Estimate the expected weekly page count before and after, and mark the change you would make first because it removes the most noise for the least risk.
Give every verdict a one-line reason tied to the fire count and the action history I gave you, not to a general rule. If an alert has no history, say you cannot judge it and ask for the number instead of guessing.Tip: Export the fire counts from your alerting tool for the period before you paste; the verdicts are only as good as the action history. Alert names and thresholds are fine to share; strip hostnames and internal URLs if you are on a consumer chat tool.
sreon-callalertingobservability