--- name: pagerduty-alerts-nobody-acts-on description: "Every Monday: last week's alerts checked against human reactions in PagerDuty, Slack and GitHub, and the rules to delete or retune. Use when the user asks for this, or when it runs on a schedule." --- # Alerts nobody acts on: PagerDuty, Slack and deploys Every Monday: last week's alerts checked against human reactions in PagerDuty, Slack and GitHub, and the rules to delete or retune. PagerDuty counts how often you were paged. It cannot tell you that 14 of last week's 22 alerts resolved on their own while nobody said a word in Slack, or that 5 fired within ten minutes of your own deploys. Those are the alerts that train the on-call person to ignore the pager, and they are invisible unless someone lines up three tools by hand. This skill is for the on-call lead or engineering manager who owns alert quality. Every Monday the agent takes every alert of the week, looks for a person reacting to it (an acknowledgement, a message in the incident channel within 30 minutes, a linked PR), puts it on the same clock as production deploys, and groups the result by alert rule. The card ends with the rules to delete and the ones to retune. ## Two ways to get the data Use the API or MCP route when you have an MCP server, a CLI or an API token for the app. Otherwise, if you can drive a browser the user is signed in to (Claude in Chrome, a Playwright or Chrome DevTools MCP server), use the browser route. Say which one you used. ## Route A: API or MCP server ### What you need - Read access to the alerts: a read-only PagerDuty API key or the PagerDuty MCP server, a Grafana Viewer service account token, or Datadog API and application keys with `events_read` and `monitors_read`. - A Slack bot invited to the incident channel with `channels:history` (`groups:history` if private), and GitHub read access to Deployments and Pull requests. - Alert posts in Slack that carry the incident number or URL, and production deploys recorded as GitHub Deployments or Vercel deploys: these are the keys the join depends on. If something is missing, say what and stop; do not invent numbers. ### Steps 1. **Pull the week's alerts** (PagerDuty): Whichever you page from. PagerDuty: `GET /incidents?since=<7 days ago>&until=&time_zone=UTC&limit=100` (page with `offset`), then `GET /incidents/{id}/log_entries?is_overview=false` for each. Grafana: `GET /api/v1/rules/history?from=&to=` for state changes into Alerting and back. Datadog: `POST /api/v2/events/search` with `filter.query: "source:alert"` and the week as `filter.from` and `filter.to`, keeping `monitor_id` and `status`. 2. **Find a human reaction in the paging tool**: In PagerDuty log entries, an `acknowledge_log_entry`, `annotate_log_entry` or `resolve_log_entry` whose `agent.type` is `user_reference` is a person acting. An incident resolved by anything else (the integration, a service, a timeout) with no acknowledgement from a user is auto-resolved. Grafana and Datadog alerts have no acknowledgement, so for them the next two checks decide. 3. **Join to Slack and GitHub on time and incident ID** (Slack): For each alert, read the incident channel with `conversations.history` from the fire time to 30 minutes later (`oldest`, `latest`) plus `conversations.replies` on the alerting bot's post. A message from a person (no `bot_id`) in that thread, or one that mentions the incident number or alert title, counts as a reaction. Then `gh search prs --repo ""` for a PR within 7 days. An alert with none of the three is unacted. 4. **Put deploys on the same clock** (GitHub): List production deploys of the week with `GET /repos/{owner}/{repo}/deployments?environment=production` and their `success` status times (or Vercel deployments that reached `READY`). An alert that fired 0 to 10 minutes after a deploy finished is deploy-aligned. Everything is compared in UTC. 5. **Group by rule and give a verdict**: Group alerts by rule: PagerDuty service plus the incident title with numbers and host names stripped, Grafana `ruleUID`, Datadog `monitor_id`. **Delete**: fired 3 or more times with no human reaction and not deploy-aligned. **Retune**: most firings deploy-aligned (add a deploy silence or a longer pending period), or a reaction on under a third of them. **Keep**: the rest. 6. **Post the card**: Headline: unacted alerts out of all alerts and the deploy-aligned count. Then the table by rule with fired, unacted, after deploy and the verdict, and one line naming the rule to change first. ## Route B: the browser ### What you need - Signed in to PagerDuty with a role that can see the services, to Slack with access to the incident channel, and to GitHub with read access to the repository. If something is missing, say what and stop; do not invent numbers. ### Steps Start at https://app.pagerduty.com/. The user is signed in already; never type passwords or codes, and ask if a sign-in page appears. 1. **List last week's incidents**: Sign-in lands on your account's PagerDuty subdomain. Open **Incidents**, choose the **All** tab and set the date range to the last 7 days. Note each incident's number, title, service and created time. 2. **Read who reacted**: Open each incident and read its timeline. An acknowledgement, note or resolve by a named person is a reaction; resolved by the integration or automatically, with no one acknowledging, is not. 3. **Check Slack for the rest**: In Slack search `in:#incidents after:` and, for each unacknowledged incident, look for a message by a person within 30 minutes of it firing or a reply in the alert bot's thread. 4. **Line up deploys**: Open `https://github.com///deployments` and filter to the production environment. Mark every alert that fired within 10 minutes after a deploy; compare in one time zone. 5. **Group and give verdicts**: Group incidents by alert rule (same service and title). Delete: 3 or more firings, no reaction, not after deploys. Retune: mostly after deploys, or reacted to under a third of the time. Keep the rest. Note: PagerDuty times show in your profile's time zone and Slack in yours, so convert before matching. Grafana and Datadog users read their alert history pages instead of Incidents. Opening every incident is slow past about 40 a week; the API route reads log entries in one pass. ## The report - **Unacted alerts**: how many of the week's alerts got no human reaction. - **Deploy-aligned alerts**: how many fired within 10 minutes of a production deploy. - **By rule**: fired, unacted, after deploy, and the verdict (delete, retune, keep). - **Rules with real reactions**, so you know which pages earn their keep. - The one rule to change first, for example "Delete Disk 80% on worker-2: 5 pages, nobody looked". ## Post it to Briefkin When `BRIEFKIN_TOKEN` is set, or `~/.config/briefkin/env` exists (the installer from https://briefkin.dev/docs writes it), also send the result to the user's Briefkin board as one card. Keep the `item_key` the same on every run so the card updates in place, use a new `event_id` per run, and put today's real numbers in place of the example ones. If the request fails, say so once and carry on. ```bash [ -n "${BRIEFKIN_TOKEN:-}" ] || . ~/.config/briefkin/env curl -sS -X POST "${BRIEFKIN_URL:-https://briefkin.dev}/v1/events" \ -H "authorization: Bearer $BRIEFKIN_TOKEN" \ -H "content-type: application/json" \ -d @- <<'JSON' { "v": 1, "event_id": "pagerduty-alerts-nobody-acts-on/", "item_key": "pagerduty-alerts-nobody-acts-on", "project": "acme-web", "source": "Alerts nobody acts on: PagerDuty, Slack and deploys skill", "kind": "update", "title": "14 of 22 alerts got no human reaction; 5 fired right after a deploy", "summary": "**14 of 22** alerts last week auto-resolved with no acknowledgement and no Slack thread. **5** fired within 10 minutes of a production deploy. Two rules account for 11 of the 14.", "widget": [ { "type": "Table", "props": { "columns": [ "Alert rule", "Fired", "No reaction", "After deploy", "Verdict" ], "align": [ "left", "right", "right", "right", "left" ], "rows": [ [ "High latency p95 > 800 ms (api)", 7, 6, 4, "Retune" ], [ "Disk 80% on worker-2", 5, 5, 0, "Delete" ], [ "5xx rate > 2% (web)", 4, 1, 1, "Keep" ], [ "Queue depth > 500 (jobs)", 3, 2, 0, "Retune" ], [ "Checkout error rate > 1%", 3, 0, 0, "Keep" ] ] } }, { "type": "Grid", "props": { "columns": 3 }, "children": [ { "type": "Stat", "props": { "label": "Alerts", "value": 22 } }, { "type": "Stat", "props": { "label": "No human reaction", "value": 14, "better": "lower", "tone": "bad" } }, { "type": "Stat", "props": { "label": "Within 10 min of a deploy", "value": 5, "better": "lower", "tone": "warn" } } ] }, { "type": "Callout", "props": { "text": "Delete \"Disk 80% on worker-2\": 5 pages, nobody acknowledged or replied, and the disk never passed 84%.", "tone": "warn" } } ] } JSON ``` Widgets come from the Briefkin catalog (https://briefkin.dev/v1/catalog). Without a token, just give the user the report. ## Run it on a schedule Suggested: Mondays at 10:00 (`0 10 * * 1`). From cron, with the agent signed in on that machine: ```bash 0 10 * * 1 claude -p "Run the pagerduty-alerts-nobody-acts-on skill" # Codex: codex exec, Gemini CLI: gemini -p, Copilot CLI: copilot -p, OpenCode: opencode run ``` Or use the agent's own scheduler (Claude Code routines, Codex automations, Claude or ChatGPT scheduled tasks): https://briefkin.dev/skills/pagerduty-alerts-nobody-acts-on#schedule Source: https://briefkin.dev/skills/pagerduty-alerts-nobody-acts-on