--- name: grafana-firing-alerts-digest description: "Every morning: Grafana alert rules that are firing or pending, grouped by folder, with how long each has fired and which are silenced. Use when the user asks for this, or when it runs on a schedule." --- # Grafana firing alerts digest Every morning: Grafana alert rules that are firing or pending, grouped by folder, with how long each has fired and which are silenced. Grafana sends a notification when an alert starts firing, and then it goes quiet until it resolves. Alerts that fire for days, the ones everyone has learned to ignore, end up hiding the new one in the same channel. The agent reads the current state of every Grafana-managed alert rule, groups the firing and pending ones by folder (usually one per team or service), sorts them by how long they have been active and marks the ones under a silence. The morning card is the list your team would see if it opened the Alert rules page and filtered by state. ## Two ways to get the data Use the API or MCP route when you have an MCP server, a CLI or an API token for the app. Otherwise, if you can drive a browser the user is signed in to (Claude in Chrome, a Playwright or Chrome DevTools MCP server), use the browser route. Say which one you used. ## Route A: API or MCP server ### What you need - A Grafana service account token with the Viewer role (enough to read alert rules and silences) in `GRAFANA_TOKEN`, and the Grafana URL. - Or the Grafana MCP server (`mcp-grafana`) connected to the same instance. If something is missing, say what and stop; do not invent numbers. ### Steps 1. **Read rule states** (Grafana): With the Grafana MCP server (`mcp-grafana`, started with `--disable-write`), `list_alert_rules` returns every rule with its state and `get_alert_rule_by_uid` the detail. Over HTTP, `GET /api/prometheus/grafana/api/v1/rules` (the endpoint the Alert rules page uses) returns groups with `file` (the folder), and rules with `state`, `health` and their active `alerts`. 2. **Keep firing and pending**: Keep rules whose `state` is `firing` or `pending`, and rules with `health` `error`, which cannot evaluate at all. For each alert instance, firing time is now minus `activeAt`; show its key labels such as `service` or `instance`. 3. **Mark silences** (Grafana): `GET /api/alertmanager/grafana/api/v2/silences` and match active silences to the alert labels, so a silenced alert is shown as such instead of as a new problem. 4. **Compare and report**: Compare with yesterday's list. Group by folder, firing before pending, longest first, and mark new ones. Rules in error get their own line, since a broken query means you have no alert at all. ## Route B: the browser ### What you need - Signed in to the Grafana instance (or the Grafana Cloud stack) with at least the Viewer role on the folders that hold the rules; Viewer can read alert rules and silences. If something is missing, say what and stop; do not invent numbers. ### Steps Start at https://grafana.com/auth/sign-in/. The user is signed in already; never type passwords or codes, and ask if a sign-in page appears. 1. **Open the Alert rules page**: For Grafana Cloud, sign in at grafana.com, open **My Account** and launch the stack; for a self-hosted Grafana, open your own URL. Then go to **Alerting** > **Alert rules** (the path is `/alerting/list`). Leave the **Grafana managed** section open and collapse data source managed rules unless you want them. 2. **Filter by state**: Use the **State** filter and pick `Firing`, read the list, then pick `Pending`. Rules are grouped by folder and evaluation group; expand a rule to see its alert instances, their labels and the **Active since** time, which gives how long it has fired. 3. **Find broken rules**: Set the **Health** filter to `Error` (or search `health:error`) to list rules that cannot evaluate; the rule row shows the error text. Clear the filters afterwards. 4. **Mark silences**: Open **Alerting** > **Silences**. Active silences list their matchers and end time; match them against the firing instances' labels and mark those alerts as silenced in the report. 5. **Compare and report**: Compare the rule names with yesterday's list to mark what is new. Group by folder, firing before pending, longest first, with silenced and errored rules called out and the one alert to look at first. Note: The Alert rules page shows the state at load time, so refresh before reading. Instances of a multi-dimensional rule can run to hundreds of rows; the page paginates them per rule. Data source managed rules (Mimir, Loki, Prometheus) appear in their own section and are not part of this report unless you ask. ## The report - **Counts**: rules firing, pending, in error, and how many are new since yesterday. - **By folder**: each firing or pending rule with its labels and how long it has been active. - **Silenced**: alerts under a silence, with when the silence ends. - **Broken rules**: rules whose query errors. - One line on the alert to look at first. ## Post it to Briefkin When `BRIEFKIN_TOKEN` is set, or `~/.config/briefkin/env` exists (the installer from https://briefkin.dev/docs writes it), also send the result to the user's Briefkin board as one card. Keep the `item_key` the same on every run so the card updates in place, use a new `event_id` per run, and put today's real numbers in place of the example ones. If the request fails, say so once and carry on. ```bash [ -n "${BRIEFKIN_TOKEN:-}" ] || . ~/.config/briefkin/env curl -sS -X POST "${BRIEFKIN_URL:-https://briefkin.dev}/v1/events" \ -H "authorization: Bearer $BRIEFKIN_TOKEN" \ -H "content-type: application/json" \ -d @- <<'JSON' { "v": 1, "event_id": "grafana-firing-alerts-digest/", "item_key": "grafana-firing-alerts-digest", "project": "Acme", "source": "Grafana firing alerts digest skill", "kind": "update", "title": "4 alert rules firing, the oldest for 3 days", "summary": "**4** rules are firing and **2** pending across 3 folders. **Payments 5xx above 2%** is new since yesterday; **1** rule cannot evaluate.", "widget": [ { "type": "Badges", "props": { "items": [ { "label": "Firing 4", "tone": "bad" }, { "label": "Pending 2", "tone": "warn" }, { "label": "New 1", "tone": "bad" }, { "label": "Silenced 1", "tone": "neutral" }, { "label": "Error 1", "tone": "warn" } ] } }, { "type": "Table", "props": { "columns": [ "Folder", "Rule", "State", "Active for" ], "rows": [ [ "Payments", "Payments 5xx above 2% (service=checkout)", "Firing", "2 h, new" ], [ "Platform", "Node disk above 85% (instance=db-2)", "Firing", "3 days" ], [ "Platform", "Certificate expires in 14 days (api.example.com)", "Firing", "2 days, silenced" ], [ "Data", "Nightly ETL did not finish", "Firing", "5 h" ], [ "Payments", "Webhook queue lag above 60 s", "Pending", "4 min" ], [ "Platform", "Pod restarts above 5 per hour", "Pending", "2 min" ] ] } }, { "type": "Callout", "props": { "text": "Rule \"Search latency p99\" is in error: its Prometheus query references a metric that no longer exists.", "tone": "warn" } } ] } JSON ``` Widgets come from the Briefkin catalog (https://briefkin.dev/v1/catalog). Without a token, just give the user the report. ## Run it on a schedule Suggested: Every day at 8:30 (`30 8 * * *`). From cron, with the agent signed in on that machine: ```bash 30 8 * * * claude -p "Run the grafana-firing-alerts-digest skill" # Codex: codex exec, Gemini CLI: gemini -p, Copilot CLI: copilot -p, OpenCode: opencode run ``` Or use the agent's own scheduler (Claude Code routines, Codex automations, Claude or ChatGPT scheduled tasks): https://briefkin.dev/skills/grafana-firing-alerts-digest#schedule Source: https://briefkin.dev/skills/grafana-firing-alerts-digest