--- name: datadog-monitors-in-alert description: "Every hour of the working day: Datadog monitors in Alert or Warn, how long each has been there, and what is new or recovered since the hour before. Use when the user asks for this, or when it runs on a schedule." --- # Datadog monitors in alert Every hour of the working day: Datadog monitors in Alert or Warn, how long each has been there, and what is new or recovered since the hour before. The Datadog Monitors page sorts by status and shows everything at once: the database alert that matters, the disk warning that has been yellow for a month, and the muted synthetic test nobody owns. On-call gets paged for the urgent ones; everyone else has no quiet way to see the state of things. This skill gives the team that view during working hours. The agent lists monitors in Alert and Warn, sorts by how long they have been firing, and compares with its previous run so the card says what is new, what is still going and what recovered. ## Two ways to get the data Use the API or MCP route when you have an MCP server, a CLI or an API token for the app. Otherwise, if you can drive a browser the user is signed in to (Claude in Chrome, a Playwright or Chrome DevTools MCP server), use the browser route. Say which one you used. ## Route A: API or MCP server ### What you need - A Datadog API key and application key in `DD_API_KEY` and `DD_APP_KEY`, with the application key scoped to `monitors_read`, and your site (for example `datadoghq.com` or `datadoghq.eu`). - Or the Datadog MCP server with the core toolset, which includes `search_datadog_monitors`. If something is missing, say what and stop; do not invent numbers. ### Steps 1. **Search monitors by state** (Datadog): `GET https://api./api/v1/monitor/search?query=status:alert muted:false&per_page=100`, then again with `status:warn`. Keep `id`, `name`, `status`, `type`, `tags` and `last_triggered_ts`. With the Datadog MCP server, `search_datadog_monitors` does the same. 2. **See which groups are firing** (Datadog): For multi-alert monitors, `GET /api/v1/monitor/{id}?group_states=alert,warn` lists the hosts or services in each state, and gives `priority` when set. Show at most three groups per monitor. 3. **Compare with the last run**: Keep the list of monitor IDs and states in a small JSON file. A monitor not in the last run is new, one in both is ongoing, one that dropped out recovered. Firing time is now minus `last_triggered_ts`. 4. **Report**: New alerts first, then ongoing ones longest first, then warnings, then recovered in one line. Mark the card as failed only when a new monitor with priority P1 or P2 is in Alert. ## Route B: the browser ### What you need - Signed in to Datadog on the right organization, with any role that can read monitors (Datadog Read Only Role is enough). If something is missing, say what and stop; do not invent numbers. ### Steps Start at https://app.datadoghq.com/monitors/triggered. The user is signed in already; never type passwords or codes, and ask if a sign-in page appears. 1. **Open Triggered Monitors on your site**: Monitors > **Triggered Monitors** lists every monitor group currently in Alert, Warn or No Data, one row per group. On another Datadog site replace the host (`app.datadoghq.eu`, `us3.datadoghq.com`, `us5.datadoghq.com`, `ap1.datadoghq.com`). 2. **Filter to alerts that are not muted**: In the search bar type `status:alert muted:false` and read the rows, then change it to `status:warn muted:false`. Each row shows the monitor name, the group (host, service, query tag), its priority and the **Triggered** time, which is how long it has been firing. 3. **Open the ones that matter**: Click a monitor name to open its status page; the **Status & History** section shows which groups are alerting and when each one changed. Keep at most three groups per monitor for the card. 4. **Compare with the last check**: Note the monitor names and states, and compare with the list from the previous hour: not seen before is new, still there is ongoing, gone is recovered. The **Manage Monitors** page (Monitors > Manage) sorted by status gives the same list per monitor instead of per group. Note: Triggered Monitors shows one row per group, so a multi-alert monitor firing on 20 hosts takes 20 rows; the Manage Monitors page shows it once. Both pages reflect the current state only, so the new/ongoing/recovered comparison needs the previous list. ## The report - **Counts**: monitors in Alert, in Warn, new since the last run, recovered. - **New**: name, groups affected, priority, when it fired. - **Ongoing**: name and how long it has been firing, longest first. - **Recovered** since the last check. - One line on which alert to look at first. ## Post it to Briefkin When `BRIEFKIN_TOKEN` is set, or `~/.config/briefkin/env` exists (the installer from https://briefkin.dev/docs writes it), also send the result to the user's Briefkin board as one card. Keep the `item_key` the same on every run so the card updates in place, use a new `event_id` per run, and put today's real numbers in place of the example ones. If the request fails, say so once and carry on. ```bash [ -n "${BRIEFKIN_TOKEN:-}" ] || . ~/.config/briefkin/env curl -sS -X POST "${BRIEFKIN_URL:-https://briefkin.dev}/v1/events" \ -H "authorization: Bearer $BRIEFKIN_TOKEN" \ -H "content-type: application/json" \ -d @- <<'JSON' { "v": 1, "event_id": "datadog-monitors-in-alert/", "item_key": "datadog-monitors-in-alert", "project": "Acme", "source": "Datadog monitors in alert skill", "kind": "update", "title": "3 monitors alerting, 1 new since 10:00", "summary": "**Checkout p95 latency** started alerting at 10:42 on 2 hosts. Two older alerts are still open and **2** monitors recovered since the last check.", "widget": [ { "type": "Badges", "props": { "items": [ { "label": "Alert 3", "tone": "bad" }, { "label": "Warn 2", "tone": "warn" }, { "label": "New 1", "tone": "bad" }, { "label": "Recovered 2", "tone": "ok" } ] } }, { "type": "Table", "props": { "columns": [ "Monitor", "State", "Firing for", "Change" ], "rows": [ [ "Checkout p95 latency above 800 ms (P2)", "Alert", "18 min", "New" ], [ "RDS acme-prod free storage below 15%", "Alert", "2 days", "Ongoing" ], [ "Queue depth on emails above 5,000", "Alert", "3 h", "Ongoing" ], [ "Disk usage above 80% on web-3", "Warn", "9 days", "Ongoing" ], [ "5xx rate on api above 1%", "Warn", "35 min", "Ongoing" ] ] } } ] } JSON ``` Widgets come from the Briefkin catalog (https://briefkin.dev/v1/catalog). Without a token, just give the user the report. ## Run it on a schedule Suggested: Hourly, 8:00 to 18:00 on weekdays (`0 8-18 * * 1-5`). From cron, with the agent signed in on that machine: ```bash 0 8-18 * * 1-5 claude -p "Run the datadog-monitors-in-alert skill" # Codex: codex exec, Gemini CLI: gemini -p, Copilot CLI: copilot -p, OpenCode: opencode run ``` Or use the agent's own scheduler (Claude Code routines, Codex automations, Claude or ChatGPT scheduled tasks): https://briefkin.dev/skills/datadog-monitors-in-alert#schedule Source: https://briefkin.dev/skills/datadog-monitors-in-alert