--- name: github-actions-flaky-ci-tax description: "Every Monday: the jobs that failed and then passed on the same commit, the PRs they held up, the hours waited and the runner minutes burned. Use when the user asks for this, or when it runs on a schedule." --- # Flaky CI tax: GitHub Actions reruns and the PRs they held up Every Monday: the jobs that failed and then passed on the same commit, the PRs they held up, the hours waited and the runner minutes burned. For developers and DevOps leads who know CI is flaky but cannot say how much it costs. GitHub Actions records every attempt of every run; the pull request page only shows the final green check. The hour between a red e2e job and its passing rerun disappears. The agent finds runs that failed and passed on retry with no code change, ties them to the PRs on that commit, and adds up how long those PRs waited and how many runner minutes the failed attempts used. The job at the top of the list is the one to quarantine or fix this week. ## Two ways to get the data Use the API or MCP route when you have an MCP server, a CLI or an API token for the app. Otherwise, if you can drive a browser the user is signed in to (Claude in Chrome, a Playwright or Chrome DevTools MCP server), use the browser route. Say which one you used. ## Route A: API or MCP server ### What you need - `gh` signed in, or a fine-grained token in `GITHUB_TOKEN` with Actions: Read, Pull requests: Read and Metadata: Read on the repositories. - CI triggered on `pull_request`, so runs carry the PR and its head SHA; reruns done with "Re-run jobs" rather than an empty commit, which changes the SHA and hides the flake. If something is missing, say what and stop; do not invent numbers. ### Steps 1. **Find runs that needed a retry** (GitHub Actions): Call `GET /repos/{owner}/{repo}/actions/runs?event=pull_request&created=>=<7 days ago>&per_page=100` and keep runs with `run_attempt` above 1 and `conclusion: success`. Those passed on a later attempt of the same run, so the same `head_sha`. 2. **Read each attempt and its jobs** (GitHub Actions): For each attempt, `GET /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{attempt_number}` for its `conclusion` and `updated_at`, and `.../attempts/{attempt_number}/jobs` for job names, conclusions, `started_at` and `completed_at`. A flake is a job that failed in one attempt and succeeded in the next on the same SHA. 3. **Tie each flake to its PRs** (GitHub): Take the run's `pull_requests` list; when it is empty (forks), call `GET /repos/{owner}/{repo}/commits/{head_sha}/pulls`. Flakes on SHAs with no PR are counted in runner minutes but not in waiting time. 4. **Add up the tax**: Waiting per flake is the passing attempt's `updated_at` minus the failed attempt's `updated_at`, capped at 24 hours so a weekend does not dominate. Runner minutes are each failed job's duration rounded up to the whole minute, as GitHub counts it. Group by job name: flakes, distinct PRs, hours waited, runner minutes. 5. **Post the trend**: Read last week's card with `GET https://briefkin.dev/v1/cards/insight-github-actions-flaky-ci-tax` to extend the history, then post total hours waited as the headline and the job table when the card opens, under the same `item_key`. ## Route B: the browser ### What you need - Signed in to GitHub with read access to the repository and its Actions runs. If something is missing, say what and stop; do not invent numbers. ### Steps Start at https://github.com/. The user is signed in already; never type passwords or codes, and ask if a sign-in page appears. 1. **Open the repository's Actions tab**: Go to the repository, click **Actions** and filter the run list by event `pull_request`. Look at the last 7 days of runs. 2. **Find runs that passed on a retry**: Open green runs; a run that was re-run shows an attempt picker at the top of its summary page. Switch to the earlier attempt and note which job failed and when it finished, then the time the passing attempt finished. 3. **Note the PR and the job time**: The run summary links the pull request. Open the failed job in the earlier attempt to read its duration, and round it up to the whole minute. 4. **Add it up**: Per job name, count the flakes, the distinct PRs, the hours between the failed and passing attempt (capped at 24 hours each), and the runner minutes. Sort by hours waited. Note: The Actions list has no filter for rerun runs, so the agent has to open each green run, which is slow past a few dozen runs a week. For busier repositories the API route is the one to use; the browser route works as a spot check on one workflow. ## The report - **Hours PRs waited on reruns** this week, against previous weeks. - **Runner minutes** spent on failed attempts and the number of PRs held up. - **Per job**: flakes, PRs, hours waited and runner minutes, worst first. - One line on what to do first, for example "Quarantine e2e-checkout: 14 flakes, 6.4 hours". ## Post it to Briefkin When `BRIEFKIN_TOKEN` is set, or `~/.config/briefkin/env` exists (the installer from https://briefkin.dev/docs writes it), also send the result to the user's Briefkin board as one card. Keep the `item_key` the same on every run so the card updates in place, use a new `event_id` per run, and put today's real numbers in place of the example ones. If the request fails, say so once and carry on. ```bash [ -n "${BRIEFKIN_TOKEN:-}" ] || . ~/.config/briefkin/env curl -sS -X POST "${BRIEFKIN_URL:-https://briefkin.dev}/v1/events" \ -H "authorization: Bearer $BRIEFKIN_TOKEN" \ -H "content-type: application/json" \ -d @- <<'JSON' { "v": 1, "event_id": "github-actions-flaky-ci-tax/", "item_key": "github-actions-flaky-ci-tax", "project": "acme-web", "source": "Flaky CI tax: GitHub Actions reruns and the PRs they held up skill", "kind": "metric", "title": "Flaky e2e-checkout cost 6.4 hours of PR waiting", "summary": "Flaky **e2e-checkout** cost **6.4 hours** of waiting and **212 runner minutes** across **9 PRs**.", "metric": { "label": "Hours PRs waited on reruns", "value": 9.1, "unit": "h", "period_label": "Last 7 days", "better": "lower", "period_start": "", "period_end": "" }, "glance": [ { "type": "Grid", "props": { "columns": 3 }, "children": [ { "type": "Stat", "props": { "label": "Runner minutes", "value": 290, "delta": 31, "better": "lower" } }, { "type": "Stat", "props": { "label": "PRs held up", "value": 12, "delta": 33, "better": "lower" } }, { "type": "Stat", "props": { "label": "Flaky reruns", "value": 21, "delta": 40, "better": "lower" } } ] } ], "widget": [ { "type": "Table", "props": { "columns": [ "Job", "Flakes", "PRs", "Hours waited", "Runner min" ], "align": [ "left", "right", "right", "right", "right" ], "rows": [ [ "e2e-checkout", 14, 9, 6.4, 212 ], [ "integration-db", 5, 4, 1.9, 58 ], [ "visual-regression", 2, 2, 0.8, 20 ], [ "All jobs", 21, 12, 9.1, 290 ] ] } }, { "type": "Callout", "props": { "text": "Quarantine or fix e2e-checkout first: 11 of its 14 failures were in the same step, Pay with test card. PRs count once in the total when two jobs held them up.", "tone": "warn" } } ], "size": "md" } JSON ``` Widgets come from the Briefkin catalog (https://briefkin.dev/v1/catalog). Without a token, just give the user the report. Set `period_start` and `period_end` to the window the number covers: the tile names it from them ("Last 7 days", "Today", "This month"). For a count right now (open items, a balance), send `"period_label": ""` so no period shows; for another name, send it as `period_label` (at most 24 characters). ## Run it on a schedule Suggested: Mondays at 9:00 (`0 9 * * 1`). From cron, with the agent signed in on that machine: ```bash 0 9 * * 1 claude -p "Run the github-actions-flaky-ci-tax skill" # Codex: codex exec, Gemini CLI: gemini -p, Copilot CLI: copilot -p, OpenCode: opencode run ``` Or use the agent's own scheduler (Claude Code routines, Codex automations, Claude or ChatGPT scheduled tasks): https://briefkin.dev/skills/github-actions-flaky-ci-tax#schedule Source: https://briefkin.dev/skills/github-actions-flaky-ci-tax