--- name: databricks-failed-jobs description: "Every morning: Databricks job runs that failed or timed out since yesterday, with the failing task, the error line and a link, and an ask about repairing. Use when the user asks for this, or when it runs on a schedule." --- # Databricks failed jobs Every morning: Databricks job runs that failed or timed out since yesterday, with the failing task, the error line and a link, and an ask about repairing. A nightly Databricks job that fails sends an email to its owner, if someone set notifications, and the dashboards that read its tables quietly show yesterday's numbers. With multi-task jobs, the run page says "Failed" while the cause sits in one task out of twelve. The agent lists runs that ended badly since yesterday morning, finds the task that failed first (the others are usually marked as upstream failures), reads its error, and checks whether a later run already succeeded. For jobs still broken it asks whether to repair the run, which reruns only the failed tasks. ## Two ways to get the data Use the API or MCP route when you have an MCP server, a CLI or an API token for the app. Otherwise, if you can drive a browser the user is signed in to (Claude in Chrome, a Playwright or Chrome DevTools MCP server), use the browser route. Say which one you used. ## Route A: API or MCP server ### What you need - A Databricks personal access token or service principal token in `DATABRICKS_TOKEN` with `DATABRICKS_HOST`, with Can View on the jobs (Can Manage Run if you want repairs). - Or the Databricks CLI configured with a profile for that workspace. If something is missing, say what and stop; do not invent numbers. ### Steps 1. **List completed runs since yesterday** (Databricks): `GET /api/2.2/jobs/runs/list?completed_only=true&start_time_from=&expand_tasks=true&limit=25`, following `next_page_token`. Or `databricks jobs list-runs --completed-only --start-time-from --expand-tasks -o json`. Keep runs whose `state.result_state` is `FAILED` or `TIMEDOUT`. 2. **Find the task that failed first**: Among the run's `tasks`, take the ones with `state.result_state` `FAILED` or `TIMEDOUT`; tasks marked `UPSTREAM_FAILED` only failed because of them. Note `task_key`, the cluster and how long it ran. 3. **Read the error** (Databricks): `GET /api/2.2/jobs/runs/get-output?run_id=` returns `error` and `error_trace`; keep the error and the last meaningful line of the trace (the exception class and message). 4. **Check for a later success** (Databricks): `GET /api/2.2/jobs/runs/list?job_id=&limit=1` gives the newest run of the job. If it succeeded, mark the failure as recovered. 5. **Ask before repairing**: Post the still-failing jobs and ask what to do. Only after a yes, call `POST /api/2.2/jobs/runs/repair` with the `run_id` and `rerun_all_failed_tasks: true`. A schema or code error needs a fix first; say so. ## Route B: the browser ### What you need - Signed in to the Databricks workspace with Can View on the jobs (Can Manage Run if you want to repair); a workspace admin sees every job. If something is missing, say what and stop; do not invent numbers. ### Steps Start at https://accounts.cloud.databricks.com/. The user is signed in already; never type passwords or codes, and ask if a sign-in page appears. 1. **Open the workspace's job runs**: The account console lists your workspaces (on Azure it is `accounts.azuredatabricks.net`, or open the workspace URL directly). In the workspace, click **Jobs & Pipelines** in the left sidebar and open the **Job runs** tab, which shows runs across every job you can view. 2. **Filter to failures since yesterday**: Set the **Status** filter to `Failed` and `Timed out` (they are separate filters; do both) and the **Start time** filter to the last 24 hours. Each row shows the job, start time, duration and the run status; note the run count per job. 3. **Find the first failing task**: Click a run to open it. In the task graph, tasks marked **Upstream failed** only failed because an earlier one did; click the task shown as **Failed** or **Timed out**, read the error at the top of its output and the last meaningful line of the stack trace. Copy the run page URL for the report. 4. **Check for a later success**: Open the job (click its name) and look at the **Runs** tab: if a newer run shows **Succeeded**, the failure recovered and only needs a line. 5. **Ask before repairing**: List the jobs still failing and ask what to do. Only after a yes, open the failed run and click **Repair run**, which reruns the failed and skipped tasks; a schema or code error needs a fix first, so say so instead of repairing. Note: The Job runs tab shows the runs the signed-in user can view; a personal account misses jobs owned by others unless it is an admin. The tab loads runs in pages while scrolling and keeps 60 days of history. Older workspaces still label the sidebar entry Workflows; it is the same page. ## The report - **Still failing**: job, failing task, error line, when it failed, link to the run (`run_page_url`). - **Recovered**: jobs that failed and later succeeded. - **Timeouts** listed separately, since they need a different fix. - The question: repair, look first, or leave. ## Post it to Briefkin When `BRIEFKIN_TOKEN` is set, or `~/.config/briefkin/env` exists (the installer from https://briefkin.dev/docs writes it), also send the result to the user's Briefkin board as one card. Keep the `item_key` the same on every run so the card updates in place, use a new `event_id` per run, and put today's real numbers in place of the example ones. If the request fails, say so once and carry on. ```bash [ -n "${BRIEFKIN_TOKEN:-}" ] || . ~/.config/briefkin/env curl -sS -X POST "${BRIEFKIN_URL:-https://briefkin.dev}/v1/events" \ -H "authorization: Bearer $BRIEFKIN_TOKEN" \ -H "content-type: application/json" \ -d @- <<'JSON' { "v": 1, "event_id": "databricks-failed-jobs/", "item_key": "databricks-failed-jobs", "project": "Acme Lakehouse", "source": "Databricks failed jobs skill", "kind": "action", "title": "2 Databricks jobs still failing", "summary": "**3** job runs failed since yesterday. **nightly_events_etl** and **ml_feature_refresh** are still broken; **export_to_s3** succeeded on its next run.", "action": { "request": "nightly_events_etl failed on a missing column, so a repair will fail again until the schema is fixed. Repair ml_feature_refresh now?", "options": [ "Repair ml_feature_refresh", "Show the full errors", "I will fix the schema first", "Leave both" ] }, "widget": { "type": "Table", "props": { "columns": [ "Job", "Task", "Result", "Error" ], "rows": [ [ "nightly_events_etl", "silver_events", "FAILED", "AnalysisException: [UNRESOLVED_COLUMN] user_agent cannot be resolved" ], [ "ml_feature_refresh", "build_features", "TIMEDOUT", "Run exceeded the 2 h timeout" ], [ "export_to_s3", "write_parquet", "Recovered", "S3 throttling, succeeded at 05:10" ] ] } } } JSON ``` Widgets come from the Briefkin catalog (https://briefkin.dev/v1/catalog). Without a token, just give the user the report. ## Run it on a schedule Suggested: Every day at 7:30 (`30 7 * * *`). From cron, with the agent signed in on that machine: ```bash 30 7 * * * claude -p "Run the databricks-failed-jobs skill" # Codex: codex exec, Gemini CLI: gemini -p, Copilot CLI: copilot -p, OpenCode: opencode run ``` Or use the agent's own scheduler (Claude Code routines, Codex automations, Claude or ChatGPT scheduled tasks): https://briefkin.dev/skills/databricks-failed-jobs#schedule Source: https://briefkin.dev/skills/databricks-failed-jobs