--- name: google-search-console-indexing-issues description: "Every weekday the agent inspects your sitemap URLs through Search Console and asks you to act when a page that was indexed no longer is, with Google's reason. Use when the user asks for this, or when it runs on a schedule." --- # Search Console indexing issues Every weekday the agent inspects your sitemap URLs through Search Console and asks you to act when a page that was indexed no longer is, with Google's reason. Pages fall out of Google's index for reasons that start in code: a `noindex` left on after staging, a robots.txt rule that matched more than intended, a template that now returns a soft 404. The Page indexing report shows it eventually, as a count that crept up, with no alert and no date. The agent inspects your important URLs one by one, remembers each verdict, and asks only when a page goes from indexed to not. Developers get the exact URL and Google's reason; the SEO lead hears about it before the traffic does. ## Two ways to get the data Use the API or MCP route when you have an MCP server, a CLI or an API token for the app. Otherwise, if you can drive a browser the user is signed in to (Claude in Chrome, a Playwright or Chrome DevTools MCP server), use the browser route. Say which one you used. ## Route A: API or MCP server ### What you need - Full or restricted user access to the Search Console property; URL inspection only works on properties you are a user of. - Search Console API access through an MCP server, a connector, or `webmasters.readonly` credentials, and a place to keep the last run's verdicts (a JSON file next to the skill is enough). If something is missing, say what and stop; do not invent numbers. ### Steps 1. **Collect the URLs that matter** (Google Search Console): List the property's sitemaps with `sitemaps.list`, fetch each file and gather its `` URLs. Add the top 200 pages by clicks from `searchanalytics.query` (dimension `page`, last 28 days) in case some are missing from the sitemap. 2. **Inspect each URL** (Google Search Console): Call `urlInspection.index.inspect` with `inspectionUrl` and `siteUrl`. From `indexStatusResult`, keep `verdict`, `coverageState`, `indexingState`, `robotsTxtState`, `pageFetchState`, `googleCanonical`, `userCanonical` and `lastCrawlTime`. 3. **Stay inside the quota**: The API allows 2,000 inspections per property per day and 600 per minute. On larger sites, inspect the top pages every day and rotate through the rest of the sitemap over the week. 4. **Compare with the last verdict**: Keep each URL's result between runs. A page is newly not indexed when its verdict was `PASS` last time and is now `NEUTRAL` or `FAIL`, or when `googleCanonical` no longer matches `userCanonical`. 5. **Ask, with the cause**: Group dropped pages by cause (`BLOCKED_BY_META_TAG`, robots.txt `DISALLOWED`, `SOFT_404`, `SERVER_ERROR`, a canonical mismatch) and post one action card. Causes in code go to developers; "Crawled - currently not indexed" is a content question. ## Route B: the browser ### What you need - Signed in to Google with at least Restricted access to the Search Console property; the URL Inspection tool works only on properties the account is a user of. If something is missing, say what and stop; do not invent numbers. ### Steps Start at https://search.google.com/search-console/index. The user is signed in already; never type passwords or codes, and ask if a sign-in page appears. 1. **Open the Page indexing report**: Pick the property; the URL lands on Indexing > Pages. Read the Not indexed and Indexed counts and the chart, then the `Why pages aren't indexed` table, which lists each reason with its page count and trend. 2. **Export the pages for each code reason**: Click `Excluded by 'noindex' tag`, `Blocked by robots.txt`, `Server error (5xx)`, `Soft 404` and `Duplicate, Google chose different canonical than user` in turn and use Export on each. Each list shows up to 1,000 URLs with the last crawl date. 3. **Compare with the last run**: Keep the exports between runs. A URL that appears in a reason list this time and was not there last time is newly dropped. Pages from your sitemap or your top pages by clicks matter most; `Crawled - currently not indexed` is a content question, not a code one. 4. **Inspect the important ones live**: Paste each newly dropped URL that matters into the `Inspect any URL` box at the top of Search Console. The result shows the verdict, the crawl date and both canonicals, which is fresher than the report. 5. **Ask with the cause**: Group dropped pages by reason, say which need a code fix and which are content issues, and ask who should take them. After a fix, the Validate fix button on that reason's page starts Google's recheck. Note: The Page indexing report refreshes every few days, so it shows a drop later than daily API inspection would, and each reason list caps at 1,000 URLs. The Inspect box handles one URL at a time, so the browser route suits sites where a few dozen pages matter, not a 10,000-URL sitemap. ## The report - **URLs inspected** and how many changed since the last run. - **Newly not indexed**: each URL with Google's coverage state, the field behind it and the last crawl date. - **Canonical changes**: pages where Google chose a canonical other than yours. - The ask: who should fix what, split into code causes and content causes. ## Post it to Briefkin When `BRIEFKIN_TOKEN` is set, or `~/.config/briefkin/env` exists (the installer from https://briefkin.dev/docs writes it), also send the result to the user's Briefkin board as one card. Keep the `item_key` the same on every run so the card updates in place, use a new `event_id` per run, and put today's real numbers in place of the example ones. If the request fails, say so once and carry on. ```bash [ -n "${BRIEFKIN_TOKEN:-}" ] || . ~/.config/briefkin/env curl -sS -X POST "${BRIEFKIN_URL:-https://briefkin.dev}/v1/events" \ -H "authorization: Bearer $BRIEFKIN_TOKEN" \ -H "content-type: application/json" \ -d @- <<'JSON' { "v": 1, "event_id": "google-search-console-indexing-issues/", "item_key": "google-search-console-indexing-issues", "project": "example.com", "source": "Search Console indexing issues skill", "kind": "action", "title": "5 pages dropped out of Google's index", "summary": "Inspected 640 sitemap URLs. **5 pages** that were indexed last week are not now, and 3 of the causes are in code: a noindex tag, a robots.txt rule and server errors.", "action": { "request": "Three of these need a code fix: /pricing/teams carries a noindex tag, /docs/api/webhooks is blocked by robots.txt and /integrations/xero returns 5xx. Should I draft a ticket for the developers?", "options": [ "Draft the ticket", "I'll check today", "Expected, ignore these" ] }, "widget": { "type": "Table", "props": { "columns": [ "URL", "Coverage state", "Last crawl" ], "rows": [ [ "/pricing/teams", "Excluded by ‘noindex’ tag", "Oct 14" ], [ "/docs/api/webhooks", "Blocked by robots.txt", "Oct 13" ], [ "/integrations/xero", "Server error (5xx)", "Oct 15" ], [ "/blog/invoice-vs-bill", "Duplicate, Google chose different canonical than user", "Oct 12" ], [ "/templates/packing-slip", "Soft 404", "Oct 14" ] ] } } } JSON ``` Widgets come from the Briefkin catalog (https://briefkin.dev/v1/catalog). Without a token, just give the user the report. ## Run it on a schedule Suggested: Weekdays at 7:00 (`0 7 * * 1-5`). From cron, with the agent signed in on that machine: ```bash 0 7 * * 1-5 claude -p "Run the google-search-console-indexing-issues skill" # Codex: codex exec, Gemini CLI: gemini -p, Copilot CLI: copilot -p, OpenCode: opencode run ``` Or use the agent's own scheduler (Claude Code routines, Codex automations, Claude or ChatGPT scheduled tasks): https://briefkin.dev/skills/google-search-console-indexing-issues#schedule Source: https://briefkin.dev/skills/google-search-console-indexing-issues