Indexing & Crawl

Crawl Efficiency

Crawl Efficiency joins Googlebot’s last crawl date for each URL with its click and impression data, then classifies every page into one of four tiers. The goal is to answer two questions at once: “Where is Googlebot wasting its crawl budget on worthless pages?” and “Which important pages has Google neglected to refresh?”

The four crawl-efficiency tiers

Zero-search crawl

Crawled in the last 14 days, zero clicks and zero impressions in the selected window. Googlebot is spending budget on pages that are invisible in search. That is not the same as worthless — checkout, support and legal pages rarely earn impressions — so check each page has no other job before acting. Then: noindex, canonical consolidation, robots.txt exclusion, or removal via 410.

Under-crawled

High clicks but a stale crawl date. These are money pages Google hasn’t refreshed recently. Content changes or schema updates may not yet be reflected in the index. Re-submit via URL Inspection or add internal links to signal freshness.

Healthy

Recently crawled and earning traffic. The expected state for your important pages. Monitor to ensure this tier grows as a share of your indexed pages.

Stale tail

Low traffic, stale crawl. Rarely crawled pages earning minimal traffic. Often old blog posts, paginated archive pages or thin category pages. Evaluate each: refresh the best, consolidate or prune the rest.

How we compute it

  1. We read the last crawl date (lastCrawlTime) from the URL Inspection records we hold. Google allows only a small number of URL inspections a day, so Metricstab inspects the pages in your running SEO tests rather than sweeping the whole site. Pages outside a running test have no inspection record here.
  2. We join each URL to its clicks and impressions from Search Console.
  3. We define recently crawled as a last-crawl date within the last 30 days. Stale means more than 30 days ago or no crawl record.
  4. Value cut-offs are percentiles of your own pages, not fixed click counts: high value is the top 10% by clicks, valuable the top half, both floored at 1 click so a page with no clicks is never called valuable.
  5. Tier assignment:
    • Crawled ≤14 days ago + zero clicks and zero impressions = Zero-search crawl
    • Crawled >30 days ago + high value = Under-crawled
    • Crawled ≤14 days ago + valuable = Healthy
    • Crawled >60 days ago + zero clicks = Stale tail
    • Everything else = OK
  6. Within each tier, rows are sorted by clicks descending.

Table columns

  • URL — links to the live page.
  • Tier — colour-coded badge.
  • Last crawl — the date Googlebot last fetched this URL. “Never” means no crawl record was returned by the URL Inspection API.
  • Clicks / Impressions — Search Console totals for the selected window.
  • Index verdict — the URL Inspection indexing verdict (Indexed, Not indexed, etc.).

What to do with it

  1. Under-crawled pages are the biggest win. For each, verify the page has fresh content, is in the sitemap, and has internal links from high-authority pages. Submit via URL Inspection and request a re-crawl.
  2. Zero-search-crawl pages are your robots/canonical review list. For pages with no impressions and no other job (no conversions, no internal-link role, not a legal or support page): add noindex, set a canonical to a better version, or remove the page with a 410.
  3. A high Zero-search crawl to Healthy ratio suggests your site is larger than it needs to be from Google’s perspective — a pruning project will likely improve rankings for the pages that remain.
  4. Re-check the Tier distribution monthly. A growing Healthy share is a direct proxy for overall crawl-budget health.

Related reports