Crawl Stats
Crawl Stats shows how often Googlebot actually crawls your site, how much data it downloads, and how quickly your server responds — captured periodically from Search Console. Crawl behaviour is upstream of indexing: if Googlebot can't crawl efficiently, your URLs can't be indexed efficiently, and a degrading crawl signal is often the earliest leading indicator of a deeper technical problem.
What it measures
- Total crawl requests — how many times Googlebot fetched something from your site in the period.
- Total download size (KiB) — total bytes served to Googlebot.
- Average response time — how long your server took to respond to Googlebot on average.
- Capture date — the snapshot timestamp.
How we compute it
- We capture the Crawl Stats summary Search Console exposes, on a recurring schedule.
- Each capture is stored as a snapshot so you can read trends over time, not just the latest reading.
Scenarios you'll see
Requests trending up steadily as your site grows. Healthy. Make sure response time stays flat as crawl volume increases.
Average response time creeping above ~500ms. Google may scale crawl back to protect your servers — that limits how quickly new content gets indexed.
Requests fell sharply with no expected cause. Possible: server errors, robots.txt change, a soft-404 epidemic, or Google determining the site has degraded in quality. Cross-check Index Coverage.
A sudden surge — usually a sitemap re-submission, a migration, or a major content release. Confirm against your release log.
Total KiB rising disproportionately to request count. Googlebot is downloading larger pages — uncompressed assets, oversized images, or new media-heavy templates. Audit page weight.
Crawl requests holding steady but a rising share of 5xx in the response-code breakdown (visible in Search Console directly). Triage with the ops team urgently — Google will reduce crawl if it persists.
What to do with it
- Read it as a trend, not a single number. Single-day swings are normal; consistent multi-week moves are the signal.
- If response time creeps up, treat it as a server-side performance project — it's both an SEO and a real-user issue.
- After a migration, expect a temporary crawl spike followed by a return to baseline; if crawl never recovers, something in the new architecture is blocking it.
Caveats & limits
- Search Console aggregates Googlebot crawl data with some delay; recent days are still finalising.
- Crawl request totals include all asset types (HTML, JS, CSS, images), not just HTML pages. A surge can be driven by a single chatty asset.
Related reports
- Index Coverage — what crawl produced in terms of indexed URLs.
- Sitemaps Status — discovery surfaces feeding the crawl.
- PSI Overview — server response time tied to lab-data performance.