Robots.txt Monitor

What it measures

• Current version — the most recent robots.txt content fetched from your site, with a red BLOCKING badge if Disallow: / is active or a green Normal badge if crawlers are allowed in.
• Last checked — the timestamp of the most recent fetch.
• Change history — every version of the file that differed from the previous one, with the date it was detected, a status badge, the short SHA-256 hash for identification, and an expandable unified diff so you can see exactly what lines were added or removed.

How we compute it

1. A Celery Beat task runs every 3 hours and fetches https://<your-domain>/robots.txt with a short timeout.
2. We SHA-256 the raw content and compare against the latest stored snapshot for your site.
3. If the hash is different (or there is no previous snapshot), we store a new RobotsTxtSnapshot row with the content, timestamp and hash.
4. We parse the new content line-by-line looking for a User-agent: * block that contains a Disallow: / directive. If found, is_blocking is set to True.
5. We compute a unified diff between the previous snapshot and the new one (identical to diff -u) so you can read exactly what changed.
6. An alert email is sent to every member of the site's team. The subject line reads [CRITICAL] Robots.txt blocking detected if the file is blocking crawlers, or [WARNING] Robots.txt changed for any other change. Email is fail_silently=True so a transient SMTP failure never crashes the task.

Scenarios you'll see

What to do when you get an alert

1. If CRITICAL (Disallow: /) — open your robots.txt immediately at https://yourdomain.com/robots.txt. If the block is live, revert your last deploy or patch the file directly. Then request a recrawl in Google Search Console.
2. If WARNING (other change) — open the Robots.txt Monitor report, expand the diff for the new snapshot, and verify each changed line against your recent deploy log or CMS release notes.
3. For intentional changes (adding a new Disallow for an admin path), no action is needed — but log it in Change Events so the annotation appears on your traffic charts.
4. If the change was unintentional, check your deployment pipeline for where robots.txt is generated — CMS plugins, environment-variable templating, and CI/CD jobs are common culprits.
5. After resolving a blocking incident, monitor Index Coverage over the following 2–3 days to confirm Google is actively recrawling.

Caveats & limits

• Fetches run every 3 hours. A deployment that briefly blocks crawlers and is reverted within 3 hours may not be captured if the window of exposure falls between two fetch cycles.
• The blocking detector checks for the literal string Disallow: / under a User-agent: * block. Non-wildcard user-agent blocks (e.g. User-agent: Googlebot) do not trigger the CRITICAL badge — review the diff manually if you're concerned about specific bots.
• Only content changes trigger a new snapshot. HTTP 5xx / connection timeouts are not stored — a missing robots.txt or server error is not the same as an explicit block directive.
• The monitor stores up to the last 50 snapshots per site for the history table.