Free Robots.txt Tester

Google retired its standalone robots.txt tester in December 2023 and folded a reduced version into Search Console, where it only covers properties you have verified. That leaves a gap when you want to test a rule change before deploying it, debug a competitor's setup, or check a client site you have no Search Console access to. This tool fills that gap with the same matching logic crawlers use, defined in RFC 9309.

The precedence rules surprise most people. First, a crawler picks exactly one group: the one whose user-agent token is the longest match for its own name, falling back to User-agent: * only when nothing else matches. Rules in other groups are ignored entirely. Second, within that group, the rule with the longest path pattern wins, regardless of the order rules appear in the file. When an Allow and a Disallow of equal length both match, Allow wins. That is why Allow: /blog/ beats Disallow: / for blog URLs, and why rule order almost never matters.

The classic footguns are worth testing for explicitly. A Disallow: / left over from a staging environment silently removes the whole site from crawling. Paths are case sensitive, so Disallow: /Admin/ does nothing for /admin/. Wildcards are greedy, so Disallow: /*? blocks every URL with a query string, including tracked campaign links. And robots.txt controls crawling only; a blocked page can still appear in search results if other sites link to it.

One newer reason to audit this file: AI assistants now fetch and cite pages through their own crawlers, and overly broad rules written years ago often block them. Run the AI crawler access checker to see how GPTBot, ClaudeBot, and PerplexityBot read your file. While you are auditing crawlability, verify your structured data with the schema validator and your titles with the title tag checker.