Skip to content
Get my AI grade

AI crawler access

Free AI crawler access checker

Enter your domain and see whether GPTBot, ClaudeBot, PerplexityBot and 11 other AI crawlers and tokens may reach your site. Rank.ai fetches your robots.txt, applies it the way crawlers do and shows the rule that decides each one.

  • Free
  • No signup
  • 14 AI crawlers and tokens

We fetch your robots.txt and test root access for 14 AI crawlers from OpenAI, Anthropic, Perplexity, Google, and others.

How it works

Your robots.txt, read the way AI crawlers read it

01

Enter your domain

Rank.ai fetches the robots.txt at the root of your site.

02

Each crawler is checked

The file is applied for 14 AI crawlers and tokens from OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, ByteDance, Meta and Amazon, following RFC 9309.

03

See who is blocked, and why

Search and user crawlers are grouped apart from training crawlers, and each shows allowed or blocked for your site root with the rule that matched.

Why it matters

Search crawlers and training crawlers

AI companies run different crawlers for different jobs, and the job decides what blocking one costs you. OpenAI says OAI-SearchBot surfaces websites in ChatGPT’s search features and that sites which block it won’t be shown in ChatGPT search answers, while GPTBot collects content that may be used to train its models, and each setting is independent. Anthropic separates ClaudeBot (training) from Claude-SearchBot (search) and Claude-User (fetching for a person), and Perplexity says PerplexityBot surfaces and links sites in its results and isn’t used to train foundation models.

So blocking a training crawler is a policy choice that doesn’t affect whether an assistant can cite you, while blocking a search or user crawler can keep your pages out of that assistant’s search answers. A robots.txt written to keep out AI training can block the search crawlers too, if its rules are broad.

The fix is usually a few lines: keep the training blocks if they reflect your policy, and add explicit groups for the search crawlers, such as User-agent: OAI-SearchBot followed by Allow: /. If a User-agent: * group disallows everything, fix that first. Test the edit with the robots.txt tester, then see whether assistants name you with the AI visibility checker.

Free tools

More free tools

Schema markup validator

Check a page's structured data for errors and missing required fields that can block Google rich results.

Title tag checker

Preview your title as a Google result and an AI citation, with length and estimated pixel width.

Robots.txt tester

Test any URL path against your robots.txt and see which rule allows or blocks each crawler.

llms.txt generator

Draft an llms.txt file from your sitemap: a curated map of your site for AI tools that read one.

Meta description generator

Get three meta descriptions of up to 160 characters, each with a character count.

AI visibility checker

Ask ChatGPT, Claude and Gemini one question and see whether each answer names your domain or cites your site.

All free tools

AI visibility checker, Google Business Profile audit, Maps rank checker, schema validator and more.

Frequently asked

Should I block GPTBot?
It depends on whether you want your content used for training. OpenAI says GPTBot collects content that may be used to train its models and that its settings are independent, so you can block GPTBot and still allow OAI-SearchBot, the crawler that surfaces sites in ChatGPT's search features, and ChatGPT-User.
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended controls whether your content may be used for training Gemini models and for grounding in some of Google's AI products, and that it doesn't affect a site's inclusion in Google Search or act as a ranking signal. Google's AI features in Search use Googlebot.
How do I unblock an AI crawler?
Edit the robots.txt at the root of your domain. If the crawler is blocked by its own group, relax that group's Disallow lines. If a wildcard group (User-agent: * with Disallow rules) catches it, add a group that names the crawler, for example User-agent: OAI-SearchBot followed by Allow: /. A crawler follows the most specific group that names it. The change applies the next time the crawler fetches your robots.txt.
Do AI companies respect robots.txt?
OpenAI, Anthropic, Google and Perplexity document robots.txt controls for their named crawlers. Fetchers that act for a person work differently: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Crawlers that don't identify themselves can ignore robots.txt entirely, so use server controls if you need hard guarantees.
What is the difference between GPTBot and ChatGPT-User?
OpenAI says GPTBot crawls to collect content that may be used for training its models. ChatGPT-User visits a page when a ChatGPT user's request needs it, isn't used to crawl the web automatically, and isn't used to decide what appears in search; OAI-SearchBot handles search.

See if AI names you when customers ask who’s best.

Enter your website. In about two minutes, Rank.ai asks ChatGPT, Claude and Gemini 12 questions your customers ask and grades how often they name you.

  • Your grade out of 100How often AI names you, cites your site, and how high it ranks you.
  • Who gets namedEvery competitor in the answers, most named first.
  • The pages AI readsThe sources behind each answer.
  • Three fixesWhat to fix first, with a brief for the first page.