Free tool · No sign-up
AI Crawler Checker: can ChatGPT, Claude and Perplexity reach your site?
Enter a page address to see which of 15 AI and search crawlers your robots.txt allows or blocks, whether the page has noindex or other robots tags, and whether your server turns AI bots away. Results take about 10 seconds.
Checks robots.txt, meta robots, X-Robots-Tag and server responses. Nothing is stored except the result for 6 hours.
Checking …
- Loading the page and following redirects
- Reading robots.txt
- Applying the rules for each AI crawler
- Testing server responses to AI user agents
Your results
Page checks
Crawler by crawler
Based on the robots.txt rules that apply to this exact page.
| Crawler | What it does | robots.txt | Server test |
|---|
Ready-to-copy robots.txt rules
Pick the setting you want and add it to your robots.txt. If your file already has a group for one of these crawlers, edit that group instead, because a crawler follows only the most specific group that names it.
Checked . robots.txt results follow RFC 9309, the standard most crawlers use. Server tests are sent from this website's server, so treat them as indicative. Crawler details are from each company's published documentation.
Why it matters
Three kinds of AI crawler, three decisions
AI companies use separate crawlers for search, for fetching a page a user asks about, and for training models. Blocking one doesn't block the others, so you can choose each one.
AI search crawlers
OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether your pages can appear and be linked in ChatGPT, Claude and Perplexity answers. Blocking them can remove you from those answers.
User-triggered fetchers
ChatGPT-User, Claude-User and Perplexity-User open a page when someone asks about it. Some of them may not follow robots.txt because a person made the request.
Training crawlers
GPTBot, ClaudeBot, CCBot and the Google-Extended and Applebot-Extended tokens control whether your content may be used to train AI models. They don't affect search results.
Being reachable is only the first step. To be recommended, your pages also need clear answers, structured data and mentions on trusted sites. See how to get your business recommended by ChatGPT and my AI SEO services.
Next step
Reachable is step one. Are you recommended?
I'll ask ChatGPT and Google five questions your customers are likely to ask, then email you what they say about your business, who they recommend instead, and three fixes to start with.
- Real answers from ChatGPT and Google, checked by me
- Who they recommend instead of you
- Three fixes you can start on straight away
Get your free AI visibility check
Takes under a minute. Results within 3 working days.
FAQ
AI crawlers, explained
Short answers about AI bots, robots.txt and what this checker can and can't tell you.
01What does the AI crawler checker test?
It reads your robots.txt file and applies the same matching rules crawlers use to see which of 15 AI and search crawlers may access the page you entered. It also checks the page's meta robots tag and X-Robots-Tag header, and sends test requests with five AI crawler user agents to spot server or firewall blocks.
02Should I block AI crawlers?
It depends on what you want. Blocking AI search crawlers such as OAI-SearchBot or PerplexityBot can stop your site appearing in those AI answers. Blocking training crawlers such as GPTBot or CCBot doesn't affect search, but it signals that you don't want your content used to train models. Many businesses allow search and decide separately on training.
03Does blocking Google-Extended remove my site from Google or AI Overviews?
No. Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls use of your content for Gemini training and grounding. Access for Google Search is controlled by Googlebot.
04Do all AI bots follow robots.txt?
Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person asked for the page, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. The checker shows this for each bot.
05Why does the server test say indicative?
The test request comes from this website's server, not from the AI company's own network. Firewalls and CDNs such as Cloudflare can treat real crawlers differently, so use the result as a warning sign to check in your firewall or CDN settings.
06Is the tool free?
Yes. There's no sign-up and no limit for normal use. If you'd like a person to look at the results, you can request a free AI visibility check below.
