AI crawler checker: can ChatGPT and Claude read your site?
One check answers three questions for ChatGPT, Claude and Perplexity: does your robots.txt let them in, do they get past your firewall, and is there text on the page without JavaScript?
Check AI access
Enter a URL. We check robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Googlebot and Bingbot, then fetch the page as GPTBot.
The three ways sites lose AI traffic
A page AI engines can't see fails at one of three layers. The checker tests all three.
1. A robots.txt line nobody remembers
Plenty of sites still carry a robots.txt copied from a template years ago, or a blanket rule added during a redesign. One forgotten Disallow keeps GPTBot or ClaudeBot out of everything, silently, with no error anywhere. The bots simply never come back, and your content never shows up in their answers.
2. A firewall toggle you didn't know about
Cloudflare and similar services now offer one-click settings that challenge every known AI crawler. Useful if that's what you want, costly if you didn't know it was on. Your site looks fine in every browser while every AI engine gets a challenge page instead of your content. This is the failure that's invisible until someone checks.
3. Content that only exists after JavaScript
GPTBot, ClaudeBot and PerplexityBot don't execute JavaScript. If your site is a React, Vue or Lovable app, they receive the empty HTML shell: a page with your scripts and nothing to read. The bot was allowed in, reached the page, and found nothing. Pre-rendering serves those same bots the full HTML without changing your site.
Who's crawling, and what each bot decides
Each engine has its own bot, its own robots.txt token, and its own role in whether you get cited.
Collects training data for OpenAI's models. Blocking it keeps your content out of future ChatGPT knowledge.
Simulate GPTBot
Fetches pages live when ChatGPT users ask about them. This is the bot behind ChatGPT citations.
Simulate ChatGPT-User
Anthropic's crawler for Claude. Feeds both answers and model improvement.
Simulate ClaudeBot
Builds Perplexity's answer index. Perplexity cites sources on nearly every answer.
Simulate PerplexityBot
Not a crawler, but a robots.txt switch that controls whether Google may use your content for Gemini.
Meta's crawler for its AI assistants across Facebook, Instagram and WhatsApp.
Simulate Meta AI
Common Crawl's bot. Its archive feeds many AI training sets, so its access ripples widely.
Simulate CCBotAllow or block AI crawlers in robots.txt
Both directions are legitimate. What matters is that it's your decision, not a leftover default.
Allow AI crawlers
For sites that want citations, AI referrals, and presence in AI answers.
# Allow the major AI crawlers User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: meta-externalagent Allow: / User-agent: CCBot Allow: /
Block AI crawlers
Keeps your content out of AI answers and training sets while Googlebot and Bingbot stay allowed for search.
# Block AI crawlers (search engines stay allowed) User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: CCBot Disallow: /
More Free SEO Tools
Continue auditing your site with these companion tools.
30 checks for crawlability, SEO, and AI readiness
See what 100+ search and AI bots get from your page
Fetch your page as Googlebot Smartphone or Desktop
Check which crawlers are allowed or blocked
Preview social media link cards
Audit meta tags and generate a fix prompt
Related Guides
Dive deeper with these step-by-step guides and tutorials.
AI crawler questions, answered
What site owners ask most about ChatGPT, Claude, and Perplexity access.
Make your site readable to every AI engine
If the checker shows AI engines hitting an empty shell, Encited pre-renders your pages into full HTML for every crawler, with no code changes, working in minutes.
