See which AI engines can read your site
One check across ChatGPT, Claude, Perplexity, Gemini and more: are they permitted by your robots.txt, can they get past your firewall, and is there actually content for them to read?
Check AI access
Enter a URL to test permission, reachability, and readability for 10 AI engines.
The three ways sites lose AI traffic
Every page that's invisible to AI engines fails at one of three layers. The checker tests all three.
1. A robots.txt line nobody remembers
Plenty of sites still carry a robots.txt copied from a template years ago, or a blanket rule added during a redesign. One forgotten Disallow keeps GPTBot or ClaudeBot out of everything — silently, with no error anywhere. The bots simply never come back, and your content never shows up in their answers.
2. A firewall toggle you didn't know about
Cloudflare and similar services now offer one-click settings that challenge every known AI crawler. Useful if that's what you want — costly if you didn't know it was on. Your site looks fine in every browser while every AI engine gets a challenge page instead of your content. This is the failure that's invisible until someone checks.
3. Content that only exists after JavaScript
GPTBot, ClaudeBot and PerplexityBot don't execute JavaScript. If your site is a React, Vue or Lovable app, they receive the empty HTML shell — a page with your scripts and nothing to read. The bot was allowed in, reached the page, and found nothing. Pre-rendering serves those same bots the full HTML without changing your site.
Who's crawling, and what each bot decides
Each engine has its own bot, its own robots.txt token, and its own role in whether you get cited.
Collects training data for OpenAI's models. Blocking it keeps your content out of future ChatGPT knowledge.
Simulate GPTBot
Fetches pages live when ChatGPT users ask about them — this is the bot behind ChatGPT citations.
Simulate ChatGPT-User
Anthropic's crawler for Claude. Feeds both answers and model improvement.
Simulate ClaudeBot
Builds Perplexity's answer index. Perplexity cites sources on nearly every answer.
Simulate PerplexityBot
Not a crawler — a robots.txt switch that controls whether Google may use your content for Gemini.
Meta's crawler for its AI assistants across Facebook, Instagram and WhatsApp.
Simulate Meta AI
Common Crawl's bot. Its archive feeds many AI training sets, so its access ripples widely.
Simulate CCBotAllow or block AI crawlers in robots.txt
Both directions are legitimate. What matters is that it's your decision, not a leftover default.
Allow AI crawlers
For sites that want citations, AI referrals, and presence in AI answers.
# Allow the major AI crawlers User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: meta-externalagent Allow: / User-agent: CCBot Allow: /
Block AI crawlers
Keeps your content out of AI answers and training sets while Googlebot and Bingbot stay allowed for search.
# Block AI crawlers (search engines stay allowed) User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: CCBot Disallow: /
More Free SEO Tools
Continue auditing your site with these companion tools.
Related Guides
Dive deeper with these step-by-step guides and tutorials.
AI crawler questions, answered
What site owners ask most about ChatGPT, Claude, and Perplexity access.
Make your site readable to every AI engine
If the checker shows AI engines hitting an empty shell, Encited pre-renders your pages into full HTML for every crawler — no code changes, working in minutes.
