--- title: "AI Crawl Checker - Can ChatGPT, Claude & Perplexity Read Your Site? | Encited" description: "Check whether GPTBot, ClaudeBot, PerplexityBot and 7 more AI engines can crawl, reach, and read your website. Free checker covering robots.txt, firewalls, and JavaScript rendering." lang: en json-ld: --- [![Encited logo](/brand/logo.svg)](/) - Features - Resources [Pricing](/pricing)Login[Contact Sales](/contact-sales)Get started free Get started free AI crawl checker # See which AI engines can read your site One check across ChatGPT, Claude, Perplexity, Gemini and more: are they permitted by your robots.txt, can they get past your firewall, and is there actually content for them to read? Check ## Check AI access Enter a URL to test permission, reachability, and readability for 10 AI engines. Check AI access Why sites disappear ## The three ways sites lose AI traffic Every page that's invisible to AI engines fails at one of three layers. The checker tests all three. ### 1\. A robots.txt line nobody remembers Plenty of sites still carry a robots.txt copied from a template years ago, or a blanket rule added during a redesign. One forgotten Disallow keeps GPTBot or ClaudeBot out of everything — silently, with no error anywhere. The bots simply never come back, and your content never shows up in their answers. ### 2\. A firewall toggle you didn't know about Cloudflare and similar services now offer one-click settings that challenge every known AI crawler. Useful if that's what you want — costly if you didn't know it was on. Your site looks fine in every browser while every AI engine gets a challenge page instead of your content. This is the failure that's invisible until someone checks. ### 3\. Content that only exists after JavaScript GPTBot, ClaudeBot and PerplexityBot don't execute JavaScript. If your site is a React, Vue or Lovable app, they receive the empty HTML shell — a page with your scripts and nothing to read. The bot was allowed in, reached the page, and found nothing. Pre-rendering serves those same bots the full HTML without changing your site. The engines ## Who's crawling, and what each bot decides Each engine has its own bot, its own robots.txt token, and its own role in whether you get cited. ![GPTBot](/crawler-logos/chatgpt.svg) GPTBot Collects training data for OpenAI's models. Blocking it keeps your content out of future ChatGPT knowledge. [Simulate GPTBot](/free-tools/crawler-simulator/chatgpt-crawler-simulator) ![ChatGPT-User](/crawler-logos/chatgpt.svg) ChatGPT-User Fetches pages live when ChatGPT users ask about them — this is the bot behind ChatGPT citations. [Simulate ChatGPT-User](/free-tools/crawler-simulator/chatgpt-crawler-simulator) ![ClaudeBot](/crawler-logos/claude.svg) ClaudeBot Anthropic's crawler for Claude. Feeds both answers and model improvement. [Simulate ClaudeBot](/free-tools/crawler-simulator/claudebot-simulator) ![PerplexityBot](/crawler-logos/perplexity.png) PerplexityBot Builds Perplexity's answer index. Perplexity cites sources on nearly every answer. [Simulate PerplexityBot](/free-tools/crawler-simulator/perplexitybot-simulator) ![Google-Extended](/crawler-logos/gemini.png) Google-Extended Not a crawler — a robots.txt switch that controls whether Google may use your content for Gemini. ![Googlebot](/crawler-logos/google-search.svg) Googlebot Google Search and the source material for AI Overviews. [Simulate Googlebot](/free-tools/crawler-simulator/googlebot-simulator) ![Bingbot](/crawler-logos/bing.svg) Bingbot Powers Bing and grounds Microsoft Copilot's answers. [Simulate Bingbot](/free-tools/crawler-simulator/bingbot-simulator) ![Grok (xAI)](/crawler-logos/xai.svg) Grok (xAI) xAI's crawler for Grok answers on X. [Simulate Grok (xAI)](/free-tools/crawler-simulator/grok-crawler-simulator) ![Meta AI](/crawler-logos/meta.svg) Meta AI Meta's crawler for its AI assistants across Facebook, Instagram and WhatsApp. [Simulate Meta AI](/free-tools/crawler-simulator/facebookbot-simulator) ![CCBot](/crawler-logos/common-crawl.png) CCBot Common Crawl's bot. Its archive feeds many AI training sets, so its access ripples widely. [Simulate CCBot](/free-tools/crawler-simulator/ccbot-simulator) Take control ## Allow or block AI crawlers in robots.txt Both directions are legitimate. What matters is that it's your decision, not a leftover default. ### Allow AI crawlers For sites that want citations, AI referrals, and presence in AI answers. \# Allow the major AI crawlers User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: meta-externalagent Allow: / User-agent: CCBot Allow: / Copy snippet ### Block AI crawlers Keeps your content out of AI answers and training sets while Googlebot and Bingbot stay allowed for search. \# Block AI crawlers (search engines stay allowed) User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: CCBot Disallow: / Copy snippet ## More Free SEO Tools Continue auditing your site with these companion tools. [ SEO & AEO Audit Full technical SEO and AI visibility audit ](/free-tools/seo-audit)[ Crawler Simulator Test how 20+ search and AI bots see your site ](/free-tools/crawler-simulator)[ Robots.txt Analyzer Check which crawlers are allowed or blocked ](/free-tools/robots-txt-analyzer)[ OG Image Previewer Preview social media link cards ](/free-tools/og-image-previewer)[ Meta Tag Audit Audit meta tags and generate a fix prompt ](/free-tools/meta-tag-audit) ## Related Guides Dive deeper with these step-by-step guides and tutorials. [Get Indexed by ChatGPT, Perplexity & AI Search ](/blog/how-to-get-pages-indexed-by-openai-chatgpt)[Is Lovable SEO Friendly? A Practical Guide ](/blog/is-lovable-seo-friendly)[What Is Pre-rendering for SPAs? ](/docs/guides/what-is-prerendering-for-spas)[Debugging robots.txt & Noindex Issues ](/docs/guides/debugging-robots-txt) ## AI crawler questions, answered What site owners ask most about ChatGPT, Claude, and Perplexity access. ### Why doesn't ChatGPT cite my website? ### Do AI crawlers run JavaScript? ### Does robots.txt actually stop AI bots? ### What's the difference between GPTBot and ChatGPT-User? ### What is Google-Extended? ### Should I block CCBot? ### How do I know if Cloudflare is blocking AI bots? ### Does noindex keep my pages out of AI answers? Get started ## Make your site readable to every AI engine If the checker shows AI engines hitting an empty shell, Encited pre-renders your pages into full HTML for every crawler — no code changes, working in minutes. [Start your 14-day free trial](/?authModal=true)[Book a demo](/book-a-demo) ![Encited logo](/brand/logo.svg) Follow us: [](https://www.facebook.com/lovablehtml/)[](https://linkedin.com/company/lovablehtml) [LaunchFast.shop](https://launchfast.shop) product ❤︎ ### Product - [Track Brand Visibility](/ai-visibility-tracker) - [Create Content](/create-content) - [Get Indexed Faster](/get-indexed-faster) - [Site Audit](/site-audit) - [Pre-rendering](/prerender) - [Encited MCP](/seo-mcp) ### Free SEO Tools - [Free SEO & AEO Audit](/free-tools/seo-audit) - [Crawler Simulator](/free-tools/crawler-simulator) - [Robots.txt Analyzer](/free-tools/robots-txt-analyzer) - [OG Image Previewer](/free-tools/og-image-previewer) - [Meta Tag Audit](/free-tools/meta-tag-audit) ### Resources - [Blog](/blog) - [Guides and API docs](/docs) - [Prerender.io Alternative](/prerender-io-alternative) ### Company - [Affiliate Program 30% ](/become-partner) - [About](/about) - [Media Kit](/media-kit) - [Contact](mailto:team@encited.com) ### Legal - [Privacy Policy](/privacy) - [Terms of Service](/terms) ![Avatar](https://replymaven.com/api/uploads/Su6Q0nK63iZW0ZFJBRNWpmRuppVDLuab/98df5262-f992-45fa-a750-00a6c919baf9.png) How can we help? Get instant answers to your questions or leave a message for an engineer will reach out Ask AI about Encited [See our docs ](https://lovablehtml.com/docs) Contact support Leave a message We'll get back to you soon Your name\* Your email\* Your message\* Send message ![Avatar](https://replymaven.com/api/uploads/Su6Q0nK63iZW0ZFJBRNWpmRuppVDLuab/98df5262-f992-45fa-a750-00a6c919baf9.png) Ask AI about Encited Product questions, troubleshooting and Thinking Drop an image to attach Powered by [ReplyMaven](https://replymaven.com)