Robots.txt Checker
Test any robots.txt file. See which crawlers are allowed or blocked, and every rule for Googlebot, GPTBot, ClaudeBot and the other search and AI bots.
Check any robots.txt
Enter your domain to see which crawlers are allowed or blocked.
Robots.txt Examples
Copy these templates to control how crawlers access your site.
Allow All Crawlers
Best for most websites wanting maximum visibility
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Block AI Training Only
Allow search engines but block AI training crawlers
User-agent: * Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / Sitemap: https://example.com/sitemap.xml
Allow Search + AI Citations
Be indexed by search engines and cited by AI assistants
User-agent: * Allow: / # Allow AI assistants to cite your content User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / Sitemap: https://example.com/sitemap.xml
Block Specific Paths
Allow most content but protect admin/private areas
User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /*.json$ Sitemap: https://example.com/sitemap.xml
More Free SEO Tools
Continue auditing your site with these companion tools.
30 checks for crawlability, SEO, and AI readiness
See what 100+ search and AI bots get from your page
Fetch your page as Googlebot Smartphone or Desktop
Check OG images and preview social link cards
Audit meta tags and generate a fix prompt
Check if ChatGPT, Claude, and Perplexity can read your site
Related Guides
Dive deeper with these step-by-step guides and tutorials.
Allowing crawlers is just step one.
If your site is built with React, Vue, or Angular, crawlers may see an empty page even with robots.txt configured correctly. Encited pre-renders your JavaScript into static HTML every crawler can read.
