Encited logo
PricingTalk to sales
Robots.txt checker

Robots.txt Checker

Test any robots.txt file. See which crawlers are allowed or blocked, and every rule for Googlebot, GPTBot, ClaudeBot and the other search and AI bots.

Check

Check any robots.txt

Enter your domain to see which crawlers are allowed or blocked.

Robots.txt Examples

Copy these templates to control how crawlers access your site.

Allow All Crawlers

Best for most websites wanting maximum visibility

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Block AI Training Only

Allow search engines but block AI training crawlers

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Allow Search + AI Citations

Be indexed by search engines and cited by AI assistants

User-agent: *
Allow: /

# Allow AI assistants to cite your content
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

Block Specific Paths

Allow most content but protect admin/private areas

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /*.json$

Sitemap: https://example.com/sitemap.xml
Google
Gemini
ChatGPT
Claude
Grok
Meta
Perplexity

Allowing crawlers is just step one.

If your site is built with React, Vue, or Angular, crawlers may see an empty page even with robots.txt configured correctly. Encited pre-renders your JavaScript into static HTML every crawler can read.

Book a demo

Robots.txt FAQ

robots.txt is a text file placed at the root of your website that tells web crawlers which pages they can or cannot access. It's part of the Robots Exclusion Protocol, a standard used by most search engines and AI crawlers. For a deeper dive, read Debugging robots.txt & Noindex Issues.

No, robots.txt is a suggestion, not a security measure. Well-behaved crawlers will respect it, but malicious bots may ignore it. For sensitive content, use authentication or password protection instead.

It depends on your goals. GPTBot collects training data for OpenAI's models. ChatGPT search uses a different crawler, OAI-SearchBot, so you can block GPTBot and still appear in ChatGPT search answers. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Learn more in SEO for ChatGPT: 12 Steps to Get Indexed and Cited.

Disallow: / blocks access to your entire website for the specified crawler. It's the most restrictive rule you can set. Use it carefully as it will prevent that crawler from seeing any of your pages.

Use this checker to see which crawlers your rules allow or block. In Google Search Console, the robots.txt report shows the robots.txt files Google found, when it last crawled them, and any warnings or errors. Use the Crawler Simulator to see what each bot actually sees. For a full technical audit, try the SEO Audit tool.

Crawl-delay specifies how many seconds a crawler should wait between requests. Not all crawlers support it (Googlebot ignores it), but it can help reduce server load from bots that do respect it.
Avatar
How can we help?
Get instant answers to your questions or leave a message for an engineer will reach out
Ask AI about Encited
See our docs
Contact support
Leave a message
We'll get back to you soon
Avatar
Ask AI about Encited
Product questions, troubleshooting and
Thinking
Preview
Drop an image to attach
Powered by ReplyMaven