What Is Robots.txt? How It Works, Syntax and Examples
Robots.txt tells crawlers which URLs they may fetch. Learn the User-agent, Disallow, Allow and Sitemap syntax, with examples, AI bot rules and how to test it.
A robots.txt file is a plain text file at the root of your site, such as example.com/robots.txt, that tells crawlers which URLs they may fetch. Googlebot, Bingbot, GPTBot and every other well-behaved crawler read it before they crawl anything else. Its rules control crawling only. A page you block in robots.txt can still show up in Google, and a page robots.txt allows can still reach a crawler empty or get refused by a firewall.
TL;DR
- robots.txt lives at the root of each host and covers only that host, protocol and port.
- A crawler follows the one group that names it, then the most specific rule that matches the URL. On a tie, Google picks the least restrictive rule.
- Google reads four fields:
user-agent,allow,disallowandsitemap. It ignorescrawl-delayandnoindex. Disallowstops crawling. To keep a page out of Google, usenoindexand leave the page crawlable.- AI companies run separate bots for training, search and user requests, so you can block training and stay in ChatGPT, Claude and Perplexity search.
What is a robots.txt file?
robots.txt follows the Robots Exclusion Protocol, a convention from 1994 that the IETF published as RFC 9309 in September 2022. Search engines, AI companies and SEO tools read it.
The file is public. Anyone can open yours by adding /robots.txt to your homepage URL, and Google's own file is at google.com/robots.txt. It uses a common pattern: a User-agent: * group with Disallow: /search, followed by Allow: /search/about to reopen one page inside the blocked folder.
Where the file has to be:
- At the root of the host. Google's spec says "Crawlers don't check for robots.txt files in subdirectories." A file at
example.com/blog/robots.txtis never read. - On each host it should cover. The rules "apply only to the host, protocol, and port number" that served the file, so
example.com/robots.txtdoesn't covershop.example.comor thehttp://version of the site. - Named
robots.txt, all lowercase, as plain text. Paths inside it are case-sensitive.
A site with no robots.txt at all is fine. The request returns a 404, and crawlers treat that as permission to crawl everything.
Google describes the file as "used primarily to manage crawler traffic to your site" and adds that "it is not a mechanism for keeping a web page out of Google." Sites use it to keep crawlers off internal search results, filter and sort URLs, carts and account pages, to point crawlers at the sitemap, and to set rules for AI bots.
Because anyone can read it, don't list private URLs in it. A Disallow: /old-admin-backup/ line tells every reader where that folder is.
How does robots.txt work?
A crawler goes through the same steps on every site:
- Before it crawls a host, it requests
/robots.txt. - It reads the HTTP status of that response (the table below).
- It picks the group of rules that matches its user agent.
- For each URL it wants, it checks the path against that group's rules, then fetches the URL or skips it.
- It keeps the file for a while. Google "generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer".
The status code of the robots.txt response decides what Google does:
| robots.txt response | What Google does |
|---|---|
2xx | Reads the rules as served |
3xx | Follows at least five redirect hops, then treats the file as a 404 |
4xx, except 429 | Acts as if no robots.txt exists, so every URL may be crawled |
5xx, a timeout or a DNS error | Stops crawling the site for the first 12 hours while it retries, then uses the last good copy for up to 30 days |
A broken robots.txt that returns a server error does more damage than a missing one: Google pauses crawling of the whole site. Google also reads only the first 500 KiB of the file, so rules past that size are ignored.
Googlebot, Bingbot and the training and search crawlers of the large AI companies follow the file. Google notes that "other crawlers might not", and RFC 9309 states that the rules "are not a form of access authorization." Content that must stay private needs a login or a firewall rule.
Robots.txt syntax and directives (User-agent, Disallow, Allow)
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines, followed by the rules for those crawlers. Every line is a field: value pair, and a # starts a comment.
# Every crawler without a group of its ownUser-agent: *Disallow: /searchDisallow: /cartAllow: /search/help# Bingbot follows this group and skips the one aboveUser-agent: BingbotDisallow: /private/Sitemap: https://example.com/sitemap.xml
User-agent
User-agent names the crawler a group is for, using the crawler's short token: Googlebot, Bingbot, GPTBot. Google matches it without regard to case: "Both the user-agent field name and its value are case-insensitive."
User-agent: * addresses every crawler that has no group of its own. A crawler follows exactly one group. Google's spec says "Only one group is valid for a particular crawler", and the crawler picks the group with the most specific name that matches it. In the example above, Bingbot reads only its own group, so it may crawl /search and /cart. To keep the shared rules for Bingbot, repeat them in its group.
Two groups for the same crawler are merged into one. A specific group is never merged with *: "User agent specific groups and global groups (*) are not combined." To give several crawlers the same rules, stack their User-agent lines on top of one group.
To see whether each crawler is allowed on your live file, run your domain through the free robots.txt checker. It shows, for Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and 13 other crawlers, whether each one is allowed, blocked or partly blocked, and which ones the file has no rules for.
Disallow
Disallow names a path the crawler shouldn't fetch. The value is matched from the start of the URL path, so it works as a prefix:
Disallow: /cartblocks/cart,/cart/,/cart?item=1and also/cartoon.Disallow: /cart/blocks only URLs inside the/cart/folder.Disallow: /blocks the whole site.- An empty
Disallow:blocks nothing.
The path must start with / and is case-sensitive. Disallow: /Admin/ doesn't block /admin/.
Allow
Allow reopens a path inside a blocked one. With Disallow: /search and Allow: /search/help, crawlers skip search result pages and still fetch the help page. Google, Bing and the RFC all support it.
Sitemap
Sitemap points crawlers at your XML sitemap. It comes from the Sitemaps protocol, a separate standard that robots.txt borrows this one line from. Google requires a full address: "It must be a fully qualified URL, including the protocol and host." The line can sit anywhere in the file, applies to every crawler whatever group it's near, and can repeat for a second sitemap.
Sitemap: https://example.com/sitemap.xmlSitemap: https://example.com/news-sitemap.xml
Wildcards: * and $
Google and Bing support two special characters in paths. * "designates 0 or more instances of any valid character", and $ "designates the end of the URL".
| Rule | Blocks | Doesn't block |
|---|---|---|
Disallow: /*?sort= | /shoes?sort=price | /shoes |
Disallow: /*.pdf$ | /files/guide.pdf | /files/guide.pdf?v=2 |
Disallow: /*? | Every URL with a query string | URLs without one |
Which rule wins when two match
Google picks "the most specific rule based on the length of the rule path." If two rules are equally long, "Google uses the least restrictive rule."
User-agent: *Disallow: /shop/Allow: /shop/sale/
Here /shop/sale/boots is crawled, because /shop/sale/ is the longer match. /shop/boots is blocked. With Allow: /page and Disallow: /page in the same group, Google crawls /page.
Fields Google ignores
Google supports user-agent, allow, disallow and sitemap, and states that "other fields such as crawl-delay aren't supported." crawl-delay still works for some crawlers, with the value in seconds: Anthropic documents support for it in ClaudeBot. Search Console no longer has a setting to slow Googlebot either: Google removed its crawl rate limiter on January 8, 2024, and says "the fastest way to reduce crawl rate is to instruct Googlebot through server responses", such as a 503 or 429 status while the server is overloaded. noindex and nofollow lines in robots.txt do nothing in Google, which retired the code that handled them on September 1, 2019.
Robots.txt examples
Copy the one that matches what you need, then replace the paths and the sitemap URL.
Allow every crawler:
User-agent: *Disallow:Sitemap: https://example.com/sitemap.xml
Block the whole site. This keeps crawlers out of a staging copy, but URLs that other sites link to can still appear in Google. A password keeps a staging site out properly.
User-agent: *Disallow: /
Block a folder and allow one page inside it:
User-agent: *Disallow: /account/Allow: /account/login
Block one crawler and allow the rest:
User-agent: *Disallow:User-agent: CCBotDisallow: /
Block sort and filter parameters:
User-agent: *Disallow: /*?sort=Disallow: /*&sort=Disallow: /*?color=Disallow: /*&color=
Block one file type:
User-agent: *Disallow: /*.pdf$
Robots.txt vs noindex: blocking crawling doesn't block indexing
Google indexes URLs it finds in links even when it can't crawl them: "its URL can still appear in search results, but the search result won't have a description." Search Console reports these as Indexed, though blocked by robots.txt.
Pick the tool by what you want to happen:
| You want | Use |
|---|---|
| A page kept out of Google's results | A noindex robots meta tag or X-Robots-Tag header, with the page left crawlable |
| Crawlers to stop requesting a set of URLs | Disallow in robots.txt |
| People and bots kept out | A login or password |
| A page gone from results quickly | A 404 or 410 status, plus Search Console's Removals tool for a temporary hide |
Never block a page that carries noindex. Google has to crawl the page to read the tag, so the Disallow rule hides the one signal that would have removed it.
How to block AI crawlers with robots.txt
AI companies run separate bots for separate jobs, and each one has a token you can name in robots.txt. What a Disallow: / does depends on the bot:
| Token | Company | Job | What a Disallow: / does |
|---|---|---|---|
GPTBot | OpenAI | Collects training data | OpenAI says it "indicates a site's content should not be used in training generative AI foundation models" |
ClaudeBot | Anthropic | Collects training data | Anthropic says it "signals that the site's future materials should be excluded" from training |
Google-Extended | A setting for Gemini training and grounding | Keeps your content out of Gemini training and grounding, with no effect on Google Search | |
CCBot | Common Crawl | Builds the open web dataset many AI models train on | Keeps your pages out of new Common Crawl snapshots |
OAI-SearchBot | OpenAI | ChatGPT search | Your pages "will not be shown in ChatGPT search answers", though they "can still appear as navigational links" |
Claude-SearchBot | Anthropic | Claude's search index | Anthropic says it "prevents our system from indexing your content for search optimization" |
PerplexityBot | Perplexity | Perplexity's search results | Perplexity recommends allowing it so your pages can appear in its answers |
ChatGPT-User | OpenAI | Fetches a page a user asked about | OpenAI says "robots.txt rules may not apply" |
Claude-User | Anthropic | Fetches a page a user asked about | Anthropic says it "prevents our system from retrieving your content in response to a user query" |
Perplexity-User | Perplexity | Fetches a page a user asked about | Perplexity says it "generally ignores robots.txt rules" |
Google-Extended isn't a crawler. Google crawls with its usual bots and reads the token as a setting. Grounding is Gemini pulling content from Google's index while it answers, so a Google-Extended block takes your pages out of those answers as well as out of training.
To keep your content out of model training and stay in ChatGPT, Claude and Perplexity search, give only the training bots a group:
User-agent: *Disallow:User-agent: GPTBotUser-agent: ClaudeBotUser-agent: Google-ExtendedUser-agent: CCBotDisallow: /Sitemap: https://example.com/sitemap.xml
The search bots have no group of their own, so they follow the * rules and can crawl everything. Drop Google-Extended from the group if you want your pages used in Gemini answers. OpenAI and Perplexity say robots.txt changes can take about 24 hours to reach their systems. How to get pages indexed by ChatGPT covers what else decides whether AI assistants cite you.
robots.txt only asks. A CDN bot rule, a security plugin or your host's firewall can still answer GPTBot with a 403 while your file allows it, and Cloudflare's managed robots.txt adds its own AI rules on top of your file when it's switched on.
Don't block the files your pages need
Google renders pages the way a browser does, so it needs your JavaScript, CSS and API responses. Google warns against blocking them "if the absence of these resources make the page harder for Google's crawler to understand the page". On a site built with React, Vue, Next.js or Lovable, a rule such as Disallow: /assets/, Disallow: /_next/ or Disallow: /api/ can leave Google's render with no content at all.
AI crawlers add a second problem. GPTBot, ClaudeBot and PerplexityBot don't run JavaScript, so even with every path allowed they read only the HTML your server sends first. On a client-rendered site that HTML is an empty shell.
To see what a bot actually receives, open the page in the crawler simulator as Googlebot or GPTBot. It shows the status code and the HTML that a request with that bot's user agent gets back, so you can spot a page that robots.txt allows but a firewall refuses, or one whose text only appears after JavaScript runs.
How to create or edit robots.txt
Static sites, Vite, React and Lovable. Put robots.txt in the public folder. Vite copies everything in public to the root of the build, so the file is served at /robots.txt. In Lovable, ask for it in a prompt: the Lovable SEO guide has one that adds AI crawler rules and your sitemap.
Next.js (App Router). Add a static app/robots.txt, or generate the file from app/robots.ts:
import type { MetadataRoute } from 'next'export default function robots(): MetadataRoute.Robots {return {rules: {userAgent: '*',allow: '/',disallow: '/private/',},sitemap: 'https://acme.com/sitemap.xml',}}
Shopify. Every store gets a default robots.txt. To change it, go to Online Store, click the menu next to your theme and choose Edit code, then click Add a new template, select robots and click Create template. Shopify calls this "an unsupported customization" and warns that "Incorrect use of the feature can result in loss of all traffic." Deleting the template restores the default file.
WordPress. WordPress serves a virtual robots.txt that SEO plugins edit, and a real file in the web root replaces it. The WordPress robots.txt guide covers Yoast, Rank Math, All in One SEO and FTP step by step.
Behind a CDN. A CDN or proxy can serve its own robots.txt or add rules to yours. Whatever you edit, the response at /robots.txt is the file crawlers read.
How to test your robots.txt
- Open the live file. Run
curl -i https://example.com/robots.txt. It should return200, atext/plaincontent type and your latest rules. If you see old rules, purge your cache or CDN. - Check each crawler. Run the domain through the robots.txt checker and look for search or AI bots marked Blocked that you meant to allow.
- Check Google's copy. In Search Console, open Settings and then the robots.txt report. It "shows which robots.txt files Google found for the top 20 hosts on your site", with the last fetch time and any parsing issues. It works for a Domain property or a URL-prefix property without a path, such as
https://example.com/. After a fix, open the menu next to the file and choose Request a recrawl. - Check single URLs. URL Inspection in Search Console shows whether Google may crawl a specific page.
- Watch real visits. A crawl log shows which bots fetched which URLs after the change and what status each request got.
Common robots.txt mistakes
Disallow: /carried over from a staging site to the live one.- CSS, JavaScript or API paths blocked on a site that needs them to render.
Disallowused to hide a page from Google, where anoindextag was needed.- A page with
noindexthat is also blocked, so Google never reads the tag. - Private URLs listed in the file, which anyone can read.
- A group for one crawler written as if it adds to the
*rules. - A path in the wrong case, such as
/Admin/for/admin/. - A relative sitemap address, such as
Sitemap: /sitemap.xml. - The file placed in a subfolder or on only one of the site's hosts.
Pages that Search Console lists as Blocked by robots.txt match one of your Disallow rules. Check each against the live file, and delete or narrow the rule for the ones you want in Google.
FAQ
What is a robots.txt file?
A plain text file at the root of a site, such as example.com/robots.txt, that tells crawlers which URLs they may fetch. Search engines and AI crawlers read it before they crawl. It controls crawling, and Google says it is not a way to keep a page out of search results.
What does User-agent: * mean in robots.txt?
The asterisk addresses every crawler that has no group of its own in the file. A crawler that is named in another group follows only that group and ignores the User-agent: * rules.
How do Disallow and Allow work?
Disallow names a URL path a crawler shouldn't fetch, matched from the start of the path, so Disallow: /cart also blocks /cartoon. Allow reopens a path inside a blocked one. When two rules match, Google uses the one with the longer path, and on a tie it uses the least restrictive rule.
Does robots.txt stop a page from being indexed?
No. A blocked URL can still appear in Google if other pages link to it, shown without a description. To keep a page out of Google, add a noindex tag and leave the page crawlable so Google can read the tag.
Can robots.txt block AI bots and scrapers? It can ask them to stay out. GPTBot, ClaudeBot, PerplexityBot and Google-Extended have published tokens you can disallow. Robots.txt is a request, so a scraper can ignore it. Only a login or a firewall rule actually stops a bot.
Where should robots.txt be placed?
At the root of the host, as /robots.txt. Crawlers don't look for it in subfolders, and the rules apply only to the host, protocol and port they were served from, so each subdomain needs its own file.
Does Google support crawl-delay in robots.txt?
No. Google reads only the user-agent, allow, disallow and sitemap fields. Some other crawlers read crawl-delay as a number of seconds, and Anthropic documents support for it in ClaudeBot. To slow Googlebot, Google points to server responses such as a 503 or 429 status.
How do I know if Google can crawl my site after updating robots.txt?
Open yoursite.com/robots.txt to confirm the new rules are live, then check the robots.txt report in Search Console, which shows when Google last fetched the file. URL Inspection shows whether a single page is allowed. Google caches robots.txt for up to 24 hours, and you can request a recrawl of the file after a fix.
