WordPress Robots.txt: How to Find, Edit and Set Up the File
Where WordPress robots.txt lives, the exact default file, how to edit it in Yoast, Rank Math, AIOSEO or FTP, a safe template, AI crawler rules and how to test.
Every WordPress site has a robots.txt file, but on a fresh install you won't find it on the server. WordPress builds it in PHP each time someone requests /robots.txt, and stops the moment a real robots.txt file appears in the web root. If plugin edits to your WordPress robots.txt never go live, look for that real file first. To change your WP robots.txt, find out which file your site serves, edit that one in your SEO plugin or over FTP, then test the live result.
TL;DR
- The default file blocks
/wp-admin/, allowsadmin-ajax.phpand lists/wp-sitemap.xmlwhen the site is public. - Edit it in Yoast (Tools > File editor), Rank Math (General Settings > Edit robots.txt) or All in One SEO (Tools > Robots.txt Editor), or upload a file over FTP. Pick one method per site.
- "Discourage search engines" no longer adds
Disallow: /. It puts noindex on every page and removes the sitemap line. - Never block
/wp-content/uploads/, theme and plugin folders or/wp-includes/. Google needs those files to show your images and render your pages. - To block AI training and stay in AI search, give GPTBot, ClaudeBot and Google-Extended their own
Disallow: /groups.
Where is the robots.txt file in WordPress?
WordPress registers a rewrite rule that maps /robots.txt to index.php?robots=1. Core's do_robots() function then prints the file with a text/plain header, and any plugin can change the text through the robots_txt filter. This generated version is the virtual robots.txt.
The default WordPress robots.txt
On a public site with pretty permalinks, current WordPress core serves this file:
User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.phpSitemap: https://example.com/wp-sitemap.xml
Line by line:
User-agent: *addresses every crawler that has no group of its own.Disallow: /wp-admin/keeps crawlers out of the dashboard. WordPress takes the path from your admin URL, so it changes when WordPress runs in a subfolder.Allow: /wp-admin/admin-ajax.phpreopens the one admin file that themes and plugins call from public pages.- The
Sitemapline points at the XML sitemaps WordPress added in version 5.5. With plain permalinks it readshttps://example.com/?sitemap=index.
How crawlers match these lines, including wildcards and conflicting rules, is in what robots.txt is and how it works.
SEO plugins change this output. With Yoast SEO active, the three default rules are replaced by a block between # START YOAST BLOCK and # END YOAST BLOCK that holds User-agent: *, an empty Disallow: (which allows everything) and Yoast's sitemap_index.xml. Rank Math and All in One SEO change it once you save rules in their editors. Before you edit anything, open yoursite.com/robots.txt: that response is the file crawlers read.
Virtual robots.txt vs physical robots.txt
WordPress's default .htaccess rules on Apache hand a request to WordPress only when no real file or folder matches it (RewriteCond %{REQUEST_FILENAME} !-f), and the usual Nginx setup does the same with try_files. So when a physical robots.txt exists in the web root, the web server returns it, and WordPress, your SEO plugin and every robots_txt filter are skipped.
| Virtual robots.txt | Physical robots.txt | |
|---|---|---|
| Where it comes from | Generated by WordPress on each request | A robots.txt file in your web root |
| How you edit it | SEO plugin, WP Robots Txt, or the robots_txt filter | FTP, SFTP, your host's file manager, or Yoast's File editor |
| Plugin crawl settings | Applied | Ignored |
| Sitemap line | Added by WordPress or your SEO plugin | You write it |
| "Discourage search engines" | Removes the sitemap line | Changes nothing in the file |
To see which one you have, look for robots.txt in your web root with FTP or your host's file manager. Two plugins also tell you. Rank Math locks its editor with the message "Contents are locked because a robots.txt file is present in the root folder." All in One SEO shows a notice that it has detected a physical robots.txt file, and offers to import its rules and delete it.
robots.txt only works at the root of a host. Google's rule is that the file "cannot be placed in a subdirectory", so if your site's address is example.com/blog/, crawlers read example.com/robots.txt, and WordPress doesn't generate a robots.txt at all in that setup. Each subdomain, such as shop.example.com, needs its own file.
What "Discourage search engines from indexing this site" does to robots.txt
The checkbox is under Settings > Reading > Search engine visibility. Until WordPress 5.3, checking it added Disallow: / to the virtual robots.txt. The do_robots() changelog for 5.3 records the switch to a robots meta tag. Today the setting does three things:
- It adds
<meta name='robots' content='noindex, nofollow' />to the head of every page. - It turns off WordPress's XML sitemaps, so the
Sitemapline disappears from the virtual robots.txt and/wp-sitemap.xmlreturns a 404. - It shows "Search engines discouraged" in the At a Glance box on the dashboard.
The robots.txt rules themselves stay as they were. Rank Math stops serving your custom robots.txt while the box is checked and shows a warning in its editor. Crawlers have to fetch a page to read its noindex tag, so don't add Disallow: / on top of the setting: a blocked URL that other sites link to can stay in Google's index with no description. For a staging site, a password keeps people and crawlers out.
A box left checked after launch keeps the whole site out of Google, and a theme or plugin can still add noindex to some templates after you clear it. The site audit flags every crawled page that carries noindex in a robots meta tag or an X-Robots-Tag header.
How to edit robots.txt in WordPress with a plugin
Rank Math and All in One SEO edit the virtual file. Yoast's editor writes a physical file to your web root.
Yoast SEO
- Go to Yoast SEO > Tools and click File editor.
- If you have no robots.txt yet, click Create robots.txt file. Yoast writes a real file to your web root, filled with your current robots.txt output.
- Edit the rules and click Save changes to robots.txt.
Yoast hides the File editor when wp-config.php sets DISALLOW_FILE_EDIT, and on multisite it sits in the network admin. If the web root isn't writable, Yoast can't save; upload the file over FTP.
Yoast also adds rules from its settings, under Yoast SEO > Settings > Advanced > Crawl optimization: "Prevent crawling of internal site search URLs", "Remove WP-JSON API" and "Prevent Google AdsBot from crawling". Yoast SEO Premium adds switches for GPTBot, Google-Extended and CCBot. These switches change the virtual file only, so they stop working once the File editor has created a physical one.
Rank Math
- Go to Rank Math SEO and switch on Advanced Mode with the toggle at the top right. In Easy Mode the robots.txt tab is hidden.
- Open Rank Math SEO > General Settings > Edit robots.txt.
- Type your rules. An empty field lets WordPress serve its default file.
- Click Save Changes.
What you type replaces the whole file, so include a Sitemap: line. A link under the editor opens Rank Math's Robots.txt Tester.
All in One SEO
- Go to All in One SEO > Tools and open the Robots.txt Editor.
- Switch on Enable Custom Robots.txt.
- Click Add Rule and fill in User Agent (
*for all crawlers), Directive (Allow, Block, Clean-param or Crawl-delay) and Value (the path, with*as a wildcard). - Click Save Changes and read the result in the Custom Robots.txt Preview.
All in One SEO keeps WordPress's default rules and adds yours to them. Block writes a Disallow line.
Without an SEO plugin
The WP Robots Txt plugin adds a Robots.txt Content field to Settings > Reading. Empty the field to go back to the default.
With no plugin at all, a developer can change the virtual file with the robots_txt filter in a must-use plugin, saved as wp-content/mu-plugins/robots-txt.php:
<?php/** Plugin Name: Custom robots.txt rules*/add_filter( 'robots_txt', function ( $output ) {$output .= "\nUser-agent: GPTBot\nDisallow: /\n";return $output;}, 100 );
The filter runs after WordPress's own sitemap line is added, and it changes nothing while a physical robots.txt exists.
How to edit robots.txt via FTP or your host's file manager
- Open
yoursite.com/robots.txtand copy what's there as your starting point. - Create a plain text file named
robots.txt, all lowercase, saved as UTF-8. Use a code editor such as VS Code or Notepad++. In TextEdit, choose Format > Make Plain Text first. - Connect with an SFTP client such as FileZilla, or open the file manager in your hosting panel. Go to the web root: the folder that holds
wp-admin,wp-contentandwp-includes. Hosts often name itpublic_html,wwworhtdocs. - Upload
robots.txtto that folder. Permissions of 644 let the web server read it. - Open
yoursite.com/robots.txtagain and check that your rules show.
From then on WordPress and your SEO plugin no longer touch the file, so update the Sitemap: line yourself when you switch sitemap plugins. To go back to the virtual file, delete or rename the physical one.
A recommended WordPress robots.txt template
Start from WordPress's default and add only lines that match something on your site. Crawlers ignore lines that start with #, so comments can stay in the file for the next person who edits it.
# Every crawler without a group of its own belowUser-agent: *# The dashboardDisallow: /wp-admin/# Themes and plugins call this file from public pagesAllow: /wp-admin/admin-ajax.php# Internal search results (the same paths Yoast's search setting blocks)Disallow: /?s=Disallow: /page/*/?s=Disallow: /search/# The sitemap your site actually generates, as a full URLSitemap: https://example.com/wp-sitemap.xml
Notes on the template:
- Keep a group's lines together, with no blank lines inside it. A blank line goes between groups.
- The search lines are optional. They keep crawlers out of search result pages, which any visitor or spammer can create by typing a query.
- On a WooCommerce shop, add
Disallow: /*add-to-cart=to the*group so crawlers skip add-to-cart links. The rules for WooCommerce's filter and sort parameters are in faceted navigation SEO. - Replace the
SitemapURL with your sitemap's address (the URL for each plugin is in the sitemap section below).
Common WordPress robots.txt mistakes: what not to block
Old WordPress robots.txt templates still carry these mistakes. Fix any that your file has:
Disallow: /wp-content/uploads/. Every image and PDF on the site lives here. Google documents robots.txt as a way "to prevent image, video, and audio files from appearing in Google Search results", so this line takes your media out of Google Images.Disallow: /wp-content/themes/,/wp-content/plugins/or/wp-includes/. These folders hold the CSS and JavaScript that build your pages, and/wp-includes/holds core scripts such as jQuery that themes and plugins load. Google warns against blocking resource files "if the absence of these resources make the page harder for Google's crawler to understand the page".- A missing
Allow: /wp-admin/admin-ajax.php. Keep it whenever you block/wp-admin/. Disallow: /wp-json/on a site whose pages load content from the REST API. Yoast's "Remove WP-JSON API" setting adds this rule. Turn it on only if no public page fetches content from/wp-json/in the browser.- Blocking pages you want out of Google. Google says robots.txt "is not a mechanism for keeping a web page out of Google". Use a noindex tag and leave the page crawlable. A blocked page can show up as Indexed, though blocked by robots.txt.
- Blocking WooCommerce's cart, checkout and My account pages. WooCommerce already adds noindex to all three, and a robots.txt block hides that tag from Google.
Disallow: /underUser-agent: *. One line blocks the whole site. Check for it after every move from staging to live.Crawl-delay. Google doesn't support it. Anthropic documents support for it in ClaudeBot.
If Search Console already lists pages as Blocked by robots.txt, find the rule that matches them in your live file and remove it if you want those pages indexed.
How to allow or block AI crawlers in WordPress robots.txt
AI companies run separate bots for model training, for their search indexes, and for fetching a page a user asked about. Each vendor publishes the token to put in robots.txt:
| User-agent token | Company | What it does | If you disallow it |
|---|---|---|---|
GPTBot | OpenAI | Crawls content that may be used to train OpenAI's models | Signals that your content shouldn't be used for training |
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search results | Your pages aren't shown in ChatGPT search answers, apart from navigational links |
ChatGPT-User | OpenAI | Fetches pages for actions a ChatGPT user starts | OpenAI says robots.txt rules "may not apply" |
ClaudeBot | Anthropic | Collects content that may be used to train Claude | Signals that your future content should be excluded from training |
Claude-SearchBot | Anthropic | Indexes content for Claude's search results | Lower visibility in Claude's search results |
Claude-User | Anthropic | Fetches pages when a user asks Claude about them | Claude can't fetch your pages for its users |
PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search results | Perplexity recommends allowing it so your pages can appear in its results |
Perplexity-User | Perplexity | Fetches pages for a Perplexity user's request | Perplexity says it "generally ignores robots.txt rules" |
Google-Extended | Controls use of your content for Gemini training and grounding | No effect on Google Search: Google says it isn't a ranking signal and doesn't affect inclusion |
Google-Extended isn't a separate crawler. Google crawls with its normal user agents and reads the token as a setting.
A crawler follows only the most specific group that names it. A User-agent: GPTBot group means GPTBot skips everything under User-agent: *. Bots with no group of their own follow your * rules, so you don't need to list the AI search bots to allow them.
To block AI training and keep AI search, give only the training bots their own groups:
User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.php# AI training crawlersUser-agent: GPTBotDisallow: /User-agent: ClaudeBotDisallow: /User-agent: Google-ExtendedDisallow: /Sitemap: https://example.com/wp-sitemap.xml
OpenAI says "Each setting is independent of the others", so a GPTBot block leaves OAI-SearchBot and ChatGPT search as they were. To leave AI search too, add the same Disallow: / group for OAI-SearchBot, Claude-SearchBot and PerplexityBot; ChatGPT-User and Perplexity-User may still fetch pages for users, and only a firewall rule stops them. To keep one bot out of part of the site, copy the * rules into its group and add the extra path. OpenAI and Perplexity both say robots.txt changes can take up to about 24 hours to reach their systems.
The Yoast Premium switches above and All in One SEO's Block AI Crawlers setting write the same groups for you. If your site runs behind Cloudflare with its managed robots.txt turned on, Cloudflare puts its own rules above your WordPress file, so the live file can block AI bots your WordPress settings allow. For everything else that decides whether ChatGPT and Perplexity cite you, read how to get pages indexed by ChatGPT and AI search.
robots.txt is a request. A security plugin, your host's firewall or a CDN bot rule can still answer GPTBot or ClaudeBot with a 403 while your robots.txt allows them. The crawler simulator fetches any URL with the user agent of the bot you pick and shows the status code and HTML it gets back.
How to add your sitemap to WordPress robots.txt
In a physical file or in Rank Math's editor, you write the Sitemap line yourself. List the sitemap your site actually generates:
| Sitemap made by | URL to list |
|---|---|
| WordPress core (5.5 and later) | https://example.com/wp-sitemap.xml, or https://example.com/?sitemap=index with plain permalinks |
| Yoast SEO | https://example.com/sitemap_index.xml |
| Rank Math | https://example.com/sitemap_index.xml |
| All in One SEO | https://example.com/sitemap.xml |
Google requires a "fully-qualified URL", with the same protocol and host your site uses. The line can go anywhere in the file, applies to every crawler, and can repeat for a second sitemap such as a news sitemap. Submit the same URL under Indexing > Sitemaps in Search Console as well.
How to test your WordPress robots.txt
- Open the live file. Run
curl -i https://yoursite.com/robots.txt, or load the URL in a private window. It should return 200 and your latest rules. If it shows old rules, look for a physical file first, then purge your caching plugin, your host's cache and your CDN. - Check the status code. Google treats a 404 robots.txt as no rules at all. A server error is worse: for the first 12 hours, Google stops crawling the site while it retries the file.
- Check each crawler. The free robots.txt checker reads your live file and gives Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and other crawlers a verdict: Allowed, Blocked, Partial or Default. A WordPress default file shows Partial, because
/wp-admin/is blocked. - Check Google's copy. In Search Console, open Settings and then the robots.txt report. It lists the robots.txt files Google found, when it last checked them and any parse errors. Google caches robots.txt for up to 24 hours. After unblocking important pages or fixing an error, click the more icon next to the file and choose Request a recrawl.
- Check single URLs. URL Inspection shows "Crawl allowed?" for any page, and Test live URL checks the page against your current rules.
FAQ
Where is the robots.txt file in WordPress?
On a fresh install it isn't on the server. WordPress generates a virtual robots.txt each time someone requests yoursite.com/robots.txt. A real robots.txt file in the web root, if one exists, replaces it.
What is the difference between WordPress's virtual robots.txt and a physical robots.txt file? WordPress builds the virtual file on each request, and SEO plugins edit it. A physical file is a text file in the web root that the server returns before WordPress runs, so while it exists, plugin editors, plugin crawl settings and the automatic sitemap line have no effect.
Why are my robots.txt changes not showing up in WordPress or Google Search Console? A physical robots.txt overrides every plugin editor, a cache or CDN can serve the old file until you purge it, and Google caches robots.txt for up to 24 hours. After an urgent fix, use Request a recrawl in Search Console's robots.txt report.
Does "Discourage search engines from indexing this site" change robots.txt?
Not the rules. Since WordPress 5.3 the setting no longer adds Disallow: /. It adds a noindex, nofollow robots meta tag to every page and removes the Sitemap line from the virtual robots.txt.
Can I accidentally block all search engines with robots.txt in WordPress?
Yes. One Disallow: / under User-agent: * asks every crawler to skip the whole site. Check yoursite.com/robots.txt after every launch, migration or plugin change.
How do I restore the default WordPress robots.txt? Delete or rename the physical robots.txt in your web root. Then empty the Rank Math robots.txt field, switch off Enable Custom Robots.txt in All in One SEO, or empty the Robots.txt Content field of WP Robots Txt.
How do I check if robots.txt blocks Google or AI chatbots?
Open yoursite.com/robots.txt, run it through a robots.txt analyzer, and use URL Inspection in Search Console for Google. For AI bots, also fetch a page with the bot's user agent, since a firewall can refuse a bot that robots.txt allows.
Should I block GPTBot on my WordPress site? Block GPTBot if you don't want OpenAI to use your content for training. ChatGPT search uses OAI-SearchBot, and OpenAI says each setting is independent, so keep OAI-SearchBot allowed if you want to be cited in ChatGPT answers.
