Encited logo
PricingTalk to sales
WordPress Robots.txt: How to Find, Edit and Set Up the File

WordPress Robots.txt: How to Find, Edit and Set Up the File

Nov 9, 2026November 9, 2026·Updated Nov 10, 2026November 10, 2026·by Aki from Encited

Where WordPress robots.txt lives, the exact default file, how to edit it in Yoast, Rank Math, AIOSEO or FTP, a safe template, AI crawler rules and how to test.

Every WordPress site has a robots.txt file, but on a fresh install you won't find it on the server. WordPress builds it in PHP each time someone requests /robots.txt, and stops the moment a real robots.txt file appears in the web root. If plugin edits to your WordPress robots.txt never go live, look for that real file first. To change your WP robots.txt, find out which file your site serves, edit that one in your SEO plugin or over FTP, then test the live result.

TL;DR

  • The default file blocks /wp-admin/, allows admin-ajax.php and lists /wp-sitemap.xml when the site is public.
  • Edit it in Yoast (Tools > File editor), Rank Math (General Settings > Edit robots.txt) or All in One SEO (Tools > Robots.txt Editor), or upload a file over FTP. Pick one method per site.
  • "Discourage search engines" no longer adds Disallow: /. It puts noindex on every page and removes the sitemap line.
  • Never block /wp-content/uploads/, theme and plugin folders or /wp-includes/. Google needs those files to show your images and render your pages.
  • To block AI training and stay in AI search, give GPTBot, ClaudeBot and Google-Extended their own Disallow: / groups.

Where is the robots.txt file in WordPress?

WordPress registers a rewrite rule that maps /robots.txt to index.php?robots=1. Core's do_robots() function then prints the file with a text/plain header, and any plugin can change the text through the robots_txt filter. This generated version is the virtual robots.txt.

The default WordPress robots.txt

On a public site with pretty permalinks, current WordPress core serves this file:

ts
CopyDownload
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml

Line by line:

  • User-agent: * addresses every crawler that has no group of its own.
  • Disallow: /wp-admin/ keeps crawlers out of the dashboard. WordPress takes the path from your admin URL, so it changes when WordPress runs in a subfolder.
  • Allow: /wp-admin/admin-ajax.php reopens the one admin file that themes and plugins call from public pages.
  • The Sitemap line points at the XML sitemaps WordPress added in version 5.5. With plain permalinks it reads https://example.com/?sitemap=index.

How crawlers match these lines, including wildcards and conflicting rules, is in what robots.txt is and how it works.

SEO plugins change this output. With Yoast SEO active, the three default rules are replaced by a block between # START YOAST BLOCK and # END YOAST BLOCK that holds User-agent: *, an empty Disallow: (which allows everything) and Yoast's sitemap_index.xml. Rank Math and All in One SEO change it once you save rules in their editors. Before you edit anything, open yoursite.com/robots.txt: that response is the file crawlers read.

Virtual robots.txt vs physical robots.txt

WordPress's default .htaccess rules on Apache hand a request to WordPress only when no real file or folder matches it (RewriteCond %{REQUEST_FILENAME} !-f), and the usual Nginx setup does the same with try_files. So when a physical robots.txt exists in the web root, the web server returns it, and WordPress, your SEO plugin and every robots_txt filter are skipped.

Virtual robots.txtPhysical robots.txt
Where it comes fromGenerated by WordPress on each requestA robots.txt file in your web root
How you edit itSEO plugin, WP Robots Txt, or the robots_txt filterFTP, SFTP, your host's file manager, or Yoast's File editor
Plugin crawl settingsAppliedIgnored
Sitemap lineAdded by WordPress or your SEO pluginYou write it
"Discourage search engines"Removes the sitemap lineChanges nothing in the file

To see which one you have, look for robots.txt in your web root with FTP or your host's file manager. Two plugins also tell you. Rank Math locks its editor with the message "Contents are locked because a robots.txt file is present in the root folder." All in One SEO shows a notice that it has detected a physical robots.txt file, and offers to import its rules and delete it.

robots.txt only works at the root of a host. Google's rule is that the file "cannot be placed in a subdirectory", so if your site's address is example.com/blog/, crawlers read example.com/robots.txt, and WordPress doesn't generate a robots.txt at all in that setup. Each subdomain, such as shop.example.com, needs its own file.

What "Discourage search engines from indexing this site" does to robots.txt

The checkbox is under Settings > Reading > Search engine visibility. Until WordPress 5.3, checking it added Disallow: / to the virtual robots.txt. The do_robots() changelog for 5.3 records the switch to a robots meta tag. Today the setting does three things:

  1. It adds <meta name='robots' content='noindex, nofollow' /> to the head of every page.
  2. It turns off WordPress's XML sitemaps, so the Sitemap line disappears from the virtual robots.txt and /wp-sitemap.xml returns a 404.
  3. It shows "Search engines discouraged" in the At a Glance box on the dashboard.

The robots.txt rules themselves stay as they were. Rank Math stops serving your custom robots.txt while the box is checked and shows a warning in its editor. Crawlers have to fetch a page to read its noindex tag, so don't add Disallow: / on top of the setting: a blocked URL that other sites link to can stay in Google's index with no description. For a staging site, a password keeps people and crawlers out.

A box left checked after launch keeps the whole site out of Google, and a theme or plugin can still add noindex to some templates after you clear it. The site audit flags every crawled page that carries noindex in a robots meta tag or an X-Robots-Tag header.

How to edit robots.txt in WordPress with a plugin

Rank Math and All in One SEO edit the virtual file. Yoast's editor writes a physical file to your web root.

Yoast SEO

  1. Go to Yoast SEO > Tools and click File editor.
  2. If you have no robots.txt yet, click Create robots.txt file. Yoast writes a real file to your web root, filled with your current robots.txt output.
  3. Edit the rules and click Save changes to robots.txt.

Yoast hides the File editor when wp-config.php sets DISALLOW_FILE_EDIT, and on multisite it sits in the network admin. If the web root isn't writable, Yoast can't save; upload the file over FTP.

Yoast also adds rules from its settings, under Yoast SEO > Settings > Advanced > Crawl optimization: "Prevent crawling of internal site search URLs", "Remove WP-JSON API" and "Prevent Google AdsBot from crawling". Yoast SEO Premium adds switches for GPTBot, Google-Extended and CCBot. These switches change the virtual file only, so they stop working once the File editor has created a physical one.

Rank Math

  1. Go to Rank Math SEO and switch on Advanced Mode with the toggle at the top right. In Easy Mode the robots.txt tab is hidden.
  2. Open Rank Math SEO > General Settings > Edit robots.txt.
  3. Type your rules. An empty field lets WordPress serve its default file.
  4. Click Save Changes.

What you type replaces the whole file, so include a Sitemap: line. A link under the editor opens Rank Math's Robots.txt Tester.

All in One SEO

  1. Go to All in One SEO > Tools and open the Robots.txt Editor.
  2. Switch on Enable Custom Robots.txt.
  3. Click Add Rule and fill in User Agent (* for all crawlers), Directive (Allow, Block, Clean-param or Crawl-delay) and Value (the path, with * as a wildcard).
  4. Click Save Changes and read the result in the Custom Robots.txt Preview.

All in One SEO keeps WordPress's default rules and adds yours to them. Block writes a Disallow line.

Without an SEO plugin

The WP Robots Txt plugin adds a Robots.txt Content field to Settings > Reading. Empty the field to go back to the default.

With no plugin at all, a developer can change the virtual file with the robots_txt filter in a must-use plugin, saved as wp-content/mu-plugins/robots-txt.php:

php
CopyDownload
<?php
/*
* Plugin Name: Custom robots.txt rules
*/
add_filter( 'robots_txt', function ( $output ) {
$output .= "\nUser-agent: GPTBot\nDisallow: /\n";
return $output;
}, 100 );

The filter runs after WordPress's own sitemap line is added, and it changes nothing while a physical robots.txt exists.

How to edit robots.txt via FTP or your host's file manager

  1. Open yoursite.com/robots.txt and copy what's there as your starting point.
  2. Create a plain text file named robots.txt, all lowercase, saved as UTF-8. Use a code editor such as VS Code or Notepad++. In TextEdit, choose Format > Make Plain Text first.
  3. Connect with an SFTP client such as FileZilla, or open the file manager in your hosting panel. Go to the web root: the folder that holds wp-admin, wp-content and wp-includes. Hosts often name it public_html, www or htdocs.
  4. Upload robots.txt to that folder. Permissions of 644 let the web server read it.
  5. Open yoursite.com/robots.txt again and check that your rules show.

From then on WordPress and your SEO plugin no longer touch the file, so update the Sitemap: line yourself when you switch sitemap plugins. To go back to the virtual file, delete or rename the physical one.

Start from WordPress's default and add only lines that match something on your site. Crawlers ignore lines that start with #, so comments can stay in the file for the next person who edits it.

ts
CopyDownload
# Every crawler without a group of its own below
User-agent: *
# The dashboard
Disallow: /wp-admin/
# Themes and plugins call this file from public pages
Allow: /wp-admin/admin-ajax.php
# Internal search results (the same paths Yoast's search setting blocks)
Disallow: /?s=
Disallow: /page/*/?s=
Disallow: /search/
# The sitemap your site actually generates, as a full URL
Sitemap: https://example.com/wp-sitemap.xml

Notes on the template:

  • Keep a group's lines together, with no blank lines inside it. A blank line goes between groups.
  • The search lines are optional. They keep crawlers out of search result pages, which any visitor or spammer can create by typing a query.
  • On a WooCommerce shop, add Disallow: /*add-to-cart= to the * group so crawlers skip add-to-cart links. The rules for WooCommerce's filter and sort parameters are in faceted navigation SEO.
  • Replace the Sitemap URL with your sitemap's address (the URL for each plugin is in the sitemap section below).

Common WordPress robots.txt mistakes: what not to block

Old WordPress robots.txt templates still carry these mistakes. Fix any that your file has:

  • Disallow: /wp-content/uploads/. Every image and PDF on the site lives here. Google documents robots.txt as a way "to prevent image, video, and audio files from appearing in Google Search results", so this line takes your media out of Google Images.
  • Disallow: /wp-content/themes/, /wp-content/plugins/ or /wp-includes/. These folders hold the CSS and JavaScript that build your pages, and /wp-includes/ holds core scripts such as jQuery that themes and plugins load. Google warns against blocking resource files "if the absence of these resources make the page harder for Google's crawler to understand the page".
  • A missing Allow: /wp-admin/admin-ajax.php. Keep it whenever you block /wp-admin/.
  • Disallow: /wp-json/ on a site whose pages load content from the REST API. Yoast's "Remove WP-JSON API" setting adds this rule. Turn it on only if no public page fetches content from /wp-json/ in the browser.
  • Blocking pages you want out of Google. Google says robots.txt "is not a mechanism for keeping a web page out of Google". Use a noindex tag and leave the page crawlable. A blocked page can show up as Indexed, though blocked by robots.txt.
  • Blocking WooCommerce's cart, checkout and My account pages. WooCommerce already adds noindex to all three, and a robots.txt block hides that tag from Google.
  • Disallow: / under User-agent: *. One line blocks the whole site. Check for it after every move from staging to live.
  • Crawl-delay. Google doesn't support it. Anthropic documents support for it in ClaudeBot.

If Search Console already lists pages as Blocked by robots.txt, find the rule that matches them in your live file and remove it if you want those pages indexed.

How to allow or block AI crawlers in WordPress robots.txt

AI companies run separate bots for model training, for their search indexes, and for fetching a page a user asked about. Each vendor publishes the token to put in robots.txt:

User-agent tokenCompanyWhat it doesIf you disallow it
GPTBotOpenAICrawls content that may be used to train OpenAI's modelsSignals that your content shouldn't be used for training
OAI-SearchBotOpenAISurfaces sites in ChatGPT search resultsYour pages aren't shown in ChatGPT search answers, apart from navigational links
ChatGPT-UserOpenAIFetches pages for actions a ChatGPT user startsOpenAI says robots.txt rules "may not apply"
ClaudeBotAnthropicCollects content that may be used to train ClaudeSignals that your future content should be excluded from training
Claude-SearchBotAnthropicIndexes content for Claude's search resultsLower visibility in Claude's search results
Claude-UserAnthropicFetches pages when a user asks Claude about themClaude can't fetch your pages for its users
PerplexityBotPerplexitySurfaces and links sites in Perplexity search resultsPerplexity recommends allowing it so your pages can appear in its results
Perplexity-UserPerplexityFetches pages for a Perplexity user's requestPerplexity says it "generally ignores robots.txt rules"
Google-ExtendedGoogleControls use of your content for Gemini training and groundingNo effect on Google Search: Google says it isn't a ranking signal and doesn't affect inclusion

Google-Extended isn't a separate crawler. Google crawls with its normal user agents and reads the token as a setting.

A crawler follows only the most specific group that names it. A User-agent: GPTBot group means GPTBot skips everything under User-agent: *. Bots with no group of their own follow your * rules, so you don't need to list the AI search bots to allow them.

To block AI training and keep AI search, give only the training bots their own groups:

ts
CopyDownload
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://example.com/wp-sitemap.xml

OpenAI says "Each setting is independent of the others", so a GPTBot block leaves OAI-SearchBot and ChatGPT search as they were. To leave AI search too, add the same Disallow: / group for OAI-SearchBot, Claude-SearchBot and PerplexityBot; ChatGPT-User and Perplexity-User may still fetch pages for users, and only a firewall rule stops them. To keep one bot out of part of the site, copy the * rules into its group and add the extra path. OpenAI and Perplexity both say robots.txt changes can take up to about 24 hours to reach their systems.

The Yoast Premium switches above and All in One SEO's Block AI Crawlers setting write the same groups for you. If your site runs behind Cloudflare with its managed robots.txt turned on, Cloudflare puts its own rules above your WordPress file, so the live file can block AI bots your WordPress settings allow. For everything else that decides whether ChatGPT and Perplexity cite you, read how to get pages indexed by ChatGPT and AI search.

robots.txt is a request. A security plugin, your host's firewall or a CDN bot rule can still answer GPTBot or ClaudeBot with a 403 while your robots.txt allows them. The crawler simulator fetches any URL with the user agent of the bot you pick and shows the status code and HTML it gets back.

How to add your sitemap to WordPress robots.txt

In a physical file or in Rank Math's editor, you write the Sitemap line yourself. List the sitemap your site actually generates:

Sitemap made byURL to list
WordPress core (5.5 and later)https://example.com/wp-sitemap.xml, or https://example.com/?sitemap=index with plain permalinks
Yoast SEOhttps://example.com/sitemap_index.xml
Rank Mathhttps://example.com/sitemap_index.xml
All in One SEOhttps://example.com/sitemap.xml

Google requires a "fully-qualified URL", with the same protocol and host your site uses. The line can go anywhere in the file, applies to every crawler, and can repeat for a second sitemap such as a news sitemap. Submit the same URL under Indexing > Sitemaps in Search Console as well.

How to test your WordPress robots.txt

  1. Open the live file. Run curl -i https://yoursite.com/robots.txt, or load the URL in a private window. It should return 200 and your latest rules. If it shows old rules, look for a physical file first, then purge your caching plugin, your host's cache and your CDN.
  2. Check the status code. Google treats a 404 robots.txt as no rules at all. A server error is worse: for the first 12 hours, Google stops crawling the site while it retries the file.
  3. Check each crawler. The free robots.txt checker reads your live file and gives Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and other crawlers a verdict: Allowed, Blocked, Partial or Default. A WordPress default file shows Partial, because /wp-admin/ is blocked.
  4. Check Google's copy. In Search Console, open Settings and then the robots.txt report. It lists the robots.txt files Google found, when it last checked them and any parse errors. Google caches robots.txt for up to 24 hours. After unblocking important pages or fixing an error, click the more icon next to the file and choose Request a recrawl.
  5. Check single URLs. URL Inspection shows "Crawl allowed?" for any page, and Test live URL checks the page against your current rules.

FAQ

Where is the robots.txt file in WordPress? On a fresh install it isn't on the server. WordPress generates a virtual robots.txt each time someone requests yoursite.com/robots.txt. A real robots.txt file in the web root, if one exists, replaces it.

What is the difference between WordPress's virtual robots.txt and a physical robots.txt file? WordPress builds the virtual file on each request, and SEO plugins edit it. A physical file is a text file in the web root that the server returns before WordPress runs, so while it exists, plugin editors, plugin crawl settings and the automatic sitemap line have no effect.

Why are my robots.txt changes not showing up in WordPress or Google Search Console? A physical robots.txt overrides every plugin editor, a cache or CDN can serve the old file until you purge it, and Google caches robots.txt for up to 24 hours. After an urgent fix, use Request a recrawl in Search Console's robots.txt report.

Does "Discourage search engines from indexing this site" change robots.txt? Not the rules. Since WordPress 5.3 the setting no longer adds Disallow: /. It adds a noindex, nofollow robots meta tag to every page and removes the Sitemap line from the virtual robots.txt.

Can I accidentally block all search engines with robots.txt in WordPress? Yes. One Disallow: / under User-agent: * asks every crawler to skip the whole site. Check yoursite.com/robots.txt after every launch, migration or plugin change.

How do I restore the default WordPress robots.txt? Delete or rename the physical robots.txt in your web root. Then empty the Rank Math robots.txt field, switch off Enable Custom Robots.txt in All in One SEO, or empty the Robots.txt Content field of WP Robots Txt.

How do I check if robots.txt blocks Google or AI chatbots? Open yoursite.com/robots.txt, run it through a robots.txt analyzer, and use URL Inspection in Search Console for Google. For AI bots, also fetch a page with the bot's user agent, since a firewall can refuse a bot that robots.txt allows.

Should I block GPTBot on my WordPress site? Block GPTBot if you don't want OpenAI to use your content for training. ChatGPT search uses OAI-SearchBot, and OpenAI says each setting is independent, so keep OAI-SearchBot allowed if you want to be cited in ChatGPT answers.

Google
Gemini
ChatGPT
Claude
Grok
Meta
Perplexity

Get your store ranked in AI

See how ChatGPT, Gemini, Claude and Perplexity recommend your products and which stores they pick instead.

Talk to sales
Avatar
How can we help?
Get instant answers to your questions or leave a message for an engineer will reach out
Ask AI about Encited
See our docs
Contact support
Leave a message
We'll get back to you soon
Avatar
Ask AI about Encited
Product questions, troubleshooting and
Thinking
Preview
Drop an image to attach
Powered by ReplyMaven