Encited

Write a robots.txt that lets crawlers in

robots.txt is the first file a crawler asks for. If it is missing, most crawlers carry on anyway. If it is wrong, they stop, and nothing you do to the pages afterwards matters.

This prompt writes a permissive robots.txt, points it at your sitemap, and keeps dashboard and API paths out of the index.

When you need this#

  • You have no robots.txt, or it returns your app's HTML instead of plain text.
  • Search Console reports pages as "Blocked by robots.txt".
  • Your dashboard, settings, or API routes are showing up in search results.
  • You just generated a sitemap and nothing references it.

Check first with the Robots.txt Analyzer, which shows which crawlers your current file allows or blocks, including GPTBot, ClaudeBot, and PerplexityBot.

The prompt#

Create a robots.txt file in the public folder that:
1. Allows all user agents to crawl the site
2. Points to my sitemap at https://YOURDOMAIN.com/sitemap.xml (replace YOURDOMAIN with my actual domain)
3. Blocks any sensitive paths like /api/, /dashboard/, /settings/ from being crawled

If a robots.txt already exists, update it to include these rules without removing any existing valid rules.

Replace YOURDOMAIN.com before pasting, or the builder will write the placeholder into the file.

What it changes#

Creates or edits public/robots.txt with an allow-all rule, a Sitemap: line, and Disallow entries for private paths. Existing valid rules are kept rather than overwritten.

Two things to watch#

Blocking is not hiding. Disallow asks crawlers not to fetch a path. It does not remove a page already in the index, and a blocked page can still rank from inbound links alone. Use a noindex meta tag for pages that must leave the index, and note that a crawler has to fetch the page to see it, so do not block and noindex the same path.

AI crawlers are separate. GPTBot, ClaudeBot, PerplexityBot, and the rest read their own rules. An allow-all directive covers them, but if you have inherited a file with per-agent blocks, check them individually. Your crawl log shows which ones are reaching you now.

Alternatives#

  • Write the four lines yourself. It is a short file, and hand-writing it means you know exactly what it says.
  • Do nothing, for now. A missing robots.txt is not a blocker; crawlers assume they may crawl. Fix it when you have a sitemap worth pointing at.
  • Let your host serve it. Some platforms generate one for you. Check what your domain returns at /robots.txt before adding a second.