Dynamic bot rendering and bot routing: setup guide

Dynamic bot rendering, also called bot routing, sends crawlers a rendered HTML snapshot of each page while people get your JavaScript app as usual. This guide covers the routing rule itself: which user agents to route, what to leave alone, how to cache it safely, and how to test it. For the concept, see What is dynamic rendering.

If you use Encited, the setup guide for your host already includes a working rule. Use this page to understand what that rule does or to adjust it.

How the routing rule works#

Every request goes through one check before it reaches your site:

1. Read the User-Agent header.
2. If it matches a crawler, return the rendered snapshot for that URL.
3. Otherwise, pass the request to your host unchanged.

const CRAWLERS =
  /googlebot|google-inspectiontool|bingbot|applebot|duckduckbot|gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexitybot|perplexity-user|facebookexternalhit|twitterbot|linkedinbot|slackbot|discordbot|whatsapp|telegrambot/i;

export function isCrawler(request) {
  const ua = request.headers.get("user-agent") ?? "";
  return CRAWLERS.test(ua);
}

Which user agents to route#

Check your logs every few months for crawlers that aren't on the list. Any crawler missing from the rule gets your app shell.

Google-Extended doesn't belong here. It's a robots.txt token for Gemini training, and no crawler uses it as a user agent.

What not to route#

Only route page requests. Pass these through untouched:

• Static assets: .js, .css, images, fonts, video.
• API routes and anything under paths like /api/.
• Files that are already plain HTML or text: robots.txt, sitemap.xml, /.well-known/*.
• Logged-in areas, checkout and account pages.

A simple way to do this is to route only GET requests whose path has no file extension, plus any paths you list explicitly.

Verify that crawlers are real#

Anyone can send a request with a Googlebot user agent. If your snapshots contain nothing that differs from the public page, a fake crawler gets nothing special, so most sites don't need verification. If you want it anyway:

• Google and Bing: do a reverse DNS lookup on the IP, check that the hostname ends in googlebot.com, google.com or search.msn.com, then do a forward lookup on that hostname and confirm it returns the same IP.
• Published IP ranges: Google and OpenAI publish the IP ranges their crawlers use. Check each provider's crawler documentation for the current files.

Caching#

The same URL now has two versions, so caches need to keep them apart:

• Send Vary: User-Agent on routed responses, or put a crawler flag in your CDN cache key.
• Never cache the snapshot under the same key people use. If a CDN serves the snapshot to visitors, they get a page with no interactivity.
• Keep snapshot freshness separate from your CDN cache. Refresh snapshots when content changes, and on a schedule for sections that change often.

Status codes and redirects#

Return the status the page should have:

• A route that shows "not found" in the app should return 404 to crawlers.
• A redirect the app performs in the browser should be a 301 or 302 for crawlers.
• If the origin is down or returns an error, don't save that response as the snapshot.

Single-page apps return 200 for every URL by default, so this needs a signal from the page, such as a meta tag the app sets on its not-found route. With Encited, set <meta name="prerender-status-code" content="404"> on that route and crawlers get a 404.

Setup by host#

Encited setup guides include the rule above, adapted for each host:

• Cloudflare Workers
• Vercel
• Netlify Edge Functions
• AWS CloudFront with Lambda@Edge
• nginx
• Express
• No-code setup via DNS

For nginx, the matching part of the rule looks like this:

map $http_user_agent $is_crawler {
    default 0;
    "~*(googlebot|google-inspectiontool|bingbot|applebot|gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|perplexitybot|perplexity-user|facebookexternalhit|twitterbot|linkedinbot|slackbot)" 1;
}

Test the setup#

1. Compare responses. Fetch the same page as a crawler and as a browser:

   curl -s -A "Googlebot" https://yoursite.com/page | wc -c
   curl -s -A "Mozilla/5.0" https://yoursite.com/page | wc -c
   

   The crawler response should contain your page text and usually be larger than the browser response, which is the app shell.

2. Check status codes. Request a URL that doesn't exist with a crawler user agent and confirm it returns 404.

3. Check Search Console. Use URL Inspection on a page and confirm the HTML Googlebot fetched includes your content.

4. Check assets. Confirm .js and .css files return the same response for both user agents.

5. Use the crawler simulator to see what each crawler receives.

Troubleshooting#