Encited

How to Integrate Pre-rendering with Koa

How Koa pre-rendering works#

Koa.js can place Encited at the front of its middleware stack so crawlers receive rendered HTML before routers and SPA fallbacks run.

The Koa middleware handles eligible crawler HTML requests and calls next() for browser traffic, assets, and any render response that should fall back.

Prerequisites#

  • Node.js 18 or later for the native fetch API.
  • Access to register middleware before static files, authentication, and routers.
  • LOVABLEHTML_API_KEY configured in the production Node.js process.

Set up pre-rendering with Koa#

1. Create the Koa middleware#

Save the snippet as encited-prerender.js. It uses the native fetch API available in Node.js 18 and later.

encited-prerender

// encited-prerender.js - app.use before routes
const CRAWLER = /(?:(OAI-SearchBot|ChatGPT-User(?:\/\d+\.\d+)?|GPTBot|ChatGPT|OpenAI-))|(?:(anthropic-ai|ClaudeBot|claude-web|Claude-Web|Claude-User|Claude-SearchBot))|(?:\b(GrokBot|xAI-Grok|xAI-Bot|xAI-SearchBot)\b)|(?:(PerplexityBot|Perplexity-User))|(?:(Google-InspectionTool|GoogleOther(?:-Image|-Video)?|Googlebot|Google-CloudVertexBot|Storebot-Google|Google-NotebookLM|GoogleAgent-URLContext|GoogleAIOverviewRenderer|Google-Lens|Google-Jetski-Antigravity))|(?:(Microsoft-Copilot|CopilotBot|Copilot-Chat|Microsoft-CopilotPlugin))|(?:(BingBot|bingbot|BingPreview))|(?:facebookexternalhit.*Twitterbot|Twitterbot.*facebookexternalhit)|(?:(Meta-ExternalAgent|meta-externalagent|meta-externalfetcher|MetaBot|FacebookBot|facebookexternalhit))|(?:LinkedInBot)|(?:Amazonbot)|(?:(Applebot-Extended|Applebot|AppleNewsBot))|(?:Bytespider)|(?:(DuckAssistBot|DuckDuckBot))|(?:(cohere-training-data-crawler|cohere-ai|CohereAI))|(?:MistralAI-User)|(?:AI2Bot)|(?:CCBot)|(?:Diffbot)|(?:omgili)|(?:TimpiBot)|(?:YouBot)|(?:ExaSearchBot)|(?:(Bravebot|Brave-Search))|(?:Yandex)|(?:Twitterbot)|(?:Discordbot)|(?:(Slackbot|Slack-ImgProxy))|(?:TelegramBot)|(?:(Pinterest|Pinterestbot))|(?:Snapchat)|(?:Redditbot)|(?:Tumblr)|(?:(Mastodon|http\.rb))|(?:Viber)|(?:^Line\/)|(?:WhatsApp)|(?:(SkypeUriPreview|Skype))|(?:(Teams|MicrosoftPreview))|(?:Zoom)|(?:Notion)|(?:(Quora|QuoraBot|Quora-Bot|Poe-Bot))|(?:Medium)|(?:(Pocket|PocketParser|PocketImageCache))|(?:(Flipboard|FlipboardProxy))|(?:(Embedly|embed\.ly))|(?:YelpBot)|(?:(Baiduspider|Baidu))|(?:(Sogou|sogou))|(?:(Yeti|NaverBot|Naver))|(?:SeznamBot)|(?:(Qwantify|Qwant))|(?:Ecosia)|(?:MojeekBot)|(?:Mail\.RU_Bot)|(?:(Yahoo! Slurp|Y!J-))|(?:(SemrushBot|SEMrush))|(?:(AhrefsBot|AhrefsSiteAudit))|(?:(DotBot|Rogerbot|moz\.com))|(?:MJ12bot)|(?:Screaming Frog)|(?:SISTRIX)|(?:SEOkicks)|(?:serpstatbot)|(?:RSiteAuditor)|(?:(?:Encited|LvHTML)-SEOAuditBot)|(?:Barkrowler)|(?:HubSpot Crawler)|(?:(AwarioSmartBot|AwarioBot))|(?:DataForSeoBot)|(?:SERanking)|(?:Seobility)/i;
const ENCITED_FORWARD_HEADERS = [
  'accept-language', 'sec-fetch-mode', 'sec-fetch-site', 'sec-fetch-dest',
  'sec-fetch-user', 'upgrade-insecure-requests', 'referer', 'user-agent',
];
const ENCITED_RESPONSE_HEADERS = [
  'x-lovablehtml-render-cache', 'x-lovablehtml-snapshot-key', 'cache-control',
  'etag', 'content-digest', 'signature-input', 'signature',
];

export async function encitedPrerender(ctx, next) {
  const accept = ctx.get('accept') || '';
  const userAgent = ctx.get('user-agent') || '';
  const isHtml = !accept || accept === '*/*' || accept.includes('text/html');
  if (ctx.method !== 'GET' || !isHtml || !CRAWLER.test(userAgent)) return next();

  try {
    const headers = {
      // Set the LOVABLEHTML_API_KEY environment variable in your deployment.
      'x-lovablehtml-api-key': process.env.LOVABLEHTML_API_KEY,
      accept: 'text/html',
    };
    for (const name of ENCITED_FORWARD_HEADERS) {
      const value = ctx.get(name);
      if (value) headers[name] = value;
    }
    const rendered = await fetch(
      'https://encited.com/api/prerender/render?url=' + encodeURIComponent(ctx.href),
      { headers, redirect: 'manual' },
    );
    const location = rendered.headers.get('location');
    if (rendered.status === 301 && location) {
      ctx.status = 301;
      ctx.redirect(location);
      return;
    }
    if (rendered.status === 304) return next();
    if (rendered.status === 200 && (rendered.headers.get('content-type') || '').includes('text/html')) {
      ctx.status = 200;
      ctx.type = 'html';
      for (const name of ENCITED_RESPONSE_HEADERS) {
        const value = rendered.headers.get(name);
        if (value) ctx.set(name, value);
      }
      ctx.body = Buffer.from(await rendered.arrayBuffer());
      return;
    }
  } catch {
    // Fail open.
  }
  return next();
}

2. Register it first#

Call app.use(encitedPrerender) before static files, authentication, and router middleware.

3. Deploy or restart Koa#

Restart the Node.js process.

npm start

Test the integration#

1. Send a crawler request

curl -sS -D - -o /dev/null \
  -A "Googlebot" \
  -H "Accept: text/html" \
  "https://your-domain.com/"

Look for a successful HTML response and the x-lovablehtml-render-cache header.

2. Confirm browser passthrough

curl -sS -D - -o /dev/null \
  -A "Mozilla/5.0" \
  -H "Accept: text/html" \
  -H "Accept-Language: en-US" \
  -H "Sec-Fetch-Mode: navigate" \
  -H "Sec-Fetch-Dest: document" \
  -H "Upgrade-Insecure-Requests: 1" \
  "https://your-domain.com/"

This request should be served by your existing website without an Encited render-cache header.

Common errors#

The Koa router responds before Encited

Register encitedPrerender with app.use before static middleware, authentication, and every router.

The Koa process cannot authenticate the render request

Set LOVABLEHTML_API_KEY in the production process environment and restart every Koa instance.

Request and security behavior#

  • Only eligible GET requests for public HTML pages are considered for pre-rendering.
  • Along with the public page URL, only the API key and crawler-classification headers are sent to Encited.
  • Cookie and Authorization headers are never forwarded to Encited.
  • If Encited is unavailable or does not return rendered HTML, the request falls through to your website.

If your site sits behind a firewall, bot protection, or rate limiting, those rules block us before this middleware helps. Set a render token and add one firewall rule so renders and audit crawls get through.

Common questions#

How does pre-rendering work with Koa.js middleware?#

The first Koa middleware detects eligible crawlers, returns Encited rendered HTML, and awaits next() for normal application traffic.

Where should Koa.js pre-rendering middleware be registered?#

Register it before static files, sessions, authentication, routers, and the SPA fallback.

How do I test crawler pre-rendering in Koa.js?#

Send a Googlebot request with Accept: text/html and confirm the response contains x-lovablehtml-render-cache.