Encited

How to Integrate Pre-rendering with Fastly VCL

How Fastly pre-rendering works#

Fastly can classify crawler requests in custom VCL and send only eligible public HTML requests to Encited before the request reaches your origin.

Crawler GET requests for HTML are routed to Encited. Browser traffic, assets, non-GET requests, and failed render calls restart against your existing Fastly origin.

Prerequisites#

  • A Fastly service using custom VCL with permission to clone and activate service versions.
  • The generated Fastly backend name that currently serves your website.
  • An Encited API key that can be stored in the private VCL configuration.

Set up pre-rendering with Fastly#

1. Clone the active Fastly service version#

Create an editable version so the current production configuration remains available for rollback.

2. Create the custom VCL#

Start from Fastly's custom VCL boilerplate, paste the code below, and replace F_your_origin_backend with the generated backend name that currently serves your website.

encited-prerender

# Encited middleware for Fastly
# Replace F_your_origin_backend with the backend name from Fastly's generated VCL.

backend Encited_Host {
  .between_bytes_timeout = 10s;
  .connect_timeout = 1s;
  .dynamic = true;
  .first_byte_timeout = 25s;
  .host = "encited.com";
  .max_connections = 200;
  .port = "443";
  .ssl = true;
  .ssl_cert_hostname = "encited.com";
  .ssl_check_cert = always;
  .ssl_sni_hostname = "encited.com";
}

sub vcl_recv {
  # A failed or non-rendered Encited response restarts against your origin.
  if (req.restarts > 0 && req.http.X-Encited-Fallback == "1") {
    set req.backend = F_your_origin_backend;
    set req.url = req.http.X-Encited-Original-URL;
    set req.http.host = req.http.X-Encited-Original-Host;
    unset req.http.X-Encited-Request;
    unset req.http.X-Encited-Fallback;
    unset req.http.X-Encited-Original-URL;
    unset req.http.X-Encited-Original-Host;
    return(lookup);
  }

  if (
    req.method == "GET" &&
    (req.http.Accept == "" || req.http.Accept ~ "(?i)text/html|\*/\*") &&
    req.http.User-Agent ~ "(?i)(?:(OAI-SearchBot|ChatGPT-User(?:\/\d+\.\d+)?|GPTBot|ChatGPT|OpenAI-))|(?:(anthropic-ai|ClaudeBot|claude-web|Claude-Web|Claude-User|Claude-SearchBot))|(?:\b(GrokBot|xAI-Grok|xAI-Bot|xAI-SearchBot)\b)|(?:(PerplexityBot|Perplexity-User))|(?:(Google-InspectionTool|GoogleOther(?:-Image|-Video)?|Googlebot|Google-CloudVertexBot|Storebot-Google|Google-NotebookLM|GoogleAgent-URLContext|GoogleAIOverviewRenderer|Google-Lens|Google-Jetski-Antigravity))|(?:(Microsoft-Copilot|CopilotBot|Copilot-Chat|Microsoft-CopilotPlugin))|(?:(BingBot|bingbot|BingPreview))|(?:facebookexternalhit.*Twitterbot|Twitterbot.*facebookexternalhit)|(?:(Meta-ExternalAgent|meta-externalagent|meta-externalfetcher|MetaBot|FacebookBot|facebookexternalhit))|(?:LinkedInBot)|(?:Amazonbot)|(?:(Applebot-Extended|Applebot|AppleNewsBot))|(?:Bytespider)|(?:(DuckAssistBot|DuckDuckBot))|(?:(cohere-training-data-crawler|cohere-ai|CohereAI))|(?:MistralAI-User)|(?:AI2Bot)|(?:CCBot)|(?:Diffbot)|(?:omgili)|(?:TimpiBot)|(?:YouBot)|(?:ExaSearchBot)|(?:(Bravebot|Brave-Search))|(?:Yandex)|(?:Twitterbot)|(?:Discordbot)|(?:(Slackbot|Slack-ImgProxy))|(?:TelegramBot)|(?:(Pinterest|Pinterestbot))|(?:Snapchat)|(?:Redditbot)|(?:Tumblr)|(?:(Mastodon|http\.rb))|(?:Viber)|(?:^Line\/)|(?:WhatsApp)|(?:(SkypeUriPreview|Skype))|(?:(Teams|MicrosoftPreview))|(?:Zoom)|(?:Notion)|(?:(Quora|QuoraBot|Quora-Bot|Poe-Bot))|(?:Medium)|(?:(Pocket|PocketParser|PocketImageCache))|(?:(Flipboard|FlipboardProxy))|(?:(Embedly|embed\.ly))|(?:YelpBot)|(?:(Baiduspider|Baidu))|(?:(Sogou|sogou))|(?:(Yeti|NaverBot|Naver))|(?:SeznamBot)|(?:(Qwantify|Qwant))|(?:Ecosia)|(?:MojeekBot)|(?:Mail\.RU_Bot)|(?:(Yahoo! Slurp|Y!J-))|(?:(SemrushBot|SEMrush))|(?:(AhrefsBot|AhrefsSiteAudit))|(?:(DotBot|Rogerbot|moz\.com))|(?:MJ12bot)|(?:Screaming Frog)|(?:SISTRIX)|(?:SEOkicks)|(?:serpstatbot)|(?:RSiteAuditor)|(?:(?:Encited|LvHTML)-SEOAuditBot)|(?:Barkrowler)|(?:HubSpot Crawler)|(?:(AwarioSmartBot|AwarioBot))|(?:DataForSeoBot)|(?:SERanking)|(?:Seobility)" &&
    req.url.ext !~ "(?i)js|css|xml|txt|png|jpg|jpeg|gif|pdf|ico|woff|woff2|ttf|svg|webmanifest"
  ) {
    set req.http.X-Encited-Request = "1";
    set req.http.X-Encited-Original-URL = req.url;
    set req.http.X-Encited-Original-Host = req.http.host;
    set req.url = "/api/prerender/render?url=" + urlencode("https://" + req.http.host + req.url);
    set req.backend = Encited_Host;
    set req.http.host = "encited.com";
    # Replace YOUR_API_KEY with your Encited API key (Settings > API keys).
    set req.http.x-lovablehtml-api-key = "YOUR_API_KEY";
    set req.http.Accept = "text/html";
    unset req.http.Cookie;
    unset req.http.Authorization;
    return(pass);
  }

  # Preserve Fastly-generated configuration and snippets.
  #FASTLY recv
}

sub vcl_hash {
  #FASTLY hash
}

sub vcl_hit {
  #FASTLY hit
}

sub vcl_miss {
  #FASTLY miss
}

sub vcl_pass {
  #FASTLY pass
}

sub vcl_fetch {
  if (req.http.X-Encited-Request == "1") {
    if (
      (beresp.status != 200 && beresp.status != 301) ||
      (beresp.status == 200 && beresp.http.Content-Type !~ "(?i)text/html")
    ) {
      # Pass the fallback signal to vcl_deliver. Restarting there guarantees
      # request state survives Fastly clustering.
      set beresp.http.X-Encited-Fallback = "1";
    }
    set beresp.cacheable = false;
    return(deliver);
  }

  #FASTLY fetch
}

sub vcl_error {
  #FASTLY error
}

sub vcl_deliver {
  if (
    req.http.X-Encited-Request == "1" &&
    resp.http.X-Encited-Fallback == "1" &&
    req.restarts == 0
  ) {
    set req.http.X-Encited-Fallback = "1";
    unset resp.http.X-Encited-Fallback;
    restart;
  }

  unset resp.http.X-Encited-Fallback;
  unset resp.http.X-Encited-Request;
  unset resp.http.X-Encited-Original-URL;
  unset resp.http.X-Encited-Original-Host;

  #FASTLY deliver
}

sub vcl_log {
  #FASTLY log
}

3. Validate and activate the service version#

Validate the VCL, activate the new version, and allow a minute for it to propagate across Fastly.

fastly service version validate --service-id YOUR_SERVICE_ID --version latest

Test the integration#

1. Send a crawler request

curl -sS -D - -o /dev/null \
  -A "Googlebot" \
  -H "Accept: text/html" \
  "https://your-domain.com/"

Look for a successful HTML response and the x-lovablehtml-render-cache header.

2. Confirm browser passthrough

curl -sS -D - -o /dev/null \
  -A "Mozilla/5.0" \
  -H "Accept: text/html" \
  -H "Accept-Language: en-US" \
  -H "Sec-Fetch-Mode: navigate" \
  -H "Sec-Fetch-Dest: document" \
  -H "Upgrade-Insecure-Requests: 1" \
  "https://your-domain.com/"

This request should be served by your existing website without an Encited render-cache header.

Common errors#

Fastly reports an unknown origin backend

Replace F_your_origin_backend with the generated backend name from your active VCL before validating the service version.

The new VCL validates but crawler requests still reach the origin

Confirm the edited service version is active and that the API key placeholder was replaced before activation.

Request and security behavior#

  • Only eligible GET requests for public HTML pages are considered for pre-rendering.
  • Along with the public page URL, only the API key and crawler-classification headers are sent to Encited.
  • Cookie and Authorization headers are never forwarded to Encited.
  • If Encited is unavailable or does not return rendered HTML, the request falls through to your website.

If you run the Next-Gen WAF or edge rate limiting, those rules block us before this VCL helps. Set a render token and add the Fastly rule so renders and audit crawls get through.

Common questions#

How does pre-rendering work with Fastly VCL?#

Custom VCL classifies crawler HTML requests at the edge, calls Encited for rendered HTML, and keeps all other requests on the existing backend.

Does the Fastly pre-rendering integration fail open?#

Yes. An unavailable Encited endpoint, a non-render response, or an ineligible request restarts against your original Fastly backend.

How do I test crawler pre-rendering on Fastly?#

Send a GET request with a Googlebot user agent and Accept: text/html, then check for the x-lovablehtml-render-cache response header.