How to Test JavaScript SEO: Six Checks and the Tools for Each
A JavaScript SEO audit in six checks: curl with a bot user agent, the crawler simulator, URL Inspection, a rendered vs raw diff, 404 status and the crawl log.
TL;DR. A JavaScript SEO audit asks one question per page template: does each crawler get the text, the tags and the links that people see? Six checks answer it, from a curl fetch as Googlebot to the crawl log, each with the command or tool, what to read in the output, and what a pass looks like. Run them on one URL per template; a template fix covers every URL built from it.
Every JavaScript SEO problem shows up as a gap between what a person sees and what a crawler receives, and six checks find that gap with free JavaScript SEO tools: curl, the Encited crawler simulators, Search Console's URL Inspection, a diff and your crawl log. Run them in order on a product page, a category page and one guide; together they are how to test JavaScript SEO on a template. The problems the checks find, and the fixes, are in our guide to JavaScript SEO and in SPA SEO.
1. Test JavaScript SEO with curl: fetch as Googlebot, fetch as a browser, diff
Two fetches of the same URL, one with a browser user agent and one with Googlebot's, show what your server sends before any JavaScript runs and whether it answers bots differently from people.
URL="https://yoursite.com/products/trail-runner-2"TEXT="a sentence you can read on the page"curl -sL -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" "$URL" > browser.htmlcurl -sL -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "$URL" > googlebot.htmlwc -c browser.html googlebot.htmlgrep -c "$TEXT" browser.html googlebot.htmldiff browser.html googlebot.html | head -20
What to look for. The byte count and the sentence count per file, and the first lines of the diff.
What a pass looks like. Both files are large and the sentence appears in both. The server renders the page, so every crawler gets the content on the first request.
Two other results are common. Both files are a few kilobytes and the sentence count is 0: the server sends an empty shell to everyone, Google has to render it, and AI crawlers get nothing. The browser file is small and the Googlebot file is large: a pre-render layer answers bots with a snapshot. Repeat the second curl with -A "GPTBot" to confirm the layer covers AI crawlers.
These requests come from your own IP address. A firewall that checks Googlebot requests against Google's published ranges can answer yours with a challenge page, so a 403 or a CAPTCHA means "test from somewhere else", which is the next step.
2. Run the crawler simulator and the other free JavaScript SEO tools
The Googlebot simulator requests your URL as any of 14 Google crawlers and shows the status code, redirects, response headers and raw HTML that came back. The ChatGPT crawler simulator does the same as GPTBot, ChatGPT-User and OAI-SearchBot, and the full crawler simulator covers two dozen user agents, ClaudeBot, PerplexityBot and Bingbot among them.
What to look for. The status code first, then the title and a sentence of body text in the HTML view. Run each URL as Googlebot Smartphone and as GPTBot at minimum.
What a pass looks like. 200 for both, no redirect chain, and the title and body text in the HTML. GPTBot, ClaudeBot and PerplexityBot do not run JavaScript, so the raw HTML view is the whole page for them; do AI crawlers render JavaScript? has the per-crawler detail. A 403 or a challenge page here points at a bot rule in your CDN or firewall. The AI crawl checker checks robots.txt for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Googlebot and Bingbot, looks for firewall challenge pages, then fetches the page as GPTBot and looks for text without JavaScript. The robots.txt analyzer lists what each crawler may fetch and catches a Disallow left over from a redesign.
A simulator checks one URL. The site audit runs the same raw-HTML checks across every route it crawls and groups the failures by template, so a shell-only body on 400 product pages is one row you fix once.
3. Test the live URL in Search Console's URL Inspection
URL Inspection is the only place that shows the result of Google's second, rendering pass for your page. Paste the URL into the inspection bar, click "Test live URL", then "View tested page".
What to look for. The HTML tab holds the rendered HTML Google produced; search it for the sentence from step 1, the canonical and the robots meta tag. The Screenshot tab shows whether the page drew at all. The "More info" tab lists page resources that failed to load and JavaScript console messages. Then compare "User-declared canonical" with "Google-selected canonical".
What a pass looks like. The sentence, the canonical and the title are in the HTML tab, the screenshot shows the page people see, and no script, stylesheet or API call is listed as failed. A blank screenshot means the render failed, timed out or waited for a click; a resource blocked by robots.txt is the usual cause.
The live test renders your page on demand. Normal crawling puts the page in a render queue first. Vercel and MERJ measured that wait on nextjs.org in April 2024 and published the numbers on July 31, 2024: a median of 10 seconds from crawl to render, and about 18 hours for the slowest 1%. The queue and what Google waits for are covered in how Google renders JavaScript.
4. Diff the rendered HTML against the raw HTML
The raw HTML is what every crawler receives; the rendered HTML is what Google indexes after its second pass. Everything in the second file and missing from the first depends on JavaScript, so it is invisible to AI crawlers and late in Google.
How to get both. The raw file is googlebot.html from step 1. For the rendered file, use headless Chrome, or save the HTML tab from step 3:
# macOS"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --headless=new --dump-dom "$URL" > rendered.html# Linuxgoogle-chrome --headless=new --dump-dom "$URL" > rendered.html
What to compare. Pull the same items from both files:
for f in googlebot.html rendered.html; doecho "== $f"grep -o "<title>[^<]*" "$f"grep -o '<link rel="canonical"[^>]*' "$f"grep -o '<meta name="robots"[^>]*' "$f"grep -c 'application/ld+json' "$f"grep -o 'href="/[^"]*"' "$f" | sort -u | wc -ldone
What a pass looks like. The title, canonical, robots tag and JSON-LD count are identical in both files, and the internal link count is close. An item present only in rendered.html is set by JavaScript, so AI crawlers never see it. A link count much higher in the rendered file means your navigation or product grid is built in the browser, and crawlers that do not render find no path to those pages. A canonical or robots tag that differs between the files is the worst case, because Google reads both versions and can act on either one.
Per page, the free SEO audit runs a "Content without JavaScript" check among its 30 checks, and the meta tag audit reads title, description, canonical and Open Graph tags from the raw HTML and flags a page that looks like a SPA shell.
5. Check the status code of a URL that does not exist
A single-page app's server returns the same HTML file, with a 200 status, for every path, and the app draws a "not found" screen afterwards. Google calls that a soft 404, and it can index the junk URL. Request a route that cannot exist and read the status:
curl -sL -o /dev/null -w "%{http_code} %{url_effective}\n" \-A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \"https://yoursite.com/this-route-does-not-exist"
Run it on a made-up path, on a product you deleted last month, and on a real product URL with a few characters changed.
What to look for. The first number on the line, and the final URL after redirects.
What a pass looks like. 404 for all three, or 410 for the deleted product, and the final URL is the one you requested. A 200 is a soft 404, and so is a redirect to the homepage; Search Console's Page indexing report lists both under "Soft 404". Google's JavaScript SEO basics ask for meaningful HTTP status codes and say a 404 keeps the page out of the index.
The fix is a real 404 from the server or the edge for every unknown route. On a client-rendered app behind Encited's pre-render layer, the not-found route sets <meta name="prerender-status-code" content="404"> and crawlers receive a 404 status while people still see the app's own not-found screen. The same layer serves every crawler on its list the rendered snapshot from steps 1 and 2.
6. Read the crawl log: the last step of a JavaScript SEO audit
Steps 1 to 5 are requests you made. A crawl log records the requests the real bots made, from their own IP addresses, with the status and the body they received, so it settles any doubt a firewall or a CDN rule left in step 1.
Where to look. If your site runs through Encited, the crawl log lists every known crawler visit with the bot, the URL, the status code the bot received, the outcome (served from cache, freshly rendered, passed through to your origin, or redirected), the response time and the stored snapshot it was served. Filter by crawler, status and outcome. Without Encited, filter your server or CDN logs by user agent, or read Search Console's Crawl stats for Googlebot alone.
What to look for. For each template, a row for Googlebot and a row for an AI crawler on a page you changed, and the row for the junk route from step 5.
What a pass looks like. Googlebot and the AI crawlers get 200 and a snapshot on the pages you want indexed, the snapshot contains the sentence from step 1, the junk route shows a 404, and no bot you care about gets a 403 or a challenge page. A bot "passed through" to the origin on a client-rendered page received the shell. An AI crawler that never appears in the log is blocked by robots.txt or the firewall before it reaches you, which the AI crawl checker in step 2 confirms.
Run the six checks again after every deploy that touches routing, the document head or the rendering setup; the first two take a minute.
FAQ
What are the best JavaScript SEO tools?
For one page: curl, Encited's crawler simulators, Search Console's URL Inspection and Chrome's headless --dump-dom. For a whole site: a crawler that fetches raw HTML across every route, such as the Encited site audit, Screaming Frog or Sitebulb. For proof of what bots received: your crawl log.
How do I know if Googlebot can render my page? Run URL Inspection's live test and open "View tested page". If the HTML tab contains your body text and the screenshot shows the page, the render works. If the screenshot is blank, open "More info" and read the failed resources and console messages.
Why does curl show an empty page when the page works in my browser? Your browser runs the JavaScript that builds the page. curl does not, and neither do GPTBot, ClaudeBot or PerplexityBot. The curl output is what those crawlers read.
Does the crawler simulator run JavaScript? No. It shows the status code, headers and raw HTML the chosen user agent receives from your server, which is what non-rendering crawlers get and what Google sees before its render queue. For Google's rendered result, use URL Inspection.
What is the difference between "View crawled page" and "Test live URL"? "View crawled page" shows the HTML from Google's last real crawl, the version in the index. "Test live URL" fetches and renders the page now, so you can see whether a fix worked before Google comes back.
