JavaScript and AI crawlers: why server-rendered HTML matters

Most AI crawlers read only the HTML your server sends and never run your JavaScript. Vercel and MERJ, measuring crawler traffic across Vercel's network in late 2024, found that none of the major AI crawlers rendered JavaScript, so text, prices or tables that a script adds after the page loads never reach them.

Google was the main exception: its Gemini uses Googlebot's infrastructure, which renders JavaScript. Apple's AppleBot renders it too. A page can therefore look complete to Google and incomplete to the crawlers behind ChatGPT, Claude and Perplexity. If the term is new, start with what an AI crawler is.

What the crawlers actually fetch

Vercel and MERJ published "The rise of the AI crawler" on 17 December 2024. Their main data came from nextjs.org and the Vercel network, checked against two job-board sites built on different stacks. In the month they reported, OpenAI's GPTBot made 569 million fetches, Anthropic's crawler 370 million, AppleBot 314 million and PerplexityBot 24.4 million, against 4.5 billion for Googlebot.

Three details matter:

• Scripts are fetched but not run. JavaScript files made up 11.50 percent of ChatGPT's fetches and 23.84 percent of Claude's, yet neither crawler executed them.
• Data in the first response can still count. Vercel and MERJ noted that content in the initial HTML, such as JSON data, may still be indexed.
• Some names were not covered. The study left out Microsoft Copilot, which had no unique user agent to track, and it does not report on Claude-SearchBot, Claude-User or Perplexity-User, which Anthropic and Perplexity now document. Check your own server logs for those.

What that looks like on a real page

On 22 September 2026 we fetched nine AI visibility vendors' pricing pages as HTML, running no JavaScript. The results are in our crawler test of nine pricing pages.

Four gave one entry price that a crawler could read in a single fetch. One, Peec AI, named its plans in the HTML but loaded every price with JavaScript: across three fetches, the digits were never in the server's response. A person in a browser sees the prices. Going by Vercel and MERJ's findings, OpenAI's, Anthropic's and Perplexity's crawlers see plan names without numbers.

When a buyer asks one of those engines what the product costs, it cannot quote the vendor's own page, because it never received the price.

Google renders, and still recommends server rendering

Googlebot queues pages that return a 200 status for rendering and runs their JavaScript in an up-to-date version of Chromium. The page may wait in that queue for a few seconds or longer. Google's guide to its generative AI features says it can process content in JavaScript as long as it is not blocked.

Even so, Google's own JavaScript guidance says server-side rendering or pre-rendering is still a great idea, because it makes a site faster for users and crawlers and because not all bots can run JavaScript. It calls dynamic rendering, serving a separate pre-rendered version only to bots, a workaround rather than a long-term solution. It recommends server-side rendering, static rendering or hydration instead.

What to render on the server

Vercel and MERJ's advice is to server-render the content that matters and leave the rest to the browser. The table below follows their list, with two additions from this guide: prices and structured data.

Server-side rendering, static generation and incremental static regeneration all put the content in the first response. The JSON-LD row matters because markup injected by a tag manager reaches Google but not the crawlers that run no scripts, as covered in schema markup for AI search.

Check any page with curl

1. Save the HTML the server sends:

   curl -s https://www.example.com/pricing -o page.html
   

2. Search it for a fact you can see in the browser, such as a plan name:

   grep -c "Starter" page.html
   

3. List every dollar figure in it:

   grep -o '\$[0-9][0-9,.]*' page.html | sort -u
   

   If the browser shows dollar prices and this prints nothing, the prices most likely arrive by JavaScript.

4. Fetch the same page as an AI crawler, using OpenAI's published GPTBot string, and compare:

   curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
     -o bot.html -w "%{http_code}\n" https://www.example.com/pricing
   grep -o '\$[0-9][0-9,.]*' bot.html | sort -u
   

   A different status code or missing figures means your server or firewall treats that crawler differently.

5. Check for broken links and redirect chains:

   curl -s -L -o /dev/null -w "%{http_code} after %{num_redirects} redirects\n" https://www.example.com/old-page
   

Step 5 matters because the same study found ChatGPT's crawler spent 34.82 percent of its fetches on 404 pages and 14.36 percent following redirects, and Claude's 34.16 percent on 404s. Keep redirects current and sitemaps up to date.

If you cannot rebuild the page yet

Most sites do not need to change their whole stack at once. Start with the few pages an engine is most likely to be asked about: pricing, comparisons, and the guides that answer buyers' questions. If your framework lets you choose server or static rendering page by page, switch those pages first.

If you need a stopgap, keep it honest. Google says dynamic rendering is not cloaking as long as bots and users get similar content, but that serving completely different content to each can be. Whatever a crawler receives should match what a reader sees.

Treat the crawler findings as a starting point rather than a permanent rule. The study was published in December 2024, and crawlers change. Your own server logs, and the curl checks above, show what is true for your site today.

What to do first

• Run the curl checks on your pricing page and your three most important guides.
• List every fact that is visible in the browser but missing from the server's HTML.
• Move those facts, and any JSON-LD, into server-rendered HTML.
• Fix the 404s and redirect chains that crawlers hit in your logs.
• Rerun the checks after every front-end release, and confirm in your robots.txt that the crawlers are allowed in at all.