AI crawlers don't run JavaScript. Here's how to see what they see
By Roshan, founder of SearchMaxx - - 2 min read
Open your site in a browser and you see your product, your pricing and your customers. Many AI crawlers see far less. They download the HTML your server sends and stop there. Anything your JavaScript adds after the page loads never reaches them.
What the data shows
In December 2024 Vercel and MERJ looked at how AI crawlers behaved across Vercel's network. Their finding: "none of the major AI crawlers currently render JavaScript." That covered OpenAI's crawlers (OAI-SearchBot, ChatGPT-User and GPTBot), Anthropic's ClaudeBot, PerplexityBot, Meta's crawler and ByteDance's.
ChatGPT's and Claude's crawlers did download JavaScript files. JavaScript made up 11.5% of ChatGPT's requests and 23.8% of Claude's. They did not run it.
Two exceptions: Gemini uses Google's crawling setup, which renders JavaScript the way Googlebot does, and Applebot renders it too. (Vercel: The rise of the AI crawler)
So a site that needs JavaScript to show its text can do fine in Google and still be invisible to ChatGPT, Claude and Perplexity.
See your page the way they do
Fetch the page without running anything, strip the tags, and count the words:
curl -s https://yoursite.com/ \
| sed -e 's/<script[^>]*>.*<\/script>//g' -e 's/<[^>]*>/ /g' \
| wc -w
The command is rough. It misses scripts that span several lines, so the real number may be lower. As a rule of thumb:
- A few hundred words or more: crawlers can read your page.
- Under about 150: crawlers are probably getting a nearly empty shell. Your headline, product description and pricing likely arrive only after JavaScript runs.
You can also turn off JavaScript in your browser's developer tools and reload. What is left is roughly what an AI crawler gets.
How to fix it
The fix is to send real text in the first response, not to remove JavaScript.
- Next.js, Nuxt, SvelteKit, Remix, Astro: these can render pages on the server or ahead of time. Make sure your marketing pages use that. A page marked as client-only, or one that fetches its content in the browser after load, arrives empty.
- A single-page app built with Vite or Create React App: pre-render the marketing pages to static HTML at build time, or move them to a framework that does. Keep the app itself as it is. Crawlers only need the public pages.
- Webflow, Framer, WordPress, Squarespace: these send real HTML already. Check anything embedded through a script, such as pricing tables, reviews and FAQ widgets. Those often load with JavaScript and come through empty.
After the fix, run the word count again. Our free check does the same test and tells you how many words arrive without JavaScript.
Want this done for your site?
Every day we make one change to your website that helps AI engines name you, and a person checks each one. Email me your website and the question your buyers ask AI.