Wayfinder AI
Access Compass
All Guides

Preparing Your Site for AI Crawlers: JavaScript Rendering Approaches

Technical guide to rendering strategies for AI—SSR vs. SSG vs. hybrid, implementation per framework, testing, and troubleshooting.

Implementing AEO

AI crawlers face the same JavaScript rendering problem as Google. But the problem is harder and the solutions are more complex for AI agents, which typically can't maintain persistent browser sessions.

The core issue is simple: if your site relies on client-side JavaScript to load critical content, AI agents might not see it. This is a technical problem you need to solve systematically.

This is not a quick checklist. If you have a dynamic site, you need to understand what's required to make it AI-accessible. Your reader's takeaway should be clear: they understand why JavaScript rendering matters, what options exist, and what's required for each approach.

The difference between crawlers and AI agents matters here. Traditional search crawlers are like census takers—visiting every door and recording everything. AI agents are like impatient users: they land on a page, scan for clues, make a decision, and move on. Wayfinder's navigation research found that if they can't find what they need quickly, they give up. This means your rendering strategy directly impacts whether AI agents can actually access and understand your content.

Why JavaScript Rendering Matters for AI

The Browser vs. Bot Problem

Humans use browsers that execute JavaScript fully. AI agents typically don't. They lack persistent JavaScript runtimes, don't maintain state across page loads, can't handle certain JavaScript patterns like infinite scrolls or lazy loading, and face strict timeouts. A React app that loads product data via API after initial page load works fine in a browser: page loads → JavaScript executes → API call → data renders. For an AI agent: page loads → might have limited JavaScript execution → data might not render.

What Content Is "Invisible" to AI Agents

Common patterns create visibility gaps:

  • Product descriptions loaded via API
  • Pricing tables rendered client-side
  • Navigation menus built dynamically
  • Infinite scroll content (only first batch loads)
  • Content hidden until user interaction
  • Checkout processes requiring JavaScript

A page might look perfect in your browser because you have JavaScript enabled. AI agents see incomplete content. You don't notice because you test in a browser. To check what AI agents see, disable JavaScript in your browser (F12, Cmd+Option+P, "Disable JavaScript") and refresh. What's missing? That's approximately what AI agents see.

Why Not Just Execute JavaScript?

Full browser execution isn't scalable for AI crawlers. Running full browser engines (Chromium, Firefox) at scale is expensive. ChatGPT, Perplexity, and Claude handle millions of requests—full browser emulation per request would be prohibitively costly. Web code is also messy and unpredictable. JavaScript can execute malicious code, create infinite loops, consume excessive resources, or access browser-specific features like geolocation and camera. Different JavaScript versions, frameworks, and libraries behave differently—execution might not produce the same result as in a browser.

Latency adds another constraint: executing JavaScript adds 5-30+ seconds per page. For a chatbot answering questions in real-time, that's unacceptable. The result: most AI crawlers use lightweight rendering (parsing HTML, basic JavaScript, limited execution). They're not full browsers.

The Practical Implication for Your Site

You cannot rely on client-side JavaScript for content discovery. Critical content must be in the initial HTML or rendered server-side before delivery. Wayfinder's research shows that sites with high JavaScript/HTML word count gaps—where rendered content is significantly larger than raw HTML—have lower AI success rates. Pages where the rendered DOM has significantly more content than the raw HTML are JavaScript-dependent and represent risk areas.

Rendering Approaches: Trade-Offs and Comparison

You have four main approaches to handling dynamic content. Each has distinct trade-offs.

Option 1: Server-Side Rendering (SSR)

Server-side rendering generates HTML on every request, sending a complete rendered page to the client or bot.

Process: Request comes in → server executes JavaScript/code → server renders to HTML → sends HTML to requester → HTML is complete, no additional JavaScript needed.

Pros: Complete HTML available immediately; works for AI agents (no JavaScript needed client-side); SEO-friendly (traditional crawlers and AI agents both see complete HTML); good for social media previews (og:tags in HTML).

Cons: Server load increases (rendering on every request); slower initial response time (rendering takes CPU); more complex architecture (need server-side code); harder to scale (high compute per request).

Best for: Content sites (blogs, news, documentation), e-commerce (product pages need full content visible), SaaS (pricing pages, product pages), any site where content is critical for discovery.

Frameworks with SSR: Next.js (React) — getServerSideProps; Nuxt (Vue) — default with asyncData; SvelteKit — default; Remix (React).

Cost: Moderate (server renders per request, but rendering is typically fast — milliseconds per page).

Option 2: Static Site Generation (SSG)

Static site generation generates HTML for all pages once during the build step, then serves pre-rendered files.

Process: Deploy time → build process renders all pages to static HTML → static files stored on CDN/server → request comes in → server immediately sends pre-built HTML (no rendering) → HTML is complete and instant.

Pros: Fastest possible delivery (pre-built HTML, no rendering); cheapest to scale (static files, CDN); perfect for AI agents (complete HTML); high SEO performance; works without JavaScript.

Cons: Only works for static content (pages don't change per user); rebuild required for content updates; build times can be long (if you have thousands of pages); not ideal for rapidly changing content.

Best for: Blogs, documentation, marketing sites, product catalogs (if not personalised), content libraries, any site where content is known at build time.

Frameworks with SSG: Next.js (React) — getStaticProps; Hugo (static site generator, very fast); Jekyll (Ruby); Gatsby (React); SvelteKit (hybrid).

Cost: Low ongoing (static file hosting is cheap), build-time cost (compiling pages takes time).

Option 3: Hybrid (SSG + ISR)

Hybrid generation creates static pages at build time but regenerates on-demand or on a schedule through Incremental Static Regeneration.

Process: Deploy → generate static pages for high-traffic content → request for stale/new page → regenerate in background → server serves existing version while regenerating → next request gets updated version.

Pros: Fast delivery (usually static); can handle dynamic content (regenerates on change); build times reasonable (not all pages at once); good balance of performance and freshness.

Cons: More complex to set up; stale content possible (brief window); background regeneration adds cost.

Best for: E-commerce sites with changing inventory, news sites with frequent updates, SaaS product pages with occasional changes, sites with thousands of pages.

Frameworks: Next.js — getStaticProps + revalidate; SvelteKit — hybrid mode.

Cost: Moderate (static delivery + periodic regeneration).

Option 4: Client-Side Rendering (CSR) — Not Recommended for AI

Client-side rendering sends minimal HTML from the server, then downloads JavaScript to render content.

Process: Server sends shell HTML (mostly empty) → browser downloads JavaScript bundle → browser executes JavaScript → JavaScript renders content to DOM.

Pros: Low server cost (minimal rendering); good for highly interactive apps; can scale easily (no server rendering).

Cons: Initial HTML is empty (no content for AI agents); slow initial page load (JavaScript download + execution); bad for SEO and AI visibility; cannot guarantee AI agents execute your JavaScript; poor social media previews.

Not recommended for primary content. Only use for interactive secondary features.

If you must use CSR: Add <meta name="robots" content="noindex"> to force crawlers away (honest approach), or implement a fallback (see Fallbacks section below).

Cost: Low (minimal server work).

Comparison Table

ApproachSpeedAI-FriendlyFlexibilityCostBest For
SSRModerateExcellentHighModerateContent + ecommerce
SSGFastExcellentLowLowStatic sites
HybridFastExcellentModerateLow-ModerateMixed content
CSRSlowPoorExcellentLowInteractive apps only

AI agents don't reliably execute complex JavaScript, making CSR a poor choice for any content you want discovered.

Implementation by Framework

Next.js (React)

Next.js offers flexible rendering options depending on your needs.

SSR approach:

export async function getServerSideProps(context) {
  const data = await fetch('API_ENDPOINT');
  return { props: { data } };
}
export default function Page({ data }) { ... }

SSG approach:

export async function getStaticProps() {
  const data = await fetch('API_ENDPOINT');
  return { props: { data }, revalidate: 3600 };
}

ISR (Incremental Static Regeneration): Add revalidate to cache strategy. This regenerates pages on a schedule while serving cached versions.

Recommendation: Use SSG for static pages, SSR for personalised content, ISR for hybrid scenarios where content changes occasionally.

Vue / Nuxt

Nuxt handles server-side rendering as the default, making it a strong choice for AI accessibility.

SSR is default in Nuxt:

export default {
  async asyncData({ params }) {
    return fetch(`API/${params.id}`);
  }
}

Recommendation: Nuxt handles this well. Use the default SSR configuration for critical pages.

SvelteKit

SvelteKit also defaults to server-side rendering, providing sensible defaults out of the box.

SSR is default:

export async function load({ params }) {
  const data = await fetch(`/api/${params.slug}`);
  return { data };
}

SSG: Use the prerender option to generate static pages.

Recommendation: SvelteKit defaults are sensible. Use as-is for most AI accessibility scenarios.

React (Without Framework)

If you're using plain React, you face a challenge: React was designed for client-side rendering by default.

Recommendation: Don't use client-side React for primary content. Either:

  1. Switch to Next.js or Remix (frameworks with SSR built-in)
  2. Build a Node backend that renders React server-side
  3. Refactor to static HTML with islands of interactivity

If stuck: Add a static HTML fallback (see Fallbacks section below) to ensure critical content is visible.

Ruby on Rails

Rails has server-side rendering built-in through ERB templates, making it inherently AI-friendly.

Rails has SSR built-in:

<div id="products">
  <%= render 'products', products: @products %>
</div>

ERB (embedded Ruby) renders server-side automatically.

Recommendation: Rails defaults are AI-friendly. No changes needed for standard server-rendered applications.

PHP/WordPress

PHP is server-side by default. Content is rendered server-side, which works well for AI crawlers.

Ensure content is in PHP output:

<?php foreach ($posts as $post): ?>
  <article><?php echo $post->content; ?></article>
<?php endforeach; ?>

Recommendation: If using a theme that loads content via JavaScript, switch to a theme that doesn't or add a server-side rendering layer.

Fallbacks and Hybrid Approaches

If you can't implement full SSR or SSG, implement pragmatic fallbacks.

Static HTML Shell + JavaScript Enhancement

For highly interactive sites, use a compromise approach that prioritises AI accessibility:

  1. Provide complete static HTML shell with key content
  2. JavaScript enhances interactivity (not loads content)

Example:

<h1>Product: Widget X</h1>
<p>£49.99</p>
<script>
  // JavaScript makes "Add to cart" button interactive
  // But the product name and price are already in HTML
</script>

Benefits: AI agents see key content in HTML, users get interactivity.

API + Server-Side Rendering of Critical Endpoints

If you have a complex single-page application:

  1. Create server endpoints that render critical pages (pricing, contact, main products)
  2. Let JavaScript handle secondary pages (settings, dashboards)

Routing example:

  • /pricing → server-rendered (SSR)
  • /app/dashboard → client-rendered (CSR, protected)
  • /app/settings → client-rendered (CSR, protected)

Prerendering at Build Time

For single-page applications, pre-render critical pages during the build process.

In build process:

const critical_pages = ['/pricing', '/about', '/contact'];
critical_pages.forEach(page => {
  render_spa_to_html(page).then(html => {
    save_as_static_file(html);
  });
});

This is not full SSG (other pages remain CSR), but critical paths are static.

Metadata Preload with Open Graph

Even if your page uses client-side rendering, ensure metadata is in the initial HTML:

<meta property="og:title" content="Product Name">
<meta property="og:description" content="Product description">
<meta property="og:image" content="/product.jpg">

This helps social sharing and provides a fallback for AI agents, even if full content rendering fails.

Testing and Verification

You need to verify your rendering solution actually works for AI agents.

Manual Testing: Ask an AI Agent

Test method 1: Claude

Visit https://yoursite.com/pricing and tell me:
1. What's the main product?
2. What's the price?
3. What are the included features?

Don't summarise. Tell me exactly what you see.

Compare to browser view. Major discrepancies indicate rendering issues.

Test method 2: ChatGPT Search Similar approach, but ChatGPT might render differently than Claude. Test with multiple agents.

Screaming Frog Comparison

Process:

  1. Open Screaming Frog
  2. Configuration → Spider → Rendering: Set to "JavaScript"
  3. Configuration → Spider → Advanced: Check "Store HTML" and "Store Rendered HTML"
  4. Crawl your site
  5. Export results
  6. Compare word count: Raw HTML vs. Rendered HTML

What you're looking for: Pages where the rendered DOM has significantly more content than the raw HTML are JavaScript-dependent. These are your risk areas.

Red flag: Rendered HTML is 50%+ larger than raw HTML

Programmatic Testing

If you have engineering resources, test programmatically.

import requests
from bs4 import BeautifulSoup

url = "https://yoursite.com/pricing"

# Test 1: Check raw HTML
response = requests.get(url)
html_word_count = len(response.text.split())

# Check if critical keywords are in HTML
if "£" not in response.text and "price" not in response.text:
  print("WARNING: Pricing not in HTML (likely JS-loaded)")

Interpretation:

  • Ratio 1.0-1.2x: Minimal JavaScript rendering (good for AI)
  • Ratio 1.2-2.0x: Some JavaScript-loaded content (watch this)
  • Ratio 2.0x+: Heavy JavaScript rendering (risk for AI)

Compass Audit (Wayfinder)

Run a Compass audit on your site. It tests navigation success and identifies rendering issues automatically.

Quarterly Monitoring

Set up recurring checks:

  • Monthly: Ask an AI to access your key pages
  • Quarterly: Run Screaming Frog comparison
  • After site updates: Manual verification that rendering still works

Common Issues and Troubleshooting

Issue: "After switching to SSR, server load is too high"

Solution 1: Caching

  • Cache rendered pages in Redis/memory
  • Invalidate on content changes
  • Reduces re-rendering

Solution 2: CDN + ISR

  • Pre-generate common pages
  • Use Incremental Static Regeneration for updates
  • Fallback to on-demand rendering for rare pages

Solution 3: Optimise rendering

  • Profile rendering time (find bottlenecks)
  • Move expensive operations to build time
  • Use lighter templating engines

Issue: "Build times are too long (thousands of pages to pre-generate)"

Solution 1: Selective pre-generation

  • Only pre-generate high-traffic pages
  • Use on-demand rendering for rest
  • Use ISR to update pages gradually

Solution 2: Parallel builds

  • Split rendering across multiple processes/machines
  • Use serverless functions for build-time rendering

Solution 3: Dynamic routes

  • Pre-generate categories/archives
  • Render individual items on-demand

Issue: "I'm using a headless CMS that assumes CSR"

Solution 1: Switch CMS to one with SSR support (Contentful Gatsby, Strapi with Next.js, etc.)

Solution 2: Build intermediate API layer that renders server-side

Solution 3: Pre-generate static pages from headless CMS on publish webhook

Issue: "My team doesn't know how to set up SSR"

Solution 1: Use a framework that does it for you (Next.js, Nuxt, SvelteKit)

Solution 2: Hire contractor to set up SSR layer

Solution 3: Use no-code/low-code SSR platforms (Vercel, Netlify)

Issue: "AI agents are still not seeing my content after SSR"

Likely causes:

  • robots.txt blocks crawlers
  • JavaScript still running client-side despite SSR attempt
  • Content loading dynamically even in SSR (verify server actually gets data)
  • Issue is not rendering but navigation depth (content exists but unreachable)

Troubleshooting:

  • Verify robots.txt allows crawlers
  • Disable JavaScript in browser, check if content visible
  • Run Screaming Frog with JavaScript enabled/disabled, compare
  • Run Compass audit (test navigation)

Implemented SSR or SSG? Test your setup with Compass to verify AI agents see your content. Navigate your site like an agent would.