Table of Contents
A few weeks ago we ran an AI-visibility audit on our own site — CraftrUSA. In a browser, everything looked great. In a raw curl, our homepage was 31 words, zero headings, zero links, zero meta description. We were invisible to Googlebot, GPTBot, ClaudeBot, and PerplexityBot — the exact bots that decide whether a business shows up in AI Overviews, ChatGPT answers, and Perplexity citations. Here's exactly what we found and the fix we shipped so you can copy it.
1. How We Found Out Our Own Site Was Invisible
The trigger was a client asking why a competitor kept appearing in ChatGPT answers about "web design Orlando" and they didn't. We opened DevTools on their site — content everywhere. Then we ran curl -A "GPTBot" https://theirsite.com/ and got back a hollow HTML shell: a <div id="root"></div>, a script tag, and nothing else. Then, out of professional paranoia, we did the same to our homepage. Same result. That is the moment we stopped selling and started rebuilding.
If you built with React, Vue, or any Vite/CRA SPA and never added prerendering, this is almost certainly happening to you right now. Browsers execute JavaScript, so you see the site. Most AI crawlers do not — they read raw HTML and move on.
TRY THIS RIGHT NOW
Open a terminal and run curl -A "GPTBot" https://yourdomain.com/ | wc -w. If the number is under 200, you are invisible to AI search. A healthy homepage returns 500–2,500 words of real content in raw HTML.
2. Why AI Bots See Nothing on a Vite SPA
Googlebot has a rendering queue that eventually executes JavaScript, but even Google's own docs admit rendering is deferred and unreliable. AI crawlers are stricter:
- GPTBot (OpenAI): reads static HTML only. No JS execution.
- ClaudeBot (Anthropic): static HTML only.
- PerplexityBot: static HTML only.
- facebookexternalhit / Twitterbot: read only the
<head>of the initial HTML — client-side helmet updates never reach them.
A Vite React app ships an index.html that is essentially empty until React hydrates. If your content, H1, meta description, canonical, and Open Graph tags are all injected by React at runtime, every one of those bots walks away with nothing.
3. The 5-Signal Audit We Run First
Before we touch code, we run these five checks on the homepage and three deep pages. They take about ten minutes and tell you exactly what is broken.
- Raw word count:
curl -A "GPTBot" URL | sed 's/<[^>]*>//g' | wc -w. Target: 500+ on the homepage, 800+ on service and city pages. - Head tag audit: View source (Cmd+U). Confirm there is exactly one
<title>, one<meta name="description">, one<link rel="canonical">, and route-specificog:*tags. Homepage tags that persist on every route are the #1 silent killer. - Anchor count:
curl URL | grep -c '<a '. Under 10 on the homepage means crawlers cannot discover the rest of your site. - Image alt text: Every
<img>needs analt. AI bots weight alt text heavily when summarizing pages. - Robots.txt + sitemap: Make sure
GPTBot,ClaudeBot, andPerplexityBotare not blocked and that the sitemap lists every indexable route on the canonical domain.
We package all five checks into every free website audit we run — it is the cheapest way to see whether you're bleeding AI visibility.
DON'T BE CLEVER WITH USER-AGENT DETECTION
Do not serve different HTML to GPTBot than to real users. Google calls that cloaking; ChatGPT will eventually notice. The right fix is prerendering once, at build time, and serving the same HTML to everyone.
4. The Fix: Build-Time SSG for Every Route
We rebuilt our pipeline as static site generation (SSG). Every one of our 76 indexable routes now ships as a real HTML file with real content — no server, no Puppeteer, no runtime cost. Here is the exact shape of the fix:
- Split App.tsx so the
<Routes>tree exports independently of<BrowserRouter>. That lets the same tree run under<StaticRouter>on the server. - Add
entry-server.tsxthat callsrenderToString()and returns both the HTML body and the react-helmet-async head payload. - Vite SSR build (
vite build --ssr) bundles the server entry to Node-consumable JS. - A prerender script loops every route, renders it, injects the helmet output into
<head>, injects the body into<div id="root">, and writesdist/<route>/index.html. - Client uses
hydrateRootwhen awindow.__SSG__flag is present, so React adopts the static markup with no flicker. - A verify step fails the build if any generated page is missing a title, meta description, canonical, or H1. No broken SEO ever ships.
Total build time went from 12 seconds to 41 seconds. Raw-HTML word counts went from 31 to between 500 and 2,300 depending on the route. That is the entire trade.
5. Route-Specific <head> — Stop the Homepage Tag Leak
Even with SSG, we found a subtle bug: helmet was emitting the correct route-specific tags, but the static homepage tags we had left in index.html were still there too. Bots that read the first tag they encounter — Facebook is famous for this — were pulling the homepage title onto our SEO service page. Two fixes:
- Strip the static
<title>,<meta name="description">,<link rel="canonical">, and everyog:*/twitter:*tag from the template before injecting the helmet output. - Route every page through a single
<SEOHead>component so title, description, canonical, OG, Twitter, and JSON-LD always ship together and always self-reference the current route.
If you are a developer, this pattern is worth stealing outright. If you would rather have us do it, that is exactly what our web design and SEO services deliver as a package.
"The single biggest lift we got from any 2026 optimization was making our own site visible to AI crawlers. Nothing else — new backlinks, faster LCP, more content — moved the needle like being readable in raw HTML."
6. What "Good" Looks Like in Raw HTML
After the rebuild, we re-ran the audit on every route. Here is the checklist we now hit on every page — and the checklist we run against every client site before we hand off a build. If your site doesn't hit all seven, it is leaving AI visibility on the table.
- Unique
<title>, 50–60 characters, keyword up front. - Unique
<meta name="description">, 140–155 characters. - Self-referencing canonical on the apex domain — no www.
- Exactly one
<h1>containing the primary keyword. - 500+ words of body content in raw HTML (800+ for money pages).
- Real internal anchor links to parent, nearby, and service pages — see our portfolio for the pattern in action.
- Every
<img>has descriptivealttext and route-specificog:image.
KEY TAKEAWAYS
- GPTBot, ClaudeBot, and PerplexityBot read raw HTML only — no JS.
- curl -A "GPTBot" YOUR_URL tells you in 3 seconds if you're invisible.
- Vite/CRA SPAs ship an empty root div — invisible by default.
- Build-time SSG (renderToString + hydrateRoot) is the clean fix.
- Strip static head tags before injecting per-route helmet output.
- Verify every route hits the 7-point raw-HTML checklist before shipping.
Frequently Asked Questions
Does Next.js fix this automatically?
Only if you use getStaticProps/generateStaticParams or the App Router with server components. A Next.js app that ships everything as "use client" has the exact same problem as a Vite SPA. The frame does not save you — the render strategy does.
Will adding noindex to unimportant pages hurt AI visibility?
No — the opposite. AI crawlers weight signal density. Noindexing thin utility pages and cart/checkout screens concentrates crawl budget on the pages you actually want quoted.
Do I need separate content for AI bots?
No. Serve the same HTML to everyone. Ranking well in AI answers is the same job as ranking in Google — clear structure, real entities, unambiguous facts, honest schema. Cloaking will get you blocked.
How long until AI answers start citing my site?
In our own data, we saw ChatGPT and Perplexity begin citing pages roughly 3–6 weeks after they became crawlable in raw HTML. Google's AI Overviews took 4–8 weeks. Your mileage varies with authority.
Share This Article
Christopher Penn
Founder & CEO, CraftrUSA
Veteran-owned agency in Orange City, FL. We build fast, crawlable, conversion-focused sites for Central Florida businesses. Book a strategy call →