Page rendering
Can AI read your vehicle pages?
Your site refuses ChatGPT's search crawler, ChatGPT's live fetcher, Claude's live fetcher, Perplexity's search crawler at the network layer, even though your robots.txt invites them in, so those assistants can't read your pages directly and fall back to third-party listings like AutoTrader. But you still allow Google's search crawler, Bing's search crawler, so engines that read those indexes can find and cite your own pages. The fix is to allowlist the blocked crawlers (almost always a Cloudflare/WAF bot rule) so every assistant reaches you, not just some.
VIN Index republishes every car as a page a crawler reads start to finish, without running a line of JavaScript.
See the detail
What we checked
Whether your vehicles are present in the raw HTML an AI crawler receives, and whether the named AI crawlers are allowed in.
How we tested
We fetch your page with no JavaScript (the way ChatGPT’s and Claude’s crawlers read), count the readable vehicles, and probe whether each engine’s crawler gets through.
What it means
If your cars only appear after JavaScript runs, most AI crawlers see an empty page and have nothing to cite when a buyer asks.
- Perplexity's search crawler (PerplexityBot)
- Refused at the network layer
- Claude's live fetcher (Claude-User)
- Refused at the network layer
- ChatGPT's live fetcher (ChatGPT-User)
- Refused at the network layer
- ChatGPT's search crawler (OAI-SearchBot)
- Refused at the network layer
- Bytes returned to a no-JS crawler fetch
- 2,400 chars of HTML
- Vehicles readable in that HTML
- Yes
- Google's search crawler (Googlebot)
- Probe refused — inconclusive: this crawler is verified by IP, and refusing look-alikes is standard anti-spoofing
- Bing's search crawler (bingbot)
- Probe refused — inconclusive: this crawler is verified by IP, and refusing look-alikes is standard anti-spoofing
