Which AI Crawlers Should Your Dealership Allow? A robots.txt Guide

GPTBot, ClaudeBot, Googlebot, PerplexityBot, and more: what each AI crawler does, why blocking the wrong one quietly removes you from AI answers, and how to set robots.txt so you stay citable.

Updated June 2026 · 7 min read

Your website has a small file called robots.txt that tells automated visitors what they are allowed to access. For years it barely mattered to a dealer. Now it can quietly decide whether AI assistants are allowed to read, and therefore recommend, your inventory. A single wrong line can remove you from AI answers without anyone noticing.

The crawlers worth knowing

Different bots do different jobs. The important distinction for a dealer is between bots that help shoppers find you right now and bots that train models on your content. You will almost always want the first group in.

Retrieval and search bots (you want these)

  • Googlebot, Google search, and the foundation for Google’s AI Overviews.
  • Bingbot, Microsoft search and the backbone for Copilot.
  • OAI-SearchBot, OpenAI’s search crawler for ChatGPT.
  • Claude-SearchBot and ClaudeBot, Anthropic’s crawlers.
  • PerplexityBot and Perplexity-User, for Perplexity answers.
  • Applebot, Apple’s crawler, which feeds Siri and Spotlight.

Training bots (your call)

  • GPTBot (OpenAI), Google-Extended, and CCBot (Common Crawl) gather content used to train models.

For most dealers the right posture is to allow everything. Your inventory is public and the goal is to be cited; there is little to gain from blocking, and real risk in blocking the wrong bot.

The trap: blocking Googlebot blocks more than Google

Here is the mistake that bites dealers. Some crawlers do not advertise their own identity and instead inherit Googlebot’s permissions. Brave’s search crawler is one. So is the retrieval path behind some assistants. If your robots.txt disallows Googlebot, even partially, you can silently delist yourself from search engines and AI backends you did not intend to touch. The safe rule: never disallow Googlebot on a page you want shoppers to reach.

A safe, simple robots.txt for a dealer

For a public inventory site, an allow-all posture that names the major bots is both legible and safe. The essentials look like this:

User-agent: *
Allow: /

# (optionally name the major bots for clarity)
User-agent: Googlebot
Allow: /

User-agent: GPTBot
Allow: /

Sitemap: https://yourdealer.example/sitemap.xml

Keep private areas (admin, staff logins) on separate paths or hosts rather than relying on a broad Disallow that might catch a bot you need. And always point to your sitemap so crawlers can find every listing.

Allowing a crawler is not the same as being readable

A crawler you allow still has to be able to read the page once it arrives. If your inventory loads via JavaScript, a permitted-but-non-rendering bot still sees nothing. Access and readability are two separate gates, and you need both. We cover the readability side in can AI see your inventory?

How VIN Index handles it

VIN Index generates a correct robots.txt for every dealer surface automatically: allow-all, every major AI and search bot named, Googlebot never disallowed, and your sitemap advertised, on both subdomains and custom domains. You do not maintain it. To see how your current site reads to these crawlers, run the free Analyzer.

Continue the guide

AI Visibility for Car Dealers: The Complete GuideStart here: the complete overview of AI visibility for dealers.Can AI see your inventory?Most AI crawlers cannot run JavaScript, so inventory that loads in the browser is invisible to them. Here is what AI assistants actually read, why your cars may be missing, and how to test it in minutes.GEO vs SEO for dealersGenerative Engine Optimization (GEO) is about being the cited source inside an AI answer, not ranking tenth on a results page. Here is how it differs from SEO for dealers, and why you need both.Vehicle schema markupA practical reference for the schema.org Vehicle and Offer structured data that lets AI assistants read a car accurately: the fields that matter, common mistakes, and a working example.How shoppers use AI in 2026The data on how buyers now research and shortlist vehicles with AI assistants, what it means for where your inventory needs to appear, and how the funnel has shifted.Get cited by ChatGPTA practical, step-by-step checklist for becoming the source ChatGPT and other assistants name when a shopper asks about a car you have in stock.AI shopping catalogsSome AI answers come from a live crawl; others come from a pre-built shopping catalog you feed ahead of time. Here is how the AI shopping surfaces work and how dealers get their inventory in.llms.txt for dealersAn honest look at the llms.txt file: what it is, what it does and does not do for AI visibility, and where it sits on a dealer’s real priority list.Entity SEO for dealershipsAI engines cite businesses they recognize as real, consistent entities. Here is how name/address/phone consistency, structured data, and authoritative references build that recognition for your dealership.

See how AI reads your inventory today.

Run the free AI Readiness Analyzer. We show you exactly what ChatGPT, Copilot, and Google AI can and cannot read on your current site. No card.

Check your AI visibilityIndex my inventory