Allow search crawlers to reach the pages you want shoppers to discover. Then check that they receive a readable vehicle page. Your training preferences are a separate decision: OpenAI and Anthropic publish different controls for search, user-requested visits and model training.
The practical goal is public inventory that can be fetched, understood and cited. A website provider can verify the permissions, firewall response and page content together. This guide shows what to ask them to check.
Which bots help shoppers find your dealership?
| Crawler or fetcher | Job | Dealer action |
|---|---|---|
| Googlebot | Google Search crawling, including its AI search features | Allow public pages you want indexed and eligible for snippets. |
| Bingbot | Bing search discovery | Keep public inventory and important business pages accessible. |
| OAI-SearchBot | OpenAI search discovery | Allow it on pages you want available to ChatGPT search. |
| ChatGPT-User | Visits made in response to a user’s request | Check actual access; user-triggered fetch behavior differs from scheduled crawling. |
| Claude-SearchBot | Anthropic search discovery | Allow public pages intended for Claude search results. |
| Claude-User | Page retrieval for Claude users | Review it separately from Claude’s training crawler. |
| PerplexityBot | Perplexity search discovery | Allow it for public search content. |
| Perplexity-User | Retrieves a page at a user’s request | Verify the request; Perplexity says this fetcher generally ignores robots rules. |
Provider references: OpenAI bots, Anthropic bots, Perplexity crawlers and Google AI-search controls. Bot names and policies can change; use the current provider documentation when configuring a site.
Training controls: make this choice separately
GPTBot concerns OpenAI model training. ClaudeBot concerns Anthropic model development and training. ClaudeBot is not the same bot as Claude-SearchBot. Blocking either training bot is different from blocking that provider’s search crawler.
Google-Extended needs a more careful reading. It is a robots control token, not a separate crawler name in your traffic logs. It covers training of future Gemini models and grounding in Gemini Apps and specified Vertex AI services. It does not control inclusion or ranking in Google Search. For Google AI Overviews and AI Mode, review Googlebot access and Search preview controls. Google explains the distinction.
A simple starting point for public inventory
This example permits crawling of public pages and advertises a sitemap. Replace the hostname and private paths with your own. Authentication must protect private information; robots.txt is a public preference file, not a security boundary.
# Example for a public dealer site.
# Keep account and administration areas authenticated.
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/
Sitemap: https://yourdealer.example/sitemap.xmlIf you want OpenAI search access while declining training, the separate groups can look like this. They illustrate the distinction; they are not a recommendation to change your training preference.
# Optional: allow OpenAI search, decline OpenAI training.
# Add these groups to your existing file, after reviewing its rules.
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /account/
User-agent: GPTBot
Disallow: /Check every specific group. A bot-specific group can take precedence over the wildcard group rather than inheriting its restrictions. A permissive wildcard does not fix a separate search-bot group containing Disallow: /. Your dealer’s main domain and inventory subdomain each need the right file at their own /robots.txt.
Four checks your website provider can show you
| Check | Ask to see | Fix if it fails |
|---|---|---|
| Robots permission | The rules matching an actual public VIN URL and the crawler token | Correct the matching group or an overbroad inventory restriction. |
| Access response | The status and body returned through the CDN or firewall | Resolve a verified crawler’s access-denied page, challenge or repeated server error. |
| Readable content | VIN, price, mileage, status and seller in the response | Make essential vehicle facts available without a client-only inventory request. |
| Index controls | Canonical, robots meta/header and sitemap entry | Remove accidental noindex from intended public pages and align the intended URL. |
A claimed bot user agent can be copied. Before creating firewall exceptions, use the provider’s current IP-verification or authentication guidance. A test with a copied bot name shows how your rules respond to that request; it does not prove a real provider crawled the page. Keep the fix specific to legitimate public access.
Access is the beginning of the citation path
Once the page is accessible, make it worth retrieving: a stable vehicle URL, current stock facts, clear dealership identity and useful answers to the question. Follow the VIN-level discoverability checklist and the ChatGPT recommendation guide for the next steps.
VIN Index publishes your authorized inventory in server-rendered pages and advertises the dealer surface’s sitemap. To find the gaps on your existing website, run the AI Readiness Analyzer and share its findings with your provider.