GPTBot, ClaudeBot, PerplexityBot: an ecommerce robots.txt guide

Crawl access is the first thing an AI shopping agent tests, and the easiest thing to get wrong without noticing. Your ecommerce robots.txt tells AI crawlers whether they may fetch your product pages at all, and if the answer is no, nothing else about the store matters: not the schema, not the sitemap, not the price.

The five crawlers that matter for shopping

Sextant checks these five by name, because they are the ones that decide whether a store can appear in an AI answer.

| User agent | Operator | What it is for | | --- | --- | --- | | GPTBot | OpenAI | Training data collection and ChatGPT browsing | | OAI-SearchBot | OpenAI | Indexing for ChatGPT search results | | ClaudeBot | Anthropic | Retrieval for Claude | | PerplexityBot | Perplexity | Answers and shopping results | | Google-Extended | Google | Google AI use of your content |

Two different jobs are mixed into that list, and collapsing them is where merchants get into trouble.

Some of these crawlers fetch a page because a shopper is asking a question right now, and what they fetch becomes part of the answer that shopper sees. Others fetch to build a training corpus for a model released later. A merchant can reasonably feel differently about those two things. Handing over a catalogue to train a model is a business decision with arguments on both sides. Refusing to be readable while a customer is actively asking for a product like yours is a different decision, and it is the one that costs sales.

Google-Extended is the clearest training control, and OAI-SearchBot, ClaudeBot and PerplexityBot sit firmly on the retrieval side. GPTBot is the awkward one, because it covers both training collection and ChatGPT browsing, so blocking it buys a training preference at some visibility cost. That tradeoff is yours to make knowingly rather than inherit.

How robots.txt actually works

Most bad robots.txt files come from a wrong mental model, so it is worth being precise.

A robots.txt file is a set of groups. Each group opens with one or more User-agent lines and is followed by Allow and Disallow rules. A crawler selects the single most specific group that names it and obeys that group only. It does not merge that group with the wildcard User-agent: * group.

That cuts both ways. Given a permissive wildcard group and a User-agent: GPTBot group containing Disallow: /, GPTBot follows the specific group and the permissive one never applies to it. Equally, an explicit Allow group for GPTBot does nothing for ClaudeBot, which falls back to the wildcard group and whatever restrictions live there.

Disallow values are prefix matches against the URL path. Disallow: /products blocks /products, /products/anything and also /products-clearance, because the string only has to match from the start. Disallow: / matches every path on the site. An empty Disallow: blocks nothing.

Finally, robots.txt is a convention rather than an access control mechanism. It is a request that well behaved crawlers honour, and the crawlers above publish named user agents precisely so you can make that request per operator. If the goal is to stop scraping, this is the wrong layer.

The accidental block

Most blocked stores never decided to be blocked. The rule arrived some other way.

• A scraping scare. During an incident someone pastes in a blocklist of bot names with Disallow: /. The incident passes. The file does not get revisited.
• Staging rules shipped to production. A staging environment correctly carries User-agent: * and Disallow: /. Then the theme or the build gets promoted, and the rule comes with it.
• Plugin and app defaults. Security, SEO and bot-mitigation tools ship AI crawler blocking, sometimes on by default, often behind one checkbox labelled something like "block AI bots" that covers retrieval crawlers as well as training ones.
• Platform template edits. On Shopify the file is generated from robots.txt.liquid in the theme. Editing it is supported, so a rule added once by an agency or an app lives in the theme, survives updates and travels through duplications.

Nobody notices because nothing breaks. There is no error page, no alert, no failing build, no dip in a dashboard anyone already watches. Human traffic is unaffected, Googlebot is unaffected, checkout works. The store simply stops being considered, which from the inside looks identical to not being popular. That is the mechanism behind why ChatGPT doesn't show your products, and the most common single finding in a Shopify store that has gone invisible to AI. The wider shift that made this expensive is the subject of the switch was flipped.

A baseline that works

A permissive baseline that names each crawler explicitly, keeps cart and checkout out of the index, and points everyone at the sitemap:

User-agent: *
Allow: /
Disallow: /cart
Disallow: /checkout

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

The Sitemap line belongs to no group and applies to the whole file. It costs nothing and it hands every crawler a route to the catalogue instead of making it guess from internal links.

The mistake pattern, for comparison:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

That reads as "do not train on us" and functions as "do not answer with us". The ClaudeBot and PerplexityBot groups have nothing to do with training. They are what fetches a page when someone is asking about a product like yours.

If you want retrieval but not training, that is expressible: keep Allow: / for OAI-SearchBot, ClaudeBot and PerplexityBot, and set Disallow: / for Google-Extended and, accepting the browsing cost, GPTBot.

How to verify

Open https://yourstore.com/robots.txt in a browser or fetch it with curl, and read the whole file top to bottom rather than searching it for one bot name. The wildcard group governs every crawler you did not name, so a restriction there is easy to miss when you are only looking for GPTBot.

Seven checks map directly onto what you are reading. crawl.robots_present is whether the file exists and serves cleanly. crawl.blanket_disallow is whether any group applying to AI crawlers carries a site wide Disallow: /. Then one per crawler: crawl.gptbot, crawl.claudebot, crawl.oai_searchbot, crawl.perplexitybot and crawl.google_extended.

Re-check after any deploy that touches the theme, a bot-mitigation setting or an SEO plugin, since those rewrite this file without telling you. Crawl access is the first of the four layers described in what agent readiness means, and the only one where a single line undoes all the others.

If you want to see how your own file reads to the five crawlers above, run a free scan. It checks robots.txt along with your schema and discovery files, and installs nothing.