We built an outbound prospecting pipeline last week, pointed it at a sector we know well, and got a result clean enough to be worth writing down.
The short version: we discovered 48 candidate stores in fishing tackle retail across British Columbia and the Pacific Northwest, scanned 19 of them, and found that every single one lets every AI crawler in. Not one blocked GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot or Google-Extended. Not one carried a blanket disallow. All 19 served over HTTPS.
Not one of them scored above a D.
The numbers
Nineteen stores scanned and scored. Median score 55, mean 55. Thirteen graded F (68 percent), six graded D (32 percent), and no store reached C, B or A. The scores ran from 40 to 69.
The five category subscores tell you where it goes wrong, and the shape is more informative than the total:
| Category | Median score | | --- | --- | | Crawl access | 100 | | Product schema | 64 | | Discovery | 29 | | Content quality | 27 | | Merchant data | 13 |
Crawl access is a clean sweep. Merchant data, which is the machine readable returns, shipping, FAQ and business identity an agent needs to price the risk of recommending you, sits at 13.
Five checks failed on every applicable store in the sample:
- FAQPage schema. 0 of 19 passed.
- MerchantReturnPolicy. 0 passed. Seven had a human readable policy page and no structured data, which scores as a partial rather than a pass.
- OfferShippingDetails. 0 passed.
- UCP manifest at
/.well-known/ucp. 0 passed. - AggregateRating. 0 of the 10 stores where the check applied.
Another band failed at roughly 90 percent: llms.txt, agents.md and
llms-full.txt at 89.5 percent each, product description length and product
image count at 90 percent each.
The checks stores did best on, after crawl access, were the core product schema fields. Product JSON-LD failed on only 20 percent of the stores where it applied, and availability likewise. That is the one piece of good news in the data, and it is mostly a platform artifact: the stores running a mainstream ecommerce platform get a Product node emitted for them whether they know it or not.
Nobody is blocking anyone
The result that matters is the one that did not show up.
There is a widely repeated story about AI and the open web in which publishers and retailers are pulling up the drawbridge, blocking crawlers, fighting to keep their content out of models. That story is real in publishing. In this sample of retail it was entirely absent. Zero blocks across five crawlers and 19 stores.
These merchants have not made a considered decision to exclude themselves from agentic commerce. Most of them, on the evidence of their robots.txt files, have not made a considered decision about AI crawlers at all. The default is open, the default was left alone, and the crawler walks straight in.
And then finds nothing it can use.
This is the failure mode worth naming, because it is invisible from inside the business in a way that blocking is not. A blocked crawler is at least a decision someone made and could reverse. An open store with no structured data does not feel like a problem from the merchant's side. The site works. Orders come in. Nothing is broken. The agent arrives, cannot fill in the fields it needs, and routes around, and no error is raised anywhere that the merchant will ever see.
The uniformity is the interesting part. If this were a story about a few laggard shops you would expect a spread: some stores well set up, some badly, most in between. That is not what the data looks like. Thirteen Fs and six Ds, with no store clearing 69, is not a distribution of individual merchant competence. It is what a sector looks like when a platform switches agentic commerce on by default and nobody told the merchants underneath it what agents read.
What each failure actually costs
Abstract check names hide the consequence. Concretely:
No FAQPage schema. The long tail of buying questions that only an FAQ answers, the ones about sizing, compatibility, whether this fits that, are unavailable to the agent. It cannot answer a shopper's specific question from your store, so it answers from a store that published the answer as data.
No MerchantReturnPolicy. An agent recommending a purchase has to reason about what happens if the purchase is wrong. Without a returnPolicyCategory and a merchantReturnDays it cannot tell whether your returns terms are better or worse than the merchant next to you. Seven stores in the sample had a perfectly good returns page written for humans. That is worth nothing in a comparison, because prose is not a value.
No OfferShippingDetails. The agent cannot answer "when does it arrive and what does delivery cost", which for a shopper stating an intent with a deadline is frequently the deciding field.
No AggregateRating. No machine readable signal that anyone was happy. In a comparison against a merchant who publishes ratings, the absence is not neutral. Worth saying clearly: the fix is to emit this only where real ratings exist. A store with no reviews should publish nothing. Inventing the number is detectable and it is a policy problem.
No llms.txt or agents.md. The agent has no orientation file telling it what the site sells and where the useful pages are. These are conventions rather than ratified standards, and their absence is not fatal, but they are close to free to publish. Cheap wins are the right place to start.
Short product descriptions and thin images. Below the thresholds where there is enough substance for an agent to compare you on anything except price.
How we did this, so you can check it
The method matters more than the numbers, because the numbers are only worth anything if someone can go and reproduce them.
Discovery. Candidates came from OpenStreetMap, queried through Overpass.
We tried the obvious approach first and rejected it: searching for platform
fingerprints such as myshopify.com alongside niche terms is what everyone
reaches for, and DuckDuckGo answered our probe with an anti-bot challenge
while Google and Bing both forbid automated result scraping in terms we would
have been knowingly breaching. Commercial store-technology datasets have far
better coverage and no usable free tier. OpenStreetMap is open data under
ODbL, built for programmatic query, needs no key, and records the two facts
discovery needs: that a business is a fishing or outdoor shop, and its
website.
We queried six geographic tiles covering coastal and interior BC, Washington, Oregon, Idaho and western Montana. One tile, Washington, failed on this run and its area is not represented in the results. Candidates were reduced to apex domains, deduplicated, and filtered against a list of marketplaces, social platforms and site builders, because a shop whose website is a Facebook page cannot act on a schema fix.
Verification. Each candidate got one homepage fetch to confirm it was actually a store, on at least two structural signals such as a platform asset host, a cart endpoint, product URL patterns or commerce schema. This is the step that removes charter operators, guide services and brochure sites from a list that would otherwise be full of them.
Scanning. The 19 survivors were scanned with the same engine that powers
the public scanner, identifying itself as
SextantBot/1.0 (+https://sextantscan.com/bot), honouring robots.txt,
rate limited per host and globally.
48 discovered, 19 scanned, 29 excluded. The exclusions break down as eight stores whose homepage did not respond, eight with no commerce signals at all, five with only one signal, and eight lost to HTTP 429 rate limiting. That last group is the important one, and it is our problem rather than theirs.
What this sample cannot tell you
Three limitations, and the third is severe enough that we are withholding a statistic over it.
Nineteen stores in one product category in one region is a small sample. It describes fishing tackle retail in the Pacific Northwest. It does not describe ecommerce, or retail, or small business, and we are not going to write a headline that says most stores are invisible to AI, because this data does not support that sentence. What it supports is narrower and still worth knowing: in this sector, in this region, crawler access was universally open and machine readable data was almost universally absent.
OpenStreetMap only lists businesses with a physical premises. A pure online retailer with no shopfront is invisible to the source. For tackle most retailers do have a shop, so the bias is tolerable, but the sample is skewed toward bricks and mortar businesses that also sell online.
The platform breakdown from this batch is unusable, and here is why. We recorded the ecommerce platform for each store scanned, and the obvious thing to do would be to publish it. We are not going to, because when we checked the eight candidates lost to HTTP 429, every one of them was Shopify. Shopify rate limits per client IP across every store it fronts, so a batch that touches many Shopify stores in sequence gets throttled, and our sample lost Shopify stores specifically and systematically. Any platform statistic drawn from what survived would be an artifact of our own rate limiting rather than a fact about the sector.
That skew also affects the failure rates above, and in a knowable direction.
Shopify began shipping llms.txt, agents.md and UCP support natively in
May 2026, so the stores our throttling removed are exactly the ones most
likely to have passed the discovery checks. The roughly 90 percent
discovery failure rate is therefore likely to be an overstatement of the
sector, and should be read as an upper bound rather than a point estimate.
The merchant data findings, where nothing any platform generates by default
would have helped, are much less exposed to this.
We would rather publish the caveat than the cleaner number.
Methodology note
So that the figures above can be judged rather than taken on trust.
A score is a weighted percentage across five categories: crawl access and product schema at 25 percent each, discovery at 20, merchant data and content quality at 15 each. Within a category each check carries a weight, a pass earns full credit, a warning earns half, and a check that cannot apply to a store is excluded and its weight redistributed rather than counted as a failure. That last rule matters for reading the numbers above: where a check shows a count out of 10 rather than 19, it is because the check did not apply to the other nine, usually because no product pages were discoverable.
Failure rates in this piece count warnings as not passing. A store with a returns page and no structured data is a real gap, and rounding it into the pass column would understate the problem in a statistic we are publishing.
Homepage response time is measured as the median of three samples taken seconds apart, not a single reading. We changed this after our own store produced readings of 4176ms, 3059ms and around 2000ms across runs with nothing about the store changing between them, which moved the overall score by up to 10 points on noise. A single timing sample measures hosting variance, not store quality.
The full check list, with what each one looks for and how it is scored, is published on the site rather than kept as a trade secret. If you think a check is wrong, you can see exactly what it does and say so.
The worked example
We ran this same profile of failure to ground on our own store before selling the fix to anyone else. It scored 47, grade F, on a first scan: open to every crawler, no discovery files, no machine readable returns or shipping, no FAQ schema, and one product silently missing its structured data entirely because of a caching bug. It now scores 89. The full write up is here, including the two checks we left failing because passing them would have meant inventing data the store does not have.
If you want the background on what agents read and why enrolment is not the same as visibility, that is the switch was flipped. For the specific fields, the schema fields AI agents actually read goes field by field, and an ecommerce robots.txt guide covers the crawler access side that this sector, to its credit, already had right.
The uncomfortable thing about this data is not that these stores are badly run. Several of them are clearly well run. It is that being well run, on a mainstream platform, with the doors wide open to every crawler that asks, is currently not enough to be legible to the software that is increasingly doing the choosing.
Run a free scan and see which of these your own store passes.