Crawler reference
SextantBot
If you found SextantBot in your server logs, this page explains what it is, what it requested, and how to stop it.
Who it is
SextantBot is the scanner behind Sextant, operated by Adcock Stanton Limited. It reads a store the way an AI shopping agent would and produces a readiness report on what those agents can and cannot understand about the products.
It sends this user-agent string on every request:
SextantBot/1.0 (+https://sextantscan.com/bot)
Requests from other parts of the product carry the same string with a purpose appended, for example ; hosted-file-verify for the weekly check that a customer's own files are still being served.
What it requests
A scan is a single short visit, not a crawl. It reads the homepage, the discovery files at your origin root (/robots.txt, /sitemap.xml, /llms.txt, /llms-full.txt, /agents.md, /.well-known/ucp), then samples up to 5 product pages and 3 policy pages found from your sitemap or homepage links.
That is 9 to 20 requests in total, over a few seconds, with at most three in flight at once (one at a time if your robots.txt asks us to space them out - see below). It requests GET only. It does not submit forms, does not attempt to log in, does not follow checkout flows, and does not execute JavaScript.
How often
Usually once. A store is rescanned only if someone asks for a fresh report, or if its owner is a customer who has asked us to monitor it for changes.
Crawl-delay
We read Crawl-delay from whichever group in your robots.txt applies to us, and space requests by it. We do not honour it without limit, so rather than claim we comply, here is exactly what we do.
A scan is a bounded visit that ends, not a crawler that keeps coming back. Crawl-delay exists to cap the sustained load of something that does; against a one-off visit of this size, honouring a large value literally just spreads the same handful of requests over several minutes. So we bound it two ways: each individual wait is capped at 2 seconds however large the declared value, and the total a single scan will spend waiting is capped at about 3 seconds when someone is waiting on a report, or about 45 seconds for a background scan with nobody watching. Past that the remaining requests proceed unspaced.
Concretely, with Crawl-delay: 1: a background scan spaces every request by a second and honours it in full; a scan someone is waiting on spaces the first three requests and then continues without waiting. With Crawl-delay: 10, no scan honours the stated rate at all, because each wait is capped at 2 seconds. Every scan records which of these happened, so if you ask what we did on a particular visit, we can tell you precisely rather than from memory.
If what you want is for us not to visit at all, Crawl-delay is the wrong instrument for it. The one below is the right one, and we honour that without qualification.
How to block it
Add this to your robots.txt:
User-agent: SextantBot Disallow: /
Our outbound prospecting reads robots.txt before it scans anything and skips your store entirely if that rule is present, without fetching a single page of it. We also remember that you declined, so later batches do not re-ask; we re-check every 90 days, in case the rule was removed, and that re-check is a single robots.txt request.
One case it deliberately does not cover: if a person enters your store's address on our site and asks for a report, we run that scan. That is somebody requesting a document about a public storefront, the same as a browser fetching it, and robots.txt is a convention for unsolicited crawling rather than a block on anyone ever looking. If you would rather we did not serve reports on your store at all, email hello@sextantscan.com and we will honour that.
If it caused a problem
It should not have - the request volume above is smaller than a single page view in most browsers. If it did, tell us at hello@sextantscan.com with the timestamps from your log and we will look into it and stop scanning your domain in the meantime.
If your concern is about personal data rather than server load - an address of yours we hold, or an email you did not ask for - there is a complaint form that acknowledges itself immediately and needs no account.