If you found SnadderBot/1.0 in your server logs, this page explains what it was doing and how to stop it.
What it is
SnadderBot/1.0 fetches a small number of pages from a website so that the person who asked for the scan can see a summary of that site's marketing: what it sells, how it describes itself, and which analytics and advertising tags it loads. SnadderImageMirror/1.0 fetches individual images referenced by those pages.
A scan is always started by a person typing or connecting a specific address. There is no continuous or exploratory crawl of the web, and no index is built.
What it fetches
- The address given, and your sitemap.xml and robots.txt if present.
- Up to six further pages on the same host, chosen from your sitemap or from links on the first page.
- For a shop, product pages listed in your product sitemap, and a set of common product-feed paths.
- Images referenced by those pages, in order to read the dominant brand colour and the site icon.
How to stop it
Write to us with the host you want excluded and we will stop fetching it. That is the reliable method today, and we will confirm it to you.
We are adding support for robots.txt, after which a "User-agent: SnadderBot" / "Disallow: /" rule will be enough on its own. It is not yet: at present our crawler reads your robots.txt only to find your sitemap, and does not act on Disallow rules.
You should also know that if a plain request is refused — a 403, a 429, or a challenge page — we currently retry the page through a headless browser that presents an ordinary Chrome user-agent. That means a block keyed to our user-agent alone may not hold. We are removing that behaviour for sites the requester has not verified they own.
What we keep
We keep the extracted text of the pages fetched, their titles and descriptions, the list of image addresses, and a screenshot of the first page, so the person who asked can read the result. We do not keep the raw HTML.
If you believe a scan of your site was run by someone with no business doing so, write to us and we will delete the record and tell you what it contained.
Contact
Write to the address on our privacy policy. Include the host that was crawled and, if you have it, a timestamp from your logs.