S Silo SEO Scanner

Crawler information

SiloSeoScanner

SiloSeoScanner is the crawler behind Silo SEO Scanner, a website audit tool operated by Silo (silo.ge). It visits a website only when someone asks for an SEO audit of that site, and checks things such as page titles, broken links, sitemaps, mobile readiness and response times.

How to recognise it

User agent
SiloSeoScanner/1.0 (+https://seo-scanner.silo.ge/bot)
robots.txt token
SiloSeoScanner
Cryptographic identity (Web Bot Auth)
Requests are signed with HTTP Message Signatures (RFC 9421, Ed25519, tag="web-bot-auth") and carry Signature-Agent: "https://seo-scanner.silo.ge". Our public keys are published at https://seo-scanner.silo.ge/.well-known/http-message-signatures-directory. Treat any request that claims to be SiloSeoScanner without a valid signature as an impostor.

How it behaves

  • Crawls only after a person requests an audit of a specific website. There is no background crawling of the web.
  • Audits at most 25 unique pages per audit (never more than 50 page requests, counting duplicate URLs whose canonical version is also fetched), with no more than 2 requests at a time, and stops after 60 seconds.
  • Follows robots.txt (RFC 9309), including Disallow, Allow and Crawl-delay, for every request, including redirect targets on any host. With a crawl delay, it sends one request at a time and waits the full delay between them.
  • If robots.txt returns a server error or does not respond, it does not crawl that site at all.
  • Requests one deliberately non-existent address per audit (/silo-seo-scanner-404-check-…) to check that missing pages correctly return 404.
  • For sites that build their pages with JavaScript, opens up to 10 pages in a headless Chrome browser that uses the same user agent and the same robots.txt rules. It stops early if the site shows no content to automated browsers.
  • Sends only GET requests. It does not submit forms, log in, run JavaScript or download files other than HTML pages, robots.txt, llms.txt and XML sitemaps.
  • Respects bot protection: if a firewall or CDN blocks it, it stops and tells the user. It never tries to get around blocks or challenges.
  • Uses what it fetches only to build the audit report shown to the person who asked. Content is not sold and is not used to train AI models.

How to block or limit it

Add these lines to your robots.txt to block it completely:

User-agent: SiloSeoScanner
Disallow: /

Or slow it down with Crawl-delay: 10 (seconds) in the same group.

Contact

Questions or problems with our crawler? Contact the Silo team at silo.ge.