SWARM-DocReader

You are probably here because this name appeared in your server logs:

SWARM-DocReader/<version> (+https://swarm-software.ai/bot/)

SWARM is a self-hosted IT operations platform. It connects to products through their APIs, and it keeps what it knows about those APIs current by reading each vendor's own public documentation. SWARM-DocReader is the part that does the reading. Each request comes from a customer's own SWARM installation, not from a central crawler.

What it requests

  • Public documentation pages for a product the installation integrates with: the API reference, a published OpenAPI description, a changelog, release notes, deprecation notices.
  • Only addresses an administrator registered for that product, and pages on the same host that those documents link to.
  • It does not follow links across a site, index pages, or collect anything other than the text of the document it was asked for.

How it behaves

  • It fetches /robots.txt first and obeys it. If the file cannot be fetched, it reads nothing from the site.
  • At most one request a second to a host.
  • It sends If-None-Match and If-Modified-Since, so an unchanged document costs a 304.
  • It sends no cookies and no credentials, and does not run scripts.

Keeping it out

Add a group for it to your robots.txt:

User-agent: SWARM-DocReader
Disallow: /

A narrower Disallow works too. Rules for User-agent: * apply to it when no group names it.

Questions

If it misbehaves on your site, tell us and include the time and the path requested.