SenseverCrawl

SenseverCrawl is the web crawler of Sensever. If you found it in your server logs, this page explains what it does and how to stop it.

What it does

Sensever projects can have an AI assistant that answers visitors' questions. A project owner can connect their own website to it; SenseverCrawl then reads the public pages of that site so the assistant can answer from them and link to the right page.

It identifies itself with this User-Agent:

SenseverCrawl/1.0 (+https://crawl.sensever.com/)

When it opens a JavaScript-only page in its headless browser, the browser's User-Agent ends with the same SenseverCrawl/1.0 (+https://crawl.sensever.com/).

Which sites it visits

• Only sites whose owner has confirmed ownership in Sensever: a DNS TXT record, a <meta name="sensever-crawl"> tag on the home page or the file /.well-known/sensever-crawl.txt. Ownership is checked again every 30 days.
• It does not follow links to other sites and never fills in forms or logs in.
• It keeps the main text of pages, not the raw HTML. When ownership is lost, the site is no longer crawled and its index is deleted after 14 days.

How it behaves

• It reads and follows robots.txt: Disallow and Allow rules, Crawl-delay and Sitemap lines. A group for SenseverCrawl takes precedence over *.
• At most 2 requests per second and 2 connections to a site; a longer Crawl-delay wins.
• On 429 or 503 it pauses for the time given in Retry-After.
• A page that shows its text only through JavaScript may be opened once in a headless browser, under the same rules: no images, fonts or media are loaded, nothing is clicked or submitted.
• It respects noindex and canonical links, uses conditional requests (ETag, Last-Modified) and revisits a site about once a day.

How to block it

Add this to your robots.txt:

User-agent: SenseverCrawl
Disallow: /

To keep it out of part of the site only, list those paths, for example Disallow: /private/. Changes are picked up on the next visit.

Contact

Questions or a problem with the crawler: crawl@sensever.com. Please include your domain and, if you can, the time and address of the requests.

Sensever · sensever.com