This is VitalCrawlBot, a website uptime monitor
You probably found this page in your server logs: our bot requested your site and identified itself as VitalCrawlBot/1.0 (+https://vitalcrawl.com/bot).
VitalCrawl is a website monitoring service. It checks that a site loads, that its certificate is valid and that the homepage content has not been tampered with. Checks never start on their own: the site owner or the agency that maintains the site set them up. We do not index your site, collect content for search or probe random URLs.
How it behaves
- One request per check. The monitor owner sets the interval: 5 minutes or longer on most plans, and never more than once a minute on the top plan. A retry is sent only when the site does not respond, with at most two confirmation attempts so a single network glitch does not raise an alert.
-
Usually headers only.
When the status code is enough, we send a HEAD request and skip the body. When we need the body, we request it compressed (
Accept-Encoding: gzip). -
A full audit once a day.
Once a day, in addition to the homepage, we read
robots.txt,sitemap.xmland a short list of common system paths to warn the owner if something is exposed that should not be. No other check does this. -
We respect 429.
A "Too Many Requests" response pauses checks for this site until the time given in
Retry-After, and it is not counted as downtime. - No logins and no forms. The bot never tries to log in to an admin panel, guess passwords or submit forms, except those the monitor owner configured for their own site.
If your site is behind a browser check
Bot protection (browser checks, CAPTCHA) only lets through clients that run JavaScript and returns 401 or 403 to us. We do not solve these challenges or pretend to be a browser. To let checks through, open the monitor settings, click "Generate secret" and add this header to your protection rule:
X-VitalCrawl-Check: <your secret>
We send it with every request to your site, so the protection lets us, and only us, through. Do not allowlist us by User-Agent: anyone can fake it. The secret or the addresses above are more reliable. Only you know the secret, and rotating it revokes the previous one immediately.
Cloudflare. In the dashboard open Security → WAF → Custom rules and add a rule with the action “Skip” (or “Allow”) for the expression below. Bot Fight Mode is a separate switch, and with it on Cloudflare may still answer our checks with its own page; a custom Skip rule wins over it. Such an answer we report as “the shield blocked our check”, not as an outage — nobody gets woken up for it.
http.request.headers["x-vitalcrawl-check"][0] eq "<your secret>"
How to limit or block it
-
Slow it down.
Respond with
429and aRetry-Afterheader, and we will wait for the given time automatically. - Block it. Block our User-Agent or the addresses above. We do not bypass blocks or disguise ourselves as a browser; the monitor owner will see that the site is unavailable to us.
- Opt out completely. If nobody ordered monitoring for your site, email support@vitalcrawl.com with the domain and we will stop checking it.