CrawlOpsBot is the crawler used by CrawlOps to monitor technical SEO signals for projects configured by CrawlOps users.
CrawlOpsBot is an automated technical SEO crawler. It requests web pages, robots.txt files and XML sitemaps so CrawlOps can create crawl snapshots, detect SEO regressions and help website teams understand what changed between deployments.
It is not an advertising bot, scraping bot or content harvesting bot. CrawlOpsBot is used for monitoring websites that have been added to CrawlOps by their owners, developers, agencies or authorized SEO teams.
Requests from the crawler identify themselves with this user-agent:
CrawlOpsBot/1.0 (+https://crawlops.net/bot)The URL in the user-agent points to this page so server administrators can verify the crawler purpose and control access.
CrawlOpsBot focuses on technical SEO and crawlability signals. Depending on project settings, a crawl can check HTTP status codes, redirects, canonical tags, page titles, heading structure, internal links, response time, structured data markers, robots.txt rules and sitemap coverage.
robots.txt is fetched to understand crawl permissions.sitemap.xml and child sitemaps are fetched to compare listed URLs with crawled URLs.CrawlOpsBot is designed to respect applicable robots.txt rules. You can allow or block it explicitly with a dedicated user-agent group.
User-agent: CrawlOpsBot
Allow: /
User-agent: *
Disallow:If you do not want CrawlOpsBot to crawl your website, add this rule to your robots.txt file:
User-agent: CrawlOpsBot
Disallow: /After the rule is published, future CrawlOps crawls should treat the site as blocked according to the configured robots policy.
CrawlOps includes crawl speed controls to reduce server load. The default project setting is 5 hits per minute, which spaces requests by roughly 12 seconds. CrawlOps users can configure this value per project within the allowed limits.
If CrawlOpsBot appears to be requesting your website unexpectedly, check whether your team, agency or development partner configured a CrawlOps project for the domain.
If you operate a website and need help identifying CrawlOpsBot traffic, include the requested URL, timestamp, source IP and user-agent from your logs.