QuarrywiseQuarrywiseBot
If you found this address in your server logs: this is the crawler behind the Quarrywise data tools. Here is what it does on your site, and how to make it stop.
How to recognise it
Every request sends this User-Agent:
QuarrywiseBot/1.0 (+https://quarrywise.com/bot)The tools run on Apify's cloud, so requests come from Apify's infrastructure and we cannot publish a fixed list of IP addresses. The User-Agent is the reliable way to identify us.
What it fetches
- Official public APIs, such as government open-data registers.
- Feeds and job-board APIs that companies publish for their own career pages.
- Pages and files that your robots.txt allows.
It never logs in, never uses session cookies, never fills in forms and never solves CAPTCHAs or works around a block. It does not collect personal data: names, email addresses and phone numbers are removed before anything is stored.
Rate limits
- Throttled per host: no more than two requests at a time to one host, and always under the rate limit the source publishes.
- On
429 Too Many Requestsit backs off, followingRetry-Afterwhen you send one and waiting longer after each repeat. - On
403 Forbiddenor a challenge page it stops for that host. It does not retry from another address. - It caches responses and sends conditional requests where the source supports them, so unchanged data is not downloaded twice.
How to opt out
Add this to the robots.txt at the root of your site:
User-agent: QuarrywiseBot
Disallow: /You can also disallow only some paths. The bot reads robots.txt at the start of every run, so a change takes effect on the next run, and it honours rules for User-agent: * as well. If robots.txt cannot be reached because of a server error, it skips your site for that run.
Contact
Seeing traffic you did not expect, or want your data left out entirely? Email hello@quarrywise.com with your domain and, if you can, a sample log line. We reply, and we stop when you ask.