Ahrefs bot crawling issue

[mchernyavsky]

Hello KJ-K,

The Ahref bot subnets are whitelisted and maintained:

https://files.imunify360.com/static/whitelist/v2/ahrefs.txt

https://help.ahrefs.com/getting-started-with-ahrefs/about-ahrefs/what-is-the-list-of-your-ip-ranges

Was it possible to identify the corresponding access log entries and compare IPs to the list?

[KJ-K]

Same issue today with Google bot. Received 500 error for our main domain and one of our subdomains.

With high certainty there was no website downtime recently.

[KJ-K]

Hello,

Received an email from Ahrefs that it has identified pretty much every webpage of our website with status 500. From the logs we can see with certainty that we didn't have any website downtime recently.

Never had issues before with Ahrefs, Semrush, Majestic, Google, Bing, etc. So far we had been seeing this situation happening with Yandex and Imunify360 team even posted a notice to verify that. Yandex had been emailing us for about a year now that once every week their bot was getting into a 500 error on our website.

Now it's Ahrefs turn. Most likely its the same issue that's prevent both bots.

Just wanted to report that.