Imunify360 blocking Googlebot

[volex.aff]

Hi,

Weve had an issue reported which is fairly major as its affected a clients Google search engine listing, Im attaching an image to avoid this getting spidered. I assume Googlebot IPs should be automatically detected and not go through captcha? Any idea on a resolution for this as the client is understandably annoyed at this and we dont want it affecting others.

Thanks

[rshaymardanov]

Hi Darryl

Google bot should be automatically whitelisted by Imunify360 according to this document: https://support.google.com/webmaster...er/80553?hl=en

Can you please submit support ticket to https://cloudlinux.zendesk.com/ so we can check if there is any misconfiguration on your server?

[volex.aff]

Hi,

Assuming this is the whitelist used it doesnt look correct:

cat /var/imunify360/files/whitelist/v2/google.txt
# Google
# IPv4
# https://www.lifewire.com/what-is-the...-google-818153
8.8.8.8
8.8.4.4
74.125.224.72
64.233.160.0/24
66.102.0.0/24
66.249.64.0/24
72.14.192.0/24
74.125.0.0/24
209.85.128.0/24
216.239.32.0/24
64.68.90.0/24
65.249.64.0/19
# IPv6
# https://developers.google.com/speed/...dns/docs/using
# https://developers.google.com/speed/...dns/docs/dns64
# https://www.wordfence.com/blog/2015/05/ #wordfence-launches-ipv6-support-announces-wordfence-6-and-passes-6-million-downloads/
# ipv6.google.com
# 2001:4860:4860::8888
# 2001:4860:4860::8844
# 2001:4860:4860::6464
# 2001:4860:4860::64
# 2607:f8b0:4003:c04::8a
# IPv6 /64
2607:f8b0:4003:c04::/64
2001:4860:4860::/64

Based on that site it lists: 66.249.64.0 – 66.249.95.255

So it should be 66.249.64.0/19 not 66.249.64.0/24

This may actually be a typo as I can see 65.249.64.0/19 is listed but that doesnt appear to be a google IP range its a Cisco range.

After a bit of digging it looks like pretty much all the IPv4 CIDR ranges are wrong, Ive corrected these here:

64.233.160.0/19
66.102.0.0/20
66.249.64.0/19
72.14.192.0/18
74.125.0.0/16
209.85.128.0/17
216.239.32.0/18
64.68.88.0/21

[volex.aff]

Hi,

Seriously, whats going on with QA in CloudLinux? For a few weeks now there have been very serious bugs, like that one that left us without a database connection for proactive defense, but this is already the last straw.

It is understood that the software can have unexpected bugs, but do wrong up to the white lists for Google ... it is necessary to improve the QA immediately.

Imunify360 is an expensive solution, and customers who pay something like that is because we need guarantees.

Greetings,

Translated with http://www.DeepL.com/Translator

[volex.aff]

These errors do appear to be mirrored in the current live whitelist versions:

https://files.imunify360.com/static/.../v2/google.txt from https://files.imunify360.com/static/whitelist/v2/

Id suggest reviewing all the other whitelists to ensure they are correct.

[volex.aff]

Hello, main source for bots are white domains: https://files.imunify360.com/static/...te_domains.txt

and whitelist generated for resolved ips: https://files.imunify360.com/static/...earch-bots.txt

google.txt is for fallback purposes only.

[volex.aff]

I can see the changes that I made in the google.txt are reflected in the /etc/imunify360-webshield/webshield-http.conf.d/static-whitelist.conf - the https://files.imunify360.com/static/...earch-bots.txt doesnt contain the IP that was being blocked - 66.249.66.219 so this wouldnt have been covered by the original google.txt or that file.

Perhaps https://files.imunify360.com/static/...te_domains.txt would cover it if an rdns lookup is done on each request though then it shouldnt have been blocked. Where is this file stored locally so I can check if there is an issue with that file as I cant see it in /etc/imunify360, /etc/imunify360-webshield or /var/imunify360

Thanks

[iseletsk]

This issue was caused by junk DNS responses from one of the DNS servers we used.
We are changing our algorithms to re-validate DNS responses based on previous state, as well as a secondary DNS server.