How to stop malicious crawlers?

[glenn]

I've seen crawlers come through using a different IP for a different session, i.e. 47.128.*.*

How can I add this to the blacklist, and is there a way to get Imunify to better identify when someone is using 100s of IPs to get around the firewall and automatically block it? Imunify does not seem to pick up on this kind of attack.

Thanks
G

[alevchenko]

Hi G,

Imunify360 can protect from attacks coming repetitively from the same IP address. Attacks from various IPs (and activity like DDoS) are not currently identified by Imunify, unfortunately. (See for reference https://cloudlinux.zendesk.com/hc/en...ith-Imunify360)

As an option, you can add the whole subnet for addresses 47.128.*.* to the Black List: https://docs.imunify360.com/dashboar...dd-ip-manually

For crawler blocking, you can use a custom rule for blocking one: https://cloudlinux.zendesk.com/hc/en...ng-custom-rule

[alevchenko]

Hi G,

Imunify360 can protect from attacks coming repetitively from the same IP address. Attacks from various IPs (and activity like DDoS) are not currently identified by Imunify, unfortunately. (See for reference https://cloudlinux.zendesk.com/hc/en...ith-Imunify360)

As an option, you can add the whole subnet for addresses 47.128.*.* to the Black List: https://docs.imunify360.com/dashboar...dd-ip-manually

For crawler blocking, you can use a custom rule for blocking one: https://cloudlinux.zendesk.com/hc/en...ng-custom-rule

[glenn]

How do I express the IP in the IP field? Like this: 47.128.*.* ?

Thanks

[alevchenko]

If I understand you correctly and you wish to block all addresses that belong to the subnet "47.128.*.*" which is mentioned in your initial request, you can use "47.128.0.0/16" if you are blacklisting in the UI and the following command if you wish to do it via the CLI:

imunify360-agent ip-list local add --purpose drop 47.128.0.0/16 --comment "block crawlers"

Refer to https://en.wikipedia.org/wiki/Classl...v4_CIDR_blocks for corrections.

[glenn]

Thanks Alevchenko. I will use.

On a side note, it would be nice if Imunify had some intelligence built-in to detect these types of subnet attacks. I only notice them when it impacts server performance and many such attacks get through.

[alevchenko]

Thanks for the feedback!

I'd like to inform you that our dev indeed has long-term plans for implementing the HTTP DDoS & enumeration attack defence in Imunify360 but for now we either have news for this feature nor ETA.

I believe it will be definitely announced on our blog https://blog.imunify360.com/