Extremely high load

[chris]

High load, over 140

[chris]

So about 20 minutes ago I noticed WHM not responding. I checked the server status and load was around 40, by the time I got logged in via SSH it was up over 140.....

I restarted cpanel and load has came down. I have been getting alot of brute force notifications and I have been blacklisting them.

However, I was under the impression that this is exactly what Cloud Linux is for(stopping high resource utilization)

Any tips or pointers would be great.

[iseletsk]

Yes, it should have stopped high resource utilization

if you could run top and send a copy of the output when something like this happens -- it would help diagnose the issue. As it is really hard to tell what exactly might have happened.

[chris]

I was struggling to get anything done.

I guess I am wondering if I missed anything during setup.

Currently there are 375 of these processes

/usr/local/cpanel/bin/dovecot-auth /usr/libexec/dovecot/checkpassword-reply

[iseletsk]

I see what is going on. We actually don protect against this particular thing. We don limit mail, nor system services. Sounds like someone overloading the system with brute force attack against dovecot auth.

It should be possible to put it into lve-wrapper, and that would help negate the load for everyone (though will make dovecot authentication not usable when such attack persist), but we don do it out of the box.

[chris]

ok, I will look into that. Thanks!

On another note, how do I verify that I configured everything correctly?

[iseletsk]

Easiest way -- run lvetop
if you see multiple active users there -- it is configured correctly.

[chris]

Hmmm, it is only showing 1 user out of the 2 it should be showing....

[iseletsk]

Is there active traffic for both users?
Try hitting second user with
ab -c 2 -n 10000 URL_TO_SECOND_USER_SITE

and see if that user appears in lvetop

[chris]

Sorry that is 1 out of 3, and that is only showing a ssh only user, not any cpanel users.

[chris]

Cool thanks, works for all but 1 client

[john]

> Igor Seletskiy wrote:
> I see what is going on. We actually don protect against this particular thing. We don limit mail, nor system services. Sounds like someone overloading the system with brute force attack against dovecot auth.
> It should be possible to put it into lve-wrapper, and that would help negate the load for everyone (though will make dovecot authentication not usable when such attack persist), but we don do it out of the box.

Igor how do you actually add another service into an LVE wrapper? more specifically i want to add in r1soft CDP since its causing some load issues (sorry if i should have created a new thread)

Thanks
John

[chris]

/proc/lve/list

Is missing one of the clients... How do I add it?

[iseletsk]

You don need to add them. They should be picked up automatically on SSH/php via web access.

If they don show up after you hit them with ab -- check that you have mod_hostinglimits installed
If you are running plesk -- make sure you have SuexecUserGroup directive (cgi enabled) for those customers.

[iseletsk]

John,

Basically if you have a command like:

/usr/bin/runthis

And you want to run it inside LVE, execute it via:
/usr/sbin/lve_wrapper 33 /usr/bin/runthis

Where 33 is LVE id -- you can use any number here.

All children of that process will be in the same lve, and you can control them all.

If you want to continue to execute /usr/bin/runthis (for example because WHM executes it that way), do:
mv /usr/bin/runthis /usr/bin/runthis.orig
echo <<DONE > /usr/bin/runthis
#!/bin/bash
/usr/sbin/lve_wrapper 33 /usr/bin/runthis.orig
DONE

chmod +x /usr/bin/runthis

The only problem with such approach is that if software updates/overwrites runthis, you need to re-create it.