Tweet
Hi,
I've got quite a few accounts with a few GB PMEM and unlimited process and IO.
These are Wordpress accounts.
I'm finding a few where there's a server wide IOWAIT alert, usually lasting 15 to 25 minutes, although sometimes hours.
On logging in, iowait from top is jumping about from 15 ish to 80ish.
lveps shows a single problem account using max io and cpu.
There are typically a few hundred lsphp processes spawned by this account.
Strace shows most threads are doing nothing or acting very sluggishly.
Every few seconds though, iotop shows 5 to 10 at random start IO reading together for a second or so.
This increases the system wide IOWAIT.
The WordPress website typically has a small database and there are typically 75ish sleeping mysql processes associated
with it.
Inspection of the domlogs reveals the problem starting when the website is briefly bombarded with
hits from a hacker or search engine.
This creates an initial large flurry of EP, threads etc until the account hits its 2GB PMEM, starts throwing 503s
and all processes slow down and stack.
The OOM killer slowly starts picking off a few lsphp processes however this has no meaningful effect.
I've tried pkill -u and killall lsphp etc however the processes quickly exhume themselves, even though
the incoming bombardment has stopped. The only way I've found to remedy this is an httpd restart.
This always resolves the problem.
Does anyone know how these lsphp processes recreate themselves? And if there's a nicer remedy other than httpd restart?
The restart obviously affects all accounts on the server, even briefly, which is undesirable.
Thanks
I've got quite a few accounts with a few GB PMEM and unlimited process and IO.
These are Wordpress accounts.
I'm finding a few where there's a server wide IOWAIT alert, usually lasting 15 to 25 minutes, although sometimes hours.
On logging in, iowait from top is jumping about from 15 ish to 80ish.
lveps shows a single problem account using max io and cpu.
There are typically a few hundred lsphp processes spawned by this account.
Strace shows most threads are doing nothing or acting very sluggishly.
Every few seconds though, iotop shows 5 to 10 at random start IO reading together for a second or so.
This increases the system wide IOWAIT.
The WordPress website typically has a small database and there are typically 75ish sleeping mysql processes associated
with it.
Inspection of the domlogs reveals the problem starting when the website is briefly bombarded with
hits from a hacker or search engine.
This creates an initial large flurry of EP, threads etc until the account hits its 2GB PMEM, starts throwing 503s
and all processes slow down and stack.
The OOM killer slowly starts picking off a few lsphp processes however this has no meaningful effect.
I've tried pkill -u and killall lsphp etc however the processes quickly exhume themselves, even though
the incoming bombardment has stopped. The only way I've found to remedy this is an httpd restart.
This always resolves the problem.
Does anyone know how these lsphp processes recreate themselves? And if there's a nicer remedy other than httpd restart?
The restart obviously affects all accounts on the server, even briefly, which is undesirable.
Thanks
Comment