High I/O and stuck httpd procs. after ClouldLinux Upgrade

[jack]

We have been testing Cloudlinux for sometime now and overall I think its its great and going to do wonders for our customers.

That being said, I feel that there is still alot to do with having the right default/server-wide settings.

After our Cloudlinux upgrade (from Centos), we have been noticing a higher then normal I/O wait time and at times httpd processes will get stuck.

1. Each server has its own typical usage levels, but on the servers we have been testing it with, the average I/O went up about 10-30%. At times the IO is seriously impacting overall performance.

2. At times httpd processes will get stuck. The entire Apache queue will fill up and then the server stops loading sites. The overall process count per user is at the right levels (based on LVE), but the processes either don die/end or are throttled to the point that they stay in the queue.

Note: This does not happen all the time, but there seems to be spikes of this type of activity. We did not get this type of activity before the upgrade.

Some background:

Server - 8 Cores, 16GB Ram.
LVE Setup: 4 Cores, 10% CPU, 30 Procs, 2GB Ram

Any advice guys?

Thank you in advance ...

[jack]

...

[iseletsk]

Sounds like something wrong. We see it from time to time, most common reasons for #1:
1. You are running vmware, without vmware tools compiled
2. Something went wrong after reboot (we have seen mysql db getting corrupted, sa-compile running crazy)
3. Some user started to mis-behave, using too much IO (coincidence)
4. Try disabling UBC (memory limits). To do that: lvectl ubc disable --save. We had that problem in earlier kernels, but I believe it was fixed in lve0.8.29 & later kernels.
5. You are using PAE kernel, and running low on lowmem, if that is the case, switch to enterprise kernel (though it is rare to see 8 core 32bit servers)

#2 This shouldn be related to CL -- something else going on.

Either way -- contact us at helpdesk.cloudlinux.com
We will need server info to investigate.

[jack]

Thanks Igor. We are not running any vm software. We were running centos 5.x. The typical customer we have is a high usage account but the io still seems pretty high.

We will try your suggestions and let you know.

[herjanto]

I also see the IO went up crazyly although the server load is very low.
I just use lvectl ubc disable --save and I will update this thread if it works to lower the IO rate.

[iseletsk]

Did disabling UBC help?
Any progress with the IO issue?

[james.smith]

Hi,

Ive been testing Cloud Linux over the last few days and I am seeing a significant increase in I/O.

We
e running several thousand domain names within a virtualised environment, across several VMs, from one SAN. Before running CloudLinux everything runs OK, but with CloudLinux on just a handful of VMs, the SAN IO has increased by at least 20-30%, with every VM now seeing high IOWAIT.

Displaying UBC appears to help, Ill need further testing over the next few days to be confident.

Same details about our environment:

- Dom0s are running XenServer
- VMs are CentOS 5.6, with XenTools installed (XenServer enhancement, increases IO and does other improvements)
- SAN is 8 bay 2.5 inch Raptors

[macscr]

Did either of you get this resolved? I am not even trialing CL yet because of the IO concerns I have been hearing about and with myself using Xen and an iSCSI SAN, disk IO is always a concern.

[skynight]

I am having the same problem. After i disable ubc (memory limits), everything ok.

lvectl ubc disable --save