mod_hostinglimits:Error on LVE enter -38

[admin]

Hello,

Anyone can help me fix this error ? All sites got error 503, when i check error logs then i got this: mod_hostinglimits:Error on LVE enter -38

Thank you !

[iseletsk]

Sounds like LVE is disabled. Are you running CentOS kernel? or liblve RPM gone missing.

You can disable mod_hostinglimits by editing /etc/httpd/conf/conf.d/modhostinglimits.conf
and commenting out LoadModule

[security]

CloudLinux will be working nicely most of the time, but about once a day Im getting the same errors. LVE is enabled as far as I know. Once the error happens a few times, all sites go 503, sometimes for hours at a time and customers become angry. The next error that occurs immediately after is related to fcgid being out of memory.

If I comment out mod_hostinglimits will that do the trick, or is upgrading the memory the real solution? Second question, what kind of protection will I be missing out on if mod_hostinglimits is commented out?

Snippet from todays Apache error_log data (customers domain has been changed to example.com for their privacy):

[Sun Feb 05 01:47:13 2012] [error] mod_hostinglim its:Error on LVE enter: LVE(616

) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(10359) er

rno (7)

[Sun Feb 05 01:47:13 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	

run /usr/local/cpanel/cgi-sys/php5

[Sun Feb 05 01:47:13 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn p

rocess /usr/local/cpanel/cgi-sys/php5 error

[Sun Feb 05 01:47:14 2012] [error] mod_hostinglim its:Error on LVE enter: LVE(616

) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(10351) er

rno (7)

[Sun Feb 05 01:47:14 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	

run /usr/local/cpanel/cgi-sys/php5

[Sun Feb 05 01:47:14 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn p

rocess /usr/local/cpanel/cgi-sys/php5 error

[Sun Feb 05 01:47:14 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	

run /usr/local/cpanel/cgi-sys/php5

[Sun Feb 05 01:47:14 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn p

rocess /usr/local/cpanel/cgi-sys/php5 error

The memory error then continues on to fill up the error log for a while.

[brooks]

Is this CL version 5 or 6? What kernel are you running? To what values are your LVE memory limits set? How much memory does the VM/physcial server have? What does "free" show when you
e seeing this problem?

[security]

Thank you for your reply. Please forgive me for my delayed response. I was not aware right away that there was a reply. I will check to make sure I have e-mail notifications enabled.

> Kevin Brooks wrote:
> Is this CL version 5 or 6?

WHM 11.30.6 (build 3)
CLOUDLINUX 5.7 i686 standard

> Kevin Brooks wrote:
> What kernel are you running?

2.6.18-338.5.1.el5.lve0.8.29PAE

> Kevin Brooks wrote:
> To what values are your LVE memory limits set?

The default values are:

CPU: 6
MEM: 500MB
Concurrent Connections: 80
Number of Cores Per LVE: 1

The LVE ID shown in the logs uses these default values.

> Kevin Brooks wrote:
> How much memory does the VM/physcial server have?

12GB which can be upgraded to 16GB if that is the solution.

> Kevin Brooks wrote:
> What does "free" show when you
e seeing this problem?

I actually do check free when it happens. It usually says that not all of the memory has been used and there is still some free, but not much. I have not seen the server use swap when this happens.

It happened again today. Here is what I found using grep on the Apache error log.

Using grep to look for the LVE error:

[Sun Feb 12 10:44:10 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(2565) errno (7)

[Sun Feb 12 10:44:11 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(2562) errno (7)

[Sun Feb 12 10:44:11 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(2570) errno (7)

[Sun Feb 12 10:44:11 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(2602) errno (7)

[Sun Feb 12 10:44:11 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(2629) errno (7)

[Sun Feb 12 10:44:12 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(2633) errno (7)

[Sun Feb 12 10:44:12 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(2635) errno (7)

Using grep to look for the memory error which happened within the same minute:

Sun Feb 12 10:44:25 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	 run /usr/local/cpanel/cgi-sys/php5

[Sun Feb 12 10:44:25 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn process /usr/local/cpanel/cgi-sys/php5 error

[Sun Feb 12 10:44:25 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	 run /usr/local/cpanel/cgi-sys/php5

[Sun Feb 12 10:44:25 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn process /usr/local/cpanel/cgi-sys/php5 error

[Sun Feb 12 10:44:25 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	 run /usr/local/cpanel/cgi-sys/php5

[Sun Feb 12 10:44:25 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn process /usr/local/cpanel/cgi-sys/php5 error

[Sun Feb 12 10:44:25 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	 run /usr/local/cpanel/cgi-sys/php5

[Sun Feb 12 10:44:25 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn process /usr/local/cpanel/cgi-sys/php5 error

[brooks]

It looks like you
e running out of LVE entry procs and memory. Given your defaults I would recommend increasing LVE max_memory and decreasing the number of LVE entry procs. LVE max memory is accounted for cumulatively across all cron and Apache spawned CGI procs. Meaning that you
e spreading 500 MB of virtual memory across 80 LVE entry procs.

[security]

I lowered the connection limit for ID 616 fr om 80 to 60. I also increased the default memory limit for everyone fr om 500MB to 1GB. I still had server-wide 503 errors about once a day for the last few days during my observations. Today, the web server went 503 for 3 hours and the Apache error logs show the same thing fr om midnight on to this afternoon. Here is a clip:

[Thu Feb 16 10:32:35 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	 run /usr/local/cpanel/cgi-sys/php5

[Thu Feb 16 10:32:35 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn process /usr/local/cpanel/cgi-sys/php5 error

[Thu Feb 16 10:32:35 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	 run /usr/local/cpanel/cgi-sys/php5

[Thu Feb 16 10:32:35 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn process /usr/local/cpanel/cgi-sys/php5 error

[Thu Feb 16 10:32:35 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	 run /usr/local/cpanel/cgi-sys/php5

[Thu Feb 16 10:32:35 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn process /usr/local/cpanel/cgi-sys/php5 error

[Thu Feb 16 10:32:35 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	 run /usr/local/cpanel/cgi-sys/php5

[Thu Feb 16 10:32:35 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn process /usr/local/cpanel/cgi-sys/php5 error

[Thu Feb 16 10:32:35 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	 run /usr/local/cpanel/cgi-sys/php5

[Thu Feb 16 10:32:35 2012] [warn] (12)Cannot allocate memory: mod_fcgid: spawn process /usr/local/cpanel/cgi-sys/php5 error

[Thu Feb 16 10:32:35 2012] [error] (12)Cannot allocate memory: mod_fcgid: can	 run /usr/local/cpanel/cgi-sys/php5

[Thu Feb 16 10:37:53 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(574) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(12291) errno (7)

[Thu Feb 16 10:37:54 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(574) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/browse.php) TID(3006) errno (7)

[Thu Feb 16 10:37:54 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(574) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/includes/process.php) TID(4282) errno (7)

[Thu Feb 16 10:37:54 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(574) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/browse.php) TID(3006) errno (7)

[Thu Feb 16 10:37:55 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(574) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/includes/process.php) TID(26810) errno (7)

[Thu Feb 16 10:37:56 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(574) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/includes/process.php) TID(2591) errno (7)

[Thu Feb 16 10:37:56 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(574) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/browse.php) TID(333) errno (7)

[Thu Feb 16 10:37:56 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(574) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(12291)

Earlier than that was the same error but fr om a different LVE ID:

[Thu Feb 16 00:23:11 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(30999) errno (7)

[Thu Feb 16 00:23:11 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(31232) errno (7)

[Thu Feb 16 00:23:11 2012] [error] mod_hostinglimits:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(31234) errno (7)

[Thu Feb 16 00:23:11 2012] [error] mod_hostinglim its:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(30907) errno (7)

[Thu Feb 16 00:23:11 2012] [error] mod_hostinglim its:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(31237) errno (7)

[Thu Feb 16 00:23:11 2012] [error] mod_hostinglim its:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(31235) errno (7)

[Thu Feb 16 00:23:11 2012] [error] mod_hostinglim its:Error on LVE enter: LVE(616) HANDLER(fcgid-script) HOSTNAME(example.com) URL(/index.php) TID(31239) errno (7)

If you were to tail -f my error log, you would see these errors ALL day. It is becoming very frustrating. I thought the point of CL was to prevent this.

[security]

One other point. I upgraded the memory on this server yesterday, but when the "can allocate memory" error and subsequent 503 errors happened, the memory on the server was only half used up.

[security]

It seems lowering the connection limit and increasing the memory limit helped along with adding some more memory to the server. Thanks again for your help.