Clarify new CPU limits

[jules]

Richard, please stop posting.

[ventas]

As i had the same doubts, i contacted Support asking them to explain me a little more how is working now.

Here is the first response:

> Hello.
> True, NCPU abolished in the new versions.
> 1 core = 100%, 2 = 200% core, core 4 = 400%, etc.
> For example, 1 core = 2000MHz. If I set a value of 150% - that account can consume 3000MHz.
>
> For setting limits in the new versions use the CPU-parameter.

I understand that the option of SPEED (% or Mhz) its the mandatory, so the nCPU should be ignored... but, to make sure, i asked that:

> Me:
> So if i set 400 at CPU %, its full 4 Cores.
>
> Even if i set at nCPU 1?, should i just ignore nCPU?
>
> Thanks

> Support:
> Hello, Jose.
>
>> So if i set 400 at CPU %, its full 4 Cores.
> Yes.
>
>> Even if i set at nCPU 1?, should i just ignore nCPU?
> Yes, this option will be ignored.

Hope this helps a little on this topic.

Have a good day

[jules]

@Jose: Thanks for the information

I didn want to create a support ticket as I thought the information would be useful for others here in public. Shame they are able to provide the information in the ticket but not in their own documentation or here on the forums. Disappointing.

[denis]

Several years since installing cloudlinux and I still feel like I am a beta tester...

First of all, on ALL of our cpanel servers upcp got hung up after the latest lve-stats upd ate. I had to manuall kill upcp and do a /scripts/upcp --force afterwards.

This is where upcp hung..

[20140723.024207] [538983] Dependency Installed:
[20140723.024207] [538983] alt-sqlite.x86_64 0:3.8.3.1-1.el5
[20140723.024207] [538983]
[20140723.024207] [538983] Updated:
[20140723.024207] [538983] lve-stats.noarch 0:0.10-31.1.el5
[20140723.024207] [538983]
[20140723.024207] [538983] Complete!
[20140723.024213] [538983] checkyum version 21.1

ANYWAY ..now that I finally got that going again. I look at the WHM lve limits and its just random numbers such as "13" for the cpu limit and I cant change the bloody thing! Trying to edit the default settings says "settings saved" but when I goto the main page it still shows "13" as the cpu limit. WTF? How did it come up with that number to begin with?

OK ANYWAY ... so I thought maybe its just the WHM frontend thats buggy so I login to root shell and use the "lvectl" command.

lvectl limits all shows this..

[~]# lvectl limits all
ID SPEED NCPU VMEM EP IO
default 104 1 1.0G 20 25
514 104 1 1.0G 20 25
643 104 1 1.0G 20 25
.
.. and so on, same for every user.

104 WHAT ????? 104% of cpu ? 104 Mhz ?? WHAT IS IT DOING!!!!

Trying to rese t this limit with this command:

lvectl apply all --speed=25%

does NOTHING. "lvectl limits all" stil shows that "104" as SPEED as WHM still shows "13"

What the hell is going on here? I feel like I have completely lost control over the server as cloudlinux is just putting arbitrary and non-consistent limits in the cli/whm and is not accepting the new ones like "lvectl apply all --speed=25% "

THIS IS NOT GOOD ENOUGH!!! We cant run buggy software that actually has such a huge and visible impact on our servers!

Come on Igor this is not good enough and you know it. You need to test this crap thoroughly before releasing it. We are paying customers and right now I am paying for nothing but stress.
PS> The documentation is and always has been a joke. It is written in such a way that only the person who wrote it could ever understand it.

I am really getting fed up with this. We had a customer make a simple mistake writing a PHP script with an endless loop in it and it crashed the server by exhausing all ram and swap in 2 minutes. I am starting to wonder if were paying for snake oil.

Really really angry right now.

[bogdan.sh]

Delta, Igor is not able to jump in here, so I will try to help. Really sorry for troubles you achieved here, but here are they one by one:

- hanging upcp - we never saw such issue, what is your current cpanel version ( what "/usr/local/cpanel/cpanel -V" say)? If it happens all the time just create a ticket and our techs would help you.

- about LVE Manager inside WHM - it looks like some post update scriptlets were not finished (maybe due to upcp hangs), please try to reinstall it with yum, just "yum reinstall lvemanager" should works

- about lvectl updating --speed, your command is wrong, so it do nothing, the right way to update speed limit for all (non custom LVEs):
lvectl set default --speed=25%
lvectl apply all

However please be sure you understand that new parameter ( it allows you to set CPU limit in terms of % of a single core , 25% means 1/4 of one core), more http://docs.cloudlinux.com/index.html?speed_limits.html

- about 104 value in lvectl list (it shows % by the way, not MHz) I assume you use 8 core server and will try to explain the situation:
* you have 8 cores on a server
* in new format its 800%
* if we set speed=100% (or it was set with the update) this is equal to 1/8 of older limits, or 100/8 = 12.5 , however rounded to upper value = 13 .
* however calculationg vice-versa we are getting 13*4 = 104% .

Unfortunately this could not be fixed in easy way, however we are preparing updated kernel module that will handle it properly, so should be fixed in future.

[bogdan.sh]

And about PHP script with an endless loop, what are the memory limits for that user? And what lveinfo says about it?
lveinfo -u username --period=1d --show-all

[iseletsk]

I am sorry for being MIA. A lot of things required my attention in the past few weeks (and that will continue for 3-4 more weeks I am afraid), and as the result I let documentation & communication via social media to lag behind. I will try to be more responsive.
Jules -- you were right at being angree with me -- I should have paid it more attention.
Delta - I am sorry to hear about upcp being stuck -- next time you see something like that - do ps -auxw, and save the output for us so we can trace it. I havent heard other users having that problem. Sounds strange.

SPEED vs CPU limits. Documentation had been updated to explain SPEED works. http://docs.cloudlinux.com/index.html?limits.html
It overrides both CPU & NCPU options -- and the only option that matters.
LVE Manager will be updated to remove NCPU within next few weeks.
lve-stats will be updated in the next 2-3 months -- as we are working on a very big rework, that should add many cool featurs (like snapshoting of processes & queries when limits are hit, burstable limits and reseller limits).

If anyone has any more questions regarding speed & CPU -- I would be happy to answer them.

[denis]

Hi Bogdan

as it turns out the package "lvemanager" was not installed on any of my cpanel boxes so I must have been using an old WHM frontend for Lve manager ... the new one looks completely different (much nicer).

edit: fixed unrelated problem removed comments regarding it.

In regards to the "104" - it makes no sense. All the servers are 4 core (8 threads). The WHM frontend now shows SPEED as "100" which is what I want if I understood your docs correctly (ie. I want each user to be able to use max 1 core). However "lvectl list" still shows every users SPEED set to "104"
Ive even run

lvectl set default --speed=100%
lvectl apply all

and when I do lvectl list its still showing "104" in the SPEED column. is this just a problem with lvectl ?

Here is my output for rpm -qa | grep -i lve (can you tell me if I have all the packages and if they are correct versions please??)

[~]# rpm -qa | grep -i lve
kernel-2.6.18-408.8.2.el5.lve0.8.61.3
lve-1.2-1.12.el5.cloudlinux
kernel-2.6.18-448.16.1.el5.lve0.8.70
liblve-devel-1.2-1.12.el5.cloudlinux
lve-wrappers-0.6-1.el5.cloudlinux
lve-utils-1.4-18.3.el5.cloudlinux
kernel-headers-2.6.18-471.3.1.el5.lve0.8.72
liblve-1.2-1.12.el5.cloudlinux
pam_lve-0.3-7.el5.cloudlinux
lvemanager-0.8-1.32.el5.cloudlinux
kmod-lve-2.6.18-408.8.2.el5.lve1.1.65.3-1.1-9.9.el5
lve-stats-0.10-31.2.el5

I even did yum reinstall lve-utils but it didnt make any change - "list" still shows that 104 number... so random.

[denis]

Bogdan

regarding the endless loop thing ... this has happened multiple times now on different servers (all with cloudlinux and all had 25% cpu limit and 1gb memory etc)

Each time it was the Apache process that was eating all the ram even though there was an associated PHP process (simple php script with endless loop) and because apache is owned by root/nobody I guess it wasnt throttled. I just find it frustrating that a user can circumvent lve limits so easily (unintentionally). I opened a ticket about this some years ago when it happened the first time (it has happened again ..different users..different servers) and fr om memory Igor said LVE couldnt do anything about this because its Apache and because its probably due to mod_security or something.

I just find it emberassing when I have to tell a customer to check their php scripts for endless loops because I know how easily it can crash the server and still have no solution for it.

Would it not be possible to put LVE limits on Apache ? ie. atleast a memory lim it so that if it starts to consume more than X Gb of ram apache gets stopped and restarted automatically?

[bogdan.sh]

Cpanels LVE manager read /etc/container/ve.cfg where 100% are set, and show them. While lvectl ask system for users CPU limit and as a result receive 104% . We understand that is not normal and working to have this fixed.

About memory overusage - you could be right, we had two similar incidents, but the problem was not with PHP script endless loop, but with redirect loop. The problem is caused mostly by apache, however after implementing http://mail-archives.apache.org/mod_...@bastun.net%3E

This could be simply checked with the following code in .htaccess:

> RewriteEngine On
> RewriteRule .* a [N]

Accessing website with such htaccess will cause redirect loop.

Here are steps to patch apache on cpanel , tested it few monthes ago :
1. create a file /root/mod_rewrite.c.patch
2. place there following content:

+++ modules/mappers/mod_rewrite.c       2012-12-19 11:13:47.701538052 +0000

@@ -1,3 +1,4 @@

+

/* Licensed to the Apache Software Foundation (ASF) under one or more

* contributor license agreements.  See the NOTICE file distributed with

* this work for additional information regarding copyright ownership.

@@ -46,6 +47,8 @@

*      www.engelschall.com

*/

+#define CORE_PRIVATE

+

#include "apr.h"

#include "apr_strings.h"

#include "apr_hash.h"

@@ -4033,6 +4036,9 @@

int rc;

int s;

rewrite_ctx *ctx;

+       core_server_config *conf = ap_get_module_config(r->server->module_config, &core_module);

+       int rlimit = conf->redirect_limit ? conf->redirect_limit : AP_DEFAULT_MAX_INTERNAL_REDIRECTS;

+       int loop_cnt = 0;

ctx = apr_palloc(r->pool, sizeof(*ctx));

ctx->perdir = perdir;

@@ -4044,6 +4050,7 @@

entries = (rewriterule_entry *)rewriterules->elts;

changed = 0;

loop:

+       loop_cnt++;

for (i = 0; i < rewriterules->nelts; i++) {

p = &entries

;

@@ -4116,6 +4123,13 @@

*  the rewriting ruleset again.

*/

if (p->flags & RULEFLAG_NEWROUND) {

+                               if (loop_cnt > rlimit) {

+                                       ap_log_error(APLOG_MARK, APLOG_CRIT, 0, r->server,

+                                                       "mod_rewrite: Request exceeded the limit of %d internal "

+                                                       "redirects due to probable configuration error.", rlimit);

+                                       break;

+                               }

+

goto loop;

}

3. open /scripts/before_apache_make and place there following line (for apache 2.4, or edit to src/httpd-2.2 for apache 2.2):

/usr/bin/patch /home/cpeasyapache/src/httpd-2.4/modules/mappers/mod_rewrite.c   /root/mod_rewrite.c.patch

4. run /scripts/easyapache and build it.

[clm]

> as we are working on a very big rework, that should add many cool featurs (like snapshoting of processes & queries when limits are hit, burstable limits and reseller limits).

Great news ! Sounds very exciting

[denis]

Whilst I do appreciate your response and time youve taken to write that Bogdan, my issue was to do with simple endless loops written by customers (by mistake) in PHP scripts, nothing to do with rewrites/redirects.

I guess Ill just have to find some other way to stop this from happening since LVE can control apache. If anyone has any tips in which direction I should be looking Id very much appreciate it.

[iseletsk]

Do you have that PHP script? Infinite loop in php is fully controled memory limits -- as long as you are using suPHP/mod_fcgid/mod_lsapi/or php via suexec.
If you use mod_php (with ITK, ruid2 or without) -- then yes, you cannot control memory for those PHP processes. Only CPU & IO.

[denis]

Yeah I have the script somewhere .. but in bed now (1am here) so will get back to you soon with the script.

Basically what happened was from memory it was just a simple i for loop in that script .. I managed to take a screen shot of top before server crashed and it was showing apache as using 10gb ram + swap (out of 12 total ..just shortly before it crashed) and that script was one of the top processes too. But the ram was definately used up by apache for whatever reason.. Ill get back to you tommorow. Cheers

edit: all my servers run suPHP and have so for years now

[denis]

OK found them ... 2 scripts from 2 different customers that managed to crash the servers via memory exhausting (endless loops in php). Can I email them to you directly or should I open a ticket at the helpdesk? Cheers D.