reaching MaxClients

[server]

Hi there!

Recently I faced an issue with Apache httpd unable to server content due to reaching MaxClients setting. I found its caused by requests going to one of the sites hosted on this server. Initially I thought - its DDoS but later I found those are legitimate http request. The problem was the httpd stopped responding to any other request due to reaching MaxClients setting effectively taking down all sites on this server.
The server runs CloudLinux system with mod_hostinglimits and has kinda low MaxEntryProcs setting for this user - EP=200. It seems to be working good. I saw the users being limited within its LVE settings and getting 503 or 508 for most of these enormous requests. Here comes the issue - httpd still needs to serve those 503 or 508 pages and when its getting more and more requests it will eventually hit MaxClients setting.

Short term solution for this is just to rise MaxClients. Thats not a good solution though. What if we could set MaxClients per vhost?

The mod_itk provides such setting. Im not sure how it really works but if this can prevent a single tenant from taking down whole server I would say its good solution.

I wish to see mod_hostinglimits can have similar feature. Please consider adding such feature on CloudLinux roadmap.

How to recreate:
1. Setup CMS site (Wordpress, Joomla)
2. Set MaxClients for mpm you use to 2048
3. On some other, decent server run: siege -c4000 this.site.url/index.php
4. Watch server-status or error log

[iseletsk]

maxentryprocesses are basically maxclients per user. Just change to * in mod_hostinglimits for allowed handlers. The question is what is your max clients for the server in total.

[pawel.panek]

Hi Igor,

I tried setting AllowedHandlers to * but it didn change this behavior. I thought its set correct since Im requesting .php file which has been listed in allowed handlers. Besides, AllowedHandlers * didn help.

When MaxClients is set to 4096 you can kill http server with following test:

ab -c4000 -n80000 http://domainname.tld/index.php

Test site should be some kind of PHP app, eg.: CMS. In the test I used EP=200 but I think EP value doesn matter much in this case

I know its DoS but what I would expect is mod_hostinglimits could protect other sites when a single site is being DoSed. Im not sure if MaxClientsPerVhost from mod_itk can do it but it sounds like it does.

[pawel.panek]

Update -
I can even change EP to 10 for test site. The result is the same - Apache server will refuse requests for other sites due to reaching MaxClinets limit. What do you think, is it possible to isolate that big number of requests for single vhost so it doesn affect other sites?

[iseletsk]

Of course you can. EP is part of the apache. It cannot go beyond MaxClients. It is not a firewall option. It is there so that once site is slow, and requests to it start to accumulate -- it would not wait for PHP pages to finish in 20 minutes, but rejected requests to those PHP pages right away - preventing natural accumulation of such things.

Sorry, but we are not trying to solve DDoS attacks where a client can create 100,000 connections to your server, or saturate your uplink with 10gbps of traffic.

[pawel.panek]

Im intentionally trying to avoid using words - DoS or DDoS because the case Im having here is real life traffic. Yes, its DoS-like but requests are fully legit.

Also its not about reaching link capacity - bandwidth nor packet rate. That ab test I posted earlier can stop the server much earlier before reaching (D)DoS volumes.

The perfect solution I see here is to have a setting that will just reset tcp connection after reaching MaxClientsPerVhost. When this setting is higher than EP connecting clients would get 508 error while EP < current # of connections < MaxClientsPerVhost. All connections above MaxClientsPerVhost should get tcp connection reset without serving error page. Of course initial handshake is required to get Host header from client. Then module can check connections counter and eventually decide to close the connection without further processing. In case MaxClientsPerVhost < MaxClients (server wide) then the server should stay safe.

Anyways, I think feasible solution here is to keep increasing MaxClients so Apache can serve 508s or 503s for site that is already above EP.

[iseletsk]

If you have fully legit traffic with 4000 connections to a single vhost -- don use apache, as it is not feasiable to keep increasing maxclients -> you will run out of memory.
Try switching to LiteSpeed or install nginx as a proxy in front of apache.