Atomic Secure Linux

[accounts]

Hi,
We use the cpanel Unixy varnish plugin which has proved excellent and worth the money. We run suphp as its more secure but it is slower, but we noticed a big difference in page load times after installing varnish. No problems with any software or .htaccess and there options to exclude websites fr om varnish and pass them directly to Apache, theres also a "slashdot" option that serves up pages stripped of cookies etc for super fast loading if you have sites that get hit hard. One thing to note, if you give varnish say 2GB of memory it will soon clock up around 2>3 GB swap so you need to be careful when setting the memory lim it.

[wrender]

I dont know how much overhead mod security is causing, but when running Cloud Linux with suPHP to limit resources, and ASL with mod security I have only seen an improvement in overall server performance. Im running a xeon 8 core, with 16GB. Most websites are Drupal and Wordpress, but also a few Joomla. My server is running at an average of 24% CPU usage, with a Max of 52% usage according to nagios snmp monitoring.

I did try the delayed mod security rules before with mod sec control and I did notice quite a large spike in CPU usage. As well I did have to make a lot of exceptions for clients websites. This was about 2 years ago though. I think there have been a lot of improvements since with the actual mod security software itself to make it more efficient with processing.

Another thing I have noticed over the past 2 years, is that there are a lot of web crawler bots that drain server resources. For example I have a client that runs a website with over 70,000 drupal pages. Google bot kept coming and overloading it. Cloud Linux has really helped with the load/resource issues. Apparently the new ASL also has crawler protection built into their rules which I think is also going to be helpful. Also, this website gets about 600-1000 attacks a day in mod security, so I do think that the amount of attacks it defers way outbalances how much cpu usage mod security would be using. As well, a major problem for me has also been getting MySQL 5.5 tuned properly. I have been using mysqtuner.pl but it always asks to allocate more and more resources to MySQL, and when I do it my CPU then starts overloading from MySQL. I have recently reduced my table_cache from 20,000 to only 5000 which seems to have resolved MySQL using all of the resources on the server.

[accounts]

Maybe for your mysql problem you could try the Cloudlinux mysql govenor? at the very least it should tell you which database is causing the problems and from there you could try to optimise the table. We found that simply converting a table to InnoDB helped on one database.

[wrender]

thanks kernow. I was hesitant to install mysql governor because it is still in beta I think. Have you tried it? I have switched some databases over to innodb which has really improved the performance. The MySQL slow query log helped me identify problem areas of databases. The problem is I think some of the databases are getting a bit large... over 1GB that it is really hard work for the hard disks to process queries. Ive got local 15K rpm SAS drives, but I may have to look at setting up another server soon with solid state drives. last time I checked the solid state drives were about $999 per drive though. Ouch!

[accounts]

Hi,
Havent tried the mysql governor yet, but I guess you can always swap back if it doesnt work out. The support from Cloudlinux is very quick in our experience so you can always submit a ticket if problems occur. I really dont think its worth buying your own hardware any more, it seems cheaper nowadays to rent/lease. If your server is really under load mainly from MYsql and you cant tune it any more it might be worth looking at setting up a dedicated mysql server, or just move the heavy load sites onto a VPS.

[clm]

@ Wesley Render

Well be installing ASL next weekend and be checking out page load times with and without it as well as memory consumption to see if it will work for our customers. Have you had any problems with it (except excluding some rules I suppose…) ?

@ Kernow
I installed the unixy varnish plugin yesterday on a server that isnt in production yet and Ive noticed that it doesnt play well with quite alot of scripts (still waiting for an answer from unixy support…).

The first script we tried varnish on is Magento as its one of the slowest scripts out there… I connected to to client area, and changed my address, saved and the page was served from cache…

Im also not sure how well it will play with scripts that have their admin folder renamed, or scripts that have non-ajax contact forms…

Ill see what Unixys support suggest but we might have to abandon their product unless they provide a way to have opt-in instead of opt-out. If we had opt-in we could offer Varnish as an option and the customer would know he had to make his scripts compatible with varnish. A shame Unixy doesnt provide the ability for cPanel users to activate / deactivate varnish on their own…

[wrender]

Hi Richard,

Surprisingly, I have not had to exclude any rules. Here are the main problems I came across when installing/using so far. All around there are quite a few bugs with getting ASL working properly with cPanel/Cloud Linux so I would recommend installing it during the week when the ASL support is available to help you. They are really responsive and helpful when issues arise. It is well worth the time I would say as it seems to add a lot of protection.

1. Make sure you have Query Cache enabled in your /etc/my.cnf. If on cPanel you can run /usr/local/cpanel/3rdparty/mysqltuner/mysqltuner.pl for recommended settings . For example on my system i have the following variables set: (I have 48GB of ram on this machine)

query_cache_size=256M

query_cache_limit=1M

query_cache_type=1

query_cache_min_res_unit=1024

2. I would not recommend enabling the advanced PHP security, as it disabled as lot of PHP features which are needed for a lot of modern websites. I had to end up turning this off.

3. If you are running Cloud Linux you will need to make sure to go to "Configuration" -> "ASL Configuration" and set "UPDATE_TYPE" to "exclude-kernel", and then after installing cloud linux make sure to edit your /etc/grub.conf so that your Cloud Linux kernel is still set to load instead of the ASL Kernel.

4. After installing ASL, my DNS services stopped working. We had to comment out this line in /etc/named.conf "allow-query { localhost; };"

5. By default no firewall rules are loaded for cPanel. I would recommend for you to google required cPanel firewall ports, and then add those to "Firewall Configuration" under the "ASL Configuration" option. Both in the inbound and outbound fields.

6. Another issue was that I could not get the firewall to respond after making changes in the ASL GUI. Also, if you are running CFS make sure to uninstall it. A very usefully command when troubleshooting is to run the following to restart the iptables scripts involved with ASL (Originally I did not know that ASL had its own firewall script as it was not in the documentation for install, so I kept just running service iptables restart which caused a lot of confusion for me):

service iptables restart;service asl-firewall restart

7. For the Kernel Configuration I had problems getting ASL to work with R1Softs CDP Agent. I had to enable "Allow_kmod_loading" to be able to allow the CDP agent driver to load on startup, I then had to edit the order of the /etc/rc.d/rc3.d cdp-agent startup file so that it was before the ASL startup scripts. Then I was able to disable "Allow_kmod_loading" and reboot the server, and the cdp agent was able to load into the kernel properly.

8. For the new ASL 3.2 I had a lot of problems when upgrading to it. You need to make sure to disable mod security in /scripts/easyapache as ASL configures its on mod security separately. I had to rebuilt apache, and then run the commands from https://www.atomicorp.com/wiki/index...#ASL_3.2_Notes to get it working properly. I also had a mod security error when I would run "service httpd restart" so I had to rebuilt apache again using " /scripts/easyapache --build"
Cheers

[mikael]

Have anyone tried ASL+CL+LiteSpeed?
Our servers are not as powerful as most of the once that have been mentioned here.
Our servers run Xeon X3430 2.4 GHz Quad-core with 4 GB RAM.
We aim to run about 200 customers per server.

Wesley mentions problems with BIND, we use TinyDNS, have anyone had any experience with it on ASL?
@Whesley, you mention CFS, do you mean CageFS or Completely Fair Scheduler?

I would like to buy some new servers, and have received a quote for 16core AMD 6274 and 16GB RAM, but Im starting to think if even that might be too weak for running ASL.

The quote is for 2 of the above servers + 1 backup server that will run R1soft CDP server, for ~6500 USD.

[mikael]

Have anyone tried ASL+CL+LiteSpeed?
Our servers are not as powerful as most of the once that have been mentioned here.
Our servers run Xeon X3430 2.4 GHz Quad-core with 4 GB RAM.
We aim to run about 200 customers per server.

Wesley mentions problems with BIND, we use TinyDNS, have anyone had any experience with it on ASL?
@Whesley, you mention CFS, do you mean CageFS or Completely Fair Scheduler?

I would like to buy some new servers, and have received a quote for 16core AMD 6274 and 16GB RAM, but Im starting to think if even that might be too weak for running ASL.

The quote is for 2 of the above servers + 1 backup server that will run R1soft CDP server, for ~6500 USD.

[clm]

Hello,

We are currently running ASL+CL+Litespeed, our server is not busy for the moment (current load 0.4) but after quite a few issues we think most problems are solved or will soon be solved :

1) ASL will soon be releasing a bug fix to solve an isssue when the mod_security audit directory is not writable by their tortix user and their daily updates change the rights to cPanels Apache nobody:nobody user.

2) ASL doesnt currently allow multiple local IPs for their T-WAF. A fix should be comming soon, in the mean time you cant protect accounts with a dedicated IP with the T-WAF

3) We had errors fr om the T-WAF because it couldnt always access litespeed. We belive to have fixed the issue by creating a new file :

/var/asl/etc/httpd/conf.d/99_fixproxy.conf

Containing :

SetEnv proxy-initial-not-pooled 1

4) Litespeed tries to load ASLs mod_security files

To fix this issue we told litespeed to ignore mod_security commands.

5) As Litespeed compresses data we were advised to configure mod_security to request non compressed data. We have yet to activate mod_deflate in the ASL T-WAF to see if it slows everything down or not. Litespeed allows to cache gziped data wh ereas mod_deflate doesnt. We will have to decide if its OK for mod_security to not be able to read outgoing data and use litepspeeds gzip compression, to use mod_deflate in ASLs T-WAF or not not allow users to gzip their data.

6) You must disable litespeeds per client throtteling and leave that to CloudLinux to manage the limits

7) You mustnt forget to disable limits for tortix in dbgoverner, and CPU + disk limits for all processes owned by ASL/litespeed etc.

---

After alot of trial and error we have descovered that ASL needs their Apache and their Mod_security and not try and work with cPanels Apache or Mod Security.

Using ASLs T-WAF (Apache) almost no extra latency is added and their T-WAF seems to use less memory then cPanels Apache when runnig ASLs mod_security (currently using 77MB per tortixd process).

With all of the issues we had, we ended up purchasing licences to give us enough time to debug.

Our server is currently running with about 10 000 page views per day. And is using about 9GB of memory (weve got 256GB of memory on this server so we have set most limits very high). Out of the 9GB, MySQL

Out of the 9GB we have got :

MySQL : 2.5 GB
Tortixd (ASL T-WAF) : 470MB
PHP : 250 MB
Litespeed : 26 MB

We will start migrating accounts to this server soon and see how the usage increases. So far we are pleased by the CPU/Memory usage and speed using this combination. Its taken quite a long time to get things setup and working together but it should be worth it.

[clm]

We have currently deactivated ASL T-WAF. Litespeed was responding in 2ms without the T-WAF and 15ms with the T-WAF, we were not able to have the T-WAF scan outgoing data if the compression was launched. Sites were noticably slower and we had some stability issues with T-WAF crashing or not responding for long periods.

Were not sure quite where to go from here, were going to keep litespeed, much better than Apache on many points.

Were currenlty debating on either not using any mod security rules or using litespeeds engine to run a smaller set of goroots rules.

It seems here that you have the choice of less speed, less stability but fewer accounts being hacked or more speed, more stability and a few accounts with outdated scripts being hacked.

Ive seen very few up to date sites being hacked that wouldnt have been hacked running ASL.

Weve come to the conclusion that its better for most end users to not have ASLs mod security rules. ASLs rules wont stop 100% of attacks, and so far has given us more work with it then without it.

We missed CSF and LFD when running ASL, another reason to not go with ASL…

ASLs support is quite fast, and problems have been solved quite quickly but we got the impression we were beta testing their product with the combination of cPanel + litespeed due to the number of problems we came accross that needed fixing.

[mikael]

Well, our new servers arrived last week. And are right now on my desk waiting for me to install CL.
They have 2xCPU AMD 16 core with 2GB RAM/core (total of 64GB). So I think they will be more then enough to run everything we want.

What concerns me with ASL is that you say its incompatible with CageFS (if I understand you correctly). I did a search on the ASL forums, And if I understand Scotts comment correctly ... they should work together.

We definitely want CageFS on the system. At the same time I want the DDoS, Malware, IPS, mod_secure etc goodies from ASL. The servers should be able to swallow just about anything performance wise, so we can concentrate on stability, customer isolation and safety.

Another consern I have, is that we are running InterWorx Controlpanel, and they seem to have removed that from their supported list. And Im not sure if I should install ASL or InterWorx first.