Random Server Outages

[jordan]

Plesk is still responding but all sites just go offline with error 522

[jordan]

Hey Guys,

Any advice to locate the root cause of my issue would be fantastic as my web server has randomly gone offline again.

Im running the latest version of CloudLinux + Plesk 12.5 with all updates installed (stable branches).

My website can run for 18 hours, then all sites hosted on my machine just vanish off the internet. I have checked LVE and it now only reports NO ACTIVITY and according to services, Apache, Nginx and MySQL are running. I can login to my Plesk panel without any issues, the machine is up.

Eventually after say 30 minutes, the site comes back online. Then go offline, and come back again.

My logs in Plesk are showing no activity whilst the site vanishes and no errors reported.

Regards, Jordan

[bogdan.sh]

Hi,

Well, that sounds strange. Especially if you are able to connect to Plesk panel while websites are offline. So.. things to check next time it happens:
- telnet to port 80 from outside
- telnet to port 80 from same server
- check apache status page
- check apache error_logs

But... is it same for the rest of internet or is it just you anned by some smart security tool? Can you check from other locations or with websites like tools.pingdom.com ?

[jordan]

Hi,

It seems to affect the entire internet as even pingdom reports as down, I also keep getting server outage warnings from statuscake.

telnet to port 80 from outside - no response.
telnet to port 80 from same server - seems fine.
check apache status page - all up and running.
check apache error_logs - nothing apart from access records and the odd 500 warning.

However, my plesk admin panel is getting slower and slower the longer the sites are down. Its almost as if the server is about to crash completely.

I ran the op command in telnet and cpu usage and memory is fine.

What I have discovered is that Apache (HTTPD) processes are constantly starting and stopping in some crazy loop?

[bogdan.sh]

Show us some 500 warnings from error_log . But also I would like to see a top header when issue is happening.

Why do you think apache processes are constanly restarting?

[jordan]

I have rebooted the server and now the error logging has started reporting errors when accessing the site:

Extract of the error log - http://pastebin.com/3Fwtkgme (expires in 24 hours)

The last error 500 was a while back, and doesn seem to be linked with the problem:

2016-04-13 10:47:07 Error 162.158.153.4 500 GET /basketItem/create?menu_item=13&_=1460558833043 HTTP/1.0

Why do you think apache processes are constanly restarting? I noticed the httpd was the highest user of RAM and CPU and its runtime never exceeded 29 seconds before closing, then it would appear again after a few seconds, starting at 1 seconds.

[bogdan.sh]

Unfortunately really we have no ideas why this is happening. We would need to install some simple monitoring script to collect basic stats but also apache status page to identify a bottleneck. Please create support ticket at https://helpdesk.cloudlinux.com/ .