Node js apps restarts now and then

[noxious]

Hello,

I have two node js apps that are restarting now and then over the day.
I have logs and doesn't see what or why they are restarting.
Nothing in Passenger log.

Has anyone run into same problem ?

Regards

[bogdan.sh]

Hi,

At what times this was happening last time? Is there anything useful in the stderr.log inside the application directory itself, or in the main server error_log file? Is it just the application restart or maybe whole Apache is restarted at that moment? I guess the main /var/log/messages can reveal more information.

[noxious]

Hi!

Nothing in stderr.log , nothing in DirectAdmin log view. Nothing in passenger.log.
Only in my own log , that just logging when server starts.

I have now three node js apps. All of them restart approx. at the same minute.

There is not much activity on these apps.
I have a cron job that runs every 4th minute to make sure that node apps not going in IDLE mode.
That cronjob run an api call and that api just "SELECT 1" from mysql db.

Is there a chance for my to log why server is going down and make a restart?

I have even
process.on uncaughtException and unhandledRejection but no luck there either.

Here is my log for three days.

{"timestamp":"2023-07-29T01:20:06.309Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-29T04:04:06.626Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-29T04:08:06.483Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-29T10:45:54.740Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-29T10:46:04.745Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-29T10:52:06.829Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-30T01:28:06.261Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-30T01:32:06.207Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-30T09:32:06.441Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-30T09:36:06.010Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-30T19:28:06.274Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-30T19:32:06.835Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-30T19:44:06.269Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-30T20:32:05.977Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T01:16:06.905Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T06:20:06.830Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T06:24:06.019Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T06:28:06.556Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T06:32:06.259Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T06:40:07.241Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T07:08:06.154Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T07:24:06.525Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T08:32:08.310Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T08:52:06.476Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T09:36:08.801Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T09:40:09.026Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T09:48:08.005Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T10:44:08.810Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T11:32:08.845Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T12:04:08.733Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T15:20:06.060Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T17:00:08.437Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T23:52:08.934Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T23:56:08.429Z","level":"info","message":"Server up and running!"}

Regards

[bogdan.sh]

Just wondering if all apache is being restarted altogether with the passenger? Can you please check the /var/log/httpd/error_log file for the time when it happened with your apps?

[noxious]

Hi,

I have now had contact with they who runs web hotel.

Nothing in /var/log/messages
Nothing in /var/log/httpd/error_log


I noticed that when I made 3 days test that it restarts more often when all nodejs apps is running.

First day I only let 1 app running - Then it only restarts twice at night time.
Day two - I started 1 more app - More restarts. Both apps
Day three I started my last app - Alot of restarts of all apps.

*EDIT*
I have now set my cron jobs to run every 3, 4, 5 minute.
So they dont trigger at the same time.

Can I set different loglevel to each passenger log ?

Regards

[bogdan.sh]

That sounds really weird. How do you know the apps are restarted if nothing in logs? You said

Only in my own log , that just logging when server starts.

How are you getting that data?

[bogdan.sh]

Hmm, am I getting you right that the cronjob is accessing passenger app (over API) every 4th minute and it responds only in those times provided above? Meaning other minutes likely it was not available?

{"timestamp":"2023-07-31T09:40:09.026Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T09:48:08.005Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T10:44:08.810Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T11:32:08.845Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T12:04:08.733Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T15:20:06.060Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T17:00:08.437Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T23:52:08.934Z","level":"info","message":"Server up and running!"}
{"timestamp":"2023-07-31T23:56:08.429Z","level":"info","message":"Server up and running!"}​

It was not available from 17:00 till 23:52, right?

[noxious]

I have a own log when server starts up.

server.listen(process.env.APP_PORT, () => {
logger.info("Server up and running!");
});

If I do manually a restart i directAdmin a new row will be in the log file.


Yes, cron job runs every 4th minute.
Server was avaliable all the time over the day.
But if I use the app during when app restarts it takes a while to show the webpage.

They also get new process Ids when server restarts,


I have googled and noticed that they have problem with nodejs restarting.

Do you think if I removed restart.txt in tmp folder will solved this issue?

https://stackoverflow.com/questions/66488345/passenger-daily-restarts-nodejs-app-on-cloudlinux

Regards

[bogdan.sh]

The restart.txt is not related to the issue as it's usually created after some actions on a server by customers where passenger restart is actually required.

There are some options you may want to adjust, e.g.:

PassengerMaxPoolSize
PassengerMaxInstancesPerApp
PassengerMaxPreloaderIdleTime

There is no way to tune them separately, the main /etc/httpd/conf/extra/passenger.conf to be used (just please note it could be rewritten with DirectAdmin conf_rewrite command), for a longer time proper customization is needed.

Are you sure apache is not restarted same time? Maybe it's normal actions from the customers? More users - more changes to websites (ssl/redirects/subdomains) that requires apache restarting.

Is the restarting affect the application functionality overall?

[noxious]

Ok.
I dont have any chance to change those options.
And the web hotel wont change it for me.

They told me that apache was not restarted at the same time.

The problem is that website holds until the server has restarted.
So I got messeges from those who uses my system that website is slow.

I think we are stuck now and can't move on in the troubleshooting.

I have to thank you for your time!

Regards

[bogdan.sh]

You are always welcome.

And we are ready to help if they jump into this thread with more information from server logs