PHP errors when LVE limits hit?

[ryan]

Sometimes, it seems like around the time that LVE limits are hit, I'll get PHP errors. Particularly include() failures. The includes work perfectly fine when tested normally. I just want to understand if this is just a byproduct of how the LVE protections work rather that something else I'm missing. I don't see any other reason the files would fail to be accessed from include(). Thanks.

[bogdan.sh]

Well, it depends of what LVE limit is hit. In case of PMEM it will result in error 503.
The EP or nproc should not be hit with a simple include() function, if other files are not doing strange things.

The question is, what are the errors and what limits are being hit?

[ryan]

The majority of the LVE faults were PMEM with just a few EP faults. The errors in the PHP logs are: "include(): Failed opening 'FILENAME' for inclusion". Returning 503 to the requestor makes sense, but I end up with a bunch of PHP errors in the logs.

[bogdan.sh]

My guess is they are not directly related, the error in logs should be different. When the PMEM limit is reached, the server may not be able to allocate the necessary memory for PHP processes, but it should not end up with include() error.
If I were you I would increase the PMEM limit for some time to monitor if it's still happening without PMEM faults.

Is this any popular website engine or self written?

[ryan]

It's definitely related in some way. The correlation is extremely strong. It happens on multiple accounts each with different PMEM limits. So my guess is that increasing them for an account would slightly reduce the incidences, but would still happen once the limit is hit.

This is happening on sites with code I wrote myself. We do have some Wordpress Sites which I don't manage, but I did just check the logs for some of those sites that also have the PMEM limit being hit frequently. I also see include() / require() errors there too at the exact same time the limits were hit. The files it's trying to include are actual files with the correct permissions. So it's the same issue.

When a file needs included, it needs to allocate memory that would count against the limit, correct? So my guess is that the PMEM will either be hit when the script is assigning a variable or when including a file. If it gets hit when assigning a variable, that is not a PHP error (because the limit being hit is an account LVE limit, not a per-script PHP limit). So the server returns 503 and quits and I never see evidence of it. However, if it happens while including a file, that IS a PHP error that is logged, so then I see it.

So I guess if you still don't think that's the case, what would I expect if the PMEM limit was hit while trying to include a file?

[bogdan.sh]

>When a file needs included, it needs to allocate memory that would count against the limit, correct?
Yes, that is correct. When PMEM occurs, the kernel terminates the PHP process, resulting in Apache throwing a 503 error, but not the PHP include() function. Typically, includes are managed within the same process (without forking), so when the main process is terminated, it will cease to pure 503.

In short - the process is terminated due to exceeding the PMEM limit, and you may not see any detailed error messages in the logs.


Do you use apache as a web server?
Can you please post a full error message from the log, including timestamps (just escape the real user/domain name if any).

[ryan]

Here is the PHP error from a WP site:
[22-Feb-2025 06:41:18 UTC] PHP Fatal error: require(): Failed opening required '/home/ACCOUNT/public_html/wp-includes/block-editor.php' (include_path='.:/opt/alt/php74/usr/share/pear') in /home/ACCOUNT/public_html/wp-settings.php on line 361

The LVE email was received at 2025-02-22 06:47:48 UTC

Is it possible that an I/O limit was hit at around the same time that didn't trigger a fault? It does look like some sort of malicious bot traffic in significant quantity happened at that time.

[bogdan.sh]

They are unrelated, the LVE fault is immediate and would cause error in log the same second when it happened.
When the IO limit is hit it just slow down disk operations for some time, theoretically that could be related if number of IO operations grows over the next minutes. You have to check the lvestats this way:

lveinfo --user USERNAME --by-fault any --period 1d

and dive into more details:

lve-read-snapshot --user USERNAME --period 1d

[ryan]

With the first command, for the time in question, I see the PMEM faults which matches up with what the email said,.but no other faults. Average IO is a little more than 10% of the limit.

With the second command, I see a bunch of process for the entry PHP script which makes sense. The max memory for one of those is 23Mb and the max CPU is 5%. The items with high memory usage are /usr/sbin/httpd -k start - is that normal? There about 10 of them ranging between 97Mb and 6716Mb. There is also a few lsphp, but with low Mem and CPU.

Let me know if you want to see the actual output.

Is it possible to check if an IO limit was hit and was throttled?

Also, is it possible that the IO gets throttled enough that some other hardware or software starts to think something went wrong and essentially gives up?

[bogdan.sh]

The processes seems Ok, there will be always few of them with different memory usage.

You can check if IO limit was hit with the similar:

lveinfo --user USERNAME --by-fault io --period 2d

As well, the lve-read-snapshot will show the IO explicitly when it was caught:

Snapshots collected starting from 2025-03-03T14:23:00 to 2025-03-03T14:33:00 for lve id 1000 @ 250c6209-4:
>>> 2025-03-03T14:32:27, UID 1000

Faults:
* IO: 1

Processes:
​

>is it possible that the IO gets throttled enough that some other hardware or software starts to think something went wrong and essentially gives up?
I still don't think so, the include() PHP function does not fork child process, while the main one remains active but super slow. If it will be killed by any timeout - it will record some 503 error but not "Failed opening required".

Please try checking the lveinfo command for a longer period like 1 week, maybe you will find some pattern.

[ryan]

It doesn't look like IO limits have been hit at all. The only limits hit for these cases are PMEM and CPU. However there are CPU hits at times where I don't get these PHP errors. But it's a 100% matchup between getting these PHP errors and having PMEM faults. The date and times match up perfectly for every instance I've checked.

I should be able to test this, right? If I create an account that has a really low PMEM limit, then make a PHP script that consumes that memory while trying to include() files?

[bogdan.sh]

Yes, this test scenario should work. Let me know if you get the same include error.