[abrt] ea-apache24: httpd killed by SIGSEGV

[zodiac9797]

Hello,
for the last 7-8 days, I believe after one of the cPanel or CloudLinux updates, we started receiving mail reports-errors, almost every minute.

reason:         httpd killed by SIGSEGV

cmdline:        /usr/sbin/httpd -k start

executable:     /usr/sbin/httpd

package:        ea-apache24-2.4.35-1.el7.cloudlinux

component:      ea-apache24

pid:            19624

pwd:            /

hostname:       xxx.xxx.xxx

count:          1

abrt_version:   2.1.11

analyzer:       CCpp

architecture:   x86_64

event_log:

global_pid:     19624

kernel:         3.10.0-714.10.2.lve1.5.19.9.el7.x86_64

last_occurrence: 1542373288

os_release:     CloudLinux release 7.5 (Viktor Gorbatko)

pkg_arch:       x86_64

pkg_epoch:      1

pkg_fingerprint: 8C55 A662 8608 CB71

pkg_name:       ea-apache24

pkg_release:    1.el7.cloudlinux

pkg_vendor:     CloudLinux

pkg_version:    2.4.35

runlevel:       N 3

time:           Fri 16 Nov 2018 02:01:28 PM CET

type:           CCpp

uid:            99

username:       nobody

uuid:           73ef1fdc3b07853d51560f95d276a80f7c05b4da

I can post entire report if it will help.

Not sure if this is connected but under process manager I can see:

xz -1 /var/spool/abrt/ccpp-2018-11-16-13:08:59-29983/sos.qJmMDV/sosreport-CLN-47528-20181116131216.tar

/usr/bin/python /usr/sbin/sosreport --tmp-dir /var/spool/abrt/ccpp-2018-11-16-13:15:10-2699 --batch --only=anaconda --only=boot --only=devicemapper --only=filesys --only=hardware --only=kernel --only=libraries --only=memory --only=networking --only=nfsserver --only=pam --only=process --only=rpm -k rpm.rpmva=off --only=ssh --only=services --only=yum --only=general --only=x11 --only=cups --only=logs --only=grub2 --only=cron --only=pci --only=auditd --only=selinux --only=lvm2 --only=sar --only=processor

/usr/bin/abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible recursive locking detected ernel BUG at list_del corruption list_add corruption do_IRQ: stack overflow: ear stack overflow (cur: eneral protection fault nable to handle kernel ouble fault: RTNL: assertion failed eek! page_mapcount(page) went negative! adness at NETDEV WATCHDOG ysctl table check failed : nobody cared IRQ handler type mismatch Kernel panic - not syncing: Machine Check Exception: Machine check events logged divide error: bounds: coprocessor segment overrun: invalid TSS: segment not present: invalid opcode: alignment check: stack segment: fpu exception: simd exception: iret exception: /var/log/messages -- /usr/bin/abrt-dump-oops -xtD

Any advice / help is welcome!

edit:

Another mail report received, this one is little different.

[abrt] sos: sosreport.py:1463:_create_checksum:IOError: [Errno 2] No such file or directory: /var/spool/abrt/ccpp-2018-11-16-17:02:36-44257/sos.Pzc2U6/sosreport-CLN-47528-20181116170714.tar.xz

reason:         sosreport.py:1463:_create_checksum:IOError: [Errno 2] No such file or directory: /var/spool/abrt/ccpp-2018-11-16-17:02:36-44257/sos.Pzc2U6/sosreport-CLN-47528-20181116170714.tar.xz

cmdline:        /usr/bin/python /usr/sbin/sosreport --tmp-dir /var/spool/abrt/ccpp-2018-11-16-17:02:36-44257 --batch --only=anaconda --only=boot --only=devicemapper --only=filesys --only=hardware --only=kernel --only=libraries --only=memory --only=networking --only=nfsserver --only=pam --only=process --only=rpm -k rpm.rpmva=off --only=ssh --only=services --only=yum --only=general --only=x11 --only=cups --only=logs --only=grub2 --only=cron --only=pci --only=auditd --only=selinux --only=lvm2 --only=sar --only=processor

executable:     /usr/sbin/sosreport

package:        sos-3.5-9.el7_5.cloudlinux

component:      sos

pid:            4143

hostname:       xxx.xxx.xxx

count:          1

abrt_version:   2.1.11

analyzer:       Python

architecture:   x86_64

duphash:        be3103bd203e454ce0cb1061f61309e10da2108e

event_log:

kernel:         3.10.0-714.10.2.lve1.5.19.9.el7.x86_64

last_occurrence: 1542384475

os_release:     CloudLinux release 7.5 (Viktor Gorbatko)

pkg_arch:       noarch

pkg_epoch:      0

pkg_fingerprint: 8C55 A662 8608 CB71

pkg_name:       sos

pkg_release:    9.el7_5.cloudlinux

pkg_vendor:     CloudLinux

pkg_version:    3.5

runlevel:       N 3

time:           Fri 16 Nov 2018 05:07:55 PM CET

type:           Python

uid:            0

username:       root

uuid:           be3103bd203e454ce0cb1061f61309e10da2108e

Cross-posted to cpanel forum, since I am not sure if this is a CloudLinux or cPanel issue.

[ighertesco]

Hello,

If you do use mod_lsapi, try to update it to the latest beta version, which was released yesterday: https://www.cloudlinux.com/cloudlinu...i-updated-1-41

I would also recommend submitting a ticket to https://cloudlinux.zendesk.com/hc/en-us/requests/new, so we could take a closer look.

Thank you.

[zodiac9797]

Hello Igor,
just did mod_lsapi update as you suggested and server reboot and it seems that everything is ok now. At first I did update and httpd restart, but that didnt help until I did server reboot.

Is there an info page where I can see when was the last stable mod_lsapi version released?
I would like to know if this happened in the last week or two.

Thank You!

[zodiac9797]

Forgot to say that during the mod_lsapi update there were a lot of "signal Segmentation fault" messages:

[Fri Nov 16 18:48:44.855806 2018] [core:notice] [pid 35796:tid 47667546952768] AH00051: child pid 6212 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Fri Nov 16 18:48:33.844114 2018] [core:notice] [pid 35796:tid 47667546952768] AH00051: child pid 3544 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Fri Nov 16 18:48:22.831369 2018] [core:notice] [pid 35796:tid 47667546952768] AH00051: child pid 5150 exit signal Segmentation fault (11), possible coredump in /etc/apache2

[apb]

The latest mod_lsapi update was on Oct 3: https://www.cloudlinux.com/component...api?Itemid=292

[apb]

I had a lot of weird issues starting with update received on 19-20 nov (when apache was updated Nov 20 01:06:02 Updated: 1:ea-apache24-2.4.37-1.el6.cloudlinux.x86_64).

my nginx logs were full of errors like this:

2018/11/21 10:25:53 [error] 102780#0: *943815 upstream prematurely closed connection while reading response header from upstream, client: ...
2018/11/21 10:26:07 [error] 102780#0: *943820 upstream prematurely closed connection while reading response header from upstream, client: ...
2018/11/21 10:26:08 [error] 102780#0: *943835 upstream prematurely closed connection while reading response header from upstream, client: ...
2018/11/21 10:26:10 [error] 102780#0: *943876 upstream prematurely closed connection while reading response header from upstream, client: ...
2018/11/21 10:26:11 [error] 102780#0: *943876 upstream prematurely closed connection while reading response header from upstream, client: ...
2018/11/21 10:26:12 [error] 102778#0: *943898 upstream prematurely closed connection while reading response header from upstream, client: ...

I solved by updating to latest beta of mod_lsapi:

yum update liblsapi liblsapi-devel ea-apache24-mod_lsapi --enablerepo=cl-ea4-testing --enablerepo=cloudlinux-updates-testing

solved by applying this update:
Nov 21 10:26:00 Updated: liblsapi-1.1-33.el6.cloudlinux.x86_64
Nov 21 10:26:05 Updated: ea-apache24-mod_lsapi-1.1-33.el6.cloudlinux.x86_64
Nov 21 10:26:06 Updated: liblsapi-devel-1.1-33.el6.cloudlinux.x86_64

version with issues was: Oct 04 01:40:45 Updated: liblsapi-1.1-31.el6.cloudlinux.x86_64