yum / yum-complete-transaction spawns 3 more yum processes

[DigiState]

This has been happening on multiple servers for quite awhile now. Whenever we run updates all the servers that run CloudLinux spawn yum multiple times, causing yum to wait for itself to finish.
ps aux | grep tee shows 2 tee sessions to the same log file.

# ps aux | grep tee
root       76662  0.0  0.0   4364   360 pts/0    S+   11:31   0:00 tee -ia /var/log/directadmin/custombuild.1708943469.76592.cmV3cml0ZV9jb25mcwA.log
root       76664  0.0  0.0   4364   360 pts/0    S+   11:31   0:00 tee -ia /var/log/directadmin/custombuild.1708943469.76592.cmV3cml0ZV9jb25mcwA.log
root       81972  0.0  0.0 110804   892 pts/1    S+   11:47   0:00 grep --color=auto tee
​

and the log file shows

Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum-complete-tr
    Memory : 215 M RSS (841 MB VSZ)
    Started: Mon Feb 26 11:28:48 2024 - 17:26 ago
    State  : Sleeping, pid: 75671
Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum-complete-tr
    Memory : 215 M RSS (841 MB VSZ)
    Started: Mon Feb 26 11:28:48 2024 - 17:28 ago
    State  : Sleeping, pid: 75671

​

processlist:

\_ /usr/bin/python -tt /sbin/yum-complete-transaction
    \_ /bin/sh /var/tmp/rpm-tmp.0CeQAq 5
         \_ /usr/local/directadmin/custombuild/build rewrite_confs
              \_ custombuild -c source /proc/self/fd/3 build rewrite_confs
                  \_ custombuild -c source /proc/self/fd/3 build rewrite_confs
                  |   \_ tee -ia /var/log/directadmin/custombuild.1708943469.76592.cmV3cml0ZV9jb25mcwA.log
                  \_ custombuild -c source /proc/self/fd/3 build rewrite_confs
                  |   \_ tee -ia /var/log/directadmin/custombuild.1708943469.76592.cmV3cml0ZV9jb25mcwA.log
                  \_ custombuild -c source /proc/self/fd/3 build rewrite_confs
                      \_ custombuild -c source /proc/self/fd/3 build rewrite_confs
                          \_ sh ./install/da_cb_install stable 1.1-72
                              \_ /usr/bin/python /bin/yum install -y yum-utils file

​

The waiting never ends.

[bogdan.sh]

Hello,

Could you please clarify how exactly do you run updates?

Same time, please show the content of /etc/resolv.conf file. Or try setting some good external nameservers in /etc/resolv.conf (e.g. 8.8.8.8 / 8.8.4.4) then clean cache:

yum clean all; yum clean metadata

[DigiState]

We use google for dns. Updates are done with

yum -y update

But like I said, it also happens with
yum-complete-transaction (without options)

[DigiState]

yum clean all; yum clean metadata

did not make any difference.

The wait has now gone on for 17 hours.

Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum-complete-tr
    Memory : 227 M RSS (573 MB VSZ)
    Started: Mon Feb 26 15:36:16 2024 - 17:19:48 ago
    State  : Sleeping, pid: 150489

​

/etc/resolv.conf

# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 8.8.8.8
nameserver 8.8.4.4

​

[DigiState]

I aborted the yum-complete-transaction command I had running and am presented with the following message:

Another instance of custombuild (PID 152182) is holding the lock on "/usr/local/directadmin/custombuild/.custombuild.lock", exiting

Which is odd because custombuild was not started, by me at least.

[bogdan.sh]

What OS is that? I just tried on CloudLinux 8 and cannot reproduce the issue. Searched for similar tickets and seems we have none. Anything special in "yum history | head"?

[DigiState]

It's CloudLinux release 7.9 (Boris Yegorov)

This happens on all the cloudlinux servers we have. Anything you do yum related starts custombuild.

[bogdan.sh]

I am wondering, why the yum transactions become uncompleted there? Can you recall any errors from the past? Please try to clean them with a pretty safe:

yum-complete-transaction --cleanup-only

[DigiState]

That certainly got rid of the list, sure. Next round of updates is on monday, so let's see if this sticks.

[bogdan.sh]

Let's investigate more
Please try running update then perform complete-transaction to check if any unfinished transactions left?

yum update
yum-complete-transaction

[DigiState]

No updates since yesterday. We're going to have to wait awhile.

[DigiState]

Running updates today. There was an update for exim but it would not install. Had to remove old libiconv and build all. During the process I keep getting

Error: /etc/cagefs/cagefs.mp file contains incorrect path - /usr/local/lib/php is NOT a directory or does NOT exist​

whcih is weird because it does exist

]# ls -l /usr/local/bin/php
lrwxrwxrwx 1 root root 26 Sep 6 11:31 /usr/local/bin/php -> /usr/local/php72/bin/php72

[bogdan.sh]

The cagefs.mp file should include only directories since it is used to mount them into virtual file system. While seems the /usr/local/bin/php is a symlink to binary file. The file itself is already pushed to CageFS with /etc/cagefs/conf.d/php.cfg file so please remove it from /etc/cagefs/cagefs.mp .

[DigiState]

Right. I removed the line from /etc/cagefs/cagefs.mp

[DigiState]

I'll consider this matter closed.