Cloudlinux 8 and rootless Podman

[phjp]

Hi,

We run CloudLinux 8.10 without control panel on pretty much all our hosting and it works great! However, I've been exploring running Podman as a non-root user on Cloudlinux and I'm running into undocumented errors that do not seem to happen on other OS, so I find myself at a loss.

I'm trying to run Podman 4.9.4-rhel with runc 1.1.12 . I've created an user with cagefs disabled and set lve to unlimited. I am able to call podman and runc on the command line. However, when I try to start up a container, I get the following error:

Error: OCI runtime error: runc: you have no read access to runc binary file
runc create failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer

Afaik though, I DO have access to the runc binary from the user. While I've delved into strace log when booting the container, I haven't been able to find out what's exactly missing for runc to start properly. My best guess is that the error message is wrong and something else is blocking.

So, I must ask, is rootless Podman compatible with CloudLinux? Or is there a security feature that fundamentally breaks rootless Podman?

[phjp]

After further research, I've isolated a more telling error message : nsexec[15143]: could not ensure we are a cloned binary: Operation not permitted

This seems to be related to runc's process of cloning itself into memory. Does Cloudlinux prevent the execution of binaries in memory?

[bogdan.sh]

Hello,

CloudLinux does not block the Podman/runc services itself, however, there is a big part of security parameters we adjust like ptrace block which could affect the services you are trying to run. Please try disabling it per https://docs.cloudlinux.com/cloudlin.../#ptrace-block .

[phjp]

Hi,

thank you for this information. I've checked and user_ptrace protection as well as kernel.yama.ptrace_scope are not enabled (kernel.user_ptrace and kernel.user_ptrace_self = 1, kernel.yama.ptrace_scope= 0 ). I realize this might not even be a cloudlinux issue in the end, but now I'm going to look further into kernel parameters if I find anything.

[viridio]

We are in the same boat, and built our own control panel for Cloudlinux instead of running cPanel or any other commercial panel. Since we are doing it this way, we dont have easy access to Redis securely. We are looking into running either Docker, Podman, or something else to get Redis working on a per user level that is secure. Did you get this figured issue figured out with podman?
As our plan is to use a container for each user running Redis and then push that socket from that container into the users virtual FS/cage. You can set docker to not use iptables and push a socket out of the container to the actual file system.

@bogdan.sh Is it possible to take a path like /tmp/redis/USERNAME/redis.sock and push it into the specifc user ? Maybe something in /etc/cagefs/cagefs.mp like /tmp/redis/%USERNAME/redis.sock

Although this script talks about running podman as the user, so would this some how work better: https://gist.github.com/juanje/de5dd...8b0d089e837e4a

[bogdan.sh]

viridio, phjp , Please explain the benefits of running Podman on CloudLinux. What are the hosting requirements, and what packages are you offering for your cstomers to meet those needs? CloudLinux is already a containerized technology that effectively splits resources between users.


The individual directories with Redis can be mounted per user using this article.

[phjp]

viridio, phjp , Please explain the benefits of running Podman on CloudLinux. What are the hosting requirements, and what packages are you offering for your cstomers to meet those needs? CloudLinux is already a containerized technology that effectively splits resources between users.

Docker offers ready made images that you can basically just deploy and run into a fairly secure environment. While Cloudlinux is great at isolating customers workload, it doesn't exactly do much in terms of easy software distribution and installation for anything that's more complicated than installing a package. My specific goal here is to run apache SOLR into its own user without having to do an install from root, but the ability to run any type of rootless container is a net plus no matter how you look at it.

Furthermore, running Docker images as user is something we get asked about by our customers quite often.

[viridio]

@bogdan.sh thanks for replying back. for us, The key is to use Redis in an easy way while making it secure per user. You have a KB article that kind of outlines how to use Redis using CageFS here but it still is insecure (it still uses shared storage for multiple websites) and we dont want to use redis ACL. cPanel uses containers to make Redis secure, but we dont use cPanel and neither does phjp as it appears. We need a way to offer it securely and running a Redis instance for each user pushing the socket for that instance into their cagefs file system would do this, and allow wordpress redis plugins to access it.

There are alternates it appears like what Kualo is doing here using KeyDB, and it appears it does not have any ACL either, so setting this up to be a redis compatible replacement that is secure on a per user level using CageFS would require the same thing as Redis, some way to run instances within the users account or getting that socket access to their specific instance outside their account. As KeyDB also uses a unix socket, so one for each user means that in the users file system it has to be unique and connecting to the correct instance for them, since there is no login/password.

[bogdan.sh]

We are going to test this on our lab servers, unfortunately, cannot provide you with any ETA.

Same time, you might want to try the same on AlmaLinux 9.

[phjp]

If it helps, I tried popping up a virgin cloudlinux 8.8 without a license or anything setup. Rootless podman worked out of the box and I did not encounter the error I got on my previous test server, which is fully configured and has a license. So, it does work on basic CloudLinux, but something prevents it from working on my licensed, updated and configured machine. Granted, it could be something we setup ourselves that's blocking it, so I'm curious if you'll be able to run it.

That said, the error "nsexec[15143]: could not ensure we are a cloned binary: Operation not permitted" sounds to me like something is preventing non-privileged user binary execution in memory, and that would make sense on a secure platform like Cloudlinux. I'd just like to know *what* and if I can configure it.

[bogdan.sh]

phjp Do you use AppArmor?

Same time, I would try to use strace to catch where exactly it fails:

strace -f -o strace.log -s 1024 podman run X Y Z

Then search for permissions failure:

grep -5 "EPERM" strace.log

Could you please guide us how do you do this? Do you have any instructions available? We would like to replicate the same way you do.

[phjp]

We do not use apparmor.

It's a bit of a mess because I had to cross several hurdles and try a lot of things before I finally got the error message I posted previously, but let me try giving you steps based on my bash history:

1. Install podman
yum install podman slirp4netns -y

2. Create a new user, in this case I called it solr
useradd -m solr

3. My goal here was to create a single service user account that works outside most user limitations, so I tried to set cagefs and lve limits so they don't block me.
cagefsctl --disable solr
vi /etc/cagefs/exclude/solr
cagefsctl --force-update

lvectl list
lvectl set-user solr --unlimited

4. Now I go inside the user and start testing Podman. Here I'll just use an image from redhat's documentation, but feel free to pick any image, it doesn't make a difference. This is where I got the first podman error (which seems somewhat misleading) :
su - solr
podman pull ubi7/ubi
podman run ubi7/ubi
Error: OCI runtime error: runc: you have no read access to runc binary file
runc create failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer

5. We see here that it's actually runc that encounters the issue. This is when I started hunting for EPERM errors in strace and try to solve each one. I'm replicating these steps on another test server so I'll list every EPERM I get and what I did to try and fix each:

strace -f -o strace.log -s 1005 podman run ubi7/ubi

974969 prlimit64(0, RLIMIT_NPROC, {rlim_cur=4096*1024, rlim_max=4096*1024}, NULL) = -1 EPERM (Operation not permitted)
974969 prlimit64(0, RLIMIT_NPROC, {rlim_cur=1024*1024, rlim_max=1024*1024}, NULL) = -1 EPERM (Operation not permitted)
974969 prlimit64(0, RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}, NULL) = -1 EPERM (Operation not permitted)

I edited /etc/security/limits.conf and added the following lines :
solr hard nofile 1048576
solr hard nproc 4194304

975000 openat(AT_FDCWD, "/proc/self/exe", O_RDONLY|O_CLOEXEC) = -1 EPERM (Operation not permitted)

I believe I changed the group of user solr to be part of clsupergid to be able to have better visibility in /proc, but that did not fix this.

975014 mknodat(AT_FDCWD, "/home/containertest/.local/share/containers/storage/overlay/backingFsBlockDev.tmp", S_IFBLK|0600, makedev(0x8, 0x11)) = -1 EPERM (Operation not permitted)

I have not found a solution to this one

6. In parallel, seeing that podman was returning an error related to runc, I started trying to run runc directly without going through podman, for test purposes (see https://github.com/opencontainers/runc for more details on running rootless container directly with runc). This is when I finally found what seems to be the real error message.

mkdir ~/mycontainer
cd ~/mycontainer
mkdir data
mkdir rootfs
podman export $(podman create busybox) | tar -C rootfs -xvf -
runc spec --rootless
runc --root data run mycontainerid

FATA[0000] nsexec[1991327]: could not ensure we are a cloned binary: Operation not permitted
WARN[0000] unable to get oom kill count error="no directory specified for memory.oom_control"
ERRO[0000] runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer

7. Executing runc with strace gives us the following EPERM :

1991591 mount("/proc/self/exe", "/home/solr/mycontainer/data/mycontainerid/runc.KYWp91", 0x5580cdfe19bf, MS_BIND, 0x5580cdfe19bf) = -1 EPERM (Operation not permitted)
1991591 openat(AT_FDCWD, "/proc/self/exe", O_RDONLY|O_CLOEXEC) = -1 EPERM (Operation not permitted)

I believe the mount failing there is the key to the current issue.