CloudLinux 8 lve kernel rejects BPF_PROG_TYPE_LSM at bpf() syscall

[chris]

On CloudLinux 8.10 with kernel 4.18.0-553.111.1.lve.el8.x86_64 (reproduced on two independent hosts), the bpf() syscall rejects BPF_PROG_TYPE_LSM even though CONFIG_BPF_LSM=y, bpf is in /sys/kernel/security/lsm, BTF is present, and the kernel-internal symbols (bpf_trampoline_*, bpf_ringbuf_*) are exported.

Reproduction:

bpftool feature probe | grep 'program_type lsm is'
# eBPF program_type lsm is NOT available

strace shows the load syscall returning EINVAL at the dispatch layer (no verifier log). The same probe on AlmaLinux 8.10 (4.18.0-553.89.1.el8_10.x86_64, same upstream RHEL 8.10 base) returns "is available" and the verifier engages normally.

Question: is the absence of userspace BPF_PROG_TYPE_LSM support in the lve kernel intentional (e.g. dropped by an lve patch), or is it an oversight in the backport? Could it be exposed in a future lve kernel release? The internal subsystem appears to be present; only the userspace loader path is gated.

This blocks running BPF LSM-based security tooling on stock CL 8 hosts. Switching to kernel-lts since it's EL9 kernel I assume it will work ?

Thanks.

[bogdan.sh]

That's interesting, I just checked the sources and the premise here is worth checking against the kernel source firrst. The build config tells a different story than the runtime symptom.

For 4.18.0-553.111.1.lve.el8, the source branch is cl8-10_553.111.1 in the kernel-el8 repo. The x86_64 config has:

CONFIG_BPF_LSM=y
CONFIG_BPF_PROG_TYPE_LSM=y
CONFIG_LSM="yama,integrity,selinux,bpf"

The CloudLinux config overlay makes no changes to LSM or BPF symbols, and the CL patch set over the AlmaLinux 8.10 base does not touch security/bpf/, kernel/bpf/, or any of the LSM headers. So at build level this should behave identically to AlmaLinux 8.10, and bpf is in the compiled-in default LSM order.

That points at runtime LSM ordering rather than the kernel build. The usual culprit is a lsm= argument on the kernel command line overriding CONFIG_LSM, or the legacy security= parameter pinning to a single major LSM. Either of those would produce exactly the symptom you describe: BPF_PROG_TYPE_LSM rejected at dispatch with no verifier log, even though CONFIG_BPF_LSM=y is visible in the kernel.

Could you paste the output of:

cat /proc/cmdline
cat /sys/kernel/security/lsm

If /sys/kernel/security/lsm does not list bpf, that is the answer, and the fix is to either drop the lsm= override or extend it to include bpf, then reboot. If bpf is present and the load still fails, that is a different bug and we would have to troubleshoot more.

[chris]

Apologies — the premise of the original post was wrong. AlmaLinux 8.10 shows identical behaviour once you go past bpftool feature probe and actually try to attach a BPF LSM program.

Both kernels have CONFIG_BPF_LSM=y and the hooks compiled in (237 bpf_lsm_* symbols in /proc/kallsyms), but vmlinux BTF contains zero bpf_lsm_* entries on either, so attach_btf_id can't be resolved by the loader.

The bpftool probe disagreement between AlmaLinux and CL was a bpftool/libbpf version artifact — both kernels fail the actual load, the tool just narrates the failure differently.

So this isn't a CL-specific regression. Apologies.

For CL8 specifically the kernel-lts path doesn't work as an alternative either, since kernel-lts ships without LVE — so any shared-hosting box that needs LVE+CageFS is stuck on the .lve. kernel and therefore without functional BPF LSM. The path forward seems to be CL9+ where LVE is delivered as kmods against a stock EL9 kernel that has BPF LSM properly wired up.

[bogdan.sh]

Thanks for the follow-up, and no apologies needed. The BTF angle is the right diagnosis. Once attach_btf_id cannot resolve, the load fails regardless of CONFIG_BPF_LSM, and that is an upstream RHEL 8 backport gap rather than anything CloudLinux chose to remove. We do not have an active workstream to backport the missing BTF for LSM hooks into the CL8 .lve kernel either.

Your CL9 hypothesis is correct on the architectural point. CloudLinux 9 ships no custom kernel. So whatever BPF LSM behaviour the AL9 kernel exposes, a CL9 box gets exactly the same.

Worth verifying before committing to a migration: spin up a current AlmaLinux 9 host and re-run your bpftool and load test against it. If the attach succeeds there, CL9 with kmod-lve will look identical to your BPF program, because the kernel ABI it attaches to is upstream AL9's, not a CL-patched variant.

[chris]

Thanks for the follow-up, and no apologies needed. The BTF angle is the right diagnosis. Once attach_btf_id cannot resolve, the load fails regardless of CONFIG_BPF_LSM, and that is an upstream RHEL 8 backport gap rather than anything CloudLinux chose to remove. We do not have an active workstream to backport the missing BTF for LSM hooks into the CL8 .lve kernel either.

Your CL9 hypothesis is correct on the architectural point. CloudLinux 9 ships no custom kernel. So whatever BPF LSM behaviour the AL9 kernel exposes, a CL9 box gets exactly the same.

Worth verifying before committing to a migration: spin up a current AlmaLinux 9 host and re-run your bpftool and load test against it. If the attach succeeds there, CL9 with kmod-lve will look identical to your BPF program, because the kernel ABI it attaches to is upstream AL9's, not a CL-patched variant.

About kernel-lts, tried kernel-lts but couldn't load LVE/Cagefs. I thought the rpm package could take care of that with postinst or script or something.
No time to investigate (production srv) so rolled it back immediately.