Cmk-agent-ctl segmentation fault on Rocky Linux 9

Hi,

I am experiencing a problem with registering the agent:
CMK version: 2.1.0p15
OS version: Rocky Linux release 9.0 (Blue Onyx)

[root@CHECKMK services]# cmk-agent-ctl
Segmentation fault (core dumped)

The same result also with the all needed parameters for the registration.

I am not able to debug the cmk-agent-ctl since it is in binary form:
[root@CHECKMK services]# less /usr/bin/cmk-agent-ctl
“/usr/bin/cmk-agent-ctl” may be a binary file. See it anyway?

Has someone experienced the same problem?

BR, Joe

1 Like

We’ve seen it when SELinux was enabled and blocking the agent registration. Not sure why it failed with a segfault but it did :slight_smile:

Hi,

i had the same problem and did some troubleshooting.

Setting SELinux to permissive with setenforce 0 and looking into the logs with journalctl -f -t setroubleshoot gave me the following output:

Note: The package setroubleshoot-server must be present on the system. dnf install setroubleshoot-server

SELinux is preventing /usr/bin/cmk-agent-ctl from execmod access on the file /usr/bin/cmk-agent-ctl.                                                                            
                                                 
*****  Plugin catchall_boolean (89.3 confidence) suggests   ******************                                                                                                  
                                                 
If you want to allow selinuxuser to execmod
Then you must tell SELinux about this by enabling the 'selinuxuser_execmod' boolean.                                                                                            
                                                 
Do                                                                                                                                                                              
setsebool -P selinuxuser_execmod 1
                                                                                                                                                                                
*****  Plugin catchall (11.6 confidence) suggests   **************************                                                                                                  
                                           
If you believe that cmk-agent-ctl should be allowed execmod access on the cmk-agent-ctl file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.                                                                                                                    
Do                                
allow this access for now by executing:
# ausearch -c 'cmk-agent-ctl' --raw | audit2allow -M my-cmkagentctl           
# semodule -X 300 -i my-cmkagentctl.pp

After setting setsebool -P selinuxuser_execmod 1 the agent works flawlessly.

Root Cause:
RedHat changed the default value of the selinuxuser_execmod SELinux Boolean with RHEL 9.

If you search for selinuxuser_execmod on the following page you will find the notice with a reference to the bugzilla entrie.

BR
Topfi

2 Likes

Here is some context to the problem for the forum search:

With selinux enabled the rpm is installed but there are errors, and therefore
the user cmk-agent is not created:

[root@rocky9 ~]# yum install check-mk-agent-2.1.0p15-1ad8f91bfc1810cb.noarch.rpm
Last metadata expiration check: 0:06:38 ago on Fri 04 Nov 2022 12:15:57 PM CET.
Dependencies resolved.
=======================================================================================================================================================================
 Package                                Architecture                   Version                                              Repository                            Size
=======================================================================================================================================================================
Installing:
 check-mk-agent                         noarch                         2.1.0p15-2.1ad8f91bfc1810cb                          @commandline                         4.3 M
 
Transaction Summary
=======================================================================================================================================================================
Install  1 Package
 
Total size: 4.3 M
Installed size: 4.3 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                               1/1
  Running scriptlet: check-mk-agent-2.1.0p15-2.1ad8f91bfc1810cb.noarch                                                                                             1/1
  Installing       : check-mk-agent-2.1.0p15-2.1ad8f91bfc1810cb.noarch                                                                                             1/1
  Running scriptlet: check-mk-agent-2.1.0p15-2.1ad8f91bfc1810cb.noarch                                                                                             1/1
 
 
/var/lib/cmk-agent/scripts/super-server/0_systemd/setup: line 67:  1427 Segmentation fault      (core dumped) "${BIN_DIR:-/usr/bin}"/cmk-agent-ctl --version > /dev/null 2>&1
/var/lib/cmk-agent/scripts/super-server/0_systemd/setup: line 67:  1430 Segmentation fault      (core dumped) cmk-agent-ctl --version > /dev/null 2>&1
/var/lib/cmk-agent/scripts/super-server/0_systemd/setup: line 67:  1438 Segmentation fault      (core dumped) "${BIN_DIR:-/usr/bin}"/cmk-agent-ctl --version > /dev/null 2>&1
/var/lib/cmk-agent/scripts/super-server/0_systemd/setup: line 67:  1444 Segmentation fault      (core dumped) cmk-agent-ctl --version > /dev/null 2>&1
Deploying systemd units: check-mk-agent-async.service check-mk-agent.socket.fallback check-mk-agent@.service
Deployed systemd
/var/tmp/rpm-tmp.Cwgtc5: line 11:  1472 Segmentation fault      (core dumped) "/usr/bin"/cmk-agent-ctl --version > /dev/null 2>&1
Activating systemd unit 'check-mk-agent-async.service'...
Created symlink /etc/systemd/system/multi-user.target.wants/check-mk-agent-async.service → /usr/lib/systemd/system/check-mk-agent-async.service.
Activating systemd unit 'check-mk-agent.socket'...
Created symlink /etc/systemd/system/sockets.target.wants/check-mk-agent.socket → /usr/lib/systemd/system/check-mk-agent.socket.
Deactivating systemd unit 'cmk-agent-ctl-daemon.service' (if active)...
 
  Verifying        : check-mk-agent-2.1.0p15-2.1ad8f91bfc1810cb.noarch                                                                                             1/1
 
Installed:
  check-mk-agent-2.1.0p15-2.1ad8f91bfc1810cb.noarch                                                                                                                   
 
Complete!

Agent register is not possible:

`[root@rocky9 ~]` `# cmk-agent-ctl register --verbose --user cmkadmin --site mysite --server 192.168.56.1`

`ERROR [cmk_agent_ctl] Failed to run as user ` `'cmk-agent'` `. Please execute with sufficient permissions (maybe try ` `'sudo'` `).`

This could be importent for a policy, you get different source contexts if you

  • call cmk-agent-ctl as root in the terminal and/or by ssh
  • run it via systemd
  • run it via xinetd

Audit log shows something like:

type=AVC msg=audit(1667562203.971:306): avc:  denied  { execmod } for  pid=12704 comm="cmk-agent-ctl" path="/usr/bin/cmk-agent-ctl" dev="dm-0" ino=4744305 scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=file permissive=0
 
type=AVC msg=audit(1667565253.484:140): avc:  denied  { execmod } for  pid=3551 comm="cmk-agent-ctl" path="/usr/bin/cmk-agent-ctl" dev="dm-0" ino=4744305 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:object_r:bin_t:s0 tclass=file permissive=0

I did not check invocation via xinetd.

Even as root it segfaults:

# as root this sefaults
[root@rocky9 ~]# cmk-agent-ctl --version
Segmentation fault (core dumped)

Why execmod ?
How is it labeled ?

[root@rocky9 ~]# ldd /usr/bin/cmk-agent-ctl
not a dynamic executable
 
[root@rocky9 ~]# ls -Z /usr/bin/cmk-agent-ctl
 
system_u:object_r:bin_t:s0 /usr/bin/cmk-agent-ctl
\ /\ /\ /\ /
 ------ ------- ---- --
 user role type MLS (multi layered security)
 
Only type is relevant in this context.

@topfi this is a solution, thanks for that but two things to notice:

  1. You have to reinstall the check-mk-agent rpm package because it also faileld to add a user when this selinux boolean is not set and selinux is enabled.

  2. This sebool will not only allow cmk-agent-ctl to work but also weakens the selinux policy for other binaries !

setsebool -P selinuxuser_execmod 1

But I think this is still better than disabling selinux completely.

PS: the exemod is needed because cmk-agent-ctl is a compressed binary.
To uncompress itself it needs this permission/syscall)

What we need is perhaps a dedicated selinux label for the cmk-agent-ctl and a policy that only allows binaries with that label to to execmod.

Proposal: We label

/usr/bin/cmk-agent-ctl as system_u:object_r:cmk_agent_bin_t:s0 instead of
system_u:object_r:bin_t:s0 and then write a policy to allow:

allow unconfined_service_t cmk_agent_bin_t:file execmod;
allow unconfined_t cmk_agent_bin_t:file execmod;

@fbotte can you jump in :wink:

Sorry I’m not on RHEL 9 and CMK 2.1.
Right now i have no time to look into this - Sorry again.

But what you could do is set selinux to permissive.

Install the client and the run audit2allow to generate a policy extension that will give all the needed permissions to checkmk.

The best solution would of course be to have a dedicated security context for the agent, but that is a little more work.

The generated policy extension can be compiled and inserted into the policy like described in my original post about selinux.

Be sure to check the generated policy extension before compiling and using it. audit2allow creates a extension that allows everything that has been blocked by selinux. Might even be a completely unrelated process that was started at the same time as the agent installation.

cheers
Frank

We had a look onto this issue and we finally managed to fix this problem.
See Werk #14962: Running cmk-agent-ctl leads to segfault on SELinux systems

Technical backgound: The cmk-agent-ctl executable is statically linked to maximize compatibility. As a downside, it is 8MB in size. To mitigate this, we used to compress the cmk-agent-ctl executable with UPX and ended up with a 4MB sized executable. But it turned out that running the UPX-packed executable uses text relocation, leading to the issue with SELinux.

Starting with the mentioned Werk, we compress cmk-agent-ctl with gzip (again leading to a file of 4MB), and decompress it during package installation. This way, we end up with the untouched executable that SELinux does not complain about.

That said, please let us know if you encounter additional issues with this new approach!

Cheers
Andi

2 Likes

Hi,

thanks for the fix. I will apply the update as soon it is released and will report back any issues.

br
Topfi

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.