Podman (non root) Checkmk check_icmp operation not permitted

[julius]

CMK version : 2.0.0b3 RAW (container version), also tested with 1.6.0p19
Container OS version : Debian GNU/Linux 9 (stretch)
Host OS version : Arch Linux (5.9.14-arch1-1)
Podman version : 2.2.1

Error message : check_icmp: Failed to obtain ICMP socket: Operation not permitted

Hello there,

I can successful run the Checkmk container version with a non root account with Podman. When I add a host only the check_icmp fails and Checkmk marks the host as down, but the services are up.

Description of the problem : In the events of recent 4 hours window you can see the following message:
“check_icmp: Failed to obtain ICMP socket: Operation not permitted”

My first question is, which check_icmp is used, because there are multiple:

[monitoring@molecule-instance-arch ~]$ podman exec -it checkmk bash
root@molecule-instance-arch:/# cd /opt
root@molecule-instance-arch:/opt# find . -name check_icmp
./omd/sites/cmk/var/check_mk/precompiled_checks/builtin/check_icmp
./omd/sites/cmk/tmp/check_mk/check_includes/builtin/check_icmp
./omd/versions/2.0.0b3.cre/share/check_mk/checkman/check_icmp
./omd/versions/2.0.0b3.cre/share/check_mk/checks/check_icmp
./omd/versions/2.0.0b3.cre/lib/nagios/plugins/check_icmp

My guess is ./omd/versions/2.0.0b3.cre/lib/nagios/plugins/check_icmp since I get a permission denied on the others.

My second question is, what is going on? It’s obvious a permission issue, when I run the container as root the check_icmp works.

What i did: Since /bin/ping also didn’t work:

1. I did a sysctl -w net.ipv4.ping_group_range="0 429496729" on the container host, see this Redhat post
2. Added the --cap-add NET_RAW [1] option to the podman run command since Checkmk switched from setuid to linux capabilities.

[1] Capability Names and Descriptions https://en.wikibooks.org/wiki/Grsecurity/Appendix/Capability_Names_and_Descriptions

After that, /bin/ping and /opt/omd/sites/cmk/lib/nagios/plugins/check_ping works, but the check_icmp doesn’t.

A confusing thing is that /bin/ping and /opt/omd/sites/cmk/lib/nagios/plugins/check_icmp do have the cap_net_raw+ep capability configured, but the /opt/omd/sites/cmk/lib/nagios/plugins/check_ping doesn’t:

[monitoring@molecule-instance-arch ~]$ podman exec -it checkmk bash
root@molecule-instance-arch:/# getcap /bin/ping
/bin/ping = cap_net_raw+ep
root@molecule-instance-arch:/# getcap /opt/omd/sites/cmk/lib/nagios/plugins/check_ping
root@molecule-instance-arch:/# getcap /opt/omd/sites/cmk/lib/nagios/plugins/check_icmp
/opt/omd/sites/cmk/lib/nagios/plugins/check_icmp = cap_net_raw+ep

Also running it with --cap-add ALL for testing purposes doesn’t solve the issue.

Running it with strace I still get a permission denied on obtaining a ICMP socket:

[monitoring@molecule-instance-arch ~]$ podman exec -it checkmk bash
root@molecule-instance-arch:/# strace -e capget,socket /opt/omd/sites/cmk/lib/nagios/plugins/check_icmp -H 127.0.0.1
socket(AF_INET, SOCK_RAW, IPPROTO_ICMP) = -1 EPERM (Operation not permitted)
check_icmp: Failed to obtain ICMP socket: Operation not permitted
+++ exited with 3 +++

When I run a a strace with check_ping I do not see any capget or socket activity but it works?
When I run a a strace with /bin/ping I see both capget and socket activity and it works.

I thought that the check_ping and check_icmp binaries are not capability aware, but why is check_ping working then? I also saw on the net that check_ping is a wrapper for /bin/ping, but strace does not calls to /bin/ping.

Extra info:

My podman set*id setup:

[monitoring@molecule-instance-arch ~]$ grep monitoring /etc/sub*
/etc/subgid:monitoring:1000000:65536
/etc/subuid:monitoring:1000000:65536

Podman run command:

#!/bin/bash

podman run -d \
  --name checkmk \
  --cap-add NET_RAW \
  --cap-add=SYS_PTRACE \
  -v "/opt/podman/monitoring/non-root-checkmk/:/omd/sites" \
  -v /etc/localtime:/etc/localtime:ro \
  -e CMK_PASSWORD="cmkadmin" \
  --ulimit nofile=1024 \
  --no-healthcheck \
  --network host \
  docker.io/checkmk/check-mk-raw:2.0.0-latest

Troubleshoot commands inside Checkmk container:

#!/bin/bash

apt update
apt install -y strace
capsh --print
getcap /bin/ping
getcap /opt/omd/sites/cmk/lib/nagios/plugins/check_ping
getcap /opt/omd/sites/cmk/lib/nagios/plugins/check_icmp
strace -e capget,socket /bin/ping -c1 127.0.0.1
strace -e capget,socket /opt/omd/sites/cmk/lib/nagios/plugins/check_ping -4 -H 127.0.0.1 -w '100,5%' -c '200,15%'
strace -e capget,socket /opt/omd/sites/cmk/lib/nagios/plugins/check_icmp -H 127.0.0.1

Someone got an idea?

Regards, Julius

[julius]

I discovered that the --network host option is causing this issue. When I run the container without that option the check_icmp works.

The reason I used the --network host option is that I can use Individual program call instead of agent access rule to ssh into the container host and monitor that system as well.

Piece from the podman-run manpage:

–network=mode, --net
Set the network mode for the container.

• host: use the Podman host network stack. Note: the host mode gives the container full access to local system services such as D-bus and is therefore considered insecure;

[julius]

I solved the problem,

You can run the podman run command with the --network slirp4netns:allow_host_loopback=true option. After that you can reach your podman host from the Checkmk container on 10.0.2.2 and use monitoring over ssh for example to monitor the podman host (and check_icmp keeps working).

Part from the podman-run manpage:

• slirp4netns[:OPTIONS,...]: use slirp4netns(1) to create a user network stack. This is the default for rootless containers. It is possible to specify
            these additional options:

  • allow_host_loopback=true|false: Allow the slirp4netns to reach the host loopback IP (10.0.2.2). Default is false.

Still wondering why the check_icmp is not permitted when using the --network host option though …

The final non-root run script:

#!/bin/bash

mkdir -p /opt/podman/monitoring/non-root-checkmk

podman run -d \
  --name checkmk \
  --cap-add NET_RAW \
  -v "/opt/podman/monitoring/non-root-checkmk/:/omd/sites" \
  -v /etc/localtime:/etc/localtime:ro \
  -e CMK_PASSWORD="cmkadmin" \
  --ulimit nofile=1024 \
  -p 5000:5000 \
  --network slirp4netns:allow_host_loopback=true \
  docker.io/checkmk/check-mk-raw:2.0.0-latest