LXC and "Failed to obtain ICMP socket: Operation not permitted"

[jhe-bzd]

CMK version: OMD - Open Monitoring Distribution Version 2.0.0-2021.12.14.cre
OS version: Ubuntu 20.04.3 LTS

Error message: check_icmp: Failed to obtain ICMP socket: Operation not permitted

Hi,

i am trying to setup checkmk2 in a LXC container and get permission issues that I cant fully solve.

The sites are setup by following the documentation

root# omd create bzd_cmk2
omd start bzd_cmk2

gives the first error:

Creating temporary filesystem /omd/sites/bzd_cmk2/tmp...mount: /opt/omd/sites/bzd_cmk2/tmp: must be superuser to use mount.
WARNING: You may continue without tmpfs, but the performance of Check_MK may be degraded.

I can get arround this by disabling tmpfs for the site. I do however not understand why it fails. As root I can mount a tmpfs file system.

mount -t tmpfs -o noauto,user,mode=755,uid=bzd_cmk2,gid=bzd_cmk2 tmpfs /opt/omd/sites/bzd_cmk2/tmp
just works

Another error is

Initializing Crontab.../var/spool/cron/: mkstemp: Permission denied
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe
ERROR

This seems to be caused by bzd_cmk2 being not part of the group crontab. Adding it solves it.

This leaves me with one Issue:
check_icmp: Failed to obtain ICMP socket: Operation not permitted

This is shown for all hosts. Adding them and doing the Ping test works, but the regular
background check fails.

I found a thread about checking for capabilities.

running gives the following output

#find /opt/omd/ -name check_icmp 
/opt/omd/sites/bzd_cmk2/var/check_mk/precompiled_checks/builtin/check_icmp
/opt/omd/versions/2.0.0-2021.12.14.cre/share/check_mk/checkman/check_icmp
/opt/omd/versions/2.0.0-2021.12.14.cre/share/check_mk/checks/check_icmp
/opt/omd/versions/2.0.0-2021.12.14.cre/lib/nagios/plugins/check_icmp

and checking for capabilities shows that the capability is missing

# find / -name check_icmp -exec getcap {} \;
/opt/omd/versions/2.0.0-2021.12.14.cre/share/check_mk/checkman/check_icmp = cap_net_raw+ep
/opt/omd/versions/2.0.0-2021.12.14.cre/share/check_mk/checks/check_icmp = cap_net_raw+ep
/opt/omd/versions/2.0.0-2021.12.14.cre/lib/nagios/plugins/check_icmp = cap_net_raw+ep

adding it can be done but makes no difference and need to be done manually for every new site

# setcap cap_net_raw+ep /opt/omd/sites/bzd_cmk2/var/check_mk/precompiled_checks/builtin/check_icmp
# find / -name check_icmp -exec getcap {} \;
/opt/omd/sites/bzd_cmk2/var/check_mk/precompiled_checks/builtin/check_icmp = cap_net_raw+ep
/opt/omd/versions/2.0.0-2021.12.14.cre/share/check_mk/checkman/check_icmp = cap_net_raw+ep
/opt/omd/versions/2.0.0-2021.12.14.cre/share/check_mk/checks/check_icmp = cap_net_raw+ep
/opt/omd/versions/2.0.0-2021.12.14.cre/lib/nagios/plugins/check_icmp = cap_net_raw+ep

stating the site now works without error

# omd start
Doing 'start' on site bzd_cmk2:
Temporary filesystem already mounted
Starting mkeventd...OK
Starting rrdcached...OK
Starting npcd...OK
Starting nagios...OK
Starting apache...OK
Starting redis...OK
Initializing Crontab...OK

I can also add hosts and services are discovered. They are however shown as down. The connection test shows working Ping

this a switch I monitored via snmp.

I also reinstalled checkmk2 multiple times and removed/added different sites.

thank you for your help.

[robin.gierse]

I cannot explain in detail, but this might be an issue with running it in a container.
Honest advice: Use a VM or dedicated hardware. Checkmk is designed to run on its own server and that just works. There are a lot of advantages in running on a dedicated host.

[jhe-bzd]

Thank you for your reply @robin.gierse

I just moved the container to a different host and it just works.
So I guess its related to the host but I don’t know how.

I am still evaluating checkmk2 so dedicated hardware is a bit much.

[robin.gierse]

I hear you, but a VM is not expensive too and might be less work than the self-made container.
We also have an official Docker image with the enterprise version if that is an option.

[zzzap (Terje)]

Sorry for bumping this old posting, but I was also trying to install checkmk on a lxc container.

As for this error
Creating temporary filesystem /omd/sites/bzd_cmk2/tmp...mount: /opt/omd/sites/bzd_cmk2/tmp: must be superuser to use mount. WARNING: You may continue without tmpfs, but the performance of Check_MK may be degraded.

This is most likely because there is no fstab in a default ubuntu lxc container.
You would normally expose any filesystems through the container config in lxc.

I created the fstab manually with a tmpfs entry and everything worked fine after that.
My guess is that it will also work fine if you create and empty fstab before you start “omd create”

[Glowsome (Michael Honkoop)]

@robin.gierse , sorry to revive this topic, but … if Check_MK can utilise TMPFS in a docker env, then what is stopping development from applying/finding a solution in a LXC -container (in particular ProxMox 7.x env)?

I can understand a/the decision being made to select a full VM compared to a LXC when setting up everything from scratch.
However as a person who is/has been using the raw-edition for quite the few years it is not a light decision to move away from current setup without any substantiated motivation other then trusting your ‘honest advice’.

Especially in a (clustered) environment where (shared-)storage is not that common, so the statement is : LXC, and if all else is no option then VM.

• Glowsome

[robin.gierse]

Quick question: Did you try to run the container in privileged mode?