Docker container won't start xinetd (for livestatus)

Troubleshooting
, ,
1

CMK version: 2.0.0-latest
OS version: Alpine Linux 3.18.0 64bit / Linux 6.1.45

Hey folks,
I’m trying to get check-mk-raw running as docker container. However when I docker exec -it -u cmk monitoring bash and then omd config to active the livestatus and do a omd start afterwards, it gets stuck trying to start xinetd.

I have tried a number of different other container versions, but to no avail. The check-mk-free image runs fine, but I cannot connect my raw master instance with this free distributed site. I also tried running my container with --privileged but it didn’t change anyhting.

Help is very much appreciated.

2

I run the Checkmk Raw Edition as a Docker container with enabled with LIVESTATUS_TCP=on and it works fine. I did it by specifiying

docker run ... -e CMK_LIVESTATUS_TCP=on ... checkmk/check-mk-raw:2.2.0-latest

Which port number do you use (variable LIVESTATUS_TCP_PORT)? Can you show the whole output of omd config show? The usual port number for livestatus is 6557. Have you changed it? Do you run other sites with active TCP livestatus in the same container? Have you tried to select another port?

When xinetd won’t start up, it should log a message into ~/var/log/xinetd.log. Have you checked that logfile?

It is not necessary to run Checkmk from the official docker image with the --privileged flag. In fact, I would avoid it; especially if you plan to run software from untrusted 3rd party vendors like MKPs within the container.

If this does not help, please show your entire docker run command.

3

Output of omd config show:

ADMIN_MAIL:
APACHE_MODE: own
APACHE_TCP_ADDR: 0.0.0.0
APACHE_TCP_PORT: 5000
AUTOSTART: on
CORE: nagios
LIVESTATUS_TCP: on
LIVESTATUS_TCP_ONLY_FROM: 0.0.0.0 ::/0
LIVESTATUS_TCP_PORT: 6557
LIVESTATUS_TCP_TLS: on
MKEVENTD: on
MKEVENTD_SNMPTRAP: off
MKEVENTD_SYSLOG: off
MKEVENTD_SYSLOG_TCP: off
MULTISITE_AUTHORISATION: on
MULTISITE_COOKIE_AUTH: on
NAGIOS_THEME: classicui
NSCA: off
PNP4NAGIOS: on
TMPFS: off

Output of omd status:

OMD[site]:~$ omd status
mkeventd:       running
rrdcached:      running
npcd:           running
nagios:         running
apache:         running
redis:          running
stunnel:        running
xinetd:         stopped
crontab:        stopped
-----------------------
Overall state:  partially running

Docker run shell script:

#!/bin/bash
container="monitoring"
site="site"
docker stop $container && echo "stopping..."
docker rm $container && echo "removing..."
if [[ "$1" == "wipe" ]]; then
        docker volume rm $container && echo "$1 ing volume..."
fi
docker run -d \
        -p 6556:6556 \
        -p 6557:6557 \
        -p 8080:5000 \
        -p 8000:8000 \
        -p 162:162/udp \
        -p 514:514/udp \
        -p 514:514/tcp \
        -v $container:/omd/sites \
        -v /etc/localtime:/etc/localtime:ro \
        --restart always \
        --tmpfs /opt/omd/sites/$site/tmp:uid=1000,gid=100 \
        -e CMK_PASSWORD='<REDACTED>' \
        -e CMK_SITE_ID=$site \
        -e CMK_LIVESTATUS_TCP=on \
        --name monitoring \
        checkmk/check-mk-raw:2.0.0-latest
echo "done"

I have changed a line inside xinetd.conf to flags = NODELAY IPv6 just like on other remote sites that I administer to enable access via v6, but I doesn’t start anyways, so that’s not the problem here.

~/var/log/xinetd.log is empty for a long time, but after a while this line appears:
23/9/12@21:29:03 xinetd[1718]: CRITICAL: 1718 {init_rw_state} out of memory

And I can see a process which uses 99% CPU:
OMD[site]:~$ ps -ef | grep xinetd root 28 1 48 21:13 ? 00:08:38 /usr/sbin/xinetd -pidfile /run/xinetd.pid -stayalive -inetd_compat -inetd_ipv6

Docker version is 23.0.6 btw.

PS: Is there any documentation where I can read up on all CMK_… Environment Variables that are available?

4

I could solve the problem myself just now.

Turns out that I had to add --ulimit nofile=1024:1024 to my start-script and then xinetd was starting without anymore complaints.

It would still be nice to get a list of all Environment Variables tho…

5

Thank you.

Would you mind explaining how you came across this solution?

6

This was 10 months ago, so I can’t remember exactly.

I tried putting myself into the situation where I would have this problem without knowing the answer and found this thread: Try to run check_mk in a docker container - Troubleshooting - Checkmk Community

Maybe I read it, saw the ulimit parameter, and tried it out for myself.

Would you mind telling me why you’re so eager to know?

7

A student mentioned your solution without explanations in:

https://www.udemy.com/course/real-world-devops-project-from-start-to-finish

The context was that the checkmk container was stuck trying o start a mail service and your solution worked.

I believe in miracles, but also want to know why things work, if possible :wink:

1 Like
8

Interesting! Thanks.

I don’t have the machine on which I am running the docker container avaiable right now, but if you also ran into this problem, you can try to run the command mentioned in docker run | Docker Docs to see how many files a single process can open at a time. (Remember to remove the explicit setting beforehand :wink: )