[CCE / Docker] Unresposive/dead after few minutes

Troubleshooting
1

CMK version: CCE 2.2.0p1
OS version: docker, Host is Linux

Error message:-

When I run the container like described on the download page, the container gets unresponsive after a few minutes. web ui slow and Apache stays at >50% load. The container cant be stopped correctly then. A thing I noticed is, that the directory /omd/sites/cmk/var/check_mk/web/cmkadmin is accessible but neither deletable (just freezes) or even listable (any df or rm or ll/ls command just freezes).

Any try to stop the container results in Error response from daemon: cannot stop container: checkmk: tried to kill container, but did not receive an exit event. I need to reboot the whole system to get rid of any checkmk process leftovers which are unkillable!

This happened on 2/2 tests. Any ideas? As soon as my system is back up from my second test just now, I try to access the mentioned directory to see whats inside and caused the system to freeze.

2

The mentioned folder was accessible again after reboot, but there is nothing special in it:
grafik

The Apache log inside the container:

[Mon Jun 05 14:33:08.598781 2023] [mpm_prefork:notice] [pid 851] AH00163: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.4 Python/3.11 mod_fcgid/2.3.9 configured -- resuming normal operations
[Mon Jun 05 14:33:08.601609 2023] [core:notice] [pid 851] AH00094: Command line: '/usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf'
[Mon Jun 05 14:43:50.282451 2023] [core:warn] [pid 851] AH00045: child process 855 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.282772 2023] [core:warn] [pid 851] AH00045: child process 1309 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.282965 2023] [core:warn] [pid 851] AH00045: child process 1727 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.283168 2023] [core:warn] [pid 851] AH00045: child process 1897 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.283563 2023] [core:warn] [pid 851] AH00045: child process 1942 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.283841 2023] [core:warn] [pid 851] AH00045: child process 2113 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.284028 2023] [core:warn] [pid 851] AH00045: child process 2184 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.284225 2023] [core:warn] [pid 851] AH00045: child process 2378 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.284439 2023] [core:warn] [pid 851] AH00045: child process 2525 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.284942 2023] [core:warn] [pid 851] AH00045: child process 2560 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.285176 2023] [core:warn] [pid 851] AH00045: child process 2733 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.285376 2023] [core:warn] [pid 851] AH00045: child process 2741 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:50.285597 2023] [core:warn] [pid 851] AH00045: child process 2913 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.287418 2023] [core:warn] [pid 851] AH00045: child process 855 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.287693 2023] [core:warn] [pid 851] AH00045: child process 1309 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.287931 2023] [core:warn] [pid 851] AH00045: child process 1727 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.288131 2023] [core:warn] [pid 851] AH00045: child process 1897 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.288302 2023] [core:warn] [pid 851] AH00045: child process 1942 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.288524 2023] [core:warn] [pid 851] AH00045: child process 2113 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.288707 2023] [core:warn] [pid 851] AH00045: child process 2184 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.288883 2023] [core:warn] [pid 851] AH00045: child process 2378 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.289060 2023] [core:warn] [pid 851] AH00045: child process 2525 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.289241 2023] [core:warn] [pid 851] AH00045: child process 2560 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.289435 2023] [core:warn] [pid 851] AH00045: child process 2733 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.289620 2023] [core:warn] [pid 851] AH00045: child process 2741 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:52.289796 2023] [core:warn] [pid 851] AH00045: child process 2913 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.291420 2023] [core:warn] [pid 851] AH00045: child process 855 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.291829 2023] [core:warn] [pid 851] AH00045: child process 1309 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.292029 2023] [core:warn] [pid 851] AH00045: child process 1727 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.292203 2023] [core:warn] [pid 851] AH00045: child process 1897 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.292455 2023] [core:warn] [pid 851] AH00045: child process 1942 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.292749 2023] [core:warn] [pid 851] AH00045: child process 2113 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.292958 2023] [core:warn] [pid 851] AH00045: child process 2184 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.293132 2023] [core:warn] [pid 851] AH00045: child process 2378 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.293289 2023] [core:warn] [pid 851] AH00045: child process 2525 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.293526 2023] [core:warn] [pid 851] AH00045: child process 2560 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.293716 2023] [core:warn] [pid 851] AH00045: child process 2733 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.293885 2023] [core:warn] [pid 851] AH00045: child process 2741 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:54.294059 2023] [core:warn] [pid 851] AH00045: child process 2913 still did not exit, sending a SIGTERM
[Mon Jun 05 14:43:56.295626 2023] [core:error] [pid 851] AH00046: child process 855 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.297496 2023] [core:error] [pid 851] AH00046: child process 1309 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.298172 2023] [core:error] [pid 851] AH00046: child process 1727 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.298500 2023] [core:error] [pid 851] AH00046: child process 1897 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.298699 2023] [core:error] [pid 851] AH00046: child process 1942 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.299031 2023] [core:error] [pid 851] AH00046: child process 2113 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.299404 2023] [core:error] [pid 851] AH00046: child process 2184 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.300427 2023] [core:error] [pid 851] AH00046: child process 2378 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.300769 2023] [core:error] [pid 851] AH00046: child process 2525 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.300990 2023] [core:error] [pid 851] AH00046: child process 2560 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.301206 2023] [core:error] [pid 851] AH00046: child process 2733 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.301615 2023] [core:error] [pid 851] AH00046: child process 2741 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:56.302322 2023] [core:error] [pid 851] AH00046: child process 2913 still did not exit, sending a SIGKILL
[Mon Jun 05 14:43:57.303404 2023] [core:error] [pid 851] AH00047: could not make child process 855 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.303791 2023] [core:error] [pid 851] AH00047: could not make child process 1309 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.304102 2023] [core:error] [pid 851] AH00047: could not make child process 1727 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.304392 2023] [core:error] [pid 851] AH00047: could not make child process 1897 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.304627 2023] [core:error] [pid 851] AH00047: could not make child process 1942 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.304846 2023] [core:error] [pid 851] AH00047: could not make child process 2113 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.305063 2023] [core:error] [pid 851] AH00047: could not make child process 2184 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.305275 2023] [core:error] [pid 851] AH00047: could not make child process 2378 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.305552 2023] [core:error] [pid 851] AH00047: could not make child process 2525 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.305759 2023] [core:error] [pid 851] AH00047: could not make child process 2560 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.305911 2023] [core:error] [pid 851] AH00047: could not make child process 2733 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.306124 2023] [core:error] [pid 851] AH00047: could not make child process 2741 exit, attempting to continue anyway
[Mon Jun 05 14:43:57.307024 2023] [mpm_prefork:notice] [pid 851] AH00169: caught SIGTERM, shutting down

I have the volume data still here, if someone would need info from it.

3

I can only say the output and setup looks a little bit strange.
Inside my environment i use the docker container for dev work since two or three years without any problem.
One point is the path you showed. How does your docker command line looks like to create the container?

4

Same as the download page (including tmpfs) except:

-net br0
—mac-address=xx:xx:xx:xx:xx:xx
—ip=192.168.178.19

The thing: the previous cmk raw 2.1.0 docker, which I was testing a week before CCE was being born, worked just fine. Wanted to wait for new deployment with CCE Free and wasnt successful.

My next step is to set it up on my dev system again. The attempts above were on my prod system.

5

Got time again to test. I disabled the removed the tmpfs parameter for testing. No change…

The container comes up like this:

### CREATING SITE 'cmk'
Adding /opt/omd/sites/cmk/tmp to /etc/fstab.
Going to set TMPFS to off.
Preparing tmp directory /omd/sites/cmk/tmp...Generating configuration for core (type cmc)...

WARNING: The number of configured checkers is higher than the number of available CPUs. To avoid unnecessary context switches, the number of checkers should be limited to the number of CPUs. Recommended number of checkers: 2
Starting full compilation for all hosts Creating global helper config...OK
 Creating cmc protobuf configuration...OK
Updating core configuration...
Executing post-create script "01_create-sample-config.py"...OK
Adding /opt/omd/sites/cmk/tmp to /etc/fstab.
Going to set TMPFS to off.
Created new site cmk with version 2.2.0p4.cce.

  The site can be started with omd start cmk.
  The default web UI is available at http://faa2d4649724/cmk/

  The admin user for the web applications is cmkadmin with password: J78HHlQ1
  For command line administration of the site, log in with 'omd su cmk'.
  After logging in, you can change the password for cmkadmin with 'cmk-passwd cmkadmin'.

WARNING: You have to execute 'omd update-apache-config cmk' as root to update and apply the configuration of the system apache.
WARNING: You have to execute 'omd update-apache-config cmk' as root to update and apply the configuration of the system apache.
### STARTING XINETD
 * Starting internet superserver xinetd
   ...done.
### STARTING SITE
Preparing tmp directory /omd/sites/cmk/tmp...Starting agent-receiver...OK
Starting mkeventd...OK
Starting liveproxyd...OK
Starting mknotifyd...OK
Starting rrdcached...OK
Starting cmc...OK
Starting apache...OK
Starting dcd...OK
Starting redis...OK
Initializing Crontab...OK
### STARTING CRON
### CONTAINER STARTED

and the whole system is slow af. The cmk website does not even open and the cpu usage was around 20% per thread. I also noticed many same processes within container:

:~# docker exec -it checkmk bash
root@faa2d4649724:/# ps -eF
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root         1     0  0  1090   516   0 18:00 ?        00:00:00 /bin/bash /docker-entrypoint.sh
root       641     1  0  1477   508   0 18:04 ?        00:00:00 /usr/sbin/xinetd -pidfile /run/xinetd.pid -stayalive -inetd_compat -inetd_ipv6
cmk        681     1  0  9494 21760   0 18:04 ?        00:00:01 gunicorn: master [agent_receiver.main:main_app()]
cmk        711   681  0 14366 38216   0 18:05 ?        00:00:11 gunicorn: worker [agent_receiver.main:main_app()]
cmk        844     1  0 44646 26128   0 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/mkeventd
cmk        863     1  0  8480 17104   1 18:05 ?        00:00:02 liveproxyd[master]
cmk        885     1  0  7542 13648   0 18:05 ?        00:00:03 python3 /omd/sites/cmk/bin/mknotifyd
cmk        891     1  0 16160  3076   1 18:05 ?        00:00:00 /omd/sites/cmk/bin/rrdcached -t 4 -w 3600 -z 1800 -f 7200 -s cmk -m 660 -l unix:/omd/sites/cmk/tmp/run/rrdcached.sock -p /omd/sites/cmk/tmp/rrdcached.pid -j /omd/sites/cmk/v
cmk        913     1  0 286114 7400   0 18:05 ?        00:00:03 /omd/sites/cmk/bin/cmc /omd/sites/cmk/var/check_mk/core/config.pb
cmk        937   913  1 38624 84580   0 18:05 ?        00:00:22 python3 /omd/sites/cmk/bin/cmk --create-rrd --keepalive
cmk        939   913  1 27213 83884   0 18:05 ?        00:00:22 python3 /omd/sites/cmk/bin/cmk --handle-alerts --keepalive
cmk        943   913  0  2327   248   1 18:05 ?        00:00:00 /omd/sites/cmk/lib/cmc/checkhelper
cmk        944   913  0  2327   248   0 18:05 ?        00:00:00 /omd/sites/cmk/lib/cmc/checkhelper
cmk        945   913  0  2327   248   1 18:05 ?        00:00:00 /omd/sites/cmk/lib/cmc/checkhelper
cmk        946   913  0  2327   248   0 18:05 ?        00:00:00 /omd/sites/cmk/lib/cmc/checkhelper
cmk        948   913  0  2327   252   0 18:05 ?        00:00:00 /omd/sites/cmk/lib/cmc/checkhelper
cmk        949   913  1 28331 85768   1 18:05 ?        00:00:22 python3 /omd/sites/cmk/bin/cmk --checker
cmk        950   913  1 28328 85624   0 18:05 ?        00:00:22 python3 /omd/sites/cmk/bin/cmk --checker
cmk        951   913  1 28306 85848   0 18:05 ?        00:00:22 python3 /omd/sites/cmk/bin/cmk --checker
cmk        952   913  1 28306 85764   1 18:05 ?        00:00:23 python3 /omd/sites/cmk/bin/cmk --checker
cmk        954   913  1 27229 83880   0 18:05 ?        00:00:22 python3 /omd/sites/cmk/bin/cmk --keepalive --real-time-checks
cmk        955   913  0 11658 24196   0 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        956   913  0 11658 24192   1 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        957   913  0 11658 24192   1 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        958   913  0 11658 24188   0 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        960   913  0 11658 24192   1 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        961   913  0 11658 24192   0 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        962   913  0 11658 24192   1 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        965   913  0 11658 24196   1 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        966   913  0 11658 24192   0 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        967   913  0 11658 24192   1 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        968   913  0 11658 24192   1 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        972   913  0 11658 24140   1 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        973   913  0 11658 24192   0 18:05 ?        00:00:02 python3 /omd/sites/cmk/bin/fetcher
cmk        974   913  0 15016 35804   0 18:05 ?        00:00:04 python3 /omd/sites/cmk/bin/cmk --notify --keepalive
cmk        976   913  0  1276   108   0 18:05 ?        00:00:00 /omd/sites/cmk/lib/cmc/icmpsender 8 0 1000
cmk        978   913  0  2788   252   0 18:05 ?        00:00:00 /omd/sites/cmk/lib/cmc/icmpreceiver
cmk       1001     1  0  3711  3664   1 18:05 ?        00:00:00 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
cmk       1004  1001  0  3705  2460   1 18:05 ?        00:00:00 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
cmk       1005  1001  1 70604 193492  0 18:05 ?        00:00:39 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
cmk       1063     1  0 28304 27348   1 18:06 ?        00:00:00 dcd
cmk       1069     1  0 13761 10224   0 18:07 ?        00:00:06 /omd/sites/cmk/bin/redis-server unixsocket:/omd/sites/cmk/tmp/run/redis
root      1091     1  0   971   508   0 18:07 ?        00:00:00 cron -f
root      1189  1091  0  1569   712   0 18:08 ?        00:00:00 CRON -f
cmk       1193  1189  0  1195   492   0 18:08 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       1209  1193  0  4952  1252   1 18:08 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
root      1355  1091  0  1569   712   0 18:09 ?        00:00:00 CRON -f
cmk       1359  1355  0  1195   492   0 18:09 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       1389  1359  0  4952  1256   0 18:09 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
root      1452  1091  0  1569   712   0 18:10 ?        00:00:00 CRON -f
cmk       1458  1452  0  1195   492   1 18:10 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       1492  1458  0  4952  1260   1 18:10 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
cmk       1643  1001  1 70576 193320  1 18:10 ?        00:00:30 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
cmk       1652     1  0 70604 193196  1 18:10 ?        00:00:00 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
root      1654  1091  0  1569   712   1 18:11 ?        00:00:00 CRON -f
cmk       1657  1654  0  1195   492   0 18:11 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       1674  1657  0  4952  1256   0 18:11 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
root      1821  1091  0  1569   712   0 18:12 ?        00:00:00 CRON -f
cmk       1823  1821  0  1195   492   0 18:12 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       1841  1823  0  4952  1264   0 18:12 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
cmk       1847  1001  1 70581 193316  0 18:12 ?        00:00:29 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
root      1993  1091  0  1569   712   1 18:13 ?        00:00:00 CRON -f
cmk       1995  1993  0  1195   488   0 18:13 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       2013  1995  0  4952  1260   0 18:13 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
cmk       2019  1001  1 70576 193320  1 18:13 ?        00:00:29 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
root      2165  1091  0  1569   712   1 18:14 ?        00:00:00 CRON -f
cmk       2168  2165  0  1195   488   1 18:14 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       2185  2168  0  4952  1260   0 18:14 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
cmk       2187  1001  1 70581 193316  0 18:14 ?        00:00:30 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
root      2338  1091  0  1569   712   0 18:15 ?        00:00:00 CRON -f
cmk       2344  2338  0  1195   492   1 18:15 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       2379  2344  0  4952  1264   0 18:15 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
cmk       2385  1001  2 70576 193328  0 18:15 ?        00:00:31 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
root      2454  1091  0  1569   712   0 18:16 ?        00:00:00 CRON -f
cmk       2459  2454  0  1195   492   1 18:16 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       2484  2459  0  4952  1256   0 18:16 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
cmk       2558  1001  2 70577 193320  0 18:16 ?        00:00:29 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
root      2640  1091  0  1569   712   0 18:17 ?        00:00:00 CRON -f
cmk       2646  2640  0  1195   488   0 18:17 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       2670  2646  0  4952  1260   0 18:17 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
cmk       2730  1001  2 70581 193320  0 18:17 ?        00:00:29 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
root      2735  1091  0  1569   712   1 18:18 ?        00:00:00 CRON -f
cmk       2739  2735  0  1195   488   0 18:18 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       2755  2739  0  4952  1256   0 18:18 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
cmk       2902  1001  2 70581 193084  0 18:18 ?        00:00:30 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
root      2908  1091  0  1569   712   1 18:19 ?        00:00:00 CRON -f
cmk       2911  2908  0  1195   488   1 18:19 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       2928  2911  0  4952  1264   1 18:19 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
root      3074  1091  0  1569   712   1 18:20 ?        00:00:00 CRON -f
root      3075  1091  0  1569   712   0 18:20 ?        00:00:00 CRON -f
root      3076  1091  0  1569   712   0 18:20 ?        00:00:00 CRON -f
cmk       3078  3075  0  3253  4228   1 18:20 ?        00:00:00 python3 /omd/sites/cmk/bin/cmk --inventorize-marked-hosts
cmk       3079  3076  0  1195   488   0 18:20 ?        00:00:00 /bin/bash -c cmk --discover-marked-hosts >/dev/null 2>&1
cmk       3081  3074  0  1195   492   1 18:20 ?        00:00:00 /bin/bash -c . $OMD_ROOT/etc/omd/site.conf ; curl http://localhost:$CONFIG_APACHE_TCP_PORT/cmk/check_mk/run_cron.py >/dev/null 2>&1
cmk       3114  3079  0  3253  4228   0 18:20 ?        00:00:00 python3 /omd/sites/cmk/bin/cmk --discover-marked-hosts
cmk       3115  3081  0  4952  1244   0 18:20 ?        00:00:00 curl http://localhost:5000/cmk/check_mk/run_cron.py
cmk       3117  1001  0  4592  5344   1 18:20 ?        00:00:00 /usr/sbin/apache2 -f /omd/sites/cmk/etc/apache/apache.conf
root      3122  1091  0  1569   712   0 18:21 ?        00:00:00 CRON -f
root      3123  1091  0  1569   712   0 18:21 ?        00:00:00 CRON -f
cmk       3124  3122  0  1569   484   0 18:21 ?        00:00:00 CRON -f
cmk       3125  3123  0  1569   484   0 18:21 ?        00:00:00 CRON -f
root      3132     1  0  9139 18388   0 18:21 ?        00:00:00 /omd/versions/2.2.0p4.cce/bin/python3 /usr/bin/omd status
root      3133  1091  0  1569   712   1 18:22 ?        00:00:00 CRON -f
root      3134  1091  0  1569   712   0 18:22 ?        00:00:00 CRON -f
cmk       3135  3133  0  1569   484   1 18:22 ?        00:00:00 CRON -f
cmk       3136  3134  0  1569   484   0 18:22 ?        00:00:00 CRON -f
root      3144     1  0  9139 18396   1 18:22 ?        00:00:00 /omd/versions/2.2.0p4.cce/bin/python3 /usr/bin/omd status
root      3145  1091  0  1569   712   0 18:23 ?        00:00:00 CRON -f
root      3146  1091  0  1569   712   1 18:23 ?        00:00:00 CRON -f
cmk       3147  3145  0  1569   484   0 18:23 ?        00:00:00 CRON -f
cmk       3148  3146  0  1569   484   1 18:23 ?        00:00:00 CRON -f
root      3149     0  0  1156   832   1 18:23 pts/0    00:00:00 bash
root      3166     1  0  9139 18396   1 18:23 ?        00:00:00 /omd/versions/2.2.0p4.cce/bin/python3 /usr/bin/omd status
root      3170  1091  0  1569   712   0 18:24 ?        00:00:00 CRON -f
root      3171  1091  0  1569   712   0 18:24 ?        00:00:00 CRON -f
cmk       3172  3170  0  1569   484   0 18:24 ?        00:00:00 CRON -f
cmk       3173  3171  0  1569   484   1 18:24 ?        00:00:00 CRON -f
root      3182     1  0  9139 18388   0 18:24 ?        00:00:00 /omd/versions/2.2.0p4.cce/bin/python3 /usr/bin/omd status
root      3183  1091  0  1569   712   1 18:25 ?        00:00:00 CRON -f
root      3184  1091  0  1569   712   1 18:25 ?        00:00:00 CRON -f
root      3185  1091  0  1569   712   0 18:25 ?        00:00:00 CRON -f
root      3186  1091  0  1569   712   1 18:25 ?        00:00:00 CRON -f
cmk       3187  3183  0  1569   484   0 18:25 ?        00:00:00 CRON -f
cmk       3188  3186  0  1569   484   0 18:25 ?        00:00:00 CRON -f
cmk       3189  3184  0  1569   484   1 18:25 ?        00:00:00 CRON -f
cmk       3190  3185  0  1569   484   0 18:25 ?        00:00:00 CRON -f
root      3191  3149  0  1891  1260   1 18:25 pts/0    00:00:05 top
root      3199     1  0  9139 18400   1 18:25 ?        00:00:00 /omd/versions/2.2.0p4.cce/bin/python3 /usr/bin/omd status
root      3200  1091  0  1569   712   1 18:26 ?        00:00:00 CRON -f
root      3201  1091  0  1569   712   0 18:26 ?        00:00:00 CRON -f
cmk       3202  3200  0  1569   484   1 18:26 ?        00:00:00 CRON -f
cmk       3203  3201  0  1569   484   1 18:26 ?        00:00:00 CRON -f
root      3210     1  0  9139 18356   1 18:26 ?        00:00:00 /omd/versions/2.2.0p4.cce/bin/python3 /usr/bin/omd status
root      3211  1091  0  1569   712   1 18:27 ?        00:00:00 CRON -f
root      3212  1091  0  1569   712   1 18:27 ?        00:00:00 CRON -f
cmk       3213  3212  0  1569   484   1 18:27 ?        00:00:00 CRON -f
cmk       3214  3211  0  1569   484   1 18:27 ?        00:00:00 CRON -f
root      3215  1091  0  1569   712   0 18:28 ?        00:00:00 CRON -f
root      3216  1091  0  1569   712   0 18:28 ?        00:00:00 CRON -f
cmk       3217  3215  0  1569   484   0 18:28 ?        00:00:00 CRON -f
cmk       3218  3216  0  1569   484   0 18:28 ?        00:00:00 CRON -f
root      3219  1091  0  1569   712   0 18:29 ?        00:00:00 CRON -f
root      3220  1091  0  1569   712   1 18:29 ?        00:00:00 CRON -f
cmk       3221  3220  0  1569   484   1 18:29 ?        00:00:00 CRON -f
cmk       3222  3219  0  1569   484   1 18:29 ?        00:00:00 CRON -f
root      3223  1091  0  1569   712   0 18:30 ?        00:00:00 CRON -f
root      3224  1091  0  1569   712   1 18:30 ?        00:00:00 CRON -f
root      3225  1091  0  1569   712   1 18:30 ?        00:00:00 CRON -f
root      3226  1091  0  1569   712   1 18:30 ?        00:00:00 CRON -f
cmk       3227  3223  0  1569   484   0 18:30 ?        00:00:00 CRON -f
cmk       3228  3225  0  1569   484   1 18:30 ?        00:00:00 CRON -f
cmk       3229  3224  0  1569   484   0 18:30 ?        00:00:00 CRON -f
cmk       3230  3226  0  1569   484   0 18:30 ?        00:00:00 CRON -f
root      3231  1091  0  1569   712   1 18:31 ?        00:00:00 CRON -f
root      3232  1091  0  1569   712   0 18:31 ?        00:00:00 CRON -f
cmk       3233  3231  0  1569   484   1 18:31 ?        00:00:00 CRON -f
cmk       3234  3232  0  1569   484   1 18:31 ?        00:00:00 CRON -f
root      3235  1091  0  1569   716   1 18:32 ?        00:00:00 CRON -f
root      3236  1091  0  1569   716   1 18:32 ?        00:00:00 CRON -f
cmk       3237  3236  0  1569   484   1 18:32 ?        00:00:00 CRON -f
cmk       3238  3235  0  1569   484   1 18:32 ?        00:00:00 CRON -f
root      3239  1091  0  1569   716   1 18:33 ?        00:00:00 CRON -f
root      3240  1091  0  1569   716   0 18:33 ?        00:00:00 CRON -f
cmk       3241  3239  0  1569   484   1 18:33 ?        00:00:00 CRON -f
cmk       3242  3240  0  1569   484   0 18:33 ?        00:00:00 CRON -f
root      3243     0  0  1156   760   1 18:33 pts/1    00:00:00 bash
root      3251     0  0  1156   804   1 18:33 pts/2    00:00:00 bash
root      3260  1091  0  1569   716   1 18:34 ?        00:00:00 CRON -f
root      3261  1091  0  1569   716   1 18:34 ?        00:00:00 CRON -f
cmk       3262  3261  0  1569   484   1 18:34 ?        00:00:00 CRON -f
cmk       3263  3260  0  1569   484   1 18:34 ?        00:00:00 CRON -f
root      3265  3251  0  1156   580   0 18:34 pts/2    00:00:00 bash
root      3266  1091  0  1569   716   1 18:35 ?        00:00:00 CRON -f
root      3267  1091  0  1569   716   0 18:35 ?        00:00:00 CRON -f
root      3268  1091  0  1569   716   0 18:35 ?        00:00:00 CRON -f
root      3269  1091  0  1569   716   0 18:35 ?        00:00:00 CRON -f
cmk       3270  3269  0  1569   484   0 18:35 ?        00:00:00 CRON -f
cmk       3271  3266  0  1569   484   0 18:35 ?        00:00:00 CRON -f
cmk       3272  3267  0  1569   484   0 18:35 ?        00:00:00 CRON -f
cmk       3273  3268  0  1569   484   0 18:35 ?        00:00:00 CRON -f
root      3274  1091  0  1569   716   1 18:36 ?        00:00:00 CRON -f
root      3275  1091  0  1569   716   1 18:36 ?        00:00:00 CRON -f
cmk       3276  3275  0  1569   484   1 18:36 ?        00:00:00 CRON -f
cmk       3277  3274  0  1569   484   1 18:36 ?        00:00:00 CRON -f
root      3278  1091  0  1569   832   0 18:37 ?        00:00:00 CRON -f
root      3279  1091  0  1569   840   1 18:37 ?        00:00:00 CRON -f
cmk       3280  3279  0  1569   484   0 18:37 ?        00:00:00 CRON -f
cmk       3281  3278  0  1569   484   1 18:37 ?        00:00:00 CRON -f
root      3282  1091  0  1569   864   1 18:38 ?        00:00:00 CRON -f
root      3283  1091  0  1569   864   0 18:38 ?        00:00:00 CRON -f
cmk       3284  3282  0  1569   484   1 18:38 ?        00:00:00 CRON -f
cmk       3285  3283  0  1569   484   1 18:38 ?        00:00:00 CRON -f
root      3287  1091  0  1569   852   1 18:39 ?        00:00:00 CRON -f
root      3288  1091  0  1569   856   1 18:39 ?        00:00:00 CRON -f
cmk       3289  3287  0  1569   484   1 18:39 ?        00:00:00 CRON -f
cmk       3291  3288  0  1569   484   0 18:39 ?        00:00:00 CRON -f
root      3332     0  0  1156  2624   0 18:39 pts/3    00:00:00 bash
root      3340  1091  0  1569  3368   1 18:40 ?        00:00:00 CRON -f
root      3341  1091  0  1569  3368   1 18:40 ?        00:00:00 CRON -f
root      3342  1091  0  1569  3340   1 18:40 ?        00:00:00 CRON -f
root      3343  1091  0  1569  3336   1 18:40 ?        00:00:00 CRON -f
cmk       3344  3341  0  1569   484   1 18:40 ?        00:00:00 CRON -f
cmk       3345  3343  0  1569   484   1 18:40 ?        00:00:00 CRON -f
cmk       3346  3342  0  1569   484   1 18:40 ?        00:00:00 CRON -f
cmk       3347  3340  0  1569   484   0 18:40 ?        00:00:00 CRON -f

Any attemp to read from /omd is very slow. I dont know what is going on.

As already noted: the previous docker versions (2.1.x) were working.

Docker also showing cmk as Up 47 minutes (unhealthy).

docker inspect - if this helps:

:~# docker inspect checkmk
[
    {
        "Id": "faa2d464972469bf4f8922d33045712cc62d2390503107b106bb2d9f014e6de5",
        "Created": "2023-07-04T16:00:51.506752429Z",
        "Path": "/docker-entrypoint.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 24899,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-07-04T16:00:54.617445009Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 39,
                "Log": [
                    {
                        "Start": "2023-07-04T18:43:35.184465387+02:00",
                        "End": "2023-07-04T18:43:40.741751603+02:00",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    },
                    {
                        "Start": "2023-07-04T18:44:43.724190762+02:00",
                        "End": "2023-07-04T18:44:48.922808115+02:00",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    },
                    {
                        "Start": "2023-07-04T18:45:51.81907558+02:00",
                        "End": "2023-07-04T18:45:58.998049001+02:00",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    },
                    {
                        "Start": "2023-07-04T18:47:12.419566162+02:00",
                        "End": "2023-07-04T18:47:12.684159746+02:00",
                        "ExitCode": -1,
                        "Output": "OCI runtime exec failed: exec failed: unable to start container process: error executing setns process: exit status 1: unknown"
                    },
                    {
                        "Start": "2023-07-04T18:48:12.87013021+02:00",
                        "End": "2023-07-04T18:48:12.98163281+02:00",
                        "ExitCode": -1,
                        "Output": "OCI runtime exec failed: exec failed: unable to start container process: error executing setns process: exit status 1: unknown"
                    }
                ]
            }
        },
        "Image": "sha256:08187f5a9ba5aae91db8b1a9545e1b0e6ef0cc3e1af1f386d02f8f001023bb62",
        "ResolvConfPath": "/var/lib/docker/containers/faa2d464972469bf4f8922d33045712cc62d2390503107b106bb2d9f014e6de5/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/faa2d464972469bf4f8922d33045712cc62d2390503107b106bb2d9f014e6de5/hostname",
        "HostsPath": "/var/lib/docker/containers/faa2d464972469bf4f8922d33045712cc62d2390503107b106bb2d9f014e6de5/hosts",
        "LogPath": "/var/lib/docker/containers/faa2d464972469bf4f8922d33045712cc62d2390503107b106bb2d9f014e6de5/faa2d464972469bf4f8922d33045712cc62d2390503107b106bb2d9f014e6de5-json.log",
        "Name": "/checkmk",
        "RestartCount": 0,
        "Driver": "btrfs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": [
            "14b8245a04dff8711c1031e8cdf7eaa0236e18208893408fc28ab3f5e3b17cb7"
        ],
        "HostConfig": {
            "Binds": [
                "/mnt/user/appdata/cmk:/omd/sites:rw",
                "/etc/localtime:/etc/localtime:ro"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "1",
                    "max-size": "50m"
                }
            },
            "NetworkMode": "br0",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": null,
            "Name": "btrfs"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/mnt/user/appdata/cmk",
                "Destination": "/omd/sites",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/localtime",
                "Destination": "/etc/localtime",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "faa2d4649724",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "5000/tcp": {},
                "6557/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "TZ=Europe/Berlin",
                "HOST_OS=Unraid",
                "HOST_HOSTNAME=Playtime",
                "HOST_CONTAINERNAME=checkmk",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "CMK_SITE_ID=cmk",
                "CMK_LIVESTATUS_TCP=",
                "CMK_PASSWORD=",
                "MAIL_RELAY_HOST=",
                "CMK_CONTAINERIZED=TRUE"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "omd status || exit 1"
                ],
                "Interval": 60000000000,
                "Timeout": 5000000000
            },
            "Image": "checkmk/check-mk-cloud:2.2.0p4",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "maintainer": "feedback@checkmk.com",
                "net.unraid.docker.managed": "dockerman",
                "org.opencontainers.image.description": "Checkmk is a leading tool for Infrastructure & Application Monitoring",
                "org.opencontainers.image.source": "https://github.com/checkmk/checkmk",
                "org.opencontainers.image.title": "Checkmk",
                "org.opencontainers.image.url": "https://checkmk.com/",
                "org.opencontainers.image.vendor": "Checkmk GmbH",
                "org.opencontainers.image.version": "2.2.0p4"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "bbd801e7323107ae29910bdf6908c6a6126e0b87be72d7e78e9102f291e82887",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/bbd801e73231",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "br0": {
                    "IPAMConfig": {
                        "IPv4Address": "192.168.178.17"
                    },
                    "Links": null,
                    "Aliases": [
                        "faa2d4649724"
                    ],
                    "NetworkID": "4a5317686d0d84b8a1d267ed444a874967ca0be40ff4f6e8629ee3685f011822",
                    "EndpointID": "dd772f63dd73256a043a965d95d244fa0e78a29eccad662138629e0e66da573e",
                    "Gateway": "192.168.178.1",
                    "IPAddress": "192.168.178.17",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "xxxxx",
                    "GlobalIPv6Address": "xxxxxx",
                    "GlobalIPv6PrefixLen": 64,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]
6

I now tried the 2.2 raw and 2.1 raw. The 2.2 has the exact same issue. The 2.1 works better, but does not immediately freeze things up. But I can always see, that the apache process is using 52% CPU for the most time (1st setup, container is up, web UI is not accessed, nothing is configured). I wonder what the apache is doing.

The cron -F commands are growing slowly. The curl commands as well.

Yes, my test system here is not that powerful, but what the hell is CMK doing with NO configs/hosts??

CMK does not accept stopping the container in ALL tests:

Removing Crontab...OK
Stopping redis...killing 717...OK

is all the docker logs shows.

Apache:

[Wed Jul 05 16:34:54.383099 2023] [core:warn] [pid 706] AH00045: child process 710 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:54.383848 2023] [core:warn] [pid 706] AH00045: child process 866 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:54.384360 2023] [core:warn] [pid 706] AH00045: child process 1725 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:54.384802 2023] [core:warn] [pid 706] AH00045: child process 1746 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:56.387129 2023] [core:warn] [pid 706] AH00045: child process 710 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:56.387749 2023] [core:warn] [pid 706] AH00045: child process 866 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:56.388160 2023] [core:warn] [pid 706] AH00045: child process 1725 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:56.388508 2023] [core:warn] [pid 706] AH00045: child process 1746 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:58.390087 2023] [core:warn] [pid 706] AH00045: child process 710 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:58.390486 2023] [core:warn] [pid 706] AH00045: child process 866 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:58.390717 2023] [core:warn] [pid 706] AH00045: child process 1725 still did not exit, sending a SIGTERM
[Wed Jul 05 16:34:58.390937 2023] [core:warn] [pid 706] AH00045: child process 1746 still did not exit, sending a SIGTERM

The Apache access_log since setup:

- - - [05/Jul/2023:16:24:02 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [05/Jul/2023:16:25:02 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [05/Jul/2023:16:26:02 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [05/Jul/2023:16:27:02 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [05/Jul/2023:16:28:02 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [05/Jul/2023:16:29:03 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 3 "-" "curl/7.81.0"
- - - [05/Jul/2023:16:31:42 +0200] "GET / HTTP/1.1" 302 215 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
- - - [05/Jul/2023:16:31:42 +0200] "GET /cmk/ HTTP/1.1" 302 224 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
- - - [05/Jul/2023:16:31:42 +0200] "GET /cmk/check_mk/ HTTP/1.1" 302 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
- - - [05/Jul/2023:16:31:43 +0200] "GET /cmk/check_mk/login.py?_origtarget=index.py HTTP/1.1" 200 1998 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
- - - [05/Jul/2023:16:31:43 +0200] "GET /cmk/check_mk/themes/modern-dark/theme-2.1.0p30.css HTTP/1.1" 200 194728 "http://192.168.178.17:5000/cmk/check_mk/login.py?_origtarget=index.py" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
- - - [05/Jul/2023:16:31:58 +0200] "GET /cmk/check_mk/js/main_min-2.1.0p30.js HTTP/1.1" 200 1681089 "http://192.168.178.17:5000/cmk/check_mk/login.py?_origtarget=index.py" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
- - - [05/Jul/2023:16:31:58 +0200] "GET /cmk/check_mk/themes/modern-dark/images/favicon.ico HTTP/1.1" 200 7323 "http://192.168.178.17:5000/cmk/check_mk/login.py?_origtarget=index.py" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
- - - [05/Jul/2023:16:32:13 +0200] "GET /cmk/check_mk/themes/facelift/images/checkmk_logo.svg HTTP/1.1" 200 2742 "http://192.168.178.17:5000/cmk/check_mk/login.py?_origtarget=index.py" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0"
- - - [05/Jul/2023:16:33:02 +0200] "GET /cmk/check_mk/run_cron.py HTTP/1.1" 200 32 "-" "curl/7.81.0"
7

This must be a problem on your environment. If the Apache process cannot spawn his child processes then the slow and unresponsive GUI is “normal”.
Happens this also on another Docker host?
How does the memory consumption of your container looks like?

8

How do you know it has that issue?

9

it was too early :slight_smile:

exit != exists

But on the other hand it looks like an Apache problem as your core is running and only the GUI is not working.
In your first post you have many curl’s from the cron job against the web interface. That is also an indicator for myself that there is an problem with your webinterface only.

10

I believe my NAS has not enough CPU power to solve all requests from Apache and the requests build up and keep it busy. this does not happen on better system.

Well, I believe I have to move cmk to another system. But CMK should detect, that a cron job is already running.

But what is CMK doing on a non-set up state? The cronjob should be done quickly.

11

I have the exact same problem. It seems like the apache2 processes freezes. They can’t be killed, only a reboot works.
I use docker compose with traefik as reverse proxy.

version: '2.4'

services:
  checkmk:
    image: checkmk/check-mk-raw:2.2.0-latest
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./data/sites:/omd/sites
    tmpfs:
      - /opt/omd/sites/cmk/tmp:mode=770,uid=1000,gid=1000
    networks:
      - traefik-net
      - default
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.checkmk.loadbalancer.server.port=5000"
      - "traefik.http.routers.checkmk-http.entrypoints=http"
      - "traefik.http.routers.checkmk-http.rule=Host(`checkmk.internal`)"
      - "traefik.http.routers.checkmk-http.middlewares=https-redirect@file"
      - "traefik.http.routers.checkmk-https.entrypoints=https"
      - "traefik.http.routers.checkmk-https.rule=Host(`checkmk.internal`)"
      - "traefik.http.routers.checkmk-https.middlewares=websocket-header@file"

networks:
  traefik-net:
    external: true
    name: traefik-net
  default:
    enable_ipv6: true
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.20.13.0/24
        - subnet: fd20:13::/64
12

What are the ulimit settings for the container? If they are extremely high, some Checkmk processes try to close phantasy file handles which slows everything down. As a rule of thumb, set the ulimit to thrice the service count to start with.

13

I tested ulimits (soft and hard) 1024 and 65535, both with the same result.
After login, the container freezes and I need to hard-reset the host machine.

docker-compose.yml ulimits:

    ulimits:
      nofile:
        soft: "1024"
        hard: "1024"

ulimit:

root@d60b37222723:/# ulimit -Sa
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 127637
max locked memory           (kbytes, -l) unlimited
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 127637
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
root@d60b37222723:/# ulimit -Ha
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 127637
max locked memory           (kbytes, -l) unlimited
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 127637
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
14

I think I have been able to solve the problem for myself.

I use Unraid OS as host OS and there on a so-called “cache pool” (= not on the array). I use two NVMe SSDs in RAID 1 for this.
By default Unraid mounts the file system/disks with the help of shfs, which can cause problems (I have only experienced this with Checkmk so far).
With the pools, however, you still have the option of configuring them as “Exclusive Share”, which skips shfs.
Now it works for me.