Bug: CMC stopped working suddenly

StefanM · July 27, 2026, 6:02am

Sunday morning the CMC terminated it’s work suddenly and stopped sites are the result:

Update to 2.5.0p9 was performed on thursday.

Last entries from CMC.log are:

Livestatus heartbeats need a lot higher timeout now (increased to 16s/8s from 6s/3s). Otherwise the central site will flood the liveproxyd.log with failed (heartbeat) connections.

andreas-doehler · July 27, 2026, 7:52am

All sites stopped and only crontab + Apache running looks like an OOM kill somewhere in a site process and then systemd stopped all other processes.
This is the picture i had on some smaller systems if i got an OOM kill from a site process.

StefanM · July 27, 2026, 8:01am

CPU (load+utilization): <1, 2%

RAM: 2-4GB free RAM +12GB free swap

Disks: <80%

Resources look good. The first processes which die are normally the apache processes.

No killed processes in the logs/console:

The CMC is terminating it’s processes (not killing!) and then the site goes down.

andreas-doehler · July 27, 2026, 12:48pm

That’s correct with 2.5 Apache needs way more memory. One Apache that dies will cause all the other site processes to stop.
As all sites are started with systemd and one unit, systemd sees this one process dying and stops all other processes that are started with this unit (omd.service) in a normal way, that’s why the log looks like a normal shutdown.
Only log entry you will see is the system log on the machine no where else i think.

→ OOMPolicy

StefanM · July 27, 2026, 12:53pm

Sorry, no killed (apache) processes.

andreas-doehler · July 27, 2026, 1:37pm

And what says the journalctl at the time all the sites stopped?

StefanM · July 27, 2026, 2:55pm

Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 321996 (gunicorn: maste) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322236 (gunicorn: worke) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322289 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322649 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322667 (rrdcached) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322673 (redis-server) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322715 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322727 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322728 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322729 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322765 (cmk-ui-job-sche) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 322780 (cmc) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323034 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323036 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323040 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323041 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323042 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323043 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323044 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323045 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323046 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323047 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323048 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323049 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323050 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323051 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323052 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323054 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323055 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323056 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323057 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323058 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323059 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323060 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323061 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323062 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323063 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323064 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323065 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323066 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323067 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323068 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323069 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323070 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323071 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323072 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323073 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323074 (icmpsender) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323075 (icmpreceiver) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323201 (dcd) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323225 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323252 (stunnel) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323264 (xinetd) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 323280 (clickhouse) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 424422 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 3540689 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 3628347 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 3717395 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 4002337 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-31991.scope: Killing process 4130568 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433019 (gunicorn: maste) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433034 (gunicorn: worke) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433038 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433057 (liveproxyd[mast) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433073 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433077 (rrdcached) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433088 (redis-server) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433113 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433125 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433126 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433127 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433161 (cmk-ui-job-sche) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433193 (cmc) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433217 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433219 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433223 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433224 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433225 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433226 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433227 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433228 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433229 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433230 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433231 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433232 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433233 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433234 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433235 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433236 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433237 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433238 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433239 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433240 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433241 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433242 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433243 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433244 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433245 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433246 (icmpsender) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433247 (icmpreceiver) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433282 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433314 (dcd) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433365 (gunicorn: maste) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433380 (gunicorn: worke) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433382 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433391 (liveproxyd[mast) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433406 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433410 (rrdcached) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433421 (redis-server) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433447 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433459 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433460 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433461 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433484 (cmk-ui-job-sche) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433500 (cmc) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433524 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433526 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433530 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433531 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433532 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433533 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433534 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433535 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433536 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433537 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433538 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433539 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433540 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433541 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433542 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433543 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433544 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433545 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433546 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433547 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433548 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433549 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433550 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433551 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433552 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433553 (icmpsender) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433554 (icmpreceiver) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433613 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 433666 (dcd) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447066 (gunicorn: maste) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447084 (gunicorn: worke) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447086 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447095 (liveproxyd[mast) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447112 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447117 (rrdcached) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447123 (redis-server) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447153 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447165 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447166 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447167 (cmk-automation-) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447189 (cmk-ui-job-sche) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447207 (cmc) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447231 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447233 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447237 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447238 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447239 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447240 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447241 (checkhelper) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447242 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447243 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447244 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447245 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447246 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447247 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447248 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447249 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447250 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447251 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447252 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447253 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447254 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447255 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447256 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447257 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447259 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447260 (python3) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447261 (icmpsender) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447262 (icmpreceiver) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447297 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 447326 (dcd) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 542403 (otelcol) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 3540652 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 3540653 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 3540655 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 3540690 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 3540691 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 3540703 (apache2) with signal SIGTERM.
Jul 26 06:47:11 pdvmscheckmk01 systemd[1]: session-72.scope: Killing process 3540813 (apache2) with signal SIGTERM.

4 sites in total.

CPU:

Memory:

andreas-doehler · July 28, 2026, 5:18am

Is it possible that these sites where started with “omd start sitename” or “omd start” inside an SSH session and not with the systemd service? I have a similar problem if i start a OMD site inside a Proxmox LXC container, if i use the Proxmox shell inside the web GUI.
The moment i ended the shell session the site stops, but only if i started the site there manually.

StefanM · July 28, 2026, 5:47am

This could be the reason, when I look at the artifacts on that host:

I’m not sure if this file in /etc/init.d is doing right. How and when should Checkmk write the systemd service files? During “dpkg -i”? Or can it be fixed afterwards?

Last time “omd.service” said something:

Last time “check-mk-managed-2.1.0p1.service” said something:

When somebody tells me how to fix the services, I will start the instances with the service and look if the sites fail again.

andreas-doehler · July 28, 2026, 9:31am

Inside a up to date CMK installation you only need the “omd.service” nothing else.
The init.d service is a “little bit old” and from an old CMK version.
If your site is now started with the omd.service such an “auto stop” should not happen, only if you get the oom problem.