VM CPU always on 95%

Troubleshooting
,
1

VM Specs: 1 vCPU, 12 Cores Per vCPU, 16GB RAM
OS version: Rocky Linux 9.3
CMK version: Checkmk Raw Edition 2.2.0p17 (About to update to p21)
Checkmk Hosts: 414
Checkmk Services: 39.000

Hello everyone,
we have a relativly new Checkmk instance that is not optimized in its the last bits.
But one thing bothers me quite some time now.
I tried to get some hints for our high CPU usage but I could not find anything that helps.

  1. Apache number of processes and Livestatus usage:
    Do we need to take action here?

    checkmk
    checkmk2199×496 38.3 KB

  2. The service “Check_MK” has some high Perf-O-Meter stats:
    Around 10 over 20 seconds
    Around 30 under 5 seconds
    Everything else inbetween

  3. Here is glimps of top -c: (ESET Sercurity cannot be removed)

top - 07:32:47 up 1 day, 19:31,  1 user,  load average: 50.16, 44.88, 44.28
Tasks: 437 total,  18 running, 419 sleeping,   0 stopped,   0 zombie
%Cpu(s): 81.6 us, 12.7 sy,  0.1 ni,  3.2 id,  1.2 wa,  0.9 hi,  0.4 si,  0.0 st
MiB Mem :  15669.4 total,   2900.0 free,   6167.5 used,   7345.4 buff/cache
MiB Swap:   8012.0 total,   7147.4 free,    864.6 used.   9501.8 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   8418 eset-ef+  20   0 2211484 546356  20964 S  25.1   3.4 610:49.93 /opt/eset/efs/lib/scand
3491124 MYSITE    20   0 1544876 103892   6612 S  21.5   0.6  97:19.90 /omd/sites/MYSITE/bin/nagios -ud /omd/sites/MYSITE/tmp/nagios/nagios.cfg
4174316 MYSITE    20   0   62964  56748  13072 S  20.8   0.4   0:00.66 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST1
4174296 MYSITE    20   0   63072  56968  13064 S  20.1   0.4   0:00.69 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST2
4174309 MYSITE    20   0   63092  56960  13056 S  19.8   0.4   0:00.62 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST3
4174280 MYSITE    20   0   64084  57412  13044 S  19.5   0.4   0:00.70 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST4
4174311 MYSITE    20   0   63932  57180  13060 S  19.1   0.4   0:00.62 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST5
4174320 MYSITE    20   0   64404  57444  13044 S  19.1   0.4   0:00.58 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST6
4174301 MYSITE    20   0   66412  59768  13076 S  18.2   0.4   0:00.63 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST7
4174278 MYSITE    20   0   63968  57568  13196 S  17.8   0.4   0:00.67 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST8
4174298 MYSITE    20   0   62952  56688  13024 S  17.8   0.4   0:00.63 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST9
4174304 MYSITE    20   0   63928  56844  13036 S  17.8   0.4   0:00.67 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST0
4174300 MYSITE    20   0   64108  56756  13120 S  16.2   0.4   0:00.56 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST11
4174851 MYSITE    20   0   56288  50988  12912 R  15.8   0.3   0:00.48 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST12
4174275 MYSITE    20   0   64056  57080  13032 S  15.5   0.4   0:00.63 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST13
4174286 MYSITE    20   0   64144  57192  13076 S  15.5   0.4   0:00.66 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST14
4174293 MYSITE    20   0   64124  57180  13040 S  15.2   0.4   0:00.60 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST15
4174288 MYSITE    20   0   64180  57744  13136 D  14.2   0.4   0:00.60 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST16
4174822 MYSITE    20   0   46608  40748  12904 R  14.2   0.3   0:00.43 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST17
4174854 MYSITE    20   0   53252  47824  12840 R  13.5   0.3   0:00.41 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST18
4174874 MYSITE    20   0   48276  42640  12932 R  12.9   0.3   0:00.39 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST19
4174846 MYSITE    20   0   48276  42820  12848 R  12.5   0.3   0:00.38 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST20
4174859 MYSITE    20   0   48404  42840  12868 R  12.5   0.3   0:00.38 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST21
4174272 MYSITE    20   0   64812  58368  13140 S  12.2   0.4   0:00.56 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST22
4173780 MYSITE    20   0   26736  14132   8732 D  11.2   0.1   0:00.84 /usr/bin/perl /omd/sites/MYSITE/lib/pnp4nagios/process_perfdata.pl -n -c /omd/sites/MYSITE/etc/pnp4nagios/process_perfdata.cfg -b /omd/sites/MYSITE/var/pnp4nagios/spool//perfdata.1706682734
4174875 MYSITE    20   0   46812  41492  12848 R  11.2   0.3   0:00.34 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST23
4174884 MYSITE    20   0   46548  40412  12820 R  10.9   0.3   0:00.33 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST24
4174883 MYSITE    20   0   44084  38584  12840 R  10.6   0.2   0:00.32 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST25
4173781 MYSITE    20   0   26752  14432   8932 D   9.6   0.1   0:00.84 /usr/bin/perl /omd/sites/MYSITE/lib/pnp4nagios/process_perfdata.pl -n -c /omd/sites/MYSITE/etc/pnp4nagios/process_perfdata.cfg -b /omd/sites/MYSITE/var/pnp4nagios/spool//perfdata.1706682749
    117 root       0 -20       0      0      0 I   9.2   0.0 140:38.84 [kworker/0:1H-kblockd]
   4793 MYSITE    20   0  844316 303044   3000 S   6.9   1.9  73:10.23 /omd/sites/MYSITE/bin/rrdcached -t 4 -w 3600 -z 1800 -f 7200 -s MYSITE -m 660 -l unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock -p /omd/sites/MYSITE/tmp/rrdcached.pid -j /omd/sites/MYSITE/var/rrdcached -o /omd/sites/MYSITE/var/log/r+
4162665 MYSITE    20   0   67556  61592  13048 R   5.0   0.4   0:00.83 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST26
   8419 root       0 -20 1123392   6392   4796 S   4.0   0.0 101:17.15 /opt/eset/efs/lib/oaeventd
1196481 MYSITE    20   0  492176 239644  24364 S   4.0   1.5   1:53.97 /usr/sbin/httpd -f /omd/sites/MYSITE/etc/apache/apache.conf
1341813 MYSITE    20   0  498392 249092  24340 S   4.0   1.6   1:49.53 /usr/sbin/httpd -f /omd/sites/MYSITE/etc/apache/apache.conf
3710694 MYSITE    20   0  493284 245928  26560 S   4.0   1.5   1:15.82 /usr/sbin/httpd -f /omd/sites/MYSITE/etc/apache/apache.conf
4169898 MYSITE    20   0   69656  63248  13080 D   3.3   0.4   0:00.69 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST27
3825922 MYSITE    20   0  457316 211320  27012 S   2.6   1.3   0:09.81 /usr/sbin/httpd -f /omd/sites/MYSITE/etc/apache/apache.conf
3392714 root      20   0       0      0      0 D   1.3   0.0   0:02.50 [kworker/u384:5+flush-253:0]
4174113 MYSITE    20   0   64380  57968  13076 D   1.3   0.4   0:00.62 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST28
4174110 MYSITE    20   0   64136  57280  13144 S   1.0   0.4   0:00.60 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST29
4174118 MYSITE    20   0   64136  57500  13092 S   1.0   0.4   0:00.63 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST30
   4038 MYSITE    20   0  277480  46964   9804 S   0.7   0.3  10:57.07 gunicorn: worker [agent_receiver.main:main_app()]
4157230 admin-c+  20   0  226612   5020   3564 R   0.7   0.0   0:00.43 top -c
4174162 MYSITE    20   0   64112  57184  13076 S   0.7   0.4   0:00.58 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST31
4174165 MYSITE    20   0   64396  57744  13076 S   0.7   0.4   0:00.59 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST32
4174951 MYSITE    20   0    9096   6572   5684 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.2.1.2.2.1.8
4174963 MYSITE    20   0    9096   6624   5732 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.2.1.31.1.1.1.6
4174971 MYSITE    20   0    8976   6204   5416 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.2.1.31.1.1.1.7
4174975 MYSITE    20   0    8976   6156   5364 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.2.1.31.1.1.1.6
4174976 MYSITE    20   0    9096   6584   5692 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.4.1.9.9.91.1.1.1.1.2
4174977 MYSITE    20   0    8976   6152   5360 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a sha -u USERNAME -A             -x AES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.2.1.31.1.1.1.12
4174978 MYSITE    20   0    9096   6596   5708 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.4.1.9.9.91.1.2.1.1.4
4174979 MYSITE    20   0    8976   6152   5360 R   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.4.1.9.9.91.1.1.1.1.3
4174981 MYSITE    20   0    8976   6156   5360 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.2.1.31.1.1.1.15
4174982 MYSITE    20   0    8976   6180   5388 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.4.1.9.9.91.1.2.1.1.3
4174983 MYSITE    20   0    8976   6156   5364 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.4.1.9.9.91.1.1.1.1.3
4174986 MYSITE    20   0    9096   6136   5348 S   0.7   0.0   0:00.02 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.2.1.31.1.1.1.7
    461 root       0 -20       0      0      0 I   0.3   0.0   2:50.12 [kworker/2:1H-kblockd]
   2211 wazuh     20   0  246012   7500   2240 S   0.3   0.0  18:25.86 /var/ossec/bin/wazuh-agentd
   8438 eset-ef+  20   0 1918416  23248   8820 S   0.3   0.1   8:27.91 /opt/eset/efs/lib/wapd
4144619 admin-c+  20   0  222652   3764   3412 S   0.3   0.0   0:00.14 bash -c while true; do sleep 1;head -v -n 8 /proc/meminfo; head -v -n 2 /proc/stat /proc/version /proc/uptime /proc/loadavg /proc/sys/fs/file-nr /proc/sys/kernel/hostname; tail -v -n 16 /proc/net/dev;echo '==> /proc/df <==';df -l;e+
4169917 MYSITE    20   0   64688  57812  13120 S   0.3   0.4   0:00.59 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST33
4169936 MYSITE    20   0   64064  57456  13060 D   0.3   0.4   0:00.66 /omd/sites/MYSITE/bin/python3 /omd/sites/MYSITE/var/check_mk/core/helper_config/latest/host_checks/MYHOST34
4174974 MYSITE    20   0    9096   6604   5720 S   0.3   0.0   0:00.01 snmpbulkwalk -Cr10 -v3 -l authPriv -a sha -u USERNAME -A             -x AES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.2.1.2.2.1.1
4174984 MYSITE    20   0    8976   6148   5360 S   0.3   0.0   0:00.01 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.4.1.9.9.91.1.2.1.1.4
4174987 MYSITE    20   0    8976   6136   5344 R   0.3   0.0   0:00.01 snmpbulkwalk -Cr10 -v3 -l authPriv -a md5 -u USERNAME -A             -x DES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.2.1.31.1.1.1.18
4174988 MYSITE    20   0    8976   6132   5344 R   0.3   0.0   0:00.01 snmpbulkwalk -Cr10 -v3 -l authPriv -a sha -u USERNAME -A             -x AES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.4.1.12356.101.12.2.3.1.6
4174989 MYSITE    20   0    8976   6180   5388 R   0.3   0.0   0:00.01 snmpbulkwalk -Cr10 -v3 -l authPriv -a sha -u USERNAME -A             -x AES -X             -m  -M  -t 2.00 -r 5 -Cc -OQ -OU -On -Ot IPADRESS .1.3.6.1.4.1.9.9.91.1.1.1.1.3
      1 root      20   0  173420  15548  10756 S   0.0   0.1  30:16.16 /usr/lib/systemd/systemd rhgb --switched-root --system --deserialize 31
      2 root      20   0       0      0      0 S   0.0   0.0   0:01.14 [kthreadd]
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [rcu_gp]
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [rcu_par_gp]
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [slub_flushwq]
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [netns]
     10 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [mm_percpu_wq]
     12 root      20   0       0      0      0 I   0.0   0.0   0:00.00 [rcu_tasks_kthre]
     13 root      20   0       0      0      0 I   0.0   0.0   0:00.00 [rcu_tasks_rude_]
     14 root      20   0       0      0      0 I   0.0   0.0   0:00.00 [rcu_tasks_trace]
     15 root      20   0       0      0      0 S   0.0   0.0   0:27.37 [ksoftirqd/0]
     16 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [pr/tty0]

It would be great to get this fixed.
If additional information are needed I’m happy to provide more.

Thanks in advance.

2

Is that ESET process an on access virus scanner?

You need more CPUs in that system.

1 Like
3

Hello @r.sander ,
yes ESET is some kind of virus, malware protection. Need to run.

I’ve changed the VM to 16 vCPU and 1 Cores Per vCPU but that did not change anything.
There are always too many Python3 processes that run for each host.

4

Have you removed the rule in “Normal check interval for service checks” that configures the 24 hours interval for the “Check_MK HW/SW inventory” service?

5

Yes we’ve changed that already to standard.