CMK version: 2.0.0p9 (CRE)
OS version: Windows Server 2016
Error message:
If the checkmk service is running, cpu will go to 100% and slows down the system. See the screen shot. Several powershell processes will be running. If I stop checkmk, the cpu usage will go down and the powershell process will disappear. I have this level of cmk agent agent running on serveral windows systems but only one windows has this problem. I have just a standard installation of the cmk agent with only some adoptions for the plugin config. But again the same config run also on the other systems.
Now I played with the cmk agent config and disabled nearly all the plugins and all the further plugin config (like winperf, ps ,…). I still see the same situation also with this mini config.
The logfile give no hints about a long running process or an error.
How can I investigate what in detail is running in the powershell processes?
How can I investigate further the reasin for this slowdown?
regards Robert
Output of “cmk --debug -vvn hostname”:
Checkmk version 2.0.0p9 Try license usage history update. Trying to acquire lock on /omd/sites/inseo/var/check_mk/license_usage/next_run Got lock on /omd/sites/inseo/var/check_mk/license_usage/next_run Trying to acquire lock on /omd/sites/inseo/var/check_mk/license_usage/history.json Got lock on /omd/sites/inseo/var/check_mk/license_usage/history.json Next run time has not been reached yet. Abort. Releasing lock on /omd/sites/inseo/var/check_mk/license_usage/history.json Released lock on /omd/sites/inseo/var/check_mk/license_usage/history.json Releasing lock on /omd/sites/inseo/var/check_mk/license_usage/next_run Released lock on /omd/sites/inseo/var/check_mk/license_usage/next_run Loading autochecks from /omd/sites/inseo/var/check_mk/autochecks/INS-DC31.mk + FETCHING DATA Source: SourceType.HOST/FetcherType.TCP [cpu_tracking] Start [7ff47445bb20] Connecting via TCP to 10.10.34.211:6556 (50.0s timeout) [TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(base_path=PosixPath('/omd/sites/inseo/tmp/check_mk/cache/INS-DC31'), max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False) Not using cache (Too old. Age is 14 sec, allowed is 0 sec) [TCPFetcher] Execute data source Reading data from agent Output is not encrypted Write data to cache file /omd/sites/inseo/tmp/check_mk/cache/INS-DC31 Trying to acquire lock on /omd/sites/inseo/tmp/check_mk/cache/INS-DC31 Got lock on /omd/sites/inseo/tmp/check_mk/cache/INS-DC31 Releasing lock on /omd/sites/inseo/tmp/check_mk/cache/INS-DC31 Released lock on /omd/sites/inseo/tmp/check_mk/cache/INS-DC31 Closing TCP connection to 10.10.34.211:6556 [cpu_tracking] Stop [7ff47445bb20 - Snapshot(process=posix.times_result(user=0.010000000000000231, system=0.0, children_user=0.0, children_system=0.0, elapsed=5.949999999254942))] Source: SourceType.HOST/FetcherType.PIGGYBACK [cpu_tracking] Start [7ff4744b5130] No piggyback files for 'INS-DC31'. Skip processing. No piggyback files for '10.10.34.211'. Skip processing. [PiggybackFetcher] Fetch with cache settings: NoCache(base_path=PosixPath('/omd/sites/inseo/tmp/check_mk/data_source_cache/piggyback/INS-DC31'), max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False) [PiggybackFetcher] Execute data source [cpu_tracking] Stop [7ff4744b5130 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))] [cpu_tracking] Start [7ff47445bfa0] + PARSE FETCHER RESULTS Source: SourceType.HOST/FetcherType.TCP No persisted sections loaded -> Add sections: ['check_mk', 'df', 'local', 'mem', 'systemtime', 'uptime', 'win_license', 'windows_updates'] Source: SourceType.HOST/FetcherType.PIGGYBACK No persisted sections loaded -> Add sections: [] Received no piggyback data Loading item states Trying to acquire lock on /omd/sites/inseo/tmp/check_mk/counters/INS-DC31 Got lock on /omd/sites/inseo/tmp/check_mk/counters/INS-DC31 Releasing lock on /omd/sites/inseo/tmp/check_mk/counters/INS-DC31 Released lock on /omd/sites/inseo/tmp/check_mk/counters/INS-DC31 Filesystem C:/ 69.71% used (34.85 of 50.00 GB), trend: +75.62 MB / 24 hours INS_Bkup-Win:Info (V2.1) Skipped! No Event Log available INS_Bkup-Win:LastRun (V2.1) Skipped! Windows Backup Services not installed INS_Bkup-Win:List (V2.1) Skipped! No Event Log available INS_Bkup-Win:Status (V2.1) skipped! No Event Log available INS_Bkup-Win:Versions (V2.1) Skipped! Windows Backup Services not installed INS_Info:ComputerInfo (V2.1) System Inventory \ INS_SW:List (V2.1) Installed Software\ Memory RAM: 53.79% - 2.15 GB of 4.00 GB, Commit charge: 27.88% - 2.06 GB of 7.37 GB System Time Offset: -870 ms System Updates 1 important updates, (Security Intelligence-Update für Microsoft Defender Antivirus - KB2267602 (Version 1.361.1422.0)) --- , 0 optional updates Uptime Up since Mar 19 2022 19:47:14, Uptime: 17 days 18 hours + EXECUTING INVENTORY PLUGINS hp_proliant_mem: skipped (no data) hp_proliant_mem: skipped (no data) ipmi_firmware: skipped (no data) ipmi_firmware: skipped (no data) inventory_checkmk: skipped (no data) inventory_checkmk: skipped (no data) inv_cisco_vlans: skipped (no data) inv_cisco_vlans: skipped (no data) inv_esx_vsphere_hostsystem: skipped (no data) inv_esx_vsphere_hostsystem: skipped (no data) inv_if: ok inv_if: skipped (no data) inventory_oracle_tablespaces: skipped (no data) inventory_oracle_tablespaces: skipped (no data) dmidecode: skipped (no data) dmidecode: skipped (no data) docker_node_network: skipped (no data) docker_node_network: skipped (no data) inventory_esx_vsphere_clusters: skipped (no data) inventory_esx_vsphere_clusters: skipped (no data) inventory_esx_vsphere_virtual_machines: skipped (no data) inventory_esx_vsphere_virtual_machines: skipped (no data) k8s_endpoint_info: skipped (no data) k8s_endpoint_info: skipped (no data) k8s_ingress_infos: skipped (no data) k8s_ingress_infos: skipped (no data) juniper_info: skipped (no data) juniper_info: skipped (no data) mem: ok mem: skipped (no data) mem_used: skipped (no data) mem_used: skipped (no data) snmp_info: skipped (no data) snmp_info: skipped (no data) aix_baselevel: skipped (no data) aix_baselevel: skipped (no data) aix_lparstat_inventory: skipped (no data) aix_lparstat_inventory: skipped (no data) aix_packages: skipped (no data) aix_packages: skipped (no data) aix_service_packs: skipped (no data) aix_service_packs: skipped (no data) allnet_ip_sensoric: skipped (no data) allnet_ip_sensoric: skipped (no data) aruba_wlc_aps: skipped (no data) aruba_wlc_aps: skipped (no data) check_mk: ok check_mk: skipped (no data) checkpoint_inv_tunnels: skipped (no data) checkpoint_inv_tunnels: skipped (no data) citrix_controller: skipped (no data) citrix_controller: skipped (no data) citrix_state: skipped (no data) citrix_state: skipped (no data) couchbase_nodes_ports: skipped (no data) couchbase_nodes_ports: skipped (no data) dell_hw_info: skipped (no data) dell_hw_info: skipped (no data) docker_container_labels: skipped (no data) docker_container_labels: skipped (no data) docker_container_network: skipped (no data) docker_container_network: skipped (no data) docker_container_node_name: skipped (no data) docker_container_node_name: skipped (no data) docker_node_images: skipped (no data) docker_node_images: skipped (no data) docker_node_info: skipped (no data) docker_node_info: skipped (no data) esx_systeminfo: skipped (no data) esx_systeminfo: skipped (no data) fireeye_sys_status: skipped (no data) fireeye_sys_status: skipped (no data) fritz: skipped (no data) fritz: skipped (no data) hp_proliant_da_phydrv: skipped (no data) hp_proliant_da_phydrv: skipped (no data) hp_proliant_systeminfo: skipped (no data) hp_proliant_systeminfo: skipped (no data) ibm_mq_channels: skipped (no data) ibm_mq_channels: skipped (no data) ibm_mq_managers: skipped (no data) ibm_mq_managers: skipped (no data) ibm_mq_queues: skipped (no data) ibm_mq_queues: skipped (no data) infoblox_osinfo: skipped (no data) infoblox_osinfo: skipped (no data) infoblox_systeminfo: skipped (no data) infoblox_systeminfo: skipped (no data) k8s_assigned_pods: skipped (no data) k8s_assigned_pods: skipped (no data) k8s_daemon_pod_containers: skipped (no data) k8s_daemon_pod_containers: skipped (no data) k8s_job_container: skipped (no data) k8s_job_container: skipped (no data) k8s_nodes: skipped (no data) k8s_nodes: skipped (no data) k8s_pod_container: skipped (no data) k8s_pod_container: skipped (no data) k8s_pod_info: skipped (no data) k8s_pod_info: skipped (no data) k8s_roles: skipped (no data) k8s_roles: skipped (no data) k8s_selector: skipped (no data) k8s_selector: skipped (no data) k8s_service_info: skipped (no data) k8s_service_info: skipped (no data) lnx_block_devices: skipped (no data) lnx_block_devices: skipped (no data) lnx_cpuinfo: skipped (no data) lnx_cpuinfo: skipped (no data) lnx_distro: skipped (no data) lnx_distro: skipped (no data) lnx_if: skipped (no data) lnx_if: skipped (no data) lnx_ip_r: skipped (no data) lnx_ip_r: skipped (no data) lnx_packages: skipped (no data) lnx_packages: skipped (no data) lnx_sysctl: skipped (no data) lnx_sysctl: skipped (no data) lnx_uname: skipped (no data) lnx_uname: skipped (no data) lnx_video: skipped (no data) lnx_video: skipped (no data) lparstat_aix: skipped (no data) lparstat_aix: skipped (no data) mssql_clusters: skipped (no data) mssql_clusters: skipped (no data) mssql_versions: skipped (no data) mssql_versions: skipped (no data) netapp_api_disk: skipped (no data) netapp_api_disk: skipped (no data) netapp_api_info: skipped (no data) netapp_api_info: skipped (no data) oracle_dataguard_stats: skipped (no data) oracle_dataguard_stats: skipped (no data) oracle_instance: skipped (no data) oracle_instance: skipped (no data) oracle_performance: skipped (no data) oracle_performance: skipped (no data) oracle_recovery_area: skipped (no data) oracle_recovery_area: skipped (no data) oracle_systemparameter: skipped (no data) oracle_systemparameter: skipped (no data) perle_chassis: skipped (no data) perle_chassis: skipped (no data) perle_chassis_slots: skipped (no data) perle_chassis_slots: skipped (no data) perle_psmu: skipped (no data) perle_psmu: skipped (no data) prtconf: skipped (no data) prtconf: skipped (no data) snmp_extended_info: skipped (no data) snmp_extended_info: skipped (no data) snmp_os: skipped (no data) snmp_os: skipped (no data) snmp_quantum_storage_info: skipped (no data) snmp_quantum_storage_info: skipped (no data) solaris_addresses: skipped (no data) solaris_addresses: skipped (no data) solaris_pkginfo: skipped (no data) solaris_pkginfo: skipped (no data) solaris_prtdiag: skipped (no data) solaris_prtdiag: skipped (no data) solaris_prtpicl: skipped (no data) solaris_prtpicl: skipped (no data) solaris_psrinfo: skipped (no data) solaris_psrinfo: skipped (no data) solaris_routes: skipped (no data) solaris_routes: skipped (no data) solaris_uname: skipped (no data) solaris_uname: skipped (no data) statgrab_net: skipped (no data) statgrab_net: skipped (no data) suseconnect: skipped (no data) suseconnect: skipped (no data) win_bios: skipped (no data) win_bios: skipped (no data) win_computersystem: skipped (no data) win_computersystem: skipped (no data) win_cpuinfo: skipped (no data) win_cpuinfo: skipped (no data) win_disks: skipped (no data) win_disks: skipped (no data) win_exefiles: skipped (no data) win_exefiles: skipped (no data) win_ip_r: skipped (no data) win_ip_r: skipped (no data) win_networkadapter: skipped (no data) win_networkadapter: skipped (no data) win_os: skipped (no data) win_os: skipped (no data) win_reg_uninstall: skipped (no data) win_reg_uninstall: skipped (no data) win_system: skipped (no data) win_system: skipped (no data) win_video: skipped (no data) win_video: skipped (no data) win_wmi_software: skipped (no data) win_wmi_software: skipped (no data) win_wmi_updates: skipped (no data) win_wmi_updates: skipped (no data) winperf_if: skipped (no data) winperf_if: skipped (no data) No piggyback files for 'INS-DC31'. Skip processing. No piggyback files for '10.10.34.211'. Skip processing. [cpu_tracking] Stop [7ff47445bfa0 - Snapshot(process=posix.times_result(user=0.06000000000000005, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.07000000029802322))] [agent] Version: 2.0.0p9, OS: windows, Missing monitoring data for check plugins: dotnet_clrmemory, logwatch, services_summary, winperf_processor_util, wmi_cpuload(!), execution time 6.0 sec | execution_time=6.020 user_time=0.070 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=5.940
CMK Logfile:
2022-04-06 15:04:02.537 [srv 6096] [Trace] Enabled Base 2022-04-06 15:04:02.541 [srv 6096] [Trace] Setting root. service: 'CheckMkService', preset: '' 2022-04-06 15:04:02.541 [srv 6096] [Trace] Try service: 'CheckMkService' 2022-04-06 15:04:02.541 [srv 6096] [Trace] Try registry 'CheckMkService' 2022-04-06 15:04:02.542 [srv 6096] [Trace] Service is found 'C:\Program Files (x86)\checkmk\service\check_mk_agent.exe' 2022-04-06 15:04:02.542 [srv 6096] Set root 'C:\Program Files (x86)\checkmk\service' from registry 'CheckMkService' 2022-04-06 15:04:02.542 [srv 6096] [Trace] Try registry 'CheckMkService' 2022-04-06 15:04:02.543 [srv 6096] Protect file from User write 'C:\Program Files (x86)\checkmk\service\check_mk_agent.exe' 2022-04-06 15:04:02.543 [srv 6096] Protect path from User access 'C:\Program Files (x86)\checkmk\service' 2022-04-06 15:04:02.544 [srv 6096] 'data' Starting executing commands [5] 2022-04-06 15:04:02.602 [srv 6096] Process is started 'C:\Windows\TEMP\cmk_data_6096_2.cmd' with pid [6500] 2022-04-06 15:04:02.602 [srv 6096] [Trace] Using root = 'C:\Program Files (x86)\checkmk\service' and data = 'C:\ProgramData\checkmk\agent' folders 2022-04-06 15:04:02.616 [srv 6096] COM Initialized 2022-04-06 15:04:02.619 [srv 6096] COM initialized 2022-04-06 15:04:02.659 [srv 6096] Found root config on path C:\Program Files (x86)\checkmk\service\check_mk.yml 2022-04-06 15:04:02.743 [srv 6096] [Trace] Enabled Base 2022-04-06 15:04:02.757 [srv 6096] Loaded start config 'C:\Program Files (x86)\checkmk\service\check_mk.yml','C:\ProgramData\checkmk\agent\bakery','C:\ProgramData\checkmk\agent\check_mk.user.yml' 2022-04-06 15:04:02.758 [srv 6096] Firewall mode is set to configure, adding rule... 2022-04-06 15:04:03.492 [srv 6096] Firewall rule 'Checkmk Agent' had been added successfully for ports [6556] 2022-04-06 15:04:03.514 [srv 6096] Service Main 2022-04-06 15:04:03.522 [srv 6096] Service handlers registered 2022-04-06 15:04:03.524 [srv 6096] [Trace] Installing cap file 'C:\Program Files (x86)\checkmk\service\install\plugins.cap' 2022-04-06 15:04:03.524 [srv 6096] [Trace] Installing of CAP file is not required 2022-04-06 15:04:03.525 [srv 6096] [Trace] Installing yml file 'C:\Program Files (x86)\checkmk\service\install\check_mk.install.yml' 2022-04-06 15:04:03.525 [srv 6096] Reinstalling 'C:\ProgramData\checkmk\agent\install\check_mk.install.yml' with 'C:\Program Files (x86)\checkmk\service\install\check_mk.install.yml' 2022-04-06 15:04:03.525 [srv 6096] This Option/YML installation form MSI is ENABLED 2022-04-06 15:04:03.526 [srv 6096] Remove 'C:\ProgramData\checkmk\agent\install\check_mk.install.yml' [OK] 2022-04-06 15:04:03.533 [srv 6096] Supplied yaml 'C:\Program Files (x86)\checkmk\service\install\check_mk.install.yml' will not be installed 2022-04-06 15:04:03.534 [srv 6096] [Trace] Copy file 'C:\Program Files (x86)\checkmk\service\install\checkmk.dat' to 'C:\ProgramData\checkmk\agent\install\checkmk.dat' 2022-04-06 15:04:03.536 [srv 6096] [Trace] Copy is not required, the file is already exists 2022-04-06 15:04:03.537 [srv 6096] Starting upgrade(migration) process... 2022-04-06 15:04:03.537 [srv 6096] Protocol File at 'C:\ProgramData\checkmk\agent\config' exists, upgrade(migration) not required 2022-04-06 15:04:03.538 [srv 6096] Found root config on path C:\Program Files (x86)\checkmk\service\check_mk.yml 2022-04-06 15:04:03.547 [srv 6096] [Trace] Enabled Base 2022-04-06 15:04:03.548 [srv 6096] Loaded start config 'C:\Program Files (x86)\checkmk\service\check_mk.yml','C:\ProgramData\checkmk\agent\bakery','C:\ProgramData\checkmk\agent\check_mk.user.yml' 2022-04-06 15:04:03.548 [srv 6096] [Trace] Successful start of thread 2022-04-06 15:04:03.562 [srv 6096] The network is available 2022-04-06 15:04:03.568 [srv 6096] Reading module config normal 2022-04-06 15:04:03.569 [srv 6096] Finding modules 2022-04-06 15:04:03.569 [srv 6096] Installing modules 2022-04-06 15:04:03.570 [srv 6096] Pre Start actions 2022-04-06 15:04:03.570 [srv 6096] [Trace] 'openhardwaremonitor' is skipped by config 2022-04-06 15:04:03.625 [srv 6096] Pre Start actions ended 2022-04-06 15:04:10.485 [srv 6096] main Wait Loop 2022-04-06 15:04:10.521 [srv 6096] Starting IO ipv6:false, used port:6556 2022-04-06 15:04:10.522 [srv 6096] Applying config auto restart_on_crash:true error_mode: log 2022-04-06 15:04:24.295 [srv 6096] [Trace] 'openhardwaremonitor' is skipped by config 2022-04-06 15:04:24.303 [srv 6096] [Trace] 'wmi_cpuload' is skipped by config 2022-04-06 15:04:24.314 [srv 6096] [Trace] 'dotnet_clrmemory' is skipped by config 2022-04-06 15:04:24.314 [srv 6096] [Trace] 'wmi_webservices' is skipped by config 2022-04-06 15:04:24.315 [srv 6096] [Trace] 'msexch' is skipped by config 2022-04-06 15:04:24.315 [srv 6096] [Trace] 'openhardwaremonitor' is skipped by config 2022-04-06 15:04:24.318 [srv 6096] [Trace] To start [3] sync plugins 2022-04-06 15:04:24.338 [srv 6096] [Trace] To start [6] sync plugins 2022-04-06 15:04:31.514 [srv 6096] Send [3476] bytes of data 2022-04-06 15:04:31.514 [srv 6096] [Trace] 'openhardwaremonitor' is skipped by config 2022-04-06 15:04:31.530 [srv 6096] [Trace] 'wmi_cpuload' is skipped by config 2022-04-06 15:04:31.553 [srv 6096] [Trace] 'dotnet_clrmemory' is skipped by config 2022-04-06 15:04:31.553 [srv 6096] [Trace] 'wmi_webservices' is skipped by config 2022-04-06 15:04:31.553 [srv 6096] [Trace] 'msexch' is skipped by config 2022-04-06 15:04:31.554 [srv 6096] [Trace] 'openhardwaremonitor' is skipped by config 2022-04-06 15:04:31.561 [srv 6096] [Trace] To start [3] sync plugins 2022-04-06 15:04:31.592 [srv 6096] [Trace] To start [6] sync plugins 2022-04-06 15:04:39.836 [srv 6096] Send [3476] bytes of data 2022-04-06 15:04:39.836 [srv 6096] [Trace] 'openhardwaremonitor' is skipped by config 2022-04-06 15:04:39.854 [srv 6096] [Trace] 'wmi_cpuload' is skipped by config 2022-04-06 15:04:39.865 [srv 6096] [Trace] 'dotnet_clrmemory' is skipped by config 2022-04-06 15:04:39.866 [srv 6096] [Trace] 'wmi_webservices' is skipped by config 2022-04-06 15:04:39.866 [srv 6096] [Trace] 'msexch' is skipped by config 2022-04-06 15:04:39.866 [srv 6096] [Trace] 'openhardwaremonitor' is skipped by config 2022-04-06 15:04:39.885 [srv 6096] [Trace] To start [3] sync plugins 2022-04-06 15:04:39.886 [srv 6096] [Trace] To start [6] sync plugins 2022-04-06 15:04:47.842 [srv 6096] Send [3476] bytes of data
