Changing services of integrated hosts freezes

f.goessner · October 11, 2024, 9:17am

CMK version:2.2.0
OS version:Debian 12

Output of “cmk --debug -vvn hostname”:

Checkmk version 2.2.0p22
+ FETCHING DATA
  Source: SourceInfo(hostname='s1s-backup01.domico.local', ipaddress='192.168.9.98', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fb255a0d0]
Read from cache: AgentFileCache(s1s-backup01.domico.local, path_template=/omd/sites/domico/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 21 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 192.168.9.98:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b'16')
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Closing TCP connection to 192.168.9.98:6556
Write data to cache file /omd/sites/domico/tmp/check_mk/cache/s1s-backup01.domico.local
Trying to acquire lock on /omd/sites/domico/tmp/check_mk/cache/s1s-backup01.domico.local
Got lock on /omd/sites/domico/tmp/check_mk/cache/s1s-backup01.domico.local
Releasing lock on /omd/sites/domico/tmp/check_mk/cache/s1s-backup01.domico.local
Released lock on /omd/sites/domico/tmp/check_mk/cache/s1s-backup01.domico.local
[cpu_tracking] Stop [7fb255a0d0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=1.3399999998509884))]
  Source: SourceInfo(hostname='s1s-backup01.domico.local', ipaddress='192.168.9.98', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fb26bc290]
Read from cache: NoCache(s1s-backup01.domico.local, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for 's1s-backup01.domico.local'. Skip processing.
No piggyback files for '192.168.9.98'. Skip processing.
[cpu_tracking] Stop [7fb26bc290 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<wmi_cpuload:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<fileinfo:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<winperf_phydisk>>> / Transition HostSectionParser -> HostSectionParser
<<<winperf_if>>> / Transition HostSectionParser -> HostSectionParser
<<<winperf_processor>>> / Transition HostSectionParser -> HostSectionParser
<<<df:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
<<<logwatch>>> / Transition HostSectionParser -> HostSectionParser
<<<services>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_win:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<dotnet_clrmemory:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<ps:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
Transition HostSectionParser -> NOOPParser
Transition NOOPParser -> NOOPParser
<<<systemtime>>> / Transition NOOPParser -> HostSectionParser
  HostKey(hostname='s1s-backup01.domico.local', source_type=<SourceType.HOST: 1>)  -> Add sections: ['check_mk', 'checkmk_agent_plugins_win', 'cmk_agent_ctl_status', 'df', 'dotnet_clrmemory', 'fileinfo', 'logwatch', 'mem', 'ps', 'services', 'systemtime', 'uptime', 'winperf_if', 'winperf_phydisk', 'winperf_processor', 'wmi_cpuload']
  HostKey(hostname='s1s-backup01.domico.local', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
[cpu_tracking] Start [7fb18bca10]
value store: synchronizing
Trying to acquire lock on /omd/sites/domico/tmp/check_mk/counters/s1s-backup01.domico.local
Got lock on /omd/sites/domico/tmp/check_mk/counters/s1s-backup01.domico.local
value store: loading from disk
Releasing lock on /omd/sites/domico/tmp/check_mk/counters/s1s-backup01.domico.local
Released lock on /omd/sites/domico/tmp/check_mk/counters/s1s-backup01.domico.local
CPU utilization      Total CPU: 9.46%
Check_MK Agent       Version: 2.2.0p22, OS: windows, Agent plugins: 0, Local checks: 0
Disk IO SUMMARY      Read: 108 kB/s, Write: 86.5 MB/s, Latency: 3 milliseconds
DotNet Memory Management _Global_ Time in GC: 0.87%
Filesystem C:/       Used: 82.60% - 769 GiB of 931 GiB (warn/crit at 80.00%/90.00% used)(!), trend per 1 day 0 hours: -5.59 MiB, trend per 1 day 0 hours: -0.00%
Filesystem R:/       Used: 85.90% - 25.0 TiB of 29.1 TiB (warn/crit at 80.00%/90.00% used)(!), trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0%
Filesystem S:/       Used: 58.61% - 17.1 TiB of 29.1 TiB, trend per 1 day 0 hours: +987 MiB, trend per 1 day 0 hours: +<0.01%, Time left until disk full: 35 years 30 days
Interface 1          [HP Ethernet 1Gb 4-port 331FLR Adapter], (Connected), Speed: 1 GBit/s, In: 373 B/s (<0.01%), Out: 80.9 kB/s (0.06%)
Interface 2          [HP Ethernet 1Gb 4-port 331FLR Adapter 2], (Connected), Speed: 1 GBit/s, In: 87.9 MB/s (70.35%), Out: 84.1 kB/s (0.07%)
Interface 3          [HP Ethernet 1Gb 4-port 331FLR Adapter 3], (Connected), Speed: 1 GBit/s, In: 462 B/s (<0.01%), Out: 82.6 kB/s (0.07%)
Interface 4          [HP Ethernet 1Gb 4-port 331FLR Adapter 4], (Connected), Speed: 1 GBit/s, In: 32.5 kB/s (0.03%), Out: 85.3 kB/s (0.07%)
Interface 5          [isatap.{F962F483-5E97-43B8-8B35-96802BFB399B}], (Connected), Speed: 100 kBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Log Application      478 WARN, 4 CRIT messages (Last worst: "Oct 11 10:35:58 0.513 Microsoft-Windows-CAPI2 Fehler beim Kryptografiedienst während der Verarbeitung des "OnIdentity()"-Aufrufobjekts "System Writer". Details: AddLegacyDriverFiles: Unable to back up image of binary Microsoft-Verbindungsschichterkennungsprotokoll. System Error: Zugriff verweigert .")
Log HardwareEvents   No error messages
Log Internet Explorer No error messages
Log Key Management Service No error messages
Log Security         No error messages
Log System           No error messages
Log Windows PowerShell No error messages
Memory               RAM: 13.73% - 8.78 GiB of 64.0 GiB, Commit charge: 11.70% - 8.60 GiB of 73.5 GiB
Processor Queue      15 min load: 0.00, 15 min load per core: 0.00 (32 logical cores)
Service Summary      Autostart services: 72, Stopped services: 4
System Time          Offset: -692 milliseconds
Uptime               Up since Aug 07 2024 16:36:34, Uptime: 64 days 17 hours
No piggyback files for 's1s-backup01.domico.local'. Skip processing.
No piggyback files for '192.168.9.98'. Skip processing.
[cpu_tracking] Stop [7fb18bca10 - Snapshot(process=posix.times_result(user=0.020000000000000018, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.019999999552965164))]
[agent] Success, [piggyback] Success (but no data found for this host), execution time 1.4 sec | execution_time=1.360 user_time=0.030 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=1.330

Hello community,

if i try to change the services of hosts (monitored → unmonitored) the gui freezes like this;

It happen with all agent hosts (linux, windows) snmp hosts are not affected. any help would be appreciated!

MarcS · October 11, 2024, 1:22pm

We also once had the problem that the Discovery got stuck in the “Updating…” process.
I assume that the normal check for the discovered services is working?

In our case, it was the WAF that blocked a request. However, no error message or timeout was displayed in the CheckMK GUI.

Try the following:
Go to the Service Discovery page.
Open the Developer Tools in your browser and there the network analysis.
Then start the discovery and look at the answers, maybe something is blocking your request.
In our case, we saw the WAF block page.

Kind regards,
Marc