Hello,
CMK version: 2.1.0p12 EE
OS version: Linux (RHEL 8.6)
Agent version: 2.1.0p19, OS: linux, TLS is not activated on monitored host (see details), Agent plugins: 2, Local checks: 1 (We use ssh to monitor our servers.)
Error message: [agent] Success, Missing monitoring data for plugins: local, logins, logwatch, mknotifyd, omd_apache WARN, execution time 1.7 sec
Output of “cmk --debug -vvn hostname”:
Checkmk version 2.1.0p19
Try license usage history update.
Trying to acquire lock on /omd/sites/icinga/var/check_mk/license_usage/next_run
Got lock on /omd/sites/icinga/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/icinga/var/check_mk/license_usage/history.json
Got lock on /omd/sites/icinga/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/icinga/var/check_mk/license_usage/history.json
Released lock on /omd/sites/icinga/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/icinga/var/check_mk/license_usage/next_run
Released lock on /omd/sites/icinga/var/check_mk/license_usage/next_run
+ FETCHING DATA
Source: SourceType.HOST/FetcherType.PROGRAM
[cpu_tracking] Start [7fae59db8820]
[ProgramFetcher] Fetch with cache settings: DefaultAgentFileCache(host.domain.com, base_path=/omd/sites/icinga/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 37 sec, allowed is 0 sec)
[ProgramFetcher] Execute data source
Calling: ssh -i $OMD_ROOT/.ssh/id_rsa -T -o StrictHostKeyChecking=no checkmk@XXX.XXX.XXX.XXX
Write data to cache file /omd/sites/icinga/tmp/check_mk/cache/host.domain.com
Trying to acquire lock on /omd/sites/icinga/tmp/check_mk/cache/host.domain.com
Got lock on /omd/sites/icinga/tmp/check_mk/cache/host.domain.com
Releasing lock on /omd/sites/icinga/tmp/check_mk/cache/host.domain.com
Released lock on /omd/sites/icinga/tmp/check_mk/cache/host.domain.com
[cpu_tracking] Stop [7fae59db8820 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.01, children_system=0.0, elapsed=1.3599999994039536))]
Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7fae59db8ac0]
[PiggybackFetcher] Fetch with cache settings: NoCache(host.domain.com, base_path=/omd/sites/icinga/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'host.domain.com'. Skip processing.
No piggyback files for 'XXX.XXX.XXX.XX'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7fae59db8ac0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
Source: SourceType.HOST/FetcherType.PROGRAM
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<fileinfo:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_status:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_ssl_certs:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<mkeventd_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cmk_site_statistics:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1673863244,120)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_status:cached(1673863259,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<mknotifyd:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_apache:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_info:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<logins>>> / Transition HostSectionParser -> HostSectionParser
<<<logwatch>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
-> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'chrony', 'cifsmounts', 'cmk_agent_ctl_status', 'cmk_site_statistics', 'cpu', 'df', 'diskstat', 'fileinfo', 'job', 'kernel', 'labels', 'livestatus_ssl_certs', 'livestatus_status', 'lnx_if', 'local', 'logins', 'logwatch', 'md', 'mem', 'mkeventd_status', 'mknotifyd', 'mounts', 'nfsmounts', 'omd_apache', 'omd_info', 'omd_status', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
-> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7fae59deb640]
value store: synchronizing
Trying to acquire lock on /omd/sites/icinga/tmp/check_mk/counters/host.domain.com
Got lock on /omd/sites/icinga/tmp/check_mk/counters/host.domain.com
value store: loading from disk
Releasing lock on /omd/sites/icinga/tmp/check_mk/counters/host.domain.com
Released lock on /omd/sites/icinga/tmp/check_mk/counters/host.domain.com
CPU load 15 min load: 2.65, 15 min load per core: 0.44 (6 cores)
CPU utilization Total CPU: 28.68%
Check_MK Agent Version: 2.1.0p19, OS: linux, TLS is not activated on monitored host (see details), Agent plugins: 2, Local checks: 1
Disk IO SUMMARY Read: 37.1 kB/s, Write: 2.35 MB/s, Latency: 273 microseconds
File /etc/httpd/conf.d/welcome.conf.rpmnew Size: 574 B, Age: 1 year 81 days
File /etc/rhsm/rhsm.conf.rpmnew Size: 3,121 B, Age: 277 days 18 hours
File group checkmk_backup Count: 15, Size: 132,471,310,100 B, Largest size: 8,845,344,034 B, Smallest size: 8,812,794,269 B, Oldest age: 14 days 11 hours, Newest age: 11 hours 39 minutes
File group rpmnew Count: 2, Size: 3,695 B, Largest size: 3,121 B, Smallest size: 574 B, Oldest age: 1 year 81 days, Newest age: 277 days 18 hours
File group rsyslog_files Count: 0, Size: 0 B
File group scom_helper Count: 1, Size: 6,195 B, Largest size: 6,195 B, Smallest size: 6,195 B, Oldest age: 39 seconds, Newest age: 39 seconds
Filesystem / 30.08% used (3.01 of 9.99 GB), trend: +449.89 kB / 24 hours
Filesystem /boot 67.42% used (341.57 of 506.66 MB), trend: 0.00 B / 24 hours
Filesystem /home 2.28% used (93.11 MB of 3.99 GB), trend: +29.20 kB / 24 hours
Filesystem /opt/omd 51.55% used (51.01 of 98.95 GB), trend: +554.35 MB / 24 hours
Filesystem /tmp 10.12% used (413.34 MB of 3.99 GB), trend: -119.99 B / 24 hours
Filesystem /var 34.2% used (3.42 of 9.99 GB), trend: -70.77 MB / 24 hours
Filesystem /var/opt/carbonblack 12.5% used (638.74 MB of 4.99 GB), trend: +4.92 MB / 24 hours
Interface 2 [ens192], (up), MAC: 00:50:56:BA:DD:0C, Speed: 10 GBit/s, In: 674 kB/s (0.05%), Out: 65.2 kB/s (<0.01%)
Interface 3 [ens224], (up), MAC: 00:50:56:BA:09:67, Speed: 10 GBit/s, In: 272 B/s (<0.01%), Out: 236 B/s (<0.01%)
Kernel Performance Process Creations: 126.05/s, Context Switches: 33298.41/s, Major Page Faults: 9.07/s, Page Swap in: 0.00/s, Page Swap Out: 0.00/s
Log /opt/omd/sites/icinga/var/log/scom_helper.log No error messages
Logins On system: 1
Memory Total virtual memory: 23.21% - 6.34 GB of 27.33 GB, 9 additional details available
Mount options of / Mount options exactly as expected
Mount options of /boot Mount options exactly as expected
Mount options of /home Mount options exactly as expected
Mount options of /opt/omd Mount options exactly as expected
Mount options of /tmp Mount options exactly as expected
Mount options of /var Mount options exactly as expected
Mount options of /var/opt/carbonblack Mount options exactly as expected
NFS mount /var/backup 60.13% used (124.01 of 206.25 GB), trend: -954.07 MB / 24 hours
NFS_usage_/var/backup mount: /var/backup, 61% used ( 125G / 207G )
NTP Time Offset: 0.0002 ms, Stratum: 3, Time since last sync: 1 minute 32 seconds
Number of threads 824, Usage: 0.43%
OMD icinga Event Console Current events: 0, Virtual memory: 205.47 MB, Overall event limit inactive, No hosts event limit active, No rules event limit active, Received messages: 0.00/s, Rule hits: 0.00/s, Rule tries: 0.00/s, Message drops: 0.00/s, Created events: 0.00/s, Client connects: 0.09/s, Rule hit ratio: -, Processing time per message: -, Time per client request: 0.38 ms
OMD icinga Notification Spooler Version: 2.1.0p19, Spooler running
OMD icinga apache No activity since last check
OMD icinga performance Livestatus version: 2.1.0p19, Host checks: 42.8/s, Service checks: 382.7/s
OMD icinga status running
Postfix Queue Deferred queue length: 0, Active queue length: 0
Postfix status Status: the Postfix mail system is running, PID: 1674
Process Carbon Black Processes: 2, virtual: 783 MiB, physical: 113 MiB, CPU: 1.88%, Youngest running for: 79 days 17 hours, Oldest running for: 79 days 17 hours
Process crond Processes: 1, virtual: 36.2 MiB, physical: 3.22 MiB, CPU: 0%, Running for: 79 days 17 hours
Process firewalld Processes: 1, virtual: 298 MiB, physical: 40.8 MiB, CPU: 0%, Running for: 79 days 17 hours
Process goferd Processes: 1, virtual: 937 MiB, physical: 58.3 MiB, CPU: 0%, 15 min average: 0.03%, Running for: 5 hours 16 minutes
Process rhsmcertd Processes: 1, virtual: 40.8 MiB, physical: 1.77 MiB, CPU: 0%, 15 min average: 0%, Running for: 79 days 17 hours
Process rsyslogd Processes: 1, virtual: 830 MiB, physical: 148 MiB, CPU: 0%, 15 min average: <0.01%, Running for: 79 days 17 hours
Process sshd Processes: 1, virtual: 90.2 MiB, physical: 6.39 MiB, CPU: 0%, Running for: 79 days 17 hours
Process sssd Processes: 9, virtual: 1.90 GiB, physical: 169 MiB, CPU: 0.38%, 15 min average: 0.85%, Youngest running for: 2 days 16 hours, Oldest running for: 79 days 17 hours
Process sssd_kcm Processes: 1, virtual: 226 MiB, physical: 65.5 MiB, CPU: 0%, Running for: 79 days 17 hours
Process testssl.sh Processes: 0
Process vmtoolsd Processes: 1, virtual: 364 MiB, physical: 10.9 MiB, CPU: 0%, 15 min average: 0.01%, Running for: 79 days 17 hours
Site icinga statistics Total hosts: 349, Problem hosts: 8, Total services: 27398, Problem services: 3103
Systemd Service Summary Total: 145, Disabled: 14, Failed: 0
TCP Connections Established: 12
Uptime Up since Oct 28 2022 18:19:27, Uptime: 79 days 17 hours
No piggyback files for 'host.domain.com'. Skip processing.
No piggyback files for 'XXX.XXX.XXX.XXX'. Skip processing.
[cpu_tracking] Stop [7fae59deb640 - Snapshot(process=posix.times_result(user=0.040000000000000036, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.04000000096857548))]
[agent] Success, execution time 1.4 sec | execution_time=1.400 user_time=0.040 system_time=0.000 children_user_time=0.010 children_system_time=0.000 cmk_time_ds=1.350 cmk_time_agent=0.000
Since we updated our agents from 2.0.0p28 to 2.1.x, we’re getting ‘Missing monitoring data for plugins’ alerts from all our servers. Sometimes once a day per server, sometimes multiple times a day per server. No regular intervals or times.
I’ve read every topic on the forum about this issue, but most are related to wmi and/or snmp.
Updating the plugins to the same version as the agent doesn’t resolve the issue.
I’ve downgraded the agents back to 2.0.0p28 after which the errors no longer occur.
For our checkmk server itself we need the 2.1.0p19 version because this version fixes werk #14708, but this brings back the missing monitoring data errors.
Since we get these errors on our monitoring server itself, I’ve added a cronjob that runs ‘/usr/bin/check_mk_agent’ and ‘cmk --debug -vnn ’ every minute and writes the output to two separate files. With this I’m hoping get some output at the moment this issue actually occurs, because I haven’t been able to reproduce the issue so far.
I am very grateful for any suggestions.