Apt plugin not working on some hosts

[julianm1092 (Julian)]

CMK version:
CRE 2.1.0p9

OS version:
Ubuntu 18.04.6 LTS

Error message:
[agent] Success, Missing monitoring data for plugins: aptWARN, execution time 0.8 sec

Output of “cmk --debug -vvn serverxyz.domain.de”:

Checkmk version 2.1.0p9
Try license usage history update.
Trying to acquire lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Got lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Got lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Released lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Released lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7fbbcfdf4a30]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(serverxyz.domain.de, base_path=/omd/sites/mysite/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 25 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to IP:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b'16')
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Write data to cache file /omd/sites/mysite/tmp/check_mk/cache/serverxyz.domain.de
Trying to acquire lock on /omd/sites/mysite/tmp/check_mk/cache/serverxyz.domain.de
Got lock on /omd/sites/mysite/tmp/check_mk/cache/serverxyz.domain.de
Releasing lock on /omd/sites/mysite/tmp/check_mk/cache/serverxyz.domain.de
Released lock on /omd/sites/mysite/tmp/check_mk/cache/serverxyz.domain.de
Closing TCP connection to IP:6556
[cpu_tracking] Stop [7fbbcfdf4a30 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.75))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7fbbcfdf4790]
[PiggybackFetcher] Fetch with cache settings: NoCache(serverxyz.domain.de, base_path=/omd/sites/mysite/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'serverxyz.domain.de'. Skip processing.
No piggyback files for 'IP'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7fbbcfdf4790 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<apt:cached(1672730682,3600):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: ['apt', 'check_mk', 'checkmk_agent_plugins_lnx', 'cifsmounts', 'cmk_agent_ctl_status', 'cpu', 'df', 'diskstat', 'job', 'kernel', 'labels', 'lnx_if', 'local', 'md', 'mem', 'mounts', 'nfsmounts', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
  -> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7fbbcff56640]
value store: synchronizing
Trying to acquire lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
Got lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
value store: loading from disk
Releasing lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
Released lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
APT Updates          PEND - Check plugin received no monitoring data
No piggyback files for 'serverxyz.domain.de'. Skip processing.
No piggyback files for 'IP'. Skip processing.
[cpu_tracking] Stop [7fbbcff56640 - Snapshot(process=posix.times_result(user=0.16999999999999993, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.17999999970197678))]
[agent] Success, Got no information from host(!!), execution time 0.9 sec | execution_time=0.930 user_time=0.180 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.740

Additional Information
The “CheckMK”-service successfully detects the existence of a plugin on my hosts.
I can run /usr/lib/check_mk_agent/plugins/3600/mk_apt with correct output.

Output of cmk --debug -vvv --detect-plugins mk_apt serverxyz.domain.de:

OMD[mysite]:~$  cmk --debug -vvv --detect-plugins mk_apt serverxyz.domain.de
Checkmk version 2.1.0p9
Try license usage history update.
Trying to acquire lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Got lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Got lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Released lock on /omd/sites/mysite/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
Released lock on /omd/sites/mysite/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7f804478af10]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(serverxyz.domain.de, base_path=/omd/sites/mysite/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Mode Mode.FORCE_SECTIONS)
[TCPFetcher] Execute data source
Connecting via TCP to IP:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b'16')
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Not using cache (Mode Mode.FORCE_SECTIONS)
Closing TCP connection to IP:6556
[cpu_tracking] Stop [7f804478af10 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.7399999983608723))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f804478ad60]
[PiggybackFetcher] Fetch with cache settings: NoCache(serverxyz.domain.de, base_path=/omd/sites/mysite/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'serverxyz.domain.de'. Skip processing.
No piggyback files for 'IP'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7f804478ad60 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<apt:cached(1672734289,3600):sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: []
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
  -> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7f8044a30400]
value store: synchronizing
Trying to acquire lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
Got lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
value store: loading from disk
Releasing lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
Released lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
value store: synchronizing
Trying to acquire lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
Got lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
value store: already loaded
Releasing lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
Released lock on /omd/sites/mysite/tmp/check_mk/counters/serverxyz.domain.de
[cpu_tracking] Stop [7f8044a30400 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.009999997913837433))]
[agent] Success, execution time 0.7 sec | execution_time=0.750 user_time=0.000 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.740

I appreciate every help. Let me know if there are other information needed.

• Julian

[crpb]

Hi Julian,

does the plugin work directly executed on the client?
/usr/lib/check_mk_agent/plugins/mk_apt
And maybe remove the cache in /var/lib/check_mk_agent/cache/* if it seems ok. After that your could run the cmk .. again. Or just run check_mk_agent on the host if it is not “encrypted”.

[julianm1092 (Julian)]

it works when i execute it on the client. removed the cache and it still works.
after i did cmk --debug -vvn serverxyz.domain.de again, i can see this (because i enforce the apt-service for testing purposes):

APT Updates          PEND - Check plugin received no monitoring data
CPU load             15 min load: 0.04, 15 min load per core: 0.02 (2 cores)
Check_MK Agent       Version: 2.1.0p9, OS: linux, Agent plugins: 1, Local checks: 0
Filesystem /         27.78% used (16.06 of 57.79 GB), trend: +77.61 kB / 24 hours
Interface ens160     [2], (up), MAC: [...], Speed: 10 GBit/s, In: 14.7 kB/s (<0.01%), Out: 3.80 kB/s (<0.01%)
Memory               Total virtual memory: 47.2% - 1.37 GB of 2.90 GB, 9 additional details available
Uptime               Up since Jan 02 2023 14:14:28, Uptime: 1 day 0 hours

The PEND won’t resolve in OK or WARN.

[Heavy]

@julianm1092

CMK version:
CRE 2.1.0p9

OS version:
Ubuntu 18.04.6 LTS

Error message:
[agent] Success, Missing monitoring data for plugins: aptWARN, execution time 0.8 sec

There has been an update to the mk_apt checkplugin in Checkmk 2.1.0p14 to support the new behaviour on Ubuntu Linux. So ensure to update both the Checkmk server and the mk_apt agent plugin to 2.1.0p14 or higher.

[Release] Checkmk stable release 2.1.0p14

Dear friends of Checkmk, the new stable release 2.1.0p14 of Checkmk is ready for download. This maintenance release ships with 33 changes affecting all editions of Checkmk, 5 Enterprise Edition specific changes and 0 Managed Services Edition specific changes. Support periods can be found here: Checkmk versions Changes in all Checkmk Editions: Agent bakery: 14535 FIX: fix runas plugin 14808 FIX: mk_filestat crash when scanning short lived files Checks & agents: 10931 VMWare UAG (Photon…

[julianm1092 (Julian)]

Thank you. I will give feedback when i had a chance to update my instance.

[julianm1092 (Julian)]

Updated my site to the latest version 2.1.0p18. Updated every Linux Host to the new agent, placed the apt-plugin in the correct path, checked the permissions.

Rescheduled the CheckMK Service at each host, new agent version is detected and the autodiscovery of the APT-Service works like a charm.

Thank you!