Empty output from host - check_mk_agent stuck on <<<multipath>>>

Troubleshooting
1

CMK version: RAW Edition 2.2.0p2.cre
OS version: Ubuntu 22.04

Hello, I’m having an issue with a server that has been working before this without issues. This is the second time this has happened. Last time a reboot solved the problem but as this is a recurring issue it needs to be solved permanently.

I’ve tried reinstalling agent, restart OMD etc but nothing seems to solve it.

On thing I’ve found is that when i run the command check_mk_agent it gets stuck at this:

<<<multipath>>>

I’ve checked a lot of previous thread regarding the error message Empty output from host but nothing that was mention has worked in this case.

Any help/input is appreciated!

Output of “cmk --debug -vvn hostname":

`Checkmk version 2.2.0p2
+ FETCHING DATA
  Source: SourceInfo(hostname='******', ipaddress='******', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fd1941df650]
Read from cache: AgentFileCache(******, path_template=/omd/sites/monitoring/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=120, inventory=120), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (does not exist)
[TCPFetcher] Execute data source
Connecting via TCP to ******:6556 (30.0s timeout)
Closing TCP connection to ******:6556
[cpu_tracking] Stop [7fd1941df650 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
  Source: SourceInfo(hostname='******', ipaddress='******', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7fd1932cfe90]
Read from cache: NoCache(******, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for '******'. Skip processing.
No piggyback files for '******'. Skip processing.
[cpu_tracking] Stop [7fd1932cfe90 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
No persisted sections
  HostKey(hostname='******', source_type=<SourceType.HOST: 1>)  -> Not adding sections: Empty output from host ******:6556
  HostKey(hostname='******', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
[cpu_tracking] Start [7fd193162650]
value store: synchronizing
Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/counters/******
Got lock on /omd/sites/monitoring/tmp/check_mk/counters/******
value store: loading from disk
Releasing lock on /omd/sites/monitoring/tmp/check_mk/counters/******
Released lock on /omd/sites/monitoring/tmp/check_mk/counters/******
No piggyback files for '******'. Skip processing.
No piggyback files for '******'. Skip processing.
[cpu_tracking] Stop [7fd193162650 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[agent] Empty output from host ******:6556(!!), [piggyback] Success (but no data found for this host), execution time 0.0 sec | execution_time=0.000 user_time=0.000 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.000OMD[monitoring]:~$`
2

I can also add that I’ve tried a lot of different things but i just can’t find the exact reason for this issue.

This is the only host we’ve had this issue on and we currently have around 60 hosts.

It has also spawned 59 of these processes:

root      550477  0.0  0.0   6704  4336 ?        Ss   14:48   0:00 /bin/bash /usr/bin/check_mk_agent
3

The section uses multipath -l to determine the status of multipath connections.
You need to figure out why this command hangs.

5

Hello,

Thanks, will have to look into that.

Around 10 minutes before you sent this message i checked ps aux and saw there were around 15-20 of these multipath -l processes running. So will investigate this deeper next time.

6

Hello again!

After cheking on this we noticed that it was Acronis Cyber Infrastructure agent that had a mount that had hanged which caused checkmk to get stuck on that mount.

Unfortunately this is something that tends to happen with Acronis, so what I’m wondering is if it’s somehow possible to exclude the agent from checking the <<>> entirely?

7

Yes. In the raw edition you have to create a config file /etc/check_mk/exclude_sections.cfg containing

MK_SKIP_MULTIPATHING=yes

for the Linux agent to skip the <<<multipath>>> section.

8

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.