Check_MK Discovery discovery fails, no route to host

**CMK version:**Raw Edition 2.2.0p17
OS version: ubuntu 22.04, docker CE

Error message: Just one out of many remote servers bring:
Services: all up to date, Host labels: all up to date, [agent] Communication failed: [Errno 113] No route to host CRIT

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

Checkmk version 2.2.0p17
+ FETCHING DATA
  Source: SourceInfo(hostname='host2check.example.com', ipaddress='233.252.0.1', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f4da1dceb10]
Read from cache: AgentFileCache(host2check.example.com, path_template=/omd/sites/cmk/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (does not exist)
[TCPFetcher] Execute data source
Connecting via TCP to 233.252.0.1:6556 (5.0s timeout)
Closing TCP connection to 233.252.0.1:6556
[cpu_tracking] Stop [7f4da1dceb10 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
  Source: SourceInfo(hostname='host2check.example.com', ipaddress='233.252.0.1', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f4da1233490]
Read from cache: NoCache(host2check.example.com, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for 'host2check.example.com'. Skip processing.
No piggyback files for '233.252.0.1'. Skip processing.
[cpu_tracking] Stop [7f4da1233490 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  HostKey(hostname='host2check.example.com', source_type=<SourceType.HOST: 1>)  -> Not adding sections: OSError(113, 'No route to host')
  HostKey(hostname='host2check.example.com', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
[cpu_tracking] Start [7f4da1233050]
value store: synchronizing
Trying to acquire lock on /omd/sites/cmk/tmp/check_mk/counters/host2check.example.com
Got lock on /omd/sites/cmk/tmp/check_mk/counters/host2check.example.com
value store: loading from disk
Releasing lock on /omd/sites/cmk/tmp/check_mk/counters/host2check.example.com
Released lock on /omd/sites/cmk/tmp/check_mk/counters/host2check.example.com
No piggyback files for 'host2check.example.com'. Skip processing.
No piggyback files for '233.252.0.1'. Skip processing.
[cpu_tracking] Stop [7f4da1233050 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.009999997913837433))]
[agent] OSError(113, 'No route to host')(!!), [piggyback] Success (but no data found for this host), execution time 0.0 sec | execution_time=0.010 user_time=0.010 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.000

the server in question is reachable (we have some availability montitors (https, smtp) running), the check_mk host is reachable from the remote server, other, similar servers are running, we really checked for typos. it’s strange.

no route in checkmk “language” means
“me as a developer don’t know anything about networking, firewalls and all that, but no route seems to be a good error message” - now time for lunch.

try nmap from your remove checkmk server to the client on 6556, what do you get?

I think this isn’t actually a checkmk error, it’s passed on from the tcp stack, but I might be wrong.
The actual reason is very likely as @Anders said just a blocked port. (see also networking - netcat no route to host on port 80 but not on 22 - Unix & Linux Stack Exchange or

spot on. The server had a different firewall, so our dumb rollout script added the rule to open port 6556 to the wrong firewall %)

but I think the error message is a bit misleading. there is a clear route to the server, just the port is closed…

I’m sorry my answer did not help you :frowning:

nono, it did! thanks!