Checkmk agent register got ERROR [cmk_agent_ctl] Host is unreachable (os error 113)

maxrva · April 12, 2023, 6:37am

checkmk agent register got ERROR [cmk_agent_ctl] Host is unreachable (os error 113) after upgrade to version 2.1.0p25

I checked on firewall its not blocked port 8000
I ran this command below on the host that i want to register
cmk-agent-ctl register --hostname localhost --server server:8000 --site mysite --user cmkadmin -vv

In checkmk for the host service got below warning
Version: 2.1.0p25
OS: linux
The hosts agent supports TLS, but it is not being used.
We strongly recommend to enable TLS by registering the host to the site (using the cmk-agent-ctl register command on the monitored host).
NOTE: A registered host will refuse all unencrypted connections. If the host is monitored by multiple sites, you must register to all of them. This can be problematic if you are monitoring the same host from a site running Checkmk version 2.0 or earlier.
If you can not register the host, you can configure missing TLS to be OK in the setting “State in case of available but not enabled TLS” of the ruleset “Checkmk Agent installation auditing”.WARN
Agent plugins: 0
Local checks: 0

after researched i had to register agent using that command but got error.
Is there a way to fix this warning? or the error?

mschlenker · April 12, 2023, 6:50am

Could you please try without specifying the port, thus using autodetection:

cmk-agent-ctl register --hostname localhost --server server --site mysite --user cmkadmin -vv

Please tell me if this worked.

maxrva · April 12, 2023, 6:53am

got the same error, i tried that before

mschlenker · April 12, 2023, 7:26am

Is the Agent Receiver running?

ss -tulpn | grep 800

maxrva · April 12, 2023, 7:30am

I run omd config show on checkmk server

ADMIN_MAIL:
AGENT_RECEIVER: on
AGENT_RECEIVER_PORT: 8000
APACHE_MODE: own
APACHE_TCP_ADDR: 127.0.0.1
APACHE_TCP_PORT: 5000
AUTOSTART: on
CORE: nagios
LIVESTATUS_TCP: off
MKEVENTD: on
MKEVENTD_SNMPTRAP: off
MKEVENTD_SYSLOG: off
MKEVENTD_SYSLOG_TCP: off
MULTISITE_AUTHORISATION: on
MULTISITE_COOKIE_AUTH: on
NAGIOS_THEME: classicui
NSCA: off
PNP4NAGIOS: on
TMPFS: on

But when i ran the command on both host got this
ss -tulpn | grep 8000
tcp LISTEN 0 128 *:8000 :

maxrva · April 12, 2023, 7:32am

but when i ran
ss -tulpn | grep 6556
tcp LISTEN 0 128 *:6556 : users:((“cmk-agent-ctl”,pid=1304,fd=9))

mschlenker · April 12, 2023, 9:28am

Should be a gunicorn process…

root@cmkserver:~# ss -tulpn | grep 800
tcp     LISTEN   0        2048                   *:8003                *:*       users:(("gunicorn: worke",pid=1386319,fd=7),("gunicorn: maste",pid=1386314,fd=7))

When running one should be able to connect via telnet, two linefeed close the connection.

What are your IPv6 settings?

maxrva · April 12, 2023, 9:42am

no ipv6 configure on both server and host

mschlenker · April 12, 2023, 12:28pm

Can you connect to the port 8000 on the external IP address, like

telnet 12.34.56.778 8000

If not, please provide me with the output of

sysctl -a | grep ipv6

maxrva · April 14, 2023, 7:17am

# sysctl -a | grep ipv6
net.ipv6.anycast_src_echo_reply = 0
net.ipv6.auto_flowlabels = 1
net.ipv6.bindv6only = 0
net.ipv6.calipso_cache_bucket_size = 10
net.ipv6.calipso_cache_enable = 1
net.ipv6.conf.all.accept_dad = 0
net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.all.accept_ra_defrtr = 1
net.ipv6.conf.all.accept_ra_from_local = 0
net.ipv6.conf.all.accept_ra_min_hop_limit = 1
net.ipv6.conf.all.accept_ra_mtu = 1
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.all.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.all.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.all.accept_ra_rtr_pref = 1
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.addr_gen_mode = 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.all.dad_transmits = 1
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.disable_policy = 0
net.ipv6.conf.all.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.all.drop_unsolicited_na = 0
net.ipv6.conf.all.enhanced_dad = 1
net.ipv6.conf.all.force_mld_version = 0
net.ipv6.conf.all.force_tllao = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.hop_limit = 64
net.ipv6.conf.all.ignore_routes_with_linkdown = 0
net.ipv6.conf.all.keep_addr_on_down = 0
net.ipv6.conf.all.max_addresses = 16
net.ipv6.conf.all.max_desync_factor = 600
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.all.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.all.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.all.ndisc_notify = 0
net.ipv6.conf.all.ndisc_tclass = 0
net.ipv6.conf.all.optimistic_dad = 0
net.ipv6.conf.all.proxy_ndp = 0
net.ipv6.conf.all.regen_max_retry = 3
net.ipv6.conf.all.router_probe_interval = 60
net.ipv6.conf.all.router_solicitation_delay = 1
net.ipv6.conf.all.router_solicitation_interval = 4
net.ipv6.conf.all.router_solicitation_max_interval = 3600
net.ipv6.conf.all.router_solicitations = -1
net.ipv6.conf.all.seg6_enabled = 0
net.ipv6.conf.all.suppress_frag_ndisc = 1
net.ipv6.conf.all.temp_prefered_lft = 86400
net.ipv6.conf.all.temp_valid_lft = 604800
net.ipv6.conf.all.use_oif_addrs_only = 0
net.ipv6.conf.all.use_optimistic = 0
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.default.accept_dad = 1
net.ipv6.conf.default.accept_ra = 0
net.ipv6.conf.default.accept_ra_defrtr = 1
net.ipv6.conf.default.accept_ra_from_local = 0
net.ipv6.conf.default.accept_ra_min_hop_limit = 1
net.ipv6.conf.default.accept_ra_mtu = 1
net.ipv6.conf.default.accept_ra_pinfo = 1
net.ipv6.conf.default.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.default.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.default.accept_ra_rtr_pref = 1
net.ipv6.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.addr_gen_mode = 0
net.ipv6.conf.default.autoconf = 1
net.ipv6.conf.default.dad_transmits = 1
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.default.disable_policy = 0
net.ipv6.conf.default.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.default.drop_unsolicited_na = 0
net.ipv6.conf.default.enhanced_dad = 1
net.ipv6.conf.default.force_mld_version = 0
net.ipv6.conf.default.force_tllao = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.hop_limit = 64
net.ipv6.conf.default.ignore_routes_with_linkdown = 0
net.ipv6.conf.default.keep_addr_on_down = 0
net.ipv6.conf.default.max_addresses = 16
net.ipv6.conf.default.max_desync_factor = 600
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.default.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.default.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.default.ndisc_notify = 0
net.ipv6.conf.default.ndisc_tclass = 0
net.ipv6.conf.default.optimistic_dad = 0
net.ipv6.conf.default.proxy_ndp = 0
net.ipv6.conf.default.regen_max_retry = 3
net.ipv6.conf.default.router_probe_interval = 60
net.ipv6.conf.default.router_solicitation_delay = 1
net.ipv6.conf.default.router_solicitation_interval = 4
net.ipv6.conf.default.router_solicitation_max_interval = 3600
net.ipv6.conf.default.router_solicitations = -1
net.ipv6.conf.default.seg6_enabled = 0
net.ipv6.conf.default.suppress_frag_ndisc = 1
net.ipv6.conf.default.temp_prefered_lft = 86400
net.ipv6.conf.default.temp_valid_lft = 604800
net.ipv6.conf.default.use_oif_addrs_only = 0
net.ipv6.conf.default.use_optimistic = 0
net.ipv6.conf.default.use_tempaddr = 0
net.ipv6.conf.ens192.accept_dad = 1
net.ipv6.conf.ens192.accept_ra = 0
net.ipv6.conf.ens192.accept_ra_defrtr = 1
net.ipv6.conf.ens192.accept_ra_from_local = 0
net.ipv6.conf.ens192.accept_ra_min_hop_limit = 1
net.ipv6.conf.ens192.accept_ra_mtu = 1
net.ipv6.conf.ens192.accept_ra_pinfo = 1
net.ipv6.conf.ens192.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.ens192.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.ens192.accept_ra_rtr_pref = 1
net.ipv6.conf.ens192.accept_redirects = 0
net.ipv6.conf.ens192.accept_source_route = 0
net.ipv6.conf.ens192.addr_gen_mode = 0
net.ipv6.conf.ens192.autoconf = 1
net.ipv6.conf.ens192.dad_transmits = 1
net.ipv6.conf.ens192.disable_ipv6 = 0
net.ipv6.conf.ens192.disable_policy = 0
net.ipv6.conf.ens192.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.ens192.drop_unsolicited_na = 0
net.ipv6.conf.ens192.enhanced_dad = 1
net.ipv6.conf.ens192.force_mld_version = 0
net.ipv6.conf.ens192.force_tllao = 0
net.ipv6.conf.ens192.forwarding = 0
net.ipv6.conf.ens192.hop_limit = 64
net.ipv6.conf.ens192.ignore_routes_with_linkdown = 0
net.ipv6.conf.ens192.keep_addr_on_down = 0
net.ipv6.conf.ens192.max_addresses = 16
net.ipv6.conf.ens192.max_desync_factor = 600
net.ipv6.conf.ens192.mc_forwarding = 0
net.ipv6.conf.ens192.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.ens192.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.ens192.mtu = 1500
net.ipv6.conf.ens192.ndisc_notify = 0
net.ipv6.conf.ens192.ndisc_tclass = 0
net.ipv6.conf.ens192.optimistic_dad = 0
net.ipv6.conf.ens192.proxy_ndp = 0
net.ipv6.conf.ens192.regen_max_retry = 3
net.ipv6.conf.ens192.router_probe_interval = 60
net.ipv6.conf.ens192.router_solicitation_delay = 1
net.ipv6.conf.ens192.router_solicitation_interval = 4
net.ipv6.conf.ens192.router_solicitation_max_interval = 3600
net.ipv6.conf.ens192.router_solicitations = -1
net.ipv6.conf.ens192.seg6_enabled = 0
net.ipv6.conf.ens192.suppress_frag_ndisc = 1
net.ipv6.conf.ens192.temp_prefered_lft = 86400
net.ipv6.conf.ens192.temp_valid_lft = 604800
net.ipv6.conf.ens192.use_oif_addrs_only = 0
net.ipv6.conf.ens192.use_optimistic = 0
net.ipv6.conf.ens192.use_tempaddr = 0
net.ipv6.conf.lo.accept_dad = -1
net.ipv6.conf.lo.accept_ra = 1
net.ipv6.conf.lo.accept_ra_defrtr = 1
net.ipv6.conf.lo.accept_ra_from_local = 0
net.ipv6.conf.lo.accept_ra_min_hop_limit = 1
net.ipv6.conf.lo.accept_ra_mtu = 1
net.ipv6.conf.lo.accept_ra_pinfo = 1
net.ipv6.conf.lo.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.lo.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.lo.accept_ra_rtr_pref = 1
net.ipv6.conf.lo.accept_redirects = 1
net.ipv6.conf.lo.accept_source_route = 0
net.ipv6.conf.lo.addr_gen_mode = 0
net.ipv6.conf.lo.autoconf = 1
net.ipv6.conf.lo.dad_transmits = 1
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.disable_policy = 0
net.ipv6.conf.lo.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.lo.drop_unsolicited_na = 0
net.ipv6.conf.lo.enhanced_dad = 1
net.ipv6.conf.lo.force_mld_version = 0
net.ipv6.conf.lo.force_tllao = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.hop_limit = 64
net.ipv6.conf.lo.ignore_routes_with_linkdown = 0
net.ipv6.conf.lo.keep_addr_on_down = 0
net.ipv6.conf.lo.max_addresses = 16
net.ipv6.conf.lo.max_desync_factor = 600
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.lo.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.lo.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.lo.mtu = 65536
net.ipv6.conf.lo.ndisc_notify = 0
net.ipv6.conf.lo.ndisc_tclass = 0
net.ipv6.conf.lo.optimistic_dad = 0
net.ipv6.conf.lo.proxy_ndp = 0
net.ipv6.conf.lo.regen_max_retry = 3
net.ipv6.conf.lo.router_probe_interval = 60
net.ipv6.conf.lo.router_solicitation_delay = 1
net.ipv6.conf.lo.router_solicitation_interval = 4
net.ipv6.conf.lo.router_solicitation_max_interval = 3600
net.ipv6.conf.lo.router_solicitations = -1
net.ipv6.conf.lo.seg6_enabled = 0
net.ipv6.conf.lo.suppress_frag_ndisc = 1
net.ipv6.conf.lo.temp_prefered_lft = 86400
net.ipv6.conf.lo.temp_valid_lft = 604800
net.ipv6.conf.lo.use_oif_addrs_only = 0
net.ipv6.conf.lo.use_optimistic = 0
net.ipv6.conf.lo.use_tempaddr = -1
net.ipv6.fib_multipath_hash_policy = 0
net.ipv6.flowlabel_consistency = 1
net.ipv6.flowlabel_reflect = 0
net.ipv6.flowlabel_state_ranges = 0
net.ipv6.fwmark_reflect = 0
net.ipv6.icmp.ratelimit = 1000
net.ipv6.idgen_delay = 1
net.ipv6.idgen_retries = 3
net.ipv6.ip6frag_high_thresh = 4194304
net.ipv6.ip6frag_low_thresh = 3145728
net.ipv6.ip6frag_secret_interval = 0
net.ipv6.ip6frag_time = 60
net.ipv6.ip_nonlocal_bind = 0
net.ipv6.max_dst_opts_length = 2147483647
net.ipv6.max_dst_opts_number = 8
net.ipv6.max_hbh_length = 2147483647
net.ipv6.max_hbh_opts_number = 8
net.ipv6.mld_max_msf = 64
net.ipv6.mld_qrv = 2
net.ipv6.neigh.default.anycast_delay = 100
net.ipv6.neigh.default.app_solicit = 0
net.ipv6.neigh.default.base_reachable_time_ms = 30000
net.ipv6.neigh.default.delay_first_probe_time = 5
net.ipv6.neigh.default.gc_interval = 30
net.ipv6.neigh.default.gc_stale_time = 60
net.ipv6.neigh.default.gc_thresh1 = 128
net.ipv6.neigh.default.gc_thresh2 = 512
net.ipv6.neigh.default.gc_thresh3 = 1024
net.ipv6.neigh.default.locktime = 0
net.ipv6.neigh.default.mcast_resolicit = 0
net.ipv6.neigh.default.mcast_solicit = 3
net.ipv6.neigh.default.proxy_delay = 80
net.ipv6.neigh.default.proxy_qlen = 64
net.ipv6.neigh.default.retrans_time_ms = 1000
net.ipv6.neigh.default.ucast_solicit = 3
net.ipv6.neigh.default.unres_qlen = 101
net.ipv6.neigh.default.unres_qlen_bytes = 212992
net.ipv6.neigh.ens192.anycast_delay = 100
net.ipv6.neigh.ens192.app_solicit = 0
net.ipv6.neigh.ens192.base_reachable_time_ms = 30000
net.ipv6.neigh.ens192.delay_first_probe_time = 5
net.ipv6.neigh.ens192.gc_stale_time = 60
net.ipv6.neigh.ens192.locktime = 0
net.ipv6.neigh.ens192.mcast_resolicit = 0
net.ipv6.neigh.ens192.mcast_solicit = 3
net.ipv6.neigh.ens192.proxy_delay = 80
net.ipv6.neigh.ens192.proxy_qlen = 64
net.ipv6.neigh.ens192.retrans_time_ms = 1000
net.ipv6.neigh.ens192.ucast_solicit = 3
net.ipv6.neigh.ens192.unres_qlen = 101
net.ipv6.neigh.ens192.unres_qlen_bytes = 212992
net.ipv6.neigh.lo.anycast_delay = 100
net.ipv6.neigh.lo.app_solicit = 0
net.ipv6.neigh.lo.base_reachable_time_ms = 30000
net.ipv6.neigh.lo.delay_first_probe_time = 5
net.ipv6.neigh.lo.gc_stale_time = 60
net.ipv6.neigh.lo.locktime = 0
net.ipv6.neigh.lo.mcast_resolicit = 0
net.ipv6.neigh.lo.mcast_solicit = 3
net.ipv6.neigh.lo.proxy_delay = 80
net.ipv6.neigh.lo.proxy_qlen = 64
net.ipv6.neigh.lo.retrans_time_ms = 1000
net.ipv6.neigh.lo.ucast_solicit = 3
net.ipv6.neigh.lo.unres_qlen = 101
net.ipv6.neigh.lo.unres_qlen_bytes = 212992
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 4096
net.ipv6.route.min_adv_mss = 1220
net.ipv6.route.mtu_expires = 600
net.ipv6.seg6_flowlabel = 0
net.ipv6.xfrm6_gc_thresh = 32768

maxrva · April 14, 2023, 7:23am

telnet x.x.x.x 8000
Trying x.x.x.x…
telnet: connect to address x.x.x.x: No route to host

mschlenker · April 14, 2023, 7:42am

This looks like a network/routing problem. If the monitoring server can contact the host to be monitored but not the other way, you can “register by proxy”:

maxrva · April 14, 2023, 7:45am

how to register by proxy? Please guide

mschlenker · April 14, 2023, 7:47am

Just click the link. It is a deep link leading directly to the detailed guide.

maxrva · April 17, 2023, 3:22am

I ran ss -tulpn | grep 8000 under root got below output
tcp LISTEN 0 128 *:8000 : users:((“python3”,pid=889632,fd=7),(“python3”,pid=889628,fd=7))

#ps 889632
PID TTY STAT TIME COMMAND
889632 ? S 20:09 python3 /omd/sites/monitoring/bin/gunicorn -D -p /omd/sites/monitoring/tmp/run/agent-receiver.pid --error-logfile /omd/sites/monito

which mean agent recevier listening on port 8000 on checkmk server correct?

maxrva · April 17, 2023, 4:05am

cmk-agent-ctl register --hostname host.lan --server cmkserver.lan --site monitoring --user cmkadmin -vv

INFO [cmk_agent_ctl] starting
INFO [cmk_agent_ctl] Loaded config from ‘“/var/lib/cmk-agent/cmk-agent-ctl.toml”’, legacy pull ‘LegacyPullMarker(“/var/lib/cmk-agent/allow-legacy-pull”)’ exists
DEBUG [reqwest::connect] starting new connection: //cmkserver.lan/
INFO [cmk_agent_ctl::site_spec] Failed to discover agent receiver port using https, trying http.
DEBUG [cmk_agent_ctl::site_spec] https error “Failed to discover agent receiver port from cmkserver.lan/monitoring/check_mk/api/1.0/domain-types/internal/actions/discover-receiver/invoke\nerror sending request for url (cmkserver.lan/monitoring/check_mk/api/1.0/domain-types/internal/actions/discover-receiver/invoke): error trying to connect: tcp connect error: Host is unreachable (os error 113)\nerror trying to connect: tcp connect error: Host is unreachable (os error 113)\ntcp connect error: Host is unreachable (os error 113)\nHost is unreachable (os error 113)”
DEBUG [reqwest::connect] starting new connection: //cmkserver.lan/
Attempting to register at cmkserver.lan:8000/monitoring. Server certificate details:

ERROR [cmk_agent_ctl] Host is unreachable (os error 113)

maxrva · April 17, 2023, 4:07am

both server on the same vlan the firewall rules not applied but don’t know why telnet to checkmk server on port 8000 not reachable.

mschlenker · April 17, 2023, 6:40am

At this point it looks strongly like a network problem. If the problem is limited to a small number of hosts, I’d do registration by proxy as mentioned above.

maxrva · April 17, 2023, 8:50am

registered using proxy got the same error
ERROR [cmk_agent_ctl] Host is unreachable (os error 113)

I checked on firewall its reachable for between the host and check_mk server on port 8000

mschlenker · April 17, 2023, 9:02am

So does

telnet localhost 8000

or

telnet 127.0.0.1 8000

work on the server itself?