Unhandled exception: 400: Site connection not initiated ([Errno 104] Connection reset by peer)

CMK version: 2.5.0p13
OS version: Debian Trixie

Error message: Unhandled exception: 400: Site connection not initiated ([Errno 104] Connection reset by peer).

all connections are lost in distributed monitoring after updating checkmk-pro from 2.5.0p11 to 2.5.0p13. Port 6557 is open on all sites, sites components are running. in liveproxyd.log

Could not send heartbeat for 10 secs - no free channel

although Maximum concurrent Livestatus connections ist set to 100, an increase didn’t help neither, telnet from master to remote doesn’t work neither.

~$ ss -tan state established '( sport = :6557 )' | wc -l
1
~$ telnet slave01 6557
Trying 192.168.1.12...
Connected to slave01.
Escape character is '^]'.
Connection closed by foreign host.
~$ netstat -tulpn | grep 6557
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::6557                 :::*                    LISTEN      34433/xinetd

master is still logged in to all remote sites bit is connected. The community update went without any issues.

web.log has many such entries

2026-09-14 06:59:15,408 [40] [cmk.web 42824] Failed to fetch peer certificate
Traceback (most recent call last):
  File "/omd/sites/$master/lib/python3/cmk/gui/wato/pages/sites.py", line 1954, in page
    cert_details = list(self._fetch_certificate_details())
  File "/omd/sites/$master/lib/python3/cmk/utils/encryption.py", line 71, in fetch_certificate_details
    verify_chain_results = _fetch_certificate_chain_verify_results(
        trusted_ca_file, address_family, address
    )
  File "/omd/sites/$master/lib/python3/cmk/utils/encryption.py", line 109, in _fetch_certificate_chain_verify_results
    sock.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
  File "/omd/sites/$master/lib/python3.13/site-packages/OpenSSL/SSL.py", line 2487, in do_handshake
    self._raise_ssl_error(self._ssl, result)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/$master/lib/python3.13/site-packages/OpenSSL/SSL.py", line 2074, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')

and now I got this error:

Got invalid data:
Internal automation error: too many values to unpack (expected 2)
Traceback (most recent call last):
  File "/omd/sites/remote01/lib/python3/cmk/gui/wato/pages/automation.py", line 302, in _execute_automation_command
    response.set_data(repr(automation.execute(automation.get_request(config, request))))
                           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/remote01/lib/python3/cmk/gui/watolib/automation_commands.py", line 83, in execute
    "omd_status": self._get_omd_status(),
                  ~~~~~~~~~~~~~~~~~~~~^^
  File "/omd/sites/remote01/lib/python3/cmk/gui/watolib/automation_commands.py", line 76, in _get_omd_status
    return self._parse_omd_status(result.stdout)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/omd/sites/remote01/lib/python3/cmk/gui/watolib/automation_commands.py", line 61, in _parse_omd_status
    status = {key: int(val) for key, val in (el.split(" ") for el in raw_status.splitlines())}
                                ^^^^^^^^
ValueError: too many values to unpack (expected 2)

Any ideas what went wrong here?

What do you see if you test with openssl from your central site to one of the remote instances?

openssl s_client -connect slave01:6557

With your telnet test, this should not work if TLS is enabled on the remote sites.
If there was no TLS enabled on the remote site then you can check your distributed monitoring configuration for these sites and look if TLS is also disabled there.

Hi Andres, you are right, TLS is enabled, so telnet doesn’t work, with openssl s_client -connect I get this:

Connecting to 192.168.1.12
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
Negotiated TLS1.3 group: <NULL>
---
SSL handshake has read 0 bytes and written 1555 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Protocol: TLSv1.3
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent

It looks like TLS is not active at the remote site currently.

well, it is enabled all the time, everything worked until 2.5.0p11, the issue arised after the update to 2.5.0p13, this is the config from one remote:

~$ omd config show 
ADMIN_MAIL: 
AGENT_RECEIVER: on
AGENT_RECEIVER_PORT: 8000
APACHE_MODE: own
APACHE_TCP_ADDR: 127.0.0.1
APACHE_TCP_PORT: 5000
AUTOMATION_HELPER: on
AUTOSTART: on
CORE: cmc
LIVEPROXYD: on
LIVESTATUS_TCP: on
LIVESTATUS_TCP_ONLY_FROM: 192.168.1.10
LIVESTATUS_TCP_PORT: 6557
LIVESTATUS_TCP_TLS: on
MCP_SERVER: off
MCP_TRACE_FORWARD: off
MKEVENTD: on
MKEVENTD_SNMPTRAP: off
MKEVENTD_SYSLOG: off
MKEVENTD_SYSLOG_TCP: off
MULTISITE_AUTHORISATION: on
MULTISITE_COOKIE_AUTH: on
PIGGYBACK_HUB: off
RABBITMQ_DIST_PORT: 25672
RABBITMQ_MANAGEMENT_PORT: 15671
RABBITMQ_ONLY_FROM: ::
RABBITMQ_PORT: 5672
TMPFS: on
TRACE_RECEIVE: off
TRACE_SEND: off

Can you have a look on the remote site inside the ~/tmp/run folder?

srw-rw---- 1 SITENAME SITENAME    0 Sep 11 15:00 live=
lrwxrwxrwx 1 SITENAME SITENAME    8 Sep 14 10:07 live-tcp -> live-tls=
srwx------ 1 SITENAME SITENAME    0 Sep 14 10:07 live-tls=

Also on this host itself the openssl test against localhost should also result in an error or?

ls -al tmp/run show the socket is there

srw-rw---- 1 slave01 slave01   0 Sep 14 07:06 live=
lrwxrwxrwx 1 slave01 slave01   8 Sep 14 07:06 live-tcp -> live-tls=
srwx------ 1 slave01 slave01   0 Sep 14 07:06 live-tls=
~$ openssl s_client -connect localhost:6557
Connecting to 127.0.0.1
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
Negotiated TLS1.3 group: <NULL>
---
SSL handshake has read 0 bytes and written 1524 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Protocol: TLSv1.3
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

live-tcp points correctly to live-tls and not to live.
Strange. You can check if you stunnel is running correctly.

I found the issue. My bad, the IP of the master has changed, and I forgot to modify the IP in LIVESTATUS_TCP_ONLY_FROM, I fixed it now but still same issues, because probably it should be changed in many files, so I downgraded to 2.5.0p11 and it worked fine, then removed 2.5.0p13 and made a clean upgrade again. Issue disappeared.
Thanks Andreas