2.0.0p3.cre (RAW): check_mk.socket: Too many incoming connections (4) from source x.x.x.x, dropping connection

check_mk.socket: Too many incoming connections (4) from source x.x.x.x, dropping connection

Keep getting this error sporadically, hilariously.

Monitoring from a remote LTE/ADSL connection very hard to overwhelm the remote servers 1gb+ connection.

Yes I can edit the /etc/systemd/system/check_mk.socket but how come / what causes these sockets to not be released in a timely manner.

Restarting the systemctl restart check_mk.socket service doesn’t seem to fix it either.

Another 24 hours later and a bunch of hosts show check_mk inventory / service offline due to this in their agent log.

I have my checks set to 3 minutes, which I thought would be far enough to prevent this sort of issue as well.

May 01 11:06:13 ns2 systemd[1]: check_mk.socket: Too many incoming connections (31) from source 192.168.192.3, dropping connection.
May 01 11:08:59 ns2 systemd[1]: check_mk.socket: Too many incoming connections (31) from source 192.168.192.3, dropping connection.
May 01 11:09:13 ns2 systemd[1]: check_mk.socket: Too many incoming connections (31) from source 192.168.192.3, dropping connection.
May 01 11:10:43 ns2 systemd[1]: check_mk.socket: Too many incoming connections (31) from source 192.168.192.3, dropping connection.
May 01 11:11:59 ns2 systemd[1]: check_mk.socket: Too many incoming connections (31) from source 192.168.192.3, dropping connection.
May 01 11:12:13 ns2 systemd[1]: check_mk.socket: Too many incoming connections (31) from source 192.168.192.3, dropping connection.
May 01 11:14:59 ns2 systemd[1]: check_mk.socket: Too many incoming connections (31) from source 192.168.192.3, dropping connection.
May 01 11:15:13 ns2 systemd[1]: check_mk.socket: Too many incoming connections (31) from source 192.168.192.3, dropping connection.
May 01 11:15:45 ns2 systemd[1]: check_mk.socket: Too many incoming connections (31) from source 192.168.192.3, dropping connection.
May 01 11:17:59 ns2 systemd[1]: check_mk.socket: Too many incoming connections (31) from source 192.168.192.3, dropping connection.

The only thing that seems to work once a host has gone like this is

systemctl restart check_mk.socket;systemctl daemon-reload;systemctl status check_mk.socket

Its super frustrating as reliable monitoring alerts is the only reason to even monitor something. Getting blasted by crits that are caused by the monitoring system makes me sad.

I can confirm that I have the samme issue. I can add that my Raspberry Pi host that I monitor runs on a wifi connection that connects back to monitoring subnet over wireguard.

For such special devices it would tend to use the connection over SSH.
This is very stable also with VPN/DSL connections.

I can confirm this also happens on very “normal” Debian 10 server.
Reinstallations of the agent did not help. The host gets monitored for some time, then the time used to query the agent increases and eventually the agent stops responding altogether.
Wen are still looking into this, but every bit of help is appreciated.

I started using the caching agent, and setting the service to unlimited/not set concurrent connection to avoid getting blasted by crits, it has helped. But… I am having to flush out failed check_mk.socket services almost daily, no idea why, on a bunch of normal machines.

I’ve just accepted it as a fact of life, but wanted to leave the commands here that I use here for anyone else to setup a cron. Also, I am running checks from an LTE remote connection (remote servers are hosted on stable connection though.), so I understand that my connection itself is a bit shaky. Perhaps others with a better connection have less of an issue.

systemctl daemon-reload;  # This should NOT be needed, but sometimes the service just doesn't "clear" up without it. No idea why.
systemctl reset-failed;  # This to clear out all the failed services see image below, Additionally I have created a rule to ignore check_mk services failures in the systemd module
systemctl restart check_mk.socket;  # Restart the service
systemctl status check_mk.socket  # Make sure its running after.