Distributed Monitoring cmk.cee.liveproxy.Client.ClientRequestTimeoutException

Troubleshooting
1

Hi,

does somebody know where to increase the timeout value related to cmk.cee.liveproxy.Client.ClientRequestTimeoutException, which gets logged in var/log/liveproxyd.log

The setup is running on 2.0.0p5 (CEE) and has several remote sites. Some of them are located on different continents, so there are high latencies between the local and remote site, which probably relates to those issues.

Some log:

cmk.cee.liveproxy.Client.ClientRequestTimeoutException: Reading request from client took longer than 5s
2021-07-01 09:38:37,254 [40] [cmk.liveproxyd.(5055).Site(site_name_replaced).Client(36)] Reading request from client took longer than 5s
Traceback (most recent call last):
  File "/omd/sites/site_name_replaced/lib/python3/cmk/cee/liveproxy/Client.py", line 150, in receive_new_request
    request = self._receive_request()
  File "/omd/sites/site_name_replaced/lib/python3/cmk/cee/liveproxy/Client.py", line 339, in _receive_request
    raise ClientRequestTimeoutException(
cmk.cee.liveproxy.Client.ClientRequestTimeoutException: Reading request from client took longer than 5s
2021-07-01 09:38:37,255 [40] [cmk.liveproxyd.(5065).Site(site_name_replaced).Client(37)] Reading request from client took longer than 5s
Traceback (most recent call last):
  File "/omd/sites/site_name_replaced/lib/python3/cmk/cee/liveproxy/Client.py", line 150, in receive_new_request
    request = self._receive_request()
  File "/omd/sites/site_name_replaced/lib/python3/cmk/cee/liveproxy/Client.py", line 339, in _receive_request
    raise ClientRequestTimeoutException(
cmk.cee.liveproxy.Client.ClientRequestTimeoutException: Reading request from client took longer than 5s
2021-07-01 09:38:37,255 [40] [cmk.liveproxyd.(5018).Site(site_name_replaced).Client(21)] Reading request from client took longer than 5s
Traceback (most recent call last):
  File "/omd/sites/site_name_replaced/lib/python3/cmk/cee/liveproxy/Client.py", line 150, in receive_new_request
    request = self._receive_request()
  File "/omd/sites/site_name_replaced/lib/python3/cmk/cee/liveproxy/Client.py", line 339, in _receive_request
    raise ClientRequestTimeoutException(
cmk.cee.liveproxy.Client.ClientRequestTimeoutException: Reading request from client took longer than 5s
2021-07-01 09:38:37,255 [40] [cmk.liveproxyd.(5071).Site(site_name_replaced).Client(39)] Reading request from client took longer than 5s
Traceback (most recent call last):
  File "/omd/sites/site_name_replaced/lib/python3/cmk/cee/liveproxy/Client.py", line 150, in receive_new_request
    request = self._receive_request()
  File "/omd/sites/site_name_replaced/lib/python3/cmk/cee/liveproxy/Client.py", line 339, in _receive_request
    raise ClientRequestTimeoutException(
cmk.cee.liveproxy.Client.ClientRequestTimeoutException: Reading request from client took longer than 5s

What I tried already is to increase Livestatus Proxy Daemon Settings, which does not seem to be related to this issue (but ofc fixed the issue that they appear to be disconnected in the gui from time to time).

heartbeat:
every: 15s , timeout: 7s
Timeout waiting for a free channel
9s
Total query timeout
120
Cooling period after failed connect/heartbeat
10

Cheers!
René

2

We are facing the same issues with 2.0.0p8 and roughly 70 remote sites. The GUI feels very slow in such cases and the liveproxyd.log gets filled with hundreds of these entries. Before decreasing the log level to “Warning” (default is “Informational”), we got around 200k of logs every 5 seconds. Once, we even had to restart liveproxyd because accessing the GUI was no longer possible (kept loading for minutes).

Most remote sites are in the same country as the master site. RTA and packet loss are very low. Some sites even are in the same datacenter but show the same issues.

Unfortunately though, we do not have a solution either…

3

Same issue here … 10 sites and soon 70 sites.

4

Hi,

the same behavior here with 70 total sites (CEE 2.0.0p6).
Below an example:

2021-09-07 14:55:31,958 [40] [cmk.liveproxyd.(19346).Site(site_name).Client(43)] Reading request from client took longer than 5s
Traceback (most recent call last):
  File "/omd/sites/master_site_name/lib/python3/cmk/cee/liveproxy/Client.py", line 150, in receive_new_request
    request = self._receive_request()
  File "/omd/sites/master_site_name/lib/python3/cmk/cee/liveproxy/Client.py", line 339, in _receive_request
    raise ClientRequestTimeoutException(
cmk.cee.liveproxy.Client.ClientRequestTimeoutException: Reading request from client took longer than 5s

Regards

5

Hello, we are experiencing something similar here as well. We have around 30 remote sites mostly in the US and Australia. Anyone find any solutions?

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.