Service Discovery fails with "failed to fetch state [500]: INTERNAL SERVER ERROR. Retrying in 1 second"

CMK version: 2.3.0p43
OS version: Monitoring Site Ubuntu 24.04 / Central Site 20.04

We are encountering the following problem: we can no longer perform service discovery management on hosts with a large number of services (approx. 2,800 services). The hosts are using the Push-Agent in the same version (2.3.0p43). When attempting a service discovery via the GUI (e.g., “Accept all”), we receive the following error:

Failed to fetch state [500]: INTERNAL SERVER ERROR. Retrying in 1 second. In case this error persists for more than some seconds, please verify that all processes of the site are running.

I have already adjusted parameters in apache.conf:

  • Timeout 600

  • LimitRequestBody 0

And in /opt/omd/apache/site.conf:

  • ProxyPass … timeout=600

Unfortunately, these changes did not resolve the issue. On the central Checkmk site, the web.log shows the following traceback:

Python

File "/omd/sites/zentrale/lib/python3.12/site-packages/werkzeug/formparser.py", line 242, in parse
    return parse_func(stream, mimetype, content_length, options)
File "/omd/sites/zentrale/lib/python3.12/site-packages/werkzeug/formparser.py", line 282, in _parse_urlencoded
    raise RequestEntityTooLarge()
werkzeug.exceptions.RequestEntityTooLarge: 413 Request Entity Too Large: The data value transmitted exceeds the capacity limit.

Is the traceback in the web.log also related to the Service Discovery Problem?

What needs to be optimized so that larger requests are processed?

I also tried accepting only a single service via the GUI, but the error still occurs.

I tried also on the monitoring site:

cmk -II destinationhost

cmk -R

cmk -O 

Did also not lead to accepted services.

Somebody a idea? Many thanks in advance!

1 Like

Yes sounds like exactly same issue:

Workaround

Edit /omd/sites/<site>/lib/python3.12/site-packages/werkzeug/wrappers/request.py and raise the limits:

max_form_memory_size = 2_000_000   # 2 MB instead of 500 kB try what is working for you
max_form_parts = 10000             # instead of 1000

Greetz Bernd

1 Like

Werk #19169: Fix 500 internal server error when sending large requests

and it`s seamed to be fixed :rocket:

3 Likes