Error when rescan devices

altvater · March 9, 2026, 10:58am

CMK version:

CEE 2.3.0.p43

OS version:

Debian 12

Error message:

2026-03-09 11:51:15,113 [40] [cmk.gui.wsgi.app 2403] Exception on /ukm/check_mk/ajax_service_discovery.py [POST]
Traceback (most recent call last):
  File "/omd/sites/ukm/lib/python3/cmk/gui/wsgi/applications/checkmk.py", line 188, in _process_request
    resp = page_handler()
           ^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/wsgi/applications/utils.py", line 104, in _call_auth
    _localize_request()
  File "/omd/sites/ukm/lib/python3/cmk/gui/wsgi/applications/utils.py", line 204, in _localize_request
    if user_language := request.get_ascii_input("lang", user.language):
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/http.py", line 462, in get_ascii_input
    value = self.get_str_input(varname, deflt)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/http.py", line 371, in get_str_input
    return self.var(varname, deflt)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/http.py", line 157, in var
    return super().var(name, default)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/http.py", line 219, in var
    values = self.values.getlist(name)  # type: ignore[attr-defined]
             ^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/utils.py", line 100, in __get__
    value = self.fget(obj)  # type: ignore
            ^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/wrappers/request.py", line 465, in values
    sources.append(self.form)
                   ^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/utils.py", line 100, in __get__
    value = self.fget(obj)  # type: ignore
            ^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/wrappers/request.py", line 445, in form
    self._load_form_data()
  File "/omd/sites/ukm/lib/python3.12/site-packages/flask/wrappers.py", line 115, in _load_form_data
    super()._load_form_data()
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/wrappers/request.py", line 271, in _load_form_data
    data = parser.parse(
           ^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/formparser.py", line 242, in parse
    return parse_func(stream, mimetype, content_length, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/formparser.py", line 282, in _parse_urlencoded
    raise RequestEntityTooLarge()
werkzeug.exceptions.RequestEntityTooLarge: 413 Request Entity Too Large: The data value transmitted exceeds the capacity limit.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/omd/sites/ukm/lib/python3.12/site-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/flask/app.py", line 873, in full_dispatch_request
    return self.finalize_request(rv)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/flask/app.py", line 892, in finalize_request
    response = self.make_response(rv)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/flask/app.py", line 1188, in make_response
    rv = self.response_class.force_type(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/wrappers/response.py", line 237, in force_type
    response = Response(*run_wsgi_app(response, environ))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/test.py", line 1264, in run_wsgi_app
    app_rv = app(environ, start_response)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/wsgi/applications/utils.py", line 53, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/wsgi/middleware.py", line 63, in wsgi_app
    return self.app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/wsgi/applications/checkmk.py", line 167, in wsgi_app
    return _process_request(environ, start_response, debug=self.debug)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/wsgi/applications/checkmk.py", line 229, in _process_request
    resp = _render_exception(e, title=_("Request too large"))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/wsgi/applications/checkmk.py", line 116, in _render_exception
    if plain_error():
       ^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/wsgi/applications/utils.py", line 123, in plain_error
    return request.has_var("_plain_error")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/http.py", line 136, in has_var
    return super().has_var(varname)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3/cmk/gui/http.py", line 230, in has_var
    return varname in self.values  # type: ignore[attr-defined]
                      ^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/utils.py", line 100, in __get__
    value = self.fget(obj)  # type: ignore
            ^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/wrappers/request.py", line 465, in values
    sources.append(self.form)
                   ^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/utils.py", line 100, in __get__
    value = self.fget(obj)  # type: ignore
            ^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/wrappers/request.py", line 445, in form
    self._load_form_data()
  File "/omd/sites/ukm/lib/python3.12/site-packages/flask/wrappers.py", line 115, in _load_form_data
    super()._load_form_data()
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/wrappers/request.py", line 271, in _load_form_data
    data = parser.parse(
           ^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/formparser.py", line 242, in parse
    return parse_func(stream, mimetype, content_length, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/ukm/lib/python3.12/site-packages/werkzeug/formparser.py", line 282, in _parse_urlencoded
    raise RequestEntityTooLarge()
werkzeug.exceptions.RequestEntityTooLarge: 413 Request Entity Too Large: The data value transmitted exceeds the capacity limit.

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

altvater · March 9, 2026, 11:04am

This error fills up web.log every second. Only occurred after updating to 2.3.0.p43.

BH2005 · March 9, 2026, 11:35am

Hi Rainer,

we’ve seen this exact error pattern before — it’s a recurring issue between Checkmk and the werkzeug library, where a limit on the number of form parts in a single POST request gets exceeded. This typically happens on hosts with a large number of services (e.g. big switches, ESXi hosts, or SNMP devices with many OIDs).

One thing that might help as an immediate workaround: try raising the max_parts limit directly in the werkzeug library:

bash

vi /omd/sites/ukm/lib/python3.12/site-packages/werkzeug/sansio/multipart.py

Find this line:

python

if self.max_parts is not None and self._parts_decoded > self.max_parts:

and change the comparison value to something like 10000 instead of the default 1000.

Note: this will be overwritten on the next Checkmk update, so it’s a temporary fix only.

Before trying this though — a few questions that would help narrow things down:

Is this error triggered on specific hosts only, or does it happen on every rescan across the board?
Do you have Periodic Service Discovery enabled? If so, that could explain why the error floods web.log every second — something is repeatedly triggering the rescan automatically.
How many services does the affected host approximately have? The more services, the larger the POST payload sent to ajax_service_discovery.py.
Did you check whether a specific host type (e.g. SNMP device, ESXi, Windows agent) is involved, or is it mixed?

Knowing this would help confirm whether it’s purely a form-size issue or whether something else changed in p43 for your setup.

Sources: Werkzeug Docs – max_form_parts · werkzeug commit – limit multipart parts · Checkmk Werk #15939 · Checkmk Forum – same error

Greetz Bernd

altvater · March 9, 2026, 12:36pm

Hi Bernd,

Thanks for your help

The error currently only occurs on one host, a Cisco Nexus 9K. It currently has around 400 services, as do our other Nexus devices (we have 20 of them). Periodic Service Discovery is enabled, and we check the hosts exclusively via SNMP.

BH2005 · March 9, 2026, 12:53pm

I know him so well … … this BigBlock has given me also a lot of trouble in the past ..

So I thing for this host you have to ajust the parameters to more granular

BH2005 · March 9, 2026, 1:05pm

With the 9000 series, I often found that not only the size of the entries but also the long query time was a problem.

I solved this by running it twice a day via cron:

#!/usr/bin/env bash
# =============================================================================
# checkmk_discovery_reload.sh
# 
# Führt cmk -iiv aus → wartet bis wirklich fertig → dann cmk -R
# =============================================================================

set -u
set -e

# ──────────────────────────────────────────────────────────────────────────────
# Konfiguration – hier anpassen
# ──────────────────────────────────────────────────────────────────────────────

# Hostname (leer lassen = alle Hosts)
HOST=""

# oder explizit einen oder mehrere Hosts
# HOST="server01.example.com"
# HOST="switch42 server01 server02"

# Wartezeit nach cmk -iiv (in Sekunden) – nur wenn du eine feste Pause willst
# 0 = keine feste Wartezeit → wartet nur bis cmk -iiv fertig ist
FIXED_WAIT_AFTER_DISCOVERY=0          # 600 = 10 Minuten

# Log-Datei (optional)
LOGFILE="/tmp/cmk_discovery_reload_$(date +%Y%m%d_%H%M%S).log"

# ──────────────────────────────────────────────────────────────────────────────
# Funktionen
# ──────────────────────────────────────────────────────────────────────────────

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOGFILE"
}

run_cmk() {
    local cmd="$1"
    log "Starte: $cmd"
    echo "──────────────────────────────────────────────"
    echo "→ $cmd"
    echo "──────────────────────────────────────────────"
    
    if [[ -n "$HOST" ]]; then
        # mit Host(s)
        /omd/sites/*/bin/cmk $cmd $HOST 2>&1 | tee -a "$LOGFILE"
    else
        # alle Hosts
        /omd/sites/*/bin/cmk $cmd 2>&1 | tee -a "$LOGFILE"
    fi
    
    local rc=$?
    if (( rc == 0 )); then
        log "→ erfolgreich beendet (rc=$rc)"
    else
        log "→ FEHLER! Return-Code $rc"
        exit $rc
    fi
    echo ""
}

# ──────────────────────────────────────────────────────────────────────────────
# Hauptprogramm
# ──────────────────────────────────────────────────────────────────────────────

log "Skript gestartet – Checkmk Discovery + Reload"

# 1. Discovery durchführen
if [[ -n "$HOST" ]]; then
    run_cmk "-iiv $HOST"
else
    run_cmk "-iiv"
fi

# 2. Entweder feste Wartezeit ODER einfach abwarten (da cmk eh blockiert)
if (( FIXED_WAIT_AFTER_DISCOVERY > 0 )); then
    log "Warte ${FIXED_WAIT_AFTER_DISCOVERY} Sekunden (10 Minuten = 600)"
    sleep $FIXED_WAIT_AFTER_DISCOVERY
else
    log "Kein fixed sleep – cmk -iiv war schon blockierend → direkt weiter"
fi

# 3. Konfiguration neu laden
run_cmk "-R"

log "Fertig! Alle Schritte abgeschlossen."
echo "Log-Datei: $LOGFILE"

altvater · March 9, 2026, 1:06pm

Looks like the corresponding values are set in

/omd/sites/<site>/lib/python3.12/site-packages/werkzeug/wrappers/request.py

    #: the maximum form field size.  This is forwarded to the form data
    #: parsing function (:func:`parse_form_data`).  When set and the
    #: :attr:`form` or :attr:`files` attribute is accessed and the
    #: data in memory for post data is longer than the specified value a
    #: :exc:`~werkzeug.exceptions.RequestEntityTooLarge` exception is raised.
    #:
    #: .. versionchanged:: 3.1
    #:     Defaults to 500kB instead of unlimited.
    #:
    #: .. versionadded:: 0.5
    max_form_memory_size: int | None = 500_000

    #: The maximum number of multipart parts to parse, passed to
    #: :attr:`form_data_parser_class`. Parsing form data with more than this
    #: many parts will raise :exc:`~.RequestEntityTooLarge`.
    #:
    #: .. versionadded:: 2.2.3
    max_form_parts = 1000

After changing


max_form_memory_size = 2_000_000

and


max_form_parts = 10000

it now works again.

altvater · March 9, 2026, 1:09pm

Thanks for the script. I’ll check whether it would be sufficient for us to just query the boxes regularly. So far, however, we haven’t had any problems with the devices at all.

BH2005 · March 9, 2026, 1:16pm

Root cause: werkzeug 3.1.0 breaking change (released October 31, 2024)

werkzeug 3.1.0 was released on October 31, 2024 and introduced two new hard limits on form data parsing as potentially breaking changes. GitHub

The two relevant changes:

Request.max_form_memory_size now defaults to 500 kB instead of unlimited. Non-file form fields exceeding this size will raise a RequestEntityTooLarge error. GitHub
max_form_parts stops reading request data if more than 1,000 parts are sent in multipart form data — a protection against a very large number of very small parts. Werkzeug

Both limits were introduced intentionally as DoS protection — not as a bug — which is why they are unlikely to be reverted upstream.

Why it hits Checkmk during Rescan: When rescanning hosts with many services, the browser submits a large multipart POST form to the Checkmk web interface. Depending on the size of the host inventory, this can easily exceed 1,000 form parts and/or 500 kB of non-file field data, triggering HTTP 413 on both counts.

Workaround (as described by @altvater above):

Edit /omd/sites/<site>/lib/python3.12/site-packages/werkzeug/wrappers/request.py and raise the limits:

python

max_form_memory_size = 2_000_000   # 2 MB instead of 500 kB
max_form_parts = 10000             # instead of 1000

Note: This file is part of Checkmk’s bundled Python environment and will be overwritten on every omd update. The workaround must be re-applied after each Checkmk update.

Sources:

werkzeug 3.1.0 release notes: Release 3.1.0 · pallets/werkzeug · GitHub
werkzeug changelog: Changes — Werkzeug Documentation (3.1.x)
werkzeug request data docs: Dealing with Request Data — Werkzeug Documentation (3.1.x)

BH2005 · March 11, 2026, 11:46am

it`s seamed to be fixed

davix · March 11, 2026, 12:03pm

Indeed this workaround worked for us as well. We had to crank it up to

max_form_memory_size = 5_000_000

Many thanks. Should it still work with Werk #19169? Or is the Werk #19169 limited to 2_000_000?

BH2005 · March 11, 2026, 12:06pm

Just with the knowledge that this issue is fixed in the next CMK release … and keep this in your own knowledge base

Greetz Bernd

davix · July 13, 2026, 9:34am

We are now at CheckMK 2.3.0p48 since the problem should be solved with Werk #19169 which is 2.3.0p44, should I set /omd/sites//lib/python3.12/site-packages/werkzeug/wrappers/request.py back to it defaults?

So:

max_form_memory_size = 500_000

max_form_parts = 1000

Many thanks in advance!

Best regards
David

BH2005 · July 13, 2026, 9:59am

I would say test it … I`m just have 2.4 and 2.5 Sites actually

BH2005 · July 13, 2026, 10:02am

I would say test it … I`m just have 2.4 and 2.5 Sites actually

and has implement also another way for that:

see GitHub - bh2005/autopatch-system: Web-basiertes Linux-Patch-Management — Ansible, CVE-Feed, Lynis Security Audit, Staged Rollouts · FastAPI + Vue 3 + Docker · GitHub

davix · July 14, 2026, 7:44am

So I will just try it out. An thanks for the nice tool!