Request Entity Too Large: The data value transmitted exceeds the capacity limit - web.log

bridrod · July 21, 2023, 8:19pm

CMK version:2.2.0p6
OS version:Ubuntu v20.04.6 LTS

Error message: werkzeug.exceptions.RequestEntityTooLarge: 413 Request Entity Too Large: The data value transmitted exceeds the capacity limit.
2023-07-21 20:11:01,876 [40] [cmk.gui.wsgi.app 2452] Request finalizing failed with an error while handling an error

If I try to edit anything (create/modify/delete) under: [Notification configuration]
Setup | Events | Notification configuration
I also tried deleting the file notifications.mk, then it shows up empty list of notifications. Makes no difference because If I try to create a new notification and save, same error occurs.

I get the following error:
On the browser:
Internal Server Error

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

In /var/log/web.log

==> /opt/omd/sites/MYSITE/var/log/web.log <==
    self.session_interface.save_session(self, ctx.session, response)
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/session.py", line 263, in save_session
    expires = self.get_expiration_time(app, session)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/flask/sessions.py", line 277, in get_expiration_time
    if session.permanent:
       ^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'permanent'
2023-07-21 20:03:08,311 [30] [cmk.web.background-job 119349] Skip termination of background job (Job ID: get_remote_audit_logs, PID: 119349)
2023-07-21 20:03:22,012 [30] [cmk.web.background-job 119349] Skip termination of background job (Job ID: get_remote_audit_logs, PID: 119349)
2023-07-21 20:05:30,069 [40] [cmk.gui.wsgi.app 2452] Exception on /MYSITE/check_mk/wato.py [POST]
Traceback (most recent call last):
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/flask/app.py", line 2528, in wsgi_app
    ctx.push()
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/flask/ctx.py", line 375, in push
    self.session = session_interface.open_session(self.app, self.request)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/session.py", line 209, in open_session
    user_name, auth_type = check_auth()
                           ^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/auth.py", line 67, in check_auth
    if user_id := auth_method():
                  ^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/auth.py", line 324, in _check_auth_by_automation_credentials_in_request_values
    if (username := request.values.get("_username")) and (
                    ^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/utils.py", line 109, in __get__
    value = self.fget(obj)  # type: ignore
            ^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 456, in values
    sources.append(self.form)
                   ^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/utils.py", line 109, in __get__
    value = self.fget(obj)  # type: ignore
            ^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 436, in form
    self._load_form_data()
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/flask/wrappers.py", line 112, in _load_form_data
    super()._load_form_data()
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 274, in _load_form_data
    data = parser.parse(
           ^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/formparser.py", line 268, in parse
    return parse_func(self, stream, mimetype, content_length, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/formparser.py", line 140, in wrapper
    return f(self, stream, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/formparser.py", line 296, in _parse_multipart
    form, files = parser.parse(stream, boundary, content_length)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/formparser.py", line 463, in parse
    event = parser.next_event()
            ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/sansio/multipart.py", line 201, in next_event
    raise RequestEntityTooLarge()
werkzeug.exceptions.RequestEntityTooLarge: 413 Request Entity Too Large: The data value transmitted exceeds the capacity limit.
2023-07-21 20:05:30,074 [40] [cmk.gui.wsgi.app 2452] Request finalizing failed with an error while handling an error
Traceback (most recent call last):
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/flask/app.py", line 2528, in wsgi_app
    ctx.push()
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/flask/ctx.py", line 375, in push
    self.session = session_interface.open_session(self.app, self.request)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/session.py", line 209, in open_session
    user_name, auth_type = check_auth()
                           ^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/auth.py", line 67, in check_auth
    if user_id := auth_method():
                  ^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/auth.py", line 324, in _check_auth_by_automation_credentials_in_request_values
    if (username := request.values.get("_username")) and (
                    ^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/utils.py", line 109, in __get__
    value = self.fget(obj)  # type: ignore
            ^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 456, in values
    sources.append(self.form)
                   ^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/utils.py", line 109, in __get__
    value = self.fget(obj)  # type: ignore
            ^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 436, in form
    self._load_form_data()
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/flask/wrappers.py", line 112, in _load_form_data
    super()._load_form_data()
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/wrappers/request.py", line 274, in _load_form_data
    data = parser.parse(
           ^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/formparser.py", line 268, in parse
    return parse_func(self, stream, mimetype, content_length, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/formparser.py", line 140, in wrapper
    return f(self, stream, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/formparser.py", line 296, in _parse_multipart
    form, files = parser.parse(stream, boundary, content_length)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/formparser.py", line 463, in parse
    event = parser.next_event()
            ^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/werkzeug/sansio/multipart.py", line 201, in next_event
    raise RequestEntityTooLarge()
werkzeug.exceptions.RequestEntityTooLarge: 413 Request Entity Too Large: The data value transmitted exceeds the capacity limit.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/flask/app.py", line 1847, in finalize_request
    response = self.process_response(response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/flask/app.py", line 2344, in process_response
    self.session_interface.save_session(self, ctx.session, response)
  File "/omd/sites/MYSITE/lib/python3/cmk/gui/session.py", line 263, in save_session
    expires = self.get_expiration_time(app, session)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/MYSITE/lib/python3.11/site-packages/flask/sessions.py", line 277, in get_expiration_time
    if session.permanent:
       ^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'permanent'

gk998 · August 10, 2023, 10:15am

Try changing
if self.max_parts is not None and self._parts_decoded > self.max_parts:
in the file
/omd/sites/SITE/lib/python3.9/site-packages/werkzeug/sansio/multipart.py
to
if self.max_parts is not None and self._parts_decoded > 2000:

(default max_parts is 1000)

bridrod · August 10, 2023, 2:08pm

Thanks for looking into it. On mine it’s “/opt/omd/versions/2.2.0p6.cre/lib/python3.11/site-packages/werkzeug/sansio/multipart.py” but it did not work. I even tried as high as 50000, but same error occurred.

alsitn · August 17, 2023, 12:46pm

Hi,
RequestEntityTooLarge is Apache related.
Depending on your setup you have to edit Apache config on the affected host (master or slave) and add/modify this value:
https://httpd.apache.org/docs/2.0/mod/core.html#limitrequestbody

Usually something like this will work

<VirtualHost *:443>
   LimitRequestBody 102400
</VirtualHost>

Beware that this mod has to be done on the frontend Apache (the one Check_MK install on the VM as a reverse proxy) and not in the OMD chroot environment!

gk998 · August 17, 2023, 1:34pm

This is not apache related. Apache just throws a 500. The Entity too large is a WSGI Error coming from “Werkzeug” due to a recent change on their side. See: https://github.com/pallets/werkzeug/blob/main/CHANGES.rst#version-223

There is now a fix in cmk similar to mine: 15939 FIX Fix possible "Internal server error" on save of changes · Checkmk/checkmk@70085a2 · GitHub

bridrod · August 30, 2023, 2:48pm

Thanks for all the replies. I can confirm that on v2.2.0p8 I no longer have the issue! Something got fixed between v2.2.0p6 and v2.2.0p8 on the RAW version.

alz0 · October 20, 2023, 10:56am

I’m having this error.
I have 4 sites, 3 of them are 2.2.0p9.cre and one (recently installed) is 2.2.0p12.cre.
I’m having this error on the 2.2.0p9.cre version only.
It is a bit weird that on the oldest versions works fine but in the new one no…

Update: i have remove p12 and install p9 and I have the same issue.
To “solve” it I have unchecked the next option on “Edit site connection”

But I really want to replicate that… So this is a patch for the moment