CheckMK-Docker after Update to 2.2.0 => Error 500

bastimm13 · May 25, 2023, 9:46am

CMK version: 2.2.0 Docker -Raw
OS version: Ubuntu 22.04

Error message: “Internal Server Error 500” on GUI

Hello,
I am new to checkmk… yesterday I updated my running checkmk to 2.2.0
=>
docker stop checkmk
docker rm checkmk
docker container run -dit -p 8095:5000 -p 8000:8000 --tmpfs /opt/omd/sites/cmk/tmp:uid=1000,gid=1000 -v monitoring:/omd/sites --name checkmk -v /etc/localtime:/etc/localtime:ro --restart always checkmk/check-mk-raw:2.2.0
On my server there is a nginx forward to port 8095 and this worked until this update…
If I open domain.de/cmk/check_mk/ => I get a Error 500…

Can somebody help me to find this error?
Thanks in advance

Bacon-O · June 8, 2023, 2:06pm

I have ran into the same issue, after upgrading my Checkmk docker to 2.2.0p2.cre the web GUI is no longer accessible and I am receiving a HTTP Error 500.

I check the web.log in the docker container /opt/omd/sites/cmk/var/log/web.log and I get the following entry

2023-06-08 09:01:31,204 [40] [cmk.gui.wsgi.app 2208] Exception on /cmk/check_mk/ [GET]
Traceback (most recent call last):
  File "/omd/sites/cmk/lib/python3/cmk/gui/hooks.py", line 74, in call
    hook.handler(*args)
  File "/omd/sites/cmk/lib/python3/cmk/gui/watolib/search.py", line 521, in _launch_requests_processing_background
    job.start(_process_update_requests_background)
  File "/omd/sites/cmk/lib/python3/cmk/gui/watolib/search.py", line 579, in start
    return super().start(target) if redis_enabled() else None
           ^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3/cmk/gui/background_job.py", line 649, in start
    self._start(target)
  File "/omd/sites/cmk/lib/python3/cmk/gui/background_job.py", line 654, in _start
    if self.is_active():
       ^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3/cmk/gui/background_job.py", line 529, in is_active
    job_status = self.get_status()
                 ^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3/cmk/gui/background_job.py", line 622, in get_status
    status = self._jobstatus_store.read()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3/cmk/gui/background_job.py", line 826, in read
    data: JobStatusSpec = JobStatusSpec.parse_obj(raw_status_spec)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3.11/site-packages/pydantic/main.py", line 526, in parse_obj
    return cls(**obj)
           ^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3.11/site-packages/pydantic/main.py", line 342, in __init__
    raise validation_error
pydantic.error_wrappers.ValidationError: 2 validation errors for JobStatusSpec
loginfo
  field required (type=value_error.missing)
is_active
  field required (type=value_error.missing)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/omd/sites/cmk/lib/python3.11/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3.11/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3.11/site-packages/flask/app.py", line 1818, in full_dispatch_request
    rv = self.preprocess_request()
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3.11/site-packages/flask/app.py", line 2309, in preprocess_request
    rv = self.ensure_sync(before_func)()
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3/cmk/gui/wsgi/blueprints/checkmk.py", line 56, in before_request
    hooks.call("request-start")
  File "/omd/sites/cmk/lib/python3/cmk/gui/hooks.py", line 78, in call
    raise MKGeneralException(msg) from e
cmk.utils.exceptions.MKGeneralException: Traceback (most recent call last):
  File "/omd/sites/cmk/lib/python3/cmk/gui/hooks.py", line 74, in call
    hook.handler(*args)
  File "/omd/sites/cmk/lib/python3/cmk/gui/watolib/search.py", line 521, in _launch_requests_processing_background
    job.start(_process_update_requests_background)
  File "/omd/sites/cmk/lib/python3/cmk/gui/watolib/search.py", line 579, in start
    return super().start(target) if redis_enabled() else None
           ^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3/cmk/gui/background_job.py", line 649, in start
    self._start(target)
  File "/omd/sites/cmk/lib/python3/cmk/gui/background_job.py", line 654, in _start
    if self.is_active():
       ^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3/cmk/gui/background_job.py", line 529, in is_active
    job_status = self.get_status()
                 ^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3/cmk/gui/background_job.py", line 622, in get_status
    status = self._jobstatus_store.read()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3/cmk/gui/background_job.py", line 826, in read
    data: JobStatusSpec = JobStatusSpec.parse_obj(raw_status_spec)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3.11/site-packages/pydantic/main.py", line 526, in parse_obj
    return cls(**obj)
           ^^^^^^^^^^
  File "/omd/sites/cmk/lib/python3.11/site-packages/pydantic/main.py", line 342, in __init__
    raise validation_error
pydantic.error_wrappers.ValidationError: 2 validation errors for JobStatusSpec
loginfo
  field required (type=value_error.missing)
is_active
  field required (type=value_error.missing)

Checking the status through the command line it looks like everything is up as well.

OMD[cmk]:~$ omd status
agent-receiver: running
mkeventd:       running
rrdcached:      running
npcd:           running
nagios:         running
apache:         running
redis:          running
crontab:        running
-----------------------
Overall state:  running
OMD[cmk]:~$

I am still getting email notifications for events/alarms so it appears like the back end is working, just the web gui seems to be having issues.

Edit: I was successful in launching a new clean docker image with 2.2.0p2.cre so this appears to only affect upgrades. I upgraded the docker image from 2.1.0p28 to 2.2.0p2

Bacon-O · June 8, 2023, 3:57pm

Looks like I was able to get back into the web gui by going into the docker image and running

OMD[cmk]:~$ cmk-update-config -vvv --debug
parsed arguments: Namespace(debug=True, verbose=3, conflict=<ConflictMode.ASK: 'ask'>)
ATTENTION
  Some steps may take a long time depending on your installation.
  Please be patient.

Verifying Checkmk configuration...
 01/04 Rulesets...
 02/04 UI extensions...
 03/04 Agent based plugins...
 04/04 Deprecated .mk configuration of plugins...
Done (success)
...
...
...
 16/17 Update backup config...
 17/17 Removing deprecated allocated_ports file...
Done (success)
OMD[cmk]:~$

After that the web GUI came back. Had to re-run some discoveries on host but eventually everything returned back to normal.

bastimm13 · June 15, 2023, 8:38am

Hi,
just tested it again…
I get this error:

Bacon-O · June 15, 2023, 3:29pm

I would not considered myself an expect on Checkmk, but I suspect the issue is that you are still the root user inside the docker image and not the site user. Try switching to the site user and running the cmk-update-config command again.

For me my site is called “cmk”

root@checkmk-ix-chart-79587f547c-f8dfp:/# whoami
root
root@checkmk-ix-chart-79587f547c-f8dfp:/# cmk-update-config -vvv --debug
bash: cmk-update-config: command not found
root@checkmk-ix-chart-79587f547c-f8dfp:/# omd sites
SITE             VERSION          COMMENTS
cmk              2.2.0p3.cre      default version
root@checkmk-ix-chart-79587f547c-f8dfp:/# su cmk
OMD[cmk]:~$ which cmk-update-config
/omd/sites/cmk/bin/cmk-update-config
OMD[cmk]:~$

bastimm13 · June 16, 2023, 4:03am

THX , didn’t know that I had to switch to cmk user…

now I got this errror:

perhaps somebody can help?

THANKS in advance!!

bastimm13 · June 16, 2023, 4:15am

after pressing “c” I got this error:

bastimm13 · June 16, 2023, 4:38am

Perhaps it is the best to delete all my settings/Servers and restart with a clean install?

Matthew1 · August 3, 2023, 6:05am

Did you solve these issues? I have the same I would not do a clean install of my server because I have a lot of data into.