Troubleshoot internal server error 500

**CMK version:checkmk 2.2.0p5
**OS version:Debian 10

Error message:
Hello,
can anyone help me please to troubleshoot this error?
All of the sudden this error popped up, unfortunately can’t explain why.

When I specify cmk --debug -vvn monitoring_site I only get this error:

Monitoring Alerts via Teams I’m able to get but this is how it looks like when accessing the webinterface:

Plugins should not be the problem I already checked that.

Looks like that you have no DNS Record for this name.

Rg, Christian

I tried with the name of the monitoring instance than I get this error.
When using the name of the monitoring server (full name) than I get the debug but it doesn’t provide me some error why the webinterface shows me Error 500.

Hi @Matthew1,

can you please provide the output of omd status SITENAME ?

Please also check the log of your site:
/omd/sites/SITENAME/var/log/web.log

Did you updated the site before the error occur?
Any errors during the last update?

Thanks in advance!
Norm

Hello @Norm,

Here the first output of omd status sitename and without:

The web.log under your specified path is empty…
But in the web.log.1 were some error logs, I wasn’t able the code so I attach a screenshot from the errors:

One of the error’s is the msteams.py plugin (I deleted that already with the mkp command) and the other error is the one from screenshot above (line 66) and this line ends with load_web$ $ad_web_plugins

I don’t know if this helps but I hope, don’t know yet how to bring back the checkmk webinterface running.

I made an update long time ago there were some problems but also with the teams plugin I was able to solve this with this command: cmk-update-config -v

Appreaciate your help!

When I put into this command: cmk-update-config -vvv
I get this error code:

cmk-update-config -vvv 
Traceback (most recent call last):
  File "/omd/sites/sitename/bin/cmk-update-config", line 8, in <module>
    from cmk.update_config.main import main
  File "/omd/sites/sitename/lib/python3/cmk/update_config/main.py", line 33, in <module>
    from cmk.gui import main_modules
  File "/omd/sites/sitename/lib/python3/cmk/gui/main_modules.py", line 24, in <module>
    import cmk.gui.plugins.main_modules  # pylint: disable=no-name-in-module,unused-import
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/sitename/lib/python3/cmk/gui/plugins/main_modules/__init__.py", line 8, in <module>
    load_plugins(__file__, __package__)
  File "/omd/sites/sitename/lib/python3/cmk/utils/plugin_loader.py", line 124, in load_plugins
    importlib.import_module(f"{package_name}.{plugin_name}")
  File "/omd/sites/sitename/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/sitename/lib/python3/cmk/gui/plugins/main_modules/cre.py", line 13, in <module>
    import cmk.gui.dashboard
  File "/omd/sites/sitename/lib/python3/cmk/gui/dashboard/__init__.py", line 23, in <module>
    from .cre_dashboards import register_builtin_dashboards
  File "/omd/sites/sitename/lib/python3/cmk/gui/dashboard/cre_dashboards.py", line 13, in <module>
    from .dashlet import LinkedViewDashletConfig, StatsDashletConfig, ViewDashletConfig
  File "/omd/sites/sitename/lib/python3/cmk/gui/dashboard/dashlet/__init__.py", line 16, in <module>
    from .dashlets import (
  File "/omd/sites/sitename/lib/python3/cmk/gui/dashboard/dashlet/dashlets/__init__.py", line 8, in <module>
    from .failed_notifications import FailedNotificationsDashlet
  File "/omd/sites/sitename/lib/python3/cmk/gui/dashboard/dashlet/dashlets/failed_notifications.py", line 6, in <module>
    import cmk.gui.notifications as notifications
  File "/omd/sites/sitename/lib/python3/cmk/gui/notifications.py", line 42, in <module>
    from cmk.gui.wato.pages.user_profile.async_replication import user_profile_async_replication_page
  File "/omd/sites/sitename/lib/python3/cmk/gui/wato/__init__.py", line 66, in <module>
    import cmk.gui.backup as backup
  File "/omd/sites/sitename/lib/python3/cmk/gui/backup.py", line 32, in <module>
    from cmk.utils.backup.targets.aws_s3_bucket import S3Bucket, S3Target
  File "/omd/sites/sitename/lib/python3/cmk/utils/backup/targets/aws_s3_bucket.py", line 10, in <module>
    import boto3  # type: ignore[import]
    ^^^^^^^^^^^^
  File "/omd/sites/sitename/lib/python3.11/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/omd/sites/sitename/lib/python3.11/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/omd/sites/sitename/lib/python3.11/site-packages/botocore/session.py", line 30, in <module>
    import botocore.credentials
  File "/omd/sites/sitename/lib/python3.11/site-packages/botocore/credentials.py", line 34, in <module>
    from botocore.config import Config
  File "/omd/sites/sitename/lib/python3.11/site-packages/botocore/config.py", line 16, in <module>
    from botocore.endpoint import DEFAULT_TIMEOUT, MAX_POOL_CONNECTIONS
  File "/omd/sites/sitename/lib/python3.11/site-packages/botocore/endpoint.py", line 22, in <module>
    from botocore.awsrequest import create_request_object
  File "/omd/sites/sitename/lib/python3.11/site-packages/botocore/awsrequest.py", line 26, in <module>
    import botocore.utils
  File "/omd/sites/sitename/lib/python3.11/site-packages/botocore/utils.py", line 33, in <module>
    import botocore.httpsession
  File "/omd/sites/sitename/lib/python3.11/site-packages/botocore/httpsession.py", line 9, in <module>
    from urllib3.util.ssl_ import (
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/omd/sites/sitename/local/lib/python3/urllib3/util/ssl_.py) 

Has anyone an Idea I’m stuck with this problem… :sleepy:

Hallo,
does a new empty site works?
Does DNS is working on the shell as user root and as cmk user?
Ralf

Hello Ralf,

thank you for your advices.
A new empty site works!

Can you please tell me how I can check if it works or not?
The DNS error above was in omd.

try nslookup as root and site user

Got this error when doing a nslookup on the monitoring server: nslookup monitoring_server_name

GOT SERVFAIL reply from X.X.X.X trying next server
**server can’t find monitoring_server_name: SERVFAIL

On the other server nslookup monitoring_server_Name works witouth problems.

Personally I don’t know if this has to do with a DNS error because also the new site would have the same problems? right?

I think that i know what the problem is.
Is it possible that you installed one of my mkp for monitoring management interfaces with redfish?
Please have look inside “~/local/lib/python3/” if there is a folder with name “urllib3-2…” then this is the problem with the botocore library.
You can solve this with pip3 install 'urllib3<2'

I don’t know a person who left the company has let this server so… :roll_eyes:

In python3 we have this folders:

I think also that it has something to do with the python version.

When I enter this command(pip3 install ‘urllib3<2’) I get this error:

Thank you all for your help! :pray:

Then you have to write pip3 install 'urllib3<2' --upgrade
If this runs without error you can remove the folder “urllib3-2.0.4.dist-info”.

1 Like

Strange is that you only have some folders there. I don’t know any plugin that is only requesting a newer version of requests or urllib but nothing else.
It looks like someone has done a “pip3 install requests” on the system.

Thank you very much this was the solution!! :pray: :star_struck:

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.