Python Environment Variables and Errors with Script After OS Migration and Update

CMK version: 2.1.0p44.cme
OS version: RHEL 9

Looking for guidance or troubleshooting pointers. I’m not sure if I understand enough about the virtual environment setup and variables within CheckMK to identify the issue to fix it properly instead of band-aiding it.

During a recent maintenance, I migrated our production instance of Check MK from CentOS 7 to RHEL 9.
I staged the new OS and setup with version 2.1.0p28.cme and tested basic function (SSL, redirects) with a mock site ‘testcmk’.
I used omd backup which was at that version and copied it to the new server.
Restored using omd restore without issue and modified DNS entries to point to the new server.
I then installed the latest release 2.1.0p44.cme and ran omd update.
After resolving a few minor issues everything seems to have gone smoothly except for a few issues.

One of my issues I’m working on is a custom check written by a coworker in python that is no longer working.
Running the script directly is displaying a generic error in the new environment. Still works on the old one.
Googling is pointing to environment variables and virtual environment.
Error message:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/usr/bin/python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/replace-me'
  sys.base_exec_prefix = '/replace-me'
  sys.platlibdir = 'lib'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/replace-me'
  sys.exec_prefix = '/replace-me'
  sys.path = [
    '/replace-me/lib/python39.zip',
    '/replace-me/lib/python3.9',
    '/replace-me/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f96fa3a4f00 (most recent call first):
<no Python frame>

The script on both the legacy server and new one is the same.
The creator has it importing a class from a file in the same folder ~/local/lib/nagios/plugins/.
Some older posts here suggested #!/usr/bin/env python3 but that doesn’t seem to make a difference in this case.

#!/usr/bin/python3

import sys
import os
import json

path = os.path.dirname(__file__)
sys.path.append(path)
sys.path.append('./')
sys.path.append('../../../')
import Kentik as Kentik

I did notice this behavior when sudoing to the site user and running the following:

OMD[lsitename]:~$ python --version
Python 2.7.5
OMD[lsitename]:~$ echo $PYTHONPATH

OMD[lsitename]:~$ python
Python 2.7.5 (default, Nov 14 2023, 16:14:06)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

On the new server

OMD[lsitename]:~$ python --version
Python 3.9.16
OMD[lsitename]:~$ echo $PYTHONPATH

OMD[lsitename]:~$ python
<error as above>

Does this indicate something with the CheckMK installation or perhaps something I missed with prep/staging the new server?

As always, thanks for you time and any assistance.

Sincerely,
Scotsie

Hi Scotsie

silly question, but did you install the RHEL9 version of Checkmk after the migration?

Regards, Lars

Thanks for the response and not silly at all. Yes, the RPM package installed on the new destination server was check-mk-managed-2.1.0p28-el9-38.x86_64.rpm and then updated using check-mk-managed-2.1.0p44-el9-38.x86_64.rpm.

Just an update. For the short term, I went ahead and caved using the workaround of putting the explicit path for python in the file which allows it to run as expected.

#!/omd/sites/lsitename/bin/python3

I do feel this is a bandaid though unless I’m misunderstanding how the environment setup should work.

Sincerely,
Scotsie