Apache MPM prefork problem

Troubleshooting
,
1

CheckMK version: free 2.1.0p30
OS version: opensuse, kernel 6.4

Hi Community,

I started with a Commodore 64 in the 80’s but I never had this before…

I migrated from one hardware server to another one, keeping all the software packages and versions EXACTLY as before.

On my previous server check mk worked as expected, but on the new server I had some problems so I completely reinstalled checkmk.

But now I receive this error message…

[Wed Jul 26 19:26:14.104096 2023] [mpm_event:notice] [pid 3146492:tid 140531921537280] AH00489: Apache/2.4.57 (Linux/SUSE) mod_wsgi/4.9.4 Python/3.9 mod_fcgid/2.3.9 configured -- resuming normal operations
[Wed Jul 26 19:26:14.104112 2023] [core:notice] [pid 3146492:tid 140531921537280] AH00094: Command line: '/usr/sbin/httpd -f /omd/sites/monitoring/etc/apache/apache.conf'
[Wed Jul 26 19:26:16.106198 2023] [mpm_event:error] [pid 3146492:tid 140531921537280] AH10159: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting
[Wed Jul 26 19:27:02.244817 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114] mod_wsgi (pid=3146593): Exception occurred processing WSGI script '/omd/sites/monitoring/share/check_mk/web/app/index.wsgi'.
[Wed Jul 26 19:27:02.245287 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114] Traceback (most recent call last):
[Wed Jul 26 19:27:02.245313 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]   File "/omd/sites/monitoring/lib/python3/cmk/gui/wsgi/profiling.py", line 74, in __call__
[Wed Jul 26 19:27:02.245316 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]     return app(environ, start_response)
[Wed Jul 26 19:27:02.245320 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]   File "/omd/sites/monitoring/lib/python3/cmk/gui/wsgi/middleware.py", line 18, in __call__
[Wed Jul 26 19:27:02.245323 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]     response = self.app(environ, start_response)
[Wed Jul 26 19:27:02.245326 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]   File "/omd/sites/monitoring/lib/python3/cmk/gui/wsgi/middleware.py", line 32, in _add_apache_env
[Wed Jul 26 19:27:02.245328 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]     return app(environ, start_response)
[Wed Jul 26 19:27:02.245330 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]   File "/omd/sites/monitoring/lib/python3/cmk/gui/wsgi/routing.py", line 91, in router
[Wed Jul 26 19:27:02.245332 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]     return endpoint(environ, start_response)
[Wed Jul 26 19:27:02.245335 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]   File "/omd/sites/monitoring/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 187, in __call__
[Wed Jul 26 19:27:02.245337 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]     return self.app(environ, start_response)
[Wed Jul 26 19:27:02.245340 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]   File "/omd/sites/monitoring/lib/python3/cmk/gui/wsgi/applications/checkmk.py", line 196, in __call__
[Wed Jul 26 19:27:02.245342 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]     req = http.Request(environ)
[Wed Jul 26 19:27:02.245345 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]   File "/omd/sites/monitoring/lib/python3/cmk/gui/http.py", line 249, in __init__
[Wed Jul 26 19:27:02.245347 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]     self._verify_not_using_threaded_mpm()
[Wed Jul 26 19:27:02.245349 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]   File "/omd/sites/monitoring/lib/python3/cmk/gui/http.py", line 253, in _verify_not_using_threaded_mpm
[Wed Jul 26 19:27:02.245351 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114]     raise MKGeneralException(
[Wed Jul 26 19:27:02.245364 2023] [wsgi:error] [pid 3146593:tid 140531657070336] [client 127.0.0.1:45114] cmk.utils.exceptions.MKGeneralException: You are trying to Checkmk together with a threaded Apache multiprocessing module (MPM). Check_MK is only working with the prefork module. Please change the MPM module to make Check_MK work.

I think I have to take care of this entry:

“cmk.utils.exceptions.MKGeneralException: You are trying to Checkmk together with a threaded Apache multiprocessing module (MPM). Check_MK is only working with the prefork module. Please change the MPM module to make Check_MK work.”

  1. Apache installation (everything) was not touched at all - absolutely the same on the old and new machine
  2. The file /omd/sites/monitoring/lib/apache/modules/mod_mpm_prefork.so does not exist in the rpm repositories, so I commented in the apache.conf file

This is the apache.conf contents

###############################################################################
# Apache configuration file for OMD site specific webserver daemon
#
# This file is only used, if you have set WEBSERVER to 'own' (which 
# is the default). It is the configuration for a site-specific additional
# Apache server process, which is just listening on 127.0.0.1 and is
# called by the system-Apache via mod_proxy.
#
# Advantages are: No root-permissions are needed for administrating this
# Apache and priviledges are better separated. Also each OMD site has its
# own Apache access and error logs below ~/var/log/apache.
###############################################################################

# OMD OMD OMD OMD OMD OMD OMD OMD OMD OMD OMD OMD  ----------------------------
# Better do not change anything in this block or you will have to
# resolve conflicts in omd cp/mv/upate:

ServerRoot "/omd/sites/monitoring"
PidFile /omd/sites/monitoring/tmp/apache/run/apache.pid
DocumentRoot "/omd/sites/monitoring/var/www"

# Include file where TCP port number is configured. That file
# is automatically created and changed by 'omd config' if you
# change APACHE_TCP_PORT.
Include /omd/sites/monitoring/etc/apache/listen-port.conf

# User/Group to run the webserver with
User monitoring
Group monitoring

# ServerName is set in listen-port.conf exactly to the Host:Port that the
# site apache is listening on. This is also the Host:Port the reverse proxy
# is working with.
# The site apache (backend server) should always report back with this
# Host:Port combination to the system apache (reverse proxy). This enables the
# system apache, which is configured using proxy-port.conf, to translate the
# URLs reported by the site apache into the public URLs of the reverse proxy.
UseCanonicalName On

# ---------------------------------------------------------------
# 
# Here you can configure and tune Apache
# 

ServerAdmin nobody@localhost

Timeout 120
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15

DirectoryIndex index.html index.php index.html.var
AccessFileName .htaccess
ServerTokens OS
HostnameLookups Off
ServerSignature Off


##
## Server-Pool Size Regulation
## 

# StartServers:        number of server processes to start
# MinSpareServers:     minimum number of server processes which are kept spare
# MaxSpareServers:     maximum number of server processes which are kept spare
# ServerLimit:         Upper limit on configurable number of processes
#                      (maximum value for MaxClients for the lifetime of the server)
# MaxClients:          Maximum number of connections that will be processed simultaneously
#                      (maximum number of server processes allowed to start)
# MaxRequestsPerChild: maximum number of requests a server process serves
StartServers         1
#MinSpareServers      1
#MaxSpareServers      5
ServerLimit          50
MaxClients           50
MaxRequestsPerChild  4000

###############################################################################
# Load Modules
###############################################################################
# Ugly IfModules! But they seem to be needed because some distributions have
# modules compiled in. An example is the log_config module which is
# compiled in on ubuntu/debian and a module on sles/centos.

# Needs to be loaded first on some platforms to have the <IfVersion> statement
# available. On newer ones, the module is already available.
<IfModule !version_module>
    LoadModule version_module /omd/sites/monitoring/lib/apache/modules/mod_version.so
</IfModule>

# SLES12 has apache 2.4 but the module does not need to be loaded
    #<IfModule !mpm_prefork_module>
    #    LoadModule mpm_prefork_module /omd/sites/monitoring/lib/apache/modules/mod_mpm_prefork.so
    #</IfModule>

# Now load the modules needed for all versions
<IfModule !mod_log_config.c>
    LoadModule log_config_module /omd/sites/monitoring/lib/apache/modules/mod_log_config.so
</IfModule>
LoadModule auth_basic_module /omd/sites/monitoring/lib/apache/modules/mod_auth_basic.so
LoadModule authn_file_module /omd/sites/monitoring/lib/apache/modules/mod_authn_file.so
LoadModule authz_host_module /omd/sites/monitoring/lib/apache/modules/mod_authz_host.so
LoadModule authz_user_module /omd/sites/monitoring/lib/apache/modules/mod_authz_user.so
LoadModule autoindex_module /omd/sites/monitoring/lib/apache/modules/mod_autoindex.so
LoadModule env_module /omd/sites/monitoring/lib/apache/modules/mod_env.so
LoadModule expires_module /omd/sites/monitoring/lib/apache/modules/mod_expires.so
LoadModule deflate_module /omd/sites/monitoring/lib/apache/modules/mod_deflate.so
LoadModule headers_module /omd/sites/monitoring/lib/apache/modules/mod_headers.so
LoadModule setenvif_module /omd/sites/monitoring/lib/apache/modules/mod_setenvif.so
LoadModule mime_magic_module /omd/sites/monitoring/lib/apache/modules/mod_mime_magic.so
LoadModule mime_module /omd/sites/monitoring/lib/apache/modules/mod_mime.so
LoadModule negotiation_module /omd/sites/monitoring/lib/apache/modules/mod_negotiation.so
LoadModule dir_module /omd/sites/monitoring/lib/apache/modules/mod_dir.so
LoadModule alias_module /omd/sites/monitoring/lib/apache/modules/mod_alias.so
LoadModule rewrite_module /omd/sites/monitoring/lib/apache/modules/mod_rewrite.so
LoadModule cgi_module /omd/sites/monitoring/lib/apache/modules/mod_cgi.so
LoadModule status_module /omd/sites/monitoring/lib/apache/modules/mod_status.so

<Directory />
    Options FollowSymLinks
    AllowOverride None
</Directory>

#
# Configure mod_status (if available)
#
<IfModule mod_status.c>
    <Location /server-status>
        SetHandler server-status
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1 ::1
    </Location>

    # Keep track of extended status information for each request
    ExtendedStatus On
</IfModule>

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
</Files>

TypesConfig /etc/mime.types

<IfVersion < 2.4>
    DefaultType text/plain
</IfVersion>


# Loglevel and logfile configuration
LogLevel warn
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

#
# Language configuration
#

LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv zh-CN zh-TW
ForceLanguagePriority Prefer Fallback

AddDefaultCharset UTF-8

AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz

AddHandler type-map var
#
# PHP specific configuration
#
AddType text/html .php


#
# The following directives modify normal HTTP response behavior to
# handle known problems with browser implementations.
#
BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0

#
# The following directive disables redirects on non-GET requests for
# a directory that does not include the trailing slash.  This fixes a 
# problem with Microsoft WebFolders which does not appropriately handle 
# redirects for folders with DAV methods.
# Same deal with Apple's DAV filesystem and Gnome VFS support for DAV.
#
BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
BrowserMatch "MS FrontPage" redirect-carefully
BrowserMatch "^WebDrive" redirect-carefully
BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully
BrowserMatch "^gnome-vfs/1.0" redirect-carefully
BrowserMatch "^XML Spy" redirect-carefully
BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully


# OMD OMD OMD OMD OMD OMD OMD OMD OMD OMD OMD OMD  ----------------------------
# Do not change anything below here

MIMEMagicFile /omd/sites/monitoring/etc/apache/magic
ErrorLog /omd/sites/monitoring/var/log/apache/error_log
CustomLog /omd/sites/monitoring/var/log/apache/access_log combined

# fcgi settings must be site specific and cannot be used from shared apache
SharememPath "/omd/sites/monitoring/tmp/apache/fcgid_shm"
SocketPath "/omd/sites/monitoring/tmp/apache/fcgid_sock"

# Load config files from the include dir. You can put your own
# configuration there. Also addons put default files there.
Include /omd/sites/monitoring/etc/apache/conf.d/*.conf

I commented the line on the old server, too - but did not have this error before.

I tried with different checkmk raw editions, did not change matters.

Maybe this error is related to the new CPUID? (kidding but I completely lost my faith in computing software).

If “we” can solve this problem this is worth one bar of the best German chocolate (including shipping costs).

Martin.

P. S. I forgot to mention:

/usr/sbin/httpd-prefork -V
Server version: Apache/2.4.57 (Linux/SUSE)
Server built:   2023-04-07 16:20:25.000000000 +0300
Server's Module Magic Number: 20120211:127
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1, PCRE 10.36 2020-12-04
Compiled using: APR 1.6.3, APR-UTIL 1.6.1, PCRE 10.36 2020-12-04
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_PROC_PTHREAD_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/srv/www"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="/var/log/apache2/error_log"
 -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
 -D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"

P.P.S.
The old server is still running (although DNS entries are obsoleted now), and I did verify - yes, in the check mk apache error logs - there is no entry in the old server.

P.P.P.S.
I think I quadriplechecked the whole horror scenario, but…

2

Well, the httpd is not called as httpd-prefork in the new server, how come?
Will investigate…

3

I think CMK is not supported on this platform.
Please use a supported OS version.

4

Thank you for investigating.
I did some “research” as well and found the culprit:

In opensuse the apache prefork is called “httpd-prefork”, and not “httpd2-prefork”, because of that “httpd” is called, and not “httpd-prefork” which would be necessary for a proper function.

So this script needed to be changed

/opt/omd/sites/SITENAME/.version_meta/skel/etc/init.d/apache

#!/bin/bash

cd ###ROOT###
. .profile
. lib/omd/init_profile
. etc/omd/site.conf
if [ "$CONFIG_APACHE_MODE" == none ]; then
  exit 5
fi

# Start httpd in the C locale by default.
HTTPD_LANG="C"

# This will prevent initlog from swallowing up a pass-phrase prompt if
# mod_ssl needs a pass-phrase from the user.
INITLOG_ARGS=""

APACHE_RUN_USER="###SITE###"
APACHE_RUN_GROUP="###SITE###"

PID_FILE=###ROOT###/tmp/apache/run/apache.pid
CONFIG_FILE=###ROOT###/etc/apache/apache.conf

apache_bin() {
    if [ -e /usr/sbin/apache2 ]; then
        echo /usr/sbin/apache2
    elif [ -e /usr/sbin/httpd2-prefork ]; then
        echo /usr/sbin/httpd2-prefork
    elif [ -e /usr/sbin/httpd ]; then
        echo /usr/sbin/httpd
    else
        echo "ERROR: Failed to locate apache binary"
        exit 1
    fi
}

....

I suspect that I did change that in the old server installation, but forgot about that.

Thank you again, but I will eat the chocolate bar myself :grinning:

5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.