Apache failed with syntax error

kkloesener · July 10, 2024, 7:18am

CMK version:
2.2.0p29.cre
OS version:
Ubuntu 22.04.4

Error message:

[Wed Jul 10 00:00:02.531519 2024] [mpm_prefork:notice] [pid 1232] AH00171: Graceful restart requested, doing restart
apache2: Syntax error on line 126 of /omd/sites/vrf105/etc/apache/apache.conf: Cannot load /omd/sites/vrf105/lib/apache/modules/mod_headers.so into server: /omd/sites/vrf105/lib/apache/modules/mod_headers.so: undefined symbol: ap_set_content_type_ex

Today around 5 from 10 instances from our distributed setup were not available because the apache service was not started anymore. Manually starting the services with ‘omd start SITE’ worked without further error.
I’ve checked every instance and all throw the same error message in $SITE/var/log/apache/error.log . Could this be a bug in this version?

aeckstein · July 10, 2024, 7:38am

just confirming that i also saw this on ~~two~~ five different sites at different customers this night.
Ubuntu 22.04 with 2.3.0p8.cee

[Wed Jul 10 00:00:01.944365 2024] [mpm_prefork:notice] [pid 908034] AH00171: Graceful restart requested, doing restart
apache2: Syntax error on line 126 of /omd/sites/sitename/etc/apache/apache.conf: Cannot load /omd/sites/sitename/lib/apache/modules/mod_headers.so into server: /omd/sites/sitename/lib/apache/modules/mod_headers.so: undefined symbol: ap_set_content_type_ex

ale-rt · July 10, 2024, 7:44am

It happened to me as well:

OMD[monitoring]:~/etc$ omd status 
agent-receiver: running
mkeventd:       running
rrdcached:      running
npcd:           running
nagios:         running
apache:         stopped
redis:          running
crontab:        running
-----------------------
Overall state:  partially running
OMD[monitoring]:~/etc$ omd restart apache
Temporary filesystem already mounted
Stopping apache...(not running)...OK
Starting apache....OK

Same traceback:

[Wed Jul 10 00:00:02.090371 2024] [mpm_prefork:notice] [pid 2067] AH00171: Graceful restart requested, doing restart
apache2: Syntax error on line 126 of /omd/sites/monitoring/etc/apache/apache.conf: Cannot load /omd/sites/monitoring/lib/apache/modules/mod_headers.so into server: /omd/sites/monitoring/lib/apache/modules/mod_headers.so: undefined symbol: ap_set_content_type_ex

Noah · July 10, 2024, 8:24am

Hi, I also had this Problem today, however only on the Master-Site:

Only the Apache-service from the Site was stopped

The Site-Internal Apache-Errorlog showed:

[Tue Jul 09 00:00:04.863390 2024] [mpm_prefork:notice] [pid 2432] AH00163: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.4 Python/3.11 mod_fcgid/2.3.9 configured -- resuming normal operations
[Tue Jul 09 00:00:04.863470 2024] [core:notice] [pid 2432] AH00094: Command line: '/usr/sbin/apache2 -f /omd/sites/mysite/etc/apache/apache.conf'
[Wed Jul 10 00:00:02.747272 2024] [mpm_prefork:notice] [pid 2432] AH00171: Graceful restart requested, doing restart
apache2: Syntax error on line 126 of /omd/sites/mysite/etc/apache/apache.conf: Cannot load /omd/sites/mysite/lib/apache/modules/mod_headers.so into server: /omd/sites/mysite/lib/apache/modules/mod_headers.so: undefined symbol: ap_set_content_type_ex

OS: Ubuntu Server 22.04.4 LTS, CheckMK 2.2.0p4 CME

Sara · July 10, 2024, 8:36am

Hi all!

We are looking into this. For now, please try omd start – it fixed the issue for one of my colleagues.

aeckstein · July 10, 2024, 8:39am

yes, (re)starting the site apache solves the problem immediatly

robin.gierse · July 10, 2024, 10:52am

Our current working theory is, that Ubuntu (Debian?) security updates triggered the issue.

Either:
7 security updates (python3.8-dev, libpython3.8-dev, libpython3.8, python3.8, libpython3.8-stdlib, python3.8-minimal, libpython3.8-minimal)(!!)
Or:
4 security updates (apache2, apache2-bin, apache2-data, apache2-utils)(!!)

Can you guys check your update logs and ideally correlate the installation of said updates with the time the Checkmk Apache went down?

andreas-doehler · July 10, 2024, 10:54am

Yeah today at night around 20 instances had this problem here

Noah · July 10, 2024, 11:39am

Hi Robin,

Updates installed today, now over 50 instances are not accessible via https on my side, however the Monitoring core still runs and Monitoring is working on the customer site.

Start-Date: 2024-07-10  06:45:17
Commandline: /usr/bin/unattended-upgrade
Upgrade: python3.10:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10-minimal:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10-stdlib:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), python3.10-minimal:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4)
End-Date: 2024-07-10  06:45:20

Edit: Yesterday the “Apache” updates have been installed but appling changes worked fine yesterday

klaus.scholl · July 10, 2024, 11:46am

Installed the apache2 updates yesterday around 9am.
Apache went down (like all the others) shortly after midnight.

installed updates were apache2, apache2-bin, apache2-utils and apache2-data on Ubuntu 20.04 LTS

The python3 updates were installed this morning after i restarted apache.

Tralin · July 10, 2024, 11:53am

Same story here: Apache updates were automatically installed on Ubuntu 20.04 and 22.04 and all CMK instances lost their Apache interface. After doing:

sudo -iu <site>
omd start

The problem is solved.

kkloesener · July 10, 2024, 2:00pm

Yeah, unattend upgrades yesterday:

Start-Date: 2024-07-09 06:30:02
Commandline: /usr/bin/unattended-upgrade
Upgrade: apache2-data:amd64 (2.4.52-1ubuntu4.9, 2.4.52-1ubuntu4.10), apache2-bin:amd64 (2.4.52-1ubuntu4.9, 2.4.52-1ubuntu4.10), apache2-utils:amd64 (2.4.52-1ubuntu4.9, 2.4.52-1ubuntu4.10), apache2:amd64 (2.4.52-1ubuntu4.9, 2.4.52-1ubuntu4.10)
End-Date: 2024-07-09 06:30:08

darkfader · July 10, 2024, 2:03pm

Basically logrotate will only do a reload and with the update changing that function in mod_headers it fails.
A good long-term fix could be to do a graceful restart instead. (-k graceful or something like that)
Note you have to check that your ‘apachectl’ doesn’t point at the systemd wrapper, because they left out the feature.

Gaborpapp · July 11, 2024, 1:14pm

Happaned the same last night on 2 sites - checkmk is running on Ubuntu and did python and apache security updates on that day.

WalterH · July 14, 2024, 6:18am

Hi,

I have seen this issue yesterday on a Debian 12 system with 2.2.0p30.cre.
omd stop and omd start solved this issue until now.

ratfink · July 15, 2024, 10:01am

Hi All. Do we have a permanent ‘Fix’ yet for this issue? I realise that manually running a “sudo omd start” on all our VMs / Servers gets things running again, but would like to know the issue is ‘fixed’ and not coming back …

robin.gierse · July 15, 2024, 2:54pm

From what we can see, it is a one-off thing at this point, but we are still investigating.

glu · July 22, 2024, 11:19am

Same issue on our customer sites, there was an update of apache2 from 2.4.51-35.51.1 to 2.4.51-35.54.1 the day before and all apache on omd stopped working at midnight.
Why is not starting by themself if it goes down?

It did happen this weekend with check-mk-enterprise-2.2.0p29-sles12sp5-38.x86_64 on a sles12sp5.

Is there a solution for a guaranteed availabilty?

[Sat Jul 20 00:00:02.057148 2024] [mpm_prefork:notice] [pid 26230] AH00171: Graceful restart requested, doing restart
httpd2-prefork: Syntax error on line 126 of /omd/sites/…/etc/apache/apache.conf: Cannot load /omd/sites/…/lib/apache/modules/mod_headers.so into server: /omd/sites/…/lib/apache/modules/mod_headers.so: undefined symbol: ap_set_content_type_ex

samueldc · July 22, 2024, 5:40pm

Hi,

Looks like this problem is related to httpd bynaries compatibility issues. Do you use any httpd docker image and based on that install any other httpd module using apt? If so, httpd docker image has its own bynaries manually installed. And when trying to install a httpd module using apt, it will install apache2-bin package (which is already installed in another version, but apt don’t know that). Because of that, the system will mix httpd bynaries of different versions, leading to error like that.

janezb · August 23, 2024, 8:49am

Hi,

Today, I encountered the same issue on OEL 9.4 (CEE 2.2p20) and Rocky Linux 8.10 (RAW 2.3p12) after applying the latest Linux patches. Interestingly, the problem occurred at midnight when checkmk checks the Apache configuration using the command /usr/sbin/httpd -f /omd/sites/SITE_NAME/etc/apache/apache.conf:
httpd: Syntax error on line 236 of /omd/sites/SITE_NAME/etc/apache/apache.conf: Syntax error on line 2 of /omd/sites/SITE_NAME/etc/apache/conf.d/01_wsgi.conf: Cannot load /omd/sites/SITE_NAME/lib/apache/modules/mod_wsgi.so into server: libpython3.12.so.1.0: cannot open shared object file: No such file or directory

The solution was to install the newer Python libraries, for example, by running dnf install python3.12-libs. After this, the command /usr/sbin/httpd -f /omd/sites/SITE_NAME/etc/apache/apache.conf executed without any problems.