Unable to connect to Checkmk web interface after reboot

Troubleshooting
1

Dear reader,

Can you help me login back into the web interface of Checkmk?

I have installed Checkmk and was able to login. Also I have enabled the omd service to start at boot time. Now I have rebooted the Linux host and I am unable to login to the web interface anymore. I get the error: Unable to connect.

Any help is appreciated.
Do you have any idea how I can get the web interface working again?
Or how I can further troubleshoot this?

**CMK version:** OMD - Open Monitoring Distribution Version 2.2.0p17.cre
**OS version:** Rocky Linux 9.3

**Error message:** Firefox: https://checkmk-01.vandenboom.local/monitoring 
Unable to connect

**Output of “cmk --debug -vvn hostname”:** (If it is a problem with checks or plugins)
OMD[monitoring]:~$ cmk --debug -vvn checkmk-01.vandenboom.local

Checkmk version 2.2.0p17

Updating IPv4 DNS cache for checkmk-01.vandenboom.local: 192.168.178.132

Trying to acquire lock on /omd/sites/monitoring/var/check_mk/ipaddresses.cache

Got lock on /omd/sites/monitoring/var/check_mk/ipaddresses.cache

Releasing lock on /omd/sites/monitoring/var/check_mk/ipaddresses.cache

Released lock on /omd/sites/monitoring/var/check_mk/ipaddresses.cache

+ FETCHING DATA

  Source: SourceInfo(hostname='checkmk-01.vandenboom.local', ipaddress='192.168.178.132', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)

[cpu_tracking] Start [7fb5d84dbfd0]

Read from cache: NoCache(checkmk-01.vandenboom.local, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)

[PiggybackFetcher] Execute data source

No piggyback files for 'checkmk-01.vandenboom.local'. Skip processing.

No piggyback files for '192.168.178.132'. Skip processing.

[cpu_tracking] Stop [7fb5d84dbfd0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]

+ PARSE FETCHER RESULTS

  HostKey(hostname='checkmk-01.vandenboom.local', source_type=<SourceType.HOST: 1>)  -> Add sections: []

Received no piggyback data

[cpu_tracking] Start [7fb5d7b05550]

value store: synchronizing

Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/counters/checkmk-01.vandenboom.local

Got lock on /omd/sites/monitoring/tmp/check_mk/counters/checkmk-01.vandenboom.local

value store: loading from disk

Releasing lock on /omd/sites/monitoring/tmp/check_mk/counters/checkmk-01.vandenboom.local

Released lock on /omd/sites/monitoring/tmp/check_mk/counters/checkmk-01.vandenboom.local

No piggyback files for 'checkmk-01.vandenboom.local'. Skip processing.

No piggyback files for '192.168.178.132'. Skip processing.

[cpu_tracking] Stop [7fb5d7b05550 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.009999999776482582))]

[piggyback] Success (but no data found for this host), execution time 0.0 sec | execution_time=0.010 user_time=0.000 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.000

** Ouput of: omd status:
[root@checkmk-01 ~]# omd status
Doing 'status' on site monitoring:
agent-receiver: running
mkeventd:       running
rrdcached:      running
npcd:           running
nagios:         running
apache:         running
redis:          running
crontab:        running
-----------------------
Overall state:  running

** Info in logging: journalctl -u omd
Note there was a restart with: systemctl restart omd

[root@checkmk-01 ~]# journalctl -u omd
Jan 05 23:13:36 checkmk-01.vandenboom.local systemd[1]: Starting Checkmk Monitoring...
Jan 05 23:13:37 checkmk-01.vandenboom.local omd[1150]: Doing 'start' on site monitoring:
Jan 05 23:13:38 checkmk-01.vandenboom.local omd[1484]: Creating temporary filesystem /omd/sites/monitoring/tmp...
Jan 05 23:13:38 checkmk-01.vandenboom.local omd[1677]: Starting agent-receiver...OK
Jan 05 23:13:39 checkmk-01.vandenboom.local omd[1790]: Starting mkeventd...OK
Jan 05 23:13:40 checkmk-01.vandenboom.local omd[1916]: Starting rrdcached...OK
Jan 05 23:13:40 checkmk-01.vandenboom.local omd[1940]: Starting npcd...OK
Jan 05 23:13:40 checkmk-01.vandenboom.local omd[1977]: Starting nagios...OK
Jan 05 23:13:40 checkmk-01.vandenboom.local omd[2106]: Starting apache...OK
Jan 05 23:13:41 checkmk-01.vandenboom.local omd[2195]: Starting redis...OK
Jan 05 23:13:41 checkmk-01.vandenboom.local crontab[2304]: (monitoring) REPLACE (monitoring)
Jan 05 23:13:41 checkmk-01.vandenboom.local omd[2297]: Initializing Crontab...OK
Jan 05 23:13:41 checkmk-01.vandenboom.local omd[1484]: OK
Jan 05 23:13:41 checkmk-01.vandenboom.local systemd[1]: Finished Checkmk Monitoring.
Jan 05 23:24:55 checkmk-01.vandenboom.local systemd[1]: Stopping Checkmk Monitoring...
Jan 05 23:24:55 checkmk-01.vandenboom.local omd[8595]: Doing 'stop' on site monitoring:
Jan 05 23:24:56 checkmk-01.vandenboom.local crontab[8676]: (monitoring) DELETE (monitoring)
Jan 05 23:24:56 checkmk-01.vandenboom.local omd[8674]: Removing Crontab...OK
Jan 05 23:24:56 checkmk-01.vandenboom.local omd[8681]: Stopping redis...killing 2205...OK
Jan 05 23:24:57 checkmk-01.vandenboom.local omd[8685]: Stopping apache...killing 2178.................OK
Jan 05 23:24:59 checkmk-01.vandenboom.local omd[8720]: Stopping nagios....OK
Jan 05 23:24:59 checkmk-01.vandenboom.local omd[8740]: Stopping npcd...OK
Jan 05 23:24:59 checkmk-01.vandenboom.local omd[8760]: Stopping rrdcached...waiting for termination...OK
Jan 05 23:25:00 checkmk-01.vandenboom.local omd[8778]: Stopping mkeventd...killing 1893...OK
Jan 05 23:25:00 checkmk-01.vandenboom.local omd[8786]: Stopping agent-receiver...killing 1791...OK
Jan 05 23:25:00 checkmk-01.vandenboom.local omd[8634]: Stopping 1 remaining site processes...OK
Jan 05 23:25:01 checkmk-01.vandenboom.local systemd[1]: omd.service: Deactivated successfully.
Jan 05 23:25:01 checkmk-01.vandenboom.local systemd[1]: Stopped Checkmk Monitoring.
Jan 05 23:25:01 checkmk-01.vandenboom.local systemd[1]: omd.service: Consumed 22.418s CPU time.
Jan 05 23:25:01 checkmk-01.vandenboom.local systemd[1]: Starting Checkmk Monitoring...
Jan 05 23:25:01 checkmk-01.vandenboom.local omd[8794]: Doing 'start' on site monitoring:
Jan 05 23:25:01 checkmk-01.vandenboom.local omd[8873]: Starting agent-receiver...OK
Jan 05 23:25:02 checkmk-01.vandenboom.local omd[8883]: Starting mkeventd...OK
Jan 05 23:25:02 checkmk-01.vandenboom.local omd[8893]: Starting rrdcached...OK
Jan 05 23:25:02 checkmk-01.vandenboom.local omd[8897]: Starting npcd...OK
Jan 05 23:25:02 checkmk-01.vandenboom.local omd[8920]: Starting nagios...OK
Jan 05 23:25:02 checkmk-01.vandenboom.local omd[8965]: Starting apache...OK
Jan 05 23:25:02 checkmk-01.vandenboom.local omd[8988]: Starting redis...OK
Jan 05 23:25:02 checkmk-01.vandenboom.local crontab[9004]: (monitoring) REPLACE (monitoring)
Jan 05 23:25:02 checkmk-01.vandenboom.local omd[9001]: Initializing Crontab...OK
Jan 05 23:25:02 checkmk-01.vandenboom.local omd[8833]: Temporary filesystem already mounted
Jan 05 23:25:02 checkmk-01.vandenboom.local systemd[1]: Finished Checkmk Monitoring.
[root@checkmk-01 ~]#

** Netstat is not showing any open connection on port 443 or 80

2

Hi,

please run “omd update-apache-config SITE” and look if this help.

regards,
Oliver

2 Likes
3

Also, double-check, if SELinux is enabled and maybe blocking something.

4

Dear reader,

For the moment this allowes me to get the web interface back and login.
But it seems I need to run this command: “omd update-apache-config SITE” every time I have rebooted.

How can I make the settings permanent?
So that they will be reboot proof.
That the SITE comes up automatically when the system boots.

Best regards,
Martijn

5

Hello,

SELinux seems not to be the issue. For me it is Enforcing and am able to login to the web interface after running: “omd update-apache-config SITE”

Sincerely

6

For reference, this is the accompanying Werk: Fix local privilege escalation from site users

7

Thank you for your reply and for providing me the reference.

But the solution is not there for me.

It says:
To eliminate the privilege escalation, you will have to execute the command: omd update-apache-config [SITE] once for each of your sites after the omd update command.

What I understood is to do:

omd stop monitoring
omd update monitoring
omd update-apache-config monitoring
omd start monitoring
reboot

But still after the reboot the web interface is not available until I run the command again:

omd update-apache-config monitoring

Maybe I could add the command to the systemctl service config file?
But I am not sure if that is the way to go.
Any other suggestions?

1 Like
8

Definitely not. The command sudo omd update-apache-config monitoring needs to be run only once after upgrade.

It looks like something is messing with your Apache configuration. Do you use any automation or something similar, that changes the systems Apache configuration?

9

I have used a fresh installed Rocky Linux 9.3 server for this setup. With no other prior testing, development or whatsoever. I have used a perfectly clean OS installation. Also followed the installation instructions written on the website. So the only two components here that are active is Rocky Linux 9.3 and Checkmk 2.2.0p18.

For now I have added the command: omd update-apache-config monitoring, to the script: /usr/bin/check_mk_agent

I understand this is not how it is supposed to be done but it works for now.

Can you maybe direct me to solve this in a proper way?
Any trouble shooting tips for Apache?

1 Like
10

Hi,

I am having same exact issue on OMD - Open Monitoring Distribution Version 2.2.0p21.cre. Which is quiet annoying and only on centos / rocky linux 9.3. Ubuntu seems to be fine.

Please help. update-apache-config needs to be run on every boot.

[root@mon01 ~]# netstat -anp |grep LISTEN |grep httpd
tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN 1871/httpd
[root@mon01 ~]# omd update-apache-config monitoring
Reloading Apache…OK
[root@mon01 ~]# netstat -anp |grep LISTEN |grep httpd
tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN 1871/httpd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2297/httpd
unix 2 [ ACC ] STREAM LISTENING 38241 2304/httpd /etc/httpd/run/cgisock.2297

[root@mon01 ~]# omd status
Doing ‘status’ on site monitoring:
agent-receiver: running
mkeventd: running
rrdcached: running
npcd: running
nagios: running
apache: running
redis: running
crontab: running

Overall state: running

11

Honestly, I have no idea what Rocky linux is but it seems …

… that the systemwide apache (not the site specific one!) isn’t running. Maybe it isn’t started automatically after a reboot on Rocky linux? Might that be the case?

What does

systemctl status apache2.service

say?

1 Like
12

there is no apache2.service, it’s httpd on Rocky Linux (read CentOS / RHEL)

[root@mon01 ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/httpd.service.d
             └─php-fpm.conf
     Active: active (running) since Mon 2024-02-05 10:10:49 EST; 2min 38s ago
       Docs: man:httpd.service(8)
   Main PID: 2361 (httpd)
     Status: "Total requests: 13; Idle/Busy workers 100/0;Requests/sec: 0.0872; Bytes served/sec: 1.3KB/sec"
      Tasks: 213 (limit: 99949)
     Memory: 49.8M
        CPU: 210ms
     CGroup: /system.slice/httpd.service
             ├─2361 /usr/sbin/httpd -DFOREGROUND
             ├─2368 /usr/sbin/httpd -DFOREGROUND
             ├─2369 /usr/sbin/httpd -DFOREGROUND
             ├─2370 /usr/sbin/httpd -DFOREGROUND
             └─2371 /usr/sbin/httpd -DFOREGROUND

Feb 05 10:10:49 mon01 systemd[1]: Starting The Apache HTTP Server...
Feb 05 10:10:49 mon01 httpd[2361]: Server configured, listening on: port 80
Feb 05 10:10:49 mon01 systemd[1]: Started The Apache HTTP Server.
1 Like
13

and here is status after reboot before running “omd update-apache-config monitoring”

[root@mon01 ~]# systemctl status httpd
○ httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/httpd.service.d
             └─php-fpm.conf
     Active: inactive (dead)
       Docs: man:httpd.service(8)
1 Like
14

Okay, the second line looks like the culprit to me:

[root@mon01 ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; preset: disabled)
###                                                         ^^^^^^^^          ^^^^^^^^

The first disabled means that the apache won’t start automatically after reboot.
The second disabled means that this is the default setting for your host (or OS).

I would recommend to run

systemctl enable httpd.service

once as user root so that apache autostarts upon reboot. The output of the status command should then show enabled instead of the first disabled.

2 Likes
15

I gotta say, I am embarrassed that was the fix. How did it get reset or I guess it never get enabled during RPM install of checkmk? is something to check for.

thank you! server starts up fine now upon each reboot. Sometimes it’s the basics eh?

1 Like
16

Glad to hear. As mentioned, I have no idea about Rocky linux. Maybe they don’t start the system wide apache by default for security reasons. :man_shrugging: I mean, they provide disabled as the default presetting. I don’t think that checkmk fiddles around with the autostart behaviour.

Maybe this is also the solution for @Martijn .

1 Like
17

@Dirk nice to have seen your reply. Checking if Apache was setup to start at boot was indeed disabled. Now I have also enabled it. Sometimes covered in too much details. Thanks

1 Like
18

I do have a question about the intended design (or bug) here.

Why on centos/rhel deployment, webui apache/httpd start up is disabled in systemd while instance that listens on port 5000 starts automatically via omd control scripts? Seems like having webui under “omd stop/start” would only make sense and that was my assumption because of which I failed to check systemd.

# ps awux |grep httpd
root         941  0.0  0.0  50664 13704 ?        Ss   Feb06   0:05 /usr/sbin/httpd -DFOREGROUND
apache      1153  0.0  0.0  44044  8096 ?        S    Feb06   0:00 /usr/sbin/httpd -DFOREGROUND
apache      1154  0.0  0.1 2497744 26116 ?       Sl   Feb06   0:29 /usr/sbin/httpd -DFOREGROUND
apache      1156  0.0  0.1 2366608 18988 ?       Sl   Feb06   0:30 /usr/sbin/httpd -DFOREGROUND
apache      1157  0.0  0.1 2366608 21256 ?       Sl   Feb06   0:35 /usr/sbin/httpd -DFOREGROUND
apache      2171  0.0  0.1 2366608 21824 ?       Sl   Feb06   0:43 /usr/sbin/httpd -DFOREGROUND
monitor+  544106  0.0  0.0  15364  8880 ?        Ss   Feb07   0:01 /usr/sbin/httpd -f /omd/sites/monitoring/etc/apache/apache.conf
root     1049583  0.0  0.0 221796  2316 pts/0    S+   08:00   0:00 grep --color=auto httpd
monitor+ 3475482  0.0  0.0  15364  2508 ?        S    00:00   0:00 /usr/sbin/httpd -f /omd/sites/monitoring/etc/apache/apache.conf
monitor+ 3475483  0.0  1.2 444864 202336 ?       S    00:00   0:20 /usr/sbin/httpd -f /omd/sites/monitoring/etc/apache/apache.conf
monitor+ 3478304  0.0  1.2 441244 198856 ?       S    00:00   0:19 /usr/sbin/httpd -f /omd/sites/monitoring/etc/apache/apache.conf
monitor+ 3504919  0.0  1.2 441768 198832 ?       S    00:08   0:19 /usr/sbin/httpd -f /omd/sites/monitoring/etc/apache/apache.conf

# netstat -anp |grep httpd |grep LISTEN
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN      544106/httpd
tcp6       0      0 :::80                   :::*                    LISTEN      941/httpd
unix  2      [ ACC ]     STREAM     LISTENING     20935    1153/httpd           /etc/httpd/run/cgisock.941

on my test ubuntu instance, apache2 is enabled upon installation of checkmk and apache2 fired up automatically at boot. If omd is stopped you receive following msg at webui url:

Checkmk: Site Not Started
You need to start this site in order to access the web interface.
19

This is a RedHat philosophy, and you will find this behavior on all RedHat derivates. Newly installed services are disabled by default and need to be enabled explicitly.

For Checkmk we handle the site Apache, as you already said, so we control the boot behavior.

2 Likes