Alerthandler (ROOT login not permitted)

[Pixelpoint (pixelpoint multimedia)]

Dear Check_MK Community,

I’m trying to deploy alerthandlers to restart systemd-services if they’re down (works well on my test server with Ubuntu 20.04 and root-login allowed).
Now I’m faced with the situation that root login via SSH is not permitted on the target host.

Check_MK Enterprise Edition 1.6.0.p15 (will be upgraded shortly to 1.6.0p17).

I have the following Rules:
(not showing the Systemd Service rules because they work and have been tested)

Remote alert handlers (Linux)

Alerthandler configuration

For this to work though, I needed to set permissions on different

Now I have a big question mark above my head:
Why do i need to set 2 Users in 2 different places?

How do others deploy alerthandlers without root access?
Do you really need to manually change permissions to different files on all your target hosts just to get this running without root access?

alert.log (on checkmk server) output
Rule 'restart if service is down after 2nd check attempt (apache2)'...
Event Type is rc
-> matches!
Executing alert handler linux_remote for HOSTNAMEREDACTED;Systemd Service apache2
Spawned event handler with PID 25924
Handler [25924] linux_remote HOSTNAMEREDACTED;Systemd Service apache2 stopped: bash: /usr/bin/mk-remote-alert-handler: Permission denied

alert handler execution view

permissions on /usr/bin/mk-remote-alert-handler on target server

P.S.:
Yes, the screenshots say elasticsearch (i made a screenshot of the wrong rule at first), but as you see we also want to restart apache2 via alerthandler.
Also: Just for testing, I got rid of the “only start alerthandlers on 2nd check attempt” portion of the rule.
Everything else is the same and I will definitely bring that portion back once the alerthandlers work.

If you need any more information, pls tell me.
I’m looking forward to your answers.

[louis (Louis Wevers)]

Hi,

Never used it myself, but still I may be able to help. In general, on a Linux system, to restart a systemd service you need root access. So, if it’s not possible to do this as root, you’ll have to configure sudo. Allow the user pixelpoint sudo rights, without password on /usr/bin/mk-remote-alert-handler. Then, instead of calling it directly, call sudo /usr/bin/mk-remote-alert-handler.

[Pixelpoint (pixelpoint multimedia)]

Hello louis and thank you for your response.

I tested this approach today.
One needs to give permissions to the following directories / files:
/usr/lib/check_mk_agent (the folder itself)
/usr/lib/check_mk_agent/alert_handlers (again, folder)
/usr/lib/check_mk_agent/alert_handlers/* (alerthandlers inside the directory)
/usr/bin/mk-remote-alert-handler (this script seems to start the alerthandlers)

I did it with sudo chmod o+rx

Problem being:
As soon as you update the check mk agent on the target server, the folders keep the access permissions, but all the executables have their access permissions reset.

This includes mk-remote-alert-handler + all the alerthandlers inside the alert_handler folder, even the ones which have had their permissions modified.

Is there a way to automatically execute a post-agent-update script which gets called everytime the agent updates itself?

The only other solution would be to add the user to the root-group.
I tested this and it works flawlessly, but I don’t know what this would mean regarding security.

[louis (Louis Wevers)]

Hi,

Hmmm, actually I was suggesting to run your /usr/bin/mk-remote-alert-handler command using sudo so
sudo /usr/bin/mk-remote-alert-handler instead of just
/usr/bin/mk-remote-alert-handler

You can then need to give the user pixelpoint sudo rights to run /usr/bin/mk-remote-alert-handler as root without being prompted for a password.

So in your sudo configuration add something like this:

pixelpoint ALL = NOPASSWD: /usr/bin/mk-remote-alert-handler

Adding the user to the root group would basically grant this user the same privileges as the root user. From a security point of view I would highly recommend not to do this.

[Pixelpoint (pixelpoint multimedia)]

Hello louis,

thank you for your time and knowledge.

We solved this problem by allowing root login restricted to the CheckMK Server IP and NO PASSWORD (only public key authentication).
Now none of us can log into root (we don’t need to because we have our pixelpoint admin user), but CheckMK can do it because the agent transfers the SSH keys.
I didn’t know one could restrict the root login this way.

@louis

pixelpoint ALL = NOPASSWD: /usr/bin/mk-remote-alert-handler

I thought the user who would execute the check_mk alerthandlers would also need permissions to the alerthandler scripts themselves.
Now I’m interested.
I will try this as soon as I find the time to do so on my ubuntu test vm.

Thank you for bringing that to my mind.

[louis (Louis Wevers)]

Hello,

I’m glad that you at least found a working solution. But from a security point of view you may want to disable remote root logins completely. Even when you only allow public key authentication.

With the sudo option I provided you don’t restrict the root login. You allow the user pixelpoint to execute this specific command with root rights. I assumed the mk-remote-alert-handler script then starts the other scripts. Since mk-remote-alert-handler runs with root rights all scripts that fork from this process will also run with root rights.

If this is not the case, you may need to add them to the sudo list.

pixelpoint ALL = NOPASSWD: /usr/bin/mk-remote-alert-handler, some-other-script, more-scripts

Good luck with the testing.

Louis

[martin.schwarz]

If you permit root login (which may or may not be acceptable depending on your security requirements, company policy and/or auditor’s opinion, … ), you should try to limit the access as far as possible.

For /etc/ssh/sshd_config, I suggest using
PermitRootLogin forced-commands-only.

Then prefix the pubkey in /root/.ssh/authorized_keys with
command="/usr/bin/mk-remote-alert-handler".

Additional options in authorized_keys you might want to consider:
from="(_the ssh client's IP address and 127.0.0.1,::1 for debugging_)",no-pty,no-user-rc,no-port-forwarding,no-X11-forwarding,no-agent-forwarding

See the sshd(8) manpage for details, search for “AUTHORIZED_KEYS FILE FORMAT”

[Pixelpoint (pixelpoint multimedia)]

Sorry for my late response, I’ve been sick the past few days.

louis
With the sudo option I provided you don’t restrict the root login. You allow the user pixelpoint to execute this specific command with root rights. I assumed the mk-remote-alert-handler script then starts the other scripts. Since mk-remote-alert-handler runs with root rights all scripts that fork from this process will also run with root rights.

If this is not the case, you may need to add them to the sudo list.

pixelpoint ALL = NOPASSWD: /usr/bin/mk-remote-alert-handler, some-other-script, more-scripts

I tried your approach today.
The results are confusing to me.

I create a user called “cmkautomation” and put it into sudo group.

visudo entry:
cmkautomation ALL=(ALL:ALL) NOPASSWD: /usr/bin/mk-remote-alert-handler

I also tried this visudo entry:
cmkautomation ALL= NOPASSWD: /usr/bin/mk-remote-alert-handler

after a fake check (CRIT) I get this result:
bash: /usr/bin/mk-remote-alert-handler: Permission denied

When i execute sudo /usr/bin/mk-remote-alert-handler from within my test ubuntu, I do not get a permission denied.

Interestingly, when I execute /usr/bin/mk-remote-alert-handler with User cmkautomation from within the test ubuntu shell, I don’t get permission denied.
If I do chmod o+rx /usr/bin/mk-remote-alert-handler I get the permission denied error again, but this time not for /usr/bin/mk-remote-alert-handler but for the alert-handler itself inside `/usr/share

With sudo /usr/lib/check_mk_agent/alert_handlers/testbuntu-apache-restart.sh it works.

It seems like there is some kind of mysterios difference between executing sudo directly on the machine and executing sudo with Check_MK.

martin.schwarz
For /etc/ssh/sshd_config , I suggest using
PermitRootLogin forced-commands-only .

Then prefix the pubkey in /root/.ssh/authorized_keys with
command="/usr/bin/mk-remote-alert-handler" .

Sadly, I am not able to prefix the keys with something, as the keys are created and deployed automatically by Check_MK.
I would need to manually edit the .ssh/authorized_keys manually every.
Thank you for your response, I will look into the other options.
Though, to be honest, I’d rather get rid of root login permanently.

[louis (Louis Wevers)]

Hmmm, that’s interesting indeed. There should be no difference. So I’m wondering, when triggering from CheckMK, are you sure that the command runs with sudo? You may have to modify your check for this.

[Pixelpoint (pixelpoint multimedia)]

How can I be sure that mk-remote-alert-handler is called via sudo?
My alerthandler themselves all have the sudo command (sudo systemctl restart apache2 etc).

I wouldn’t know of a way to call mk-remote-alert-handler directly and to give the correct parameters to start the alerthandlers.

[EDIT]
Isn’t mk-remote-alert-handler called via root?
At least that’s the standard setting.

[louis (Louis Wevers)]

Hmmm, reading up on remote-alert-handlers I see they’re not in the raw edition, that’s why I couldn’t find them in my installation and try it out for you.

So having said that, maybe I can still help. For jour mk-remote-alert-handler script, make sure it’s owned by root and then set the setuid bit on it (chmod +s mk-remote-alert-handler). That way you’ll make sure it’s always executed as root.

[Pixelpoint (pixelpoint multimedia)]

I’ve set the setuid bit on mk-remote-alert-handler.

The following error appeared while executing the alerthandler:
/usr/bin/mk-remote-alert-handler: line 15: /usr/lib/check_mk_agent/alert_handlers/testbuntu-apache-restart.sh: Permission denied
/usr/bin/mk-remote-alert-handler: line 15: exec: /usr/lib/check_mk_agent/alert_handlers/testbuntu-apache-restart.sh: cannot execute: Permission denied

So again I’m at the point where I would need to change permissions of all the alerthandlers I deploy anywhere.

When I change the alerthandler scripts from sudo systemctl restart apache2 to systemctl restart apache2 (with the setuid bit the child processes who’re being launched by mk-remote-alert-handlers should also with elevated privileges, right?) I get an alerthandler execution without errors, but without output and nothing happens (apache does not restart).

I thank you very much for your help, but I think we’ve reached the point where putting in more and more work to bend the default to our will is not appropriate anymore (at least not when I do this at work).

Maybe I’ll do some check mk setup at home in the future, where I can take as much time as needed for everything to be exactly as I want it.

Btw: Alerthandlers are not in the RAW Edition, but you could always go for nagios eventhandlers (https://checkmk.de/cms_alert_handlers.html#Alerthandler%20in%20(CMK))

[louis (Louis Wevers)]

That’s too bad. Sorry I couldn’t be of more help. Maybe, if I find some time to spare, I’ll see if I can get something to work with the nagios event handlers. If I find something I’ll let you know.

[Pixelpoint (pixelpoint multimedia)]

Thank you very much, louis.
You’ve been plenty of help though, so don’t be sorry!