[Bug?] Check-mk-agent.socket flooding /var/log/auth.log

CMK version: 2.0.0p21 (CFE)
OS version: Docker image

Error message: May 25 10:54:32 hostname su: (to root) root on none

For the Devs: I’ve analysed everything, please take a look through the entire thread

Hey everyone,

The investigation started when I noticed the error message above every ~1 minute in /var/log/auth.log

This little issue has been blowing my head off for the past month. I have tried suppressing the error message by modifying files in /etc/pam.d (such as su or common-session, but no luck, though I shouldn’t really be making a workaround, instead I should be finding the root problem.

This happens on no other host, and although this is a host with the most modified agent, including docker, another docker host with similar agent modifications does not have this issue.

I have started disabling services one by one and looking at the log with tail -f /var/log/auth.log, and would you look at that, systemctl stop check-mk-agent.socket stops the log message from appearing, but checkMK can’t talk to this host now, so this is a bit of a problem now :slight_smile:

Does anyone know why this is happening and what can be done to stop it?

Here’s a screenshot of the problem agent in the bakery:

This is what I had in the beginning (^ every minute all these lines ^), and managed to reduce it down only to May 25 10:54:32 hostname su: (to root) root on none (every minute) by modifying files in /etc/pam.d

Although there doesn’t seem to be a DB on the docker host itself, don’t see how that could be related.

I have just tried removing each configuration from the agent, and in the end just started using the vanilla agent, no luck again. This is what it boils down to, and the log kept appearing every minute on every new agent configuration.

With the default Debian 11 /etc/pam.d configuration, these are the 3 lines I get:

May 25 11:33:24 hostname su: (to root) root on none
May 25 11:33:24 hostname su: pam_unix(su:session): session opened for user root(uid=0) by (uid=0)
May 25 11:33:24 hostname su: pam_unix(su:session): session closed for user root

This is normal linux behavior and related to the logging settings of your system and systemd. If you like to suppress such messages you need to adjust your system. take a look at the documentation of syslogd.

This log lines appears every time a user logs into the system. The agent is called by the central monitoring server every minute (default), that’s why you see this message every minute.

1 Like

It’s odd to say that this is normal linux behavior because I don’t get these logs on my other cmk monitored hosts.

Same OS (Deb 11), same template used to configure the VM, literally everything is the same.

Oddly enough, dpkg --purge check-mk-agent and reinstalling seems to have done the job for now.

Though now I have these:

Obviously it’s because the jobs didn’t run since reinstalling, which makes me wonder if mk-job has something to do with my issue. I’ll keep everyone posted here.

OKAY:

So it boils down to this, as soon as mk-job whatever_script runs once, the log messages start appearing every minute in /var/log/auth.log

May 25 11:33:24 hostname su: (to root) root on none
May 25 11:33:24 hostname su: pam_unix(su:session): session opened for user root(uid=0) by (uid=0)
May 25 11:33:24 hostname su: pam_unix(su:session): session closed for user root

I’m looking at /usr/bin/mk-job - but I don’t see it calling su anywhere. What’s the deal here?

I have noticed that only on hosts that monitor jobs with mk-job, when I manually Reschedule 'Checkmk' service
image

The infamous 3 log lines appear (And they appear every minute also), on other hosts this does not happen.

I guess we could call this a “semantic” bug.

/usr/bin/check_mk_agent:

Is this the problem?

  • Yep this is definitely the line causing the logs to appear. I just copied the else clause into the main if, so they’re both the same (both just do the head command). No annoying logs appear now, and there’s only one difference - the jobs names are now prepended with ./ in the web-ui, and they appear as new jobs.

I do understand this change presents a security issue, but I’m sure there is a proper solution.

Dev team, please help me out, I’m pretty sure most of the detective work is done by now.

Well Detective, @tosch already answered your question: This is not a Checkmk issue, but the logging of your system. I see the messages you complain about on every Linux system when a user logs in, no matter whether he logs in remotely or locally.

This cannot be considered a bug, but if you find an equal solution, that does not generate this messages, we do appreciate pull requests on GitHub. :slight_smile:

1 Like