Logwatch/Text Logfiles can't see logfile

Troubleshooting
1

CMK version: 2.4.0p20 cee
OS version: RHEL9.7

Error message: [[[/opt/splunk/var/log/splunk/splunkd.log:missing]]]

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

OMD[crlcmk]:~$ cmk --debug -vvn hostname
value store: loading from disk
Checkmk version 2.4.0p20

Hi, I’ve tried everything I can think of to resolve this including starting from scratch. The log file is there, the mk_logwatch.py runs as root, but all servers this is deployed to are unable to see the file. I’m probably missing something, but I’ve been through all the settings and nothing seems to work.

The logwatch.cfg contains:

“/opt/splunk/var/log/splunk/splunkd.log” regex=‘host_dest=(x.x.x.x|x.x.x.x.x).*has\sbeen\sblocked’ encoding=utf-8 maxlines=100 overflow=I maxoutputsize=2560000 nocontext=True

-rw-r–r–. 1 root root 6132433 Mar 27 11:14 /opt/splunk/var/log/splunk/splunkd.log

TIA

2

did you checked selinux for any errors?

3

Do you maybe have selinux enabled (check with getenforce)?

4

Yeah, one of the first things I tried was disabling selinux, but it didn’t make any difference. Though with it being a splunk server, it runs in permissive mode anyway.

5

So… can you normally “tail” this file? Would you be able
to see such an entry you’re trying to “catch” with mk_logwatch?

Additionally: How exactly is your agent installed on this machine?
Is it polled via “xinetd” or the “systemd socket”?

And finally can you perhaps check with “getfacl” what kind
of permissions this file has?

6

Hi, Yup, I have full access to the file and the regex works as expected. The agent is running via systemd socket and as the root user. Getfacl returns the expected results:

file: opt/splunk/var/log/splunk/splunkd.log
owner: root
group: root
user::rw-
group::r–
other::r–

It’s a head scratcher

edit - Ran some further tests and it seems to do this for any file regardless of location or permissions.

edit - Obliterated the entire checkmk installation and redeployed. Now it doesn’t complain about the log file missing, it just doesn’t read it :slight_smile: One step closer I guess.

7

Hi, Yup, I have full access to the file and the regex works as expected. The agent is running via systemd socket and as the root user. Getfacl returns the expected results:

file: opt/splunk/var/log/splunk/splunkd.log
owner: root
group: root
user::rw-
group::r–
other::r–

It’s a head scratcher

edit - Ran some further tests and it seems to do this for any file regardless of location or permissions.

8

Maybe you can find a hint by running
/usr/bin/check_mk_agent -d
on one of the hosts.

9

Hi, Can’t see anything wrong.

+ for script in ./*
+ is_valid_plugin './*'
+ case "${1:?No plugin defined}" in
+ '[' -f './*' ']'
+ for script in [1-9]*/*
+ is_valid_plugin '[1-9]*/*'
+ case "${1:?No plugin defined}" in
+ '[' -f '[1-9]*/*' ']'
+ run_plugins
+ cd /usr/lib/check_mk_agent/plugins
+ true
+ for script in ./*
+ is_valid_plugin ./3600
+ case "${1:?No plugin defined}" in
+ '[' -f ./3600 ']'
+ for script in ./*
+ is_valid_plugin ./mk_logwatch.py
+ case "${1:?No plugin defined}" in
+ '[' -f ./mk_logwatch.py ']'
+ '[' -x ./mk_logwatch.py ']'
++ get_plugin_interpreter ./mk_logwatch.py
++ agent_plugin=mk_logwatch.py
++ extension=py
++ filename=mk_logwatch
++ '[' py '!=' py ']'
++ '[' ch '!=' _2 ']'
++ '[' -n '' ']'
++ '[' -n '' ']'
++ '[' -n python3 ']'
++ echo python3
++ return 0
+ plugin_interpreter=python3
+ _log_section_time python3 ./mk_logwatch.py
+ python3 ./mk_logwatch.py
<<<logwatch>>>
[[[/opt/splunk/var/log/splunk/splunkd.log:missing]]]
10

After trying to reproduce your problem I probably first did the same error as you did:
I think you added the pattern, you want to check the content of the logfile for, into the field “Regular expression for logfile filtering” instead in “Regular expressions for message classification”. So you filter the log file names against the regex instead of the content.

11

Ah you are correct :smiley: Thank you so much.