We are using check-mk-agent version 2.1.0p1 and after some time (~5-6 days) it happens that the PATH variable in check-mk-agent-async seems to be completely mangled…
We’ve installed CheckMK agent in a different location (/opt/check_mk/bin) and it looks like that /opt/check_mk/bin at the end is the only place where the agent searches for programs.
Running a strace shows that any command is expected to be in /opt/check_mk/bin:
.....
stat("/opt/check_mk/bin/sleep", 0x7ffc48c731e0) = -1 ENOENT (No such file or directory)
stat("/opt/check_mk/bin/sleep", 0x7ffc48c731e0) = -1 ENOENT (No such file or directory)
stat("/opt/check_mk/bin/sleep", 0x7ffc48c731e0) = -1 ENOENT (No such file or directory)
stat("/opt/check_mk/bin/sleep", 0x7ffc48c731e0) = -1 ENOENT (No such file or directory)
stat("/opt/check_mk/bin/sleep", 0x7ffc48c731e0) = -1 ENOENT (No such file or directory)
stat("/opt/check_mk/bin/sleep", 0x7ffc48c731e0) = -1 ENOENT (No such file or directory)
...
stat("/opt/check_mk/bin/grep", 0x7ffc48c72400) = -1 ENOENT (No such file or directory)
stat("/opt/check_mk/bin/grep", 0x7ffc48c72400) = -1 ENOENT (No such file or directory)
stat("/opt/check_mk/bin/grep", 0x7ffc48c72400) = -1 ENOENT (No such file or directory)
stat("/opt/check_mk/bin/grep", 0x7ffc48c72400) = -1 ENOENT (No such file or directory)
stat("/opt/check_mk/bin/grep", 0x7ffc48c72400) = -1 ENOENT (No such file or directory)
...
( and of course the wrong PATH then is inherited to the children ):
[pid 1642946] stat("/usr/local/bin/ipmi-sensors", 0x7ffc48c70a20) = -1 ENOENT (No such file or directory)
[pid 1642946] stat("/usr/local/bin/ipmi-sensors", 0x7ffc48c70a20) = -1 ENOENT (No such file or directory)
[pid 1642946] stat("/usr/local/bin/ipmi-sensors", 0x7ffc48c70a20) = -1 ENOENT (No such file or directory)
[pid 1642946] stat("/usr/local/bin/ipmi-sensors", 0x7ffc48c70a20) = -1 ENOENT (No such file or directory)
[pid 1642946] stat("/usr/local/bin/ipmi-sensors", 0x7ffc48c70a20) = -1 ENOENT (No such file or directory)
The only cure for it then would be to restart the check-mk-agent-async service