Difference in output between telnet and check_mk

Troubleshooting
1

I have created a custom local script to check kubernetes pods. However, i am now baffled as to why when i turn off a pod the local output displays correctly when i run the script locally or run the check_mk_agent command the correct warning is displayed. Here is an example:

2 cl5-jenkins-pods varname=2;crit Warning, pods are inconsistent on namespace cl5-jenkins

However, when i run this from the telnet output im getting that everything is ok still…

0 cl5-jenkins-pods varname=0;ok There are no missing pods on namespace cl5-jenkins

could anyone explain as to why this is happening? I thought maybe it was some permission issue but i dont think thats the reason!

thank you

Nicholas

2

It is a typical user rights problem. If you query the agent over telnet with what user is the agent running then? Your local test was not with the same user i think.

3

Hi again,

so thanks for the reply.

I think i have now found the issue. Basically in my script i output the commands to a temp file on the os. When i telnet the script it generates the file but does not populate it. I thought maybe this is a permission issue but ive modified the file to be written by any user and the telnet output still does not populate the temp file. Is there something else im missing here ? I also put full paths to the commands to make sure theres no issue with missing paths.

4

Ok upon further testing it appears as if the checkmk cannot run the kubectl command! Is there any workaround for this ?

5

The CMK agent normally runs as root if not configured different. As this user it should be also possible to run a kubectl.

6

yeah its as root… is there a debug flag for the telnet to see the full output of my checkmk local scripts to actually see the error im getting ?

7

You should see normally all the output. If i make a programming mistake i see this inside the agent output.

8

Maybe you’re using an environment variable in your script that’s not set when telnetting? Had a similar issue with the mysql check. The mysqladmin was not in the standard path so I had to modify the checkmk@.service to include the updated path.

1 Like
9

i tried to put the full path suich as: /usr/bin/kubectl into the script to no avail.

10

I had an issue closely related to this. Problem was that my local script required an actual shell even though I was using subprocess with shell=True. I changed the script to build a kind of fake shell with environment variables that were required.

11

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.