Agent-initiated checks instead of server-initiated checks

Hello community,

I’m new to checkmk and have a question on deploy agents on remote networks. We have a couple of remote offices outside our data centre where I would like to monitor a few Windows computers that are behind a firewall. I could use port forwarding to 6556 but sometimes these computers change networks. Ideally, I would like to have an agent on these computers that connects to the server to send updates, rather than the server connecting to the agent to gather checked data, so that we never have to worry about exposing ports on remote firewalls, and we’re always able to gather stats from these remote systems regardless of where they are. Is this possible with the checkmk agent? Or is there a different way to achieve this sort of setup? Note that we’re using checkmk enterprise. thank you!

The proper way would be to setup distributed monitoring, that is: separate “slave” checkmk servers in your remote offices. You would then configure your “master” checkmk server in the data centre to fetch the data from the slave instances via a secured tcp connection.

Another way would be to have cronjobs (or systemd timers) on the computers in the remote offices that call the check_mk_agent every minute and push the output via ssh to the data centre into a file, ideally with the same name as the remote’s hostnames. Then setup the checkmk server so that it doesn’t fetch the data by connecting to port 6556 but to just read the file instead.

I would definitely not recommend the second approach. Been there, done that. No fun. You get problems with outdated files or even no files at all. Also, checkmk cannot properly check if a host is up or down.

A third possibility might be the dynamic host configuration.

2 Likes

Thank you Dirk! I’m with you regarding #2 - I will explore the first and third options. Appreciate the detailed response.

@Ktivdv,

You’re welcome. It was a mere overview of what came to my mind for your situation. Maybe others have a fourth or fifth idea. :thinking:

You can do this. Have your remote host run a job at a scheduled interval that:

  1. Runs the agent and captures to a file.
  2. Copies the agent data result to a named (by the remote host) file on something accessible by your checkmk monitor (the monitor system itself maybe).

(use rsync, scp or whatever you may have, you might setup some sort of key’d copy mechanism, for safer passwordless copying of the agent data)

Then on the monitor, you’ll want to use a rule for “Individual program call instead of agent access”. That local to the monitor script would check for the presence of the “named file” (above) first (return fail if not, so which would be a host like “ping” fail) and then would return the contents of the “named file” and then remove it. The local monitor script would likely need to make sure the copy from #2 above is finished somehow (look for “last” entry in agent return data being present).

So, the idea is the scheduled job on the remote host should send it’s agent data. If not, you assume the host is down. If it’s there and complete, that becomes the substitute agent output.

Use the rule “Host check command” on the monitor for the remote host to tell it to use the “status of the Check_Mk agent” for it’s host check (instead of ping).

Should work (in theory, I did not test this)

1 Like

@cjcox,

To make sure only fully transmitted files get considered on the server side, you would use code like this on the client side:

check_mk_agent | ssh user@checkmk-server \
    "cat > $(hostname -f).part && mv $(hostname -f).part $(hostname -f)"

That is: pipe the output of check_mk_agent to a temporary file directly on the server through the ssh connection and then mv that temporary file to the final file (using the local hostname).

This works because mv is an atomic operation when done in the same filesystem. No need to scan the file for some last entry. The file either exists on the server side (and was completely transferred) – or not.

2 Likes

Correct, but I was trying not to make an assumption with regards to shell access, etc. to the monitor host.

But yes, this might be a great way for those that have full shell access to the host, or you can use a invoke command on the key for a specific program on the monitor side (perhaps a bit more secure).

Edit: There are some risks to this approach when things “hang” and do not complete. YMMV.

Hi all, many thanks for all the great input. When I determine the best approach for our situation and if I have any useful updates, I will report back.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.