Multiple Check_MK Agents on same monitored host

General
, ,
1

Hello together,

I have a question regarding the installation of multiple Check_MK Agents on a same host.
Here is the situation based on my favored concept I thought through. Please let me just know if my concept is bad, if there are better ways to achieve what I want or when I’m just missing something.

Situation (my favored concept):

I have one single monitoring-server which serves two sites:

  • Site test
    • purpose: testing checkmk-configurations (rules, etc.), before manually applying the “needed
      configurations” on the prod-site; can be a playground aswell when testing stuff
    • has got a site-specific Check_MK Agent, which is configured via AgentBakery
      • uses the test-site as deployment-server for auto-updates
  • Site prod
    • config-state differs from test-site; everything here was beforhand configured and tested on test-site
    • purpose: mirrors the “real and correct configuration”, sending various important alerts etc pp.
    • has got a site-specific Check_MK Agent, which is configured via AgentBakery
      • uses the prod-site as deployment-server for auto-updates
  • important: in order to correctly test every config-adjustments, both sites need to have the exact same hosts to be monitored
  • because the agent from test-site and prod-site can differ, I need both agents to be installed on the monitored host accessed by different ports

As mentioned, both sites serve their own Check_MK Agent. Both agents, let’s call them test-agent and prod-agent, are configured via AgentBakery on their respective site. Because I think its needed that the two agents differ, I already configured the following rules to make them different:

  • Rule TCP port for connection to Checkmk agent: test-agent is accessed by 6556, prod-agent is accessed by 6557
  • Rule Installation paths for agent files (Linux, UNIX)
    • Directory for configuration files: /etc/check_mk_test (for prod-agent its _prod)
    • Base directory for plugins and local: /usr/lib/check_mk_agent_test (for prod-agent its _prod)
    • Base directory for variable data (caches, state files): /var/lib/check_mk_agent_test (for prod-agent its _prod)
  • Rule Name of agent packages (Linux, UNIX): check-mk-agent-test (for prod-agent its prod)*

So far so good - theoretically…

The Problem is that I can not install both agents at the same time.
When I install the prod-agent via dpkg -i when the test-agent is already installed,
then the agent-controller and deployed systemd units, which got installed by test-agent beforhand, gets uninstalled (orange box) and there is a problem with the super-server.cfg (red box) because it is already from the test-agent. Somehow they collide, interfere each other, even though I changed for example some paths in rules.

doubleinstallerror
doubleinstallerror3040×440 91.8 KB

And when I try to uninstall an agent, it says that “/var/lib/cmk-agent” is not empty but I can not find a rule in AgentBakerey to change it to “/var/lib/cmk-agent-test” for example.

Any ideas? Is it not the “checkmk-way” / bad practice to have two Check_MK-Agents installed simultaneously to test stuff?
Is it a dpkg-issue and I should install the second agent with dpkg -i --force-overwrite agent.deb?

I am really looking forward for help! Thanks in advance.

Greetings,

Julien

2

I understand the problem. Just call the production agent the way you always do (agent controller) and deploy a second agent script that uses a few modified variables. Call this via SSH as described in the Linux legacy article.

3

I understand what you are trying to accomplish, but I am not sure that your approach is the right one:

  • Although the Checkmk agent is quite lightweight, the resource consumption on the monitored servers is not zero.
  • If you implement the same approach for SNMP devices, some of them might break because they get overloaded. You can find some posts here in the forum discussing this issue
4

Thanks for your response.

So the prod-agent would be configured via AgentBakery rulesets: which sections to monitor? which plugins to rollout? what to monitor etc.) + auto-updates etc

And the test-agent would be just a script / binary. Is it also configured with AgentBakery to benefit from auto-updates, configure it via rulesets to tell which plugins to rollout etc? Or would that be all manually. Because I want to configure both prod-agent and test-agent “in the same way”. Prod-agent with AgentBakery for configuring seems better in my eyes…

Is there no checkmk-builtin-way to make two checkmk-agents run “as intended” with agent-controller etc. the whole package?

5

Thanks for your response.

I think the first point is OK since i will set the test-site for 60min intervall and the prod-site for a quicker intervall.

SNMP devices is something I wont have I suppose.