"Deploy custom files with agent" doesn't work properly on some Linux systems

Im using Check_MK Enterprise 1.5.0p11

I have exactly worked according to the description in Check_MK and have provided my checks in /omd/sites/vine/local/share/check_mk/agents/custom/my_checks/lib/local/600

After executing the agent bakery and signing all agents with password the check_mk web gui displays no error and new hash tags for all agents.

But not all of my new custom checks have been deployed to the target hosts !
In rare cases of successfull deployment I can find my new checks on the production host at
/usr/lib/check_mk_agent/local/600

On both of my production hosts the files sometimes have been successfully deployed sometimes not. So all my new custom checks are not deployed / transfered and the “services” page in Check_MK still displays all my old services (names)

What is the cause ?
Which mechanism does Check_MK use to exchange the custom files on the target hosts ?

Hi Feingeist,

do you have a host-/tag based filter in your “Deploy custom files with agent” rule?
If true, this could be the root of your problems.
If not, it’ll be hard to decipher what exactly is your problem with the lack of information.

Did you checked your logs, if an error occured while baking?

Sincerely Kruzgoth

Yes, I use a filter. But it’s the right filter.
Which logs can I check in this case ?

On my Linux based monitoring server there is a check_mk_agent.rpm installed.

Normaly i would always take a look into the “cmc.log” and “web.log”, because there are the most
error messages to find.
Can you say for sure, that all hosts (with the problem and without), which should get this custom files, have the same agent (same hash version)?

For bakery troubleshooting it is also good to make a cmk --debug -f -vvA hostname.
There you should see all the files used to bake the agent.
You can control if all your wanted files are included.

After this build on the command line sign the agent inside WATO and check the deployment with an cmk-update-agent -v from the target server.
Now all the included files from the build should be on your target machine.
A good point for “what happens at install time” is the log file of the agent updater.

Meanwhile I had a look to the file /var/lib/check_mk_agent/cmk-update-agent.log on one of the monitored hosts. There we found an error entry exactly in that case where a new agent hash has been detected.

The system calls cmk_update_agent.py, update the cmk_update_agent.state etc. with the new received hash and raises the following exception and do nothing more! Means that my custom files are not transferred.

2020-05-29 18:37:38,270 DEBUG: Starting Check_MK Agent Updater v1.5.0p11
2020-05-29 18:37:38,271 DEBUG: Successfully read /etc/cmk-update-agent.state.
2020-05-29 18:37:38,272 DEBUG: Successfully read /etc/check_mk/cmk-update-agent.cfg.
2020-05-29 18:37:38,273 DEBUG: Starting manual update mode.
2020-05-29 18:37:38,273 INFO: Getting target agent configuration for host 'PROD-FE1' from deployment server
2020-05-29 18:37:38,283 DEBUG: Fetching content (using requests): http://xxxxxxxxxxx/vine/check_mk/deploy_agent.py
2020-05-29 18:37:38,306 DEBUG: Response from deployment server:
AgentAvailable: True
TargetHash: df49102dcdfa8e2b
Signature: 2d2d2d2d2d.....
2020-05-29 18:37:38,307 DEBUG: Successfully read /etc/cmk-update-agent.state.
2020-05-29 18:37:38,311 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2020-05-29 18:37:38,311 INFO: Target state (from deployment server):
2020-05-29 18:37:38,311 INFO:   Agent Available:     True
2020-05-29 18:37:38,311 INFO:   Signatures:          1
2020-05-29 18:37:38,311 INFO:   Target Hash:         df49102dcdfa8e2b
2020-05-29 18:37:38,319 DEBUG: Fetching content (using requests): http://xxxxxxxxxxxx/vine/check_mk/deploy_agent.py
2020-05-29 18:37:38,480 INFO: Downloaded agent has size 9631839 bytes.
2020-05-29 18:37:38,480 INFO: Skipping signature check (as you requested).
2020-05-29 18:37:38,480 DEBUG: Successfully read /etc/cmk-update-agent.state.
2020-05-29 18:37:38,483 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2020-05-29 18:37:38,547 DEBUG: Caught Exception:
Traceback (most recent call last):
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1215, in _install_agent
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1233, in _invoke_installer
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1252, in _install_agent_linux_rpm
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1320, in _invoke_unix_pkg_manager
AttributeError: 'NoneType' object has no attribute 'read'
2020-05-29 18:37:38,548 ERROR: Failed installing Check_MK agent: 'NoneType' object has no attribute 'read'.
 Keeping /tmp/check-mk-agent-jnFbih for error diagnosis.
2020-05-29 18:37:38,548 DEBUG: Caught Exception:
Traceback (most recent call last):
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1782, in main
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 670, in run
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1029, in _run_mode
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1077, in _do_update_as_command
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1175, in _do_update_agent
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1215, in _install_agent
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1233, in _invoke_installer
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1252, in _install_agent_linux_rpm
  File "/bauwelt-1.5.0p11/enterprise/agents/plugins/cmk_update_agent.py", line 1320, in _invoke_unix_pkg_manager
AttributeError: 'NoneType' object has no attribute 'read'
2020-05-29 18:37:38,548 ERROR: 'NoneType' object has no attribute 'read'

The function in cmk_update_agent, around line 1320 is:

 def _invoke_unix_pkg_manager(self, command):
        if self._config.verbose_level:
            stdout = sys.stdout
        else:
            stdout = open(os.devnull, "w")
        proc = subprocess.Popen(
            command,
            stdin=open(os.devnull),
            stdout=stdout,
            stderr=subprocess.STDOUT,
            close_fds=True,
            env=self._env.sub_env)
        status = proc.wait()
        if status:
**1320:       raise Exception("Error during installation of package:\n%s" % proc.stdout.read())**

Additionally I think if proc.wait() return without a status code, it indicates that the subprocess has not terminated. But in the other case a status indicated not automatically an error.

Do I have found some bugs in the Check_MK file cmk-update-agent.py ?

First step i would do is upgrade you 1.5 to the latest patch level -> p24.
If the error is also there you should consider an upgrade to 1.6.
The actual agent update script uses the better way with “proc.communicate()”.

But i can think what the problem is. This problem is also documented in some werks.
The agent hash inside the filename of the package is interpreted as a version string.
This leads to an false detection that you want to make a “downgrade” of your installed agent. Result is a failed installation.

This exception is handled inside the 1.6 upgrade agent script.

1 Like

Now we have installed check_mk Enterprise 1.6.0p12 on all our Linux hosts.
It seems so that the described error before has been fixed.

But we get a new error on one of our monitored hosts: The automatic agent update failed!

2020-06-03 23:50:13,634 DEBUG: Starting Check_MK Agent Updater v1.6.0p12
2020-06-03 23:50:13,635 DEBUG: Successfully read /etc/cmk-update-agent.state.
2020-06-03 23:50:13,635 DEBUG: Successfully read /etc/check_mk/cmk-update-agent.cfg.
2020-06-03 23:50:13,637 INFO: Starting Update mode as plugin.
2020-06-03 23:50:13,637 INFO: Getting target agent configuration for host 'VINE3-FE1' from deployment server
2020-06-03 23:50:13,645 DEBUG: Fetching content (using requests): http://....../vinetest/check_mk/deploy_agent.py
2020-06-03 23:50:13,673 DEBUG: Response from deployment server:
AgentAvailable: True
TargetHash: a8af450d7c538fd6
Signature: 2d2d2d2d2......

2020-06-03 23:50:13,674 DEBUG: Successfully read /etc/cmk-update-agent.state.
2020-06-03 23:50:13,691 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2020-06-03 23:50:13,692 INFO: Target state (from deployment server):
2020-06-03 23:50:13,692 INFO:   Agent Available:     True
2020-06-03 23:50:13,692 INFO:   Signatures:          1
2020-06-03 23:50:13,692 INFO:   Target Hash:         a8af450d7c538fd6
2020-06-03 23:50:13,698 DEBUG: Fetching content (using requests): http://....../vinetest/check_mk/deploy_agent.py
2020-06-03 23:50:13,817 INFO: Downloaded agent has size 9638837 bytes.
2020-06-03 23:50:13,836 INFO: Signature check OK.
2020-06-03 23:50:13,844 INFO: Invoking package manager: rpm -vU --oldpackage --replacepkgs /tmp/check-mk-agent-EZbzvq
2020-06-03 23:50:14,031 INFO: Output from rpm:
Preparing packages for installation...

2020-06-03 23:50:14,031 INFO: Warning/Error from rpm:
error: %pre(check-mk-agent-1.6.0p12-a8af450d7c538fd6.noarch) scriptlet failed, exit status 127
error:   install: %pre scriptlet failed (2), skipping check-mk-agent-1.6.0p12-a8af450d7c538fd6

2020-06-03 23:50:14,031 DEBUG: Caught Exception:
Traceback (most recent call last):
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1394, in _install_agent
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1417, in _invoke_installer
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1433, in _install_agent_linux_rpm
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1509, in _invoke_unix_pkg_manager
Exception: Error during installation of package
2020-06-03 23:50:14,033 ERROR: Failed installing Check_MK agent: Error during installation of package.
2020-06-03 23:50:14,033 INFO: Hint: You can try to install or analyze the agent package manually by downloading it from the checkmk server. You can find it by looking for your OS's package for host VINE3-FE1 with agent hash a8af450d7c538fd6 at the agent bakery.
2020-06-03 23:50:14,036 DEBUG: Successfully read /etc/cmk-update-agent.state.
2020-06-03 23:50:14,040 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2020-06-03 23:50:14,041 DEBUG: Caught Exception:
Traceback (most recent call last):
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1080, in _do_update_as_plugin
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1160, in _do_update_agent
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1379, in _trigger_installation
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1394, in _install_agent
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1417, in _invoke_installer
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1433, in _install_agent_linux_rpm
  File "/home/jenkins/workspace/cmk_160/nightly_build_containerized/agent-dest/enterprise/agents/plugins/cmk_update_agent.py", line 1509, in _invoke_unix_pkg_manager
Exception: Error during installation of package
2020-06-03 23:50:14,041 DEBUG: Writing agent section to stdout:
<<<check_mk>>>
AgentUpdate: last_check 1591221013.67 last_update 1591200559.57 aghash da62eebebb2bc12c pending_hash None error Error during installation of package

2020-06-03 23:50:14,042 DEBUG: Sending new state data to agent bakery
2020-06-03 23:50:14,055 DEBUG: Fetching content (using requests): http://ac1vmds-vin07.devacu.ascom-sus.local/vinetest/check_mk/deploy_agent.py
2020-06-03 23:50:14,092 DEBUG: Done.

How can we solve the rpm error ?

What happens if you install the agent with the provided command from the log file manually?

rpm -vU --oldpackage --replacepkgs downloaded_package.rpm

Other thing your agent is called as root user i think or not?

Ok, the actual original call on our monitored host done by the local Check_MK update mechanism was:
rpm -vU --oldpackage --replacepkgs /tmp/check-mk-agent-EZbzvq

To simulate this, I have downloaded the corresponding rpm agent file for this host from Check_MK’s bakery page. Then I have put this file to our host’s /tmp dir, so that I (as root user) can execute the following line and got the following logs:

[root@v3-dev-wh1 tmp]# rpm -vU --oldpackage --replacepkgs /tmp/check-mk-agent-1.6.0p12-4c75ab0cca3cf3fa.noarch.rpm
Preparing packages for installation...
check-mk-agent-1.6.0p12-4c75ab0cca3cf3fa
Activating startscript of xinetd
Reloading xinetd...
Reloading configuration: [  OK  ]
Reloading xinetd...
Reloading configuration: [  OK  ]

After that operation my custom file in
/usr/lib/check_mk_agent/local/600/check_ws_DUMMY2
and the agent in
/usr/lib/check_mk_agent/plugins/600/cmk_update_agent
both have been updated!

But all following automated updates executed in 10min intervall as logged in file
/var/lib/check_mk_agent/cmk-update-agent.log
still fail as described in my previous posting.

How can I examine whether the agent is called as root user ?

This depends on your setup. Do you use the xinetd method or ssh to call your agent?
I have many systems where i cannot run the agent as root, there i need to use special users and these users have no sudo/root privilege and with this cannot install packages.

On the monitored host I found in file /etc/xinetd.d/check-mk-agent the following content:

# Created by Check_MK Agent Bakery.
# This file is managed via WATO, do not edit manually or you 
# lose your changes next time when you update the agent.

service check-mk-agent
{
        type           = UNLISTED
        port           = 6556
        socket_type    = stream
        protocol       = tcp
        wait           = no
        user           = root
        server         = /usr/bin/check_mk_agent
        log_on_success =
        disable        = no
}

So I think the xinetd method as root user is used to call the agent updater.

If on your monitoring server no other datasource (ssh) is configured then yes.
But you need to look at the monitoring server “cmk -D hostname” gives you the type of agent the host is using.

cmk -D returns a dump info. Among other data the folowing data is avialable:

Agent mode:             Normal Checkmk agent, or special agent if configured
Type of agent:
  TCP: xxx.xxx.xxx.xxx:6556
  Process piggyback data from /omd/sites/vinetest/tmp/check_mk/piggyback/VINE3-FE1

Some additional information: The monitored server VINE3-FE1 works under a Red Hat Linux and there is a Python 2.6.6. on board. But I think this does’nt matters because we have installed Check_MK’s binary agent parts. On the monitored server Check_MK comes with a Python 2.7.17

The agent is called with the normal xinetd over port 6556 and with your posted config from xinetd it should run with root user. Then i don’t know what’s the difference between your two commands.
Was the exit code also 0 as you tested it manually?

Which two commands do you mean ?

The rpm commands you tested and the rpm command the agent gave the error message.

I have no exit codes seen. But I think if I execute the rpm tool manually all things work fine.
But in the automatic case the log reported the error code from rpm:

2020-06-03 23:50:13,844 INFO: Invoking package manager: rpm -vU --oldpackage --replacepkgs /tmp/check-mk-agent-EZbzvq
2020-06-03 23:50:14,031 INFO: Output from rpm:
Preparing packages for installation...

2020-06-03 23:50:14,031 INFO: Warning/Error from rpm:
error: %pre(check-mk-agent-1.6.0p12-a8af450d7c538fd6.noarch) scriptlet failed, exit status 127
error:   install: %pre scriptlet failed (2), skipping check-mk-agent-1.6.0p12-a8af450d7c538fd6

It is this bug ?
https://bugzilla.redhat.com/show_bug.cgi?id=1054350

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.