Custom special agent not able to call file in libexec folder

Checkmk Dev&Exchange
1

CMK version: 2.5.0p3
OS version: Ubuntu 26.04 LTS

Error message: Program ‘local/lib/python3/cmk_addons/plugins/xxxxx/libexec/agent_xxxxx’ not found (exit code 127)

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

[special_hellospecial] Program ‘local/lib/python3/cmk_addons/plugins/hellospecial/libexec/agent_hellospecial’ not found (exit code 127)

The special agent call is not able to start the exec file.

I spent many hours on a custom plugin to understand why it was not working, and in the end tried to use the basic hellospecial example here Developing special agents with the exact same issue.

The file exists, has no CRLF issue, has the right executable permissions and user/group ownership, but the issue persists.

Easily reproductible when following the steps from the documentation to create the hellospecial agent.

2

Can you pls do a “ls -lR” inside your plugin folder → “~/local/lib/python3/cmk_addons/plugins/xxxx/”?
For easy troubleshooting the whole code would also be helpful.

3 
OMD[main]:~$ ls -lR ~/local/lib/python3/cmk_addons/plugins/hellospecial/libexec/
/omd/sites/main/local/lib/python3/cmk_addons/plugins/hellospecial/libexec/:
total 4
-rwxr-xr-x 1 main main 91 May 28 17:28 agent_hellospecial*

OMD[main]:~$ cmk --debug testhost
[special_hellospecial] Program 'local/lib/python3/cmk_addons/plugins/hellospecial/libexec/agent_hellospecial' not found (exit code 127)(!!)

To avoid any doubt about my custom script I am now using the example from the documentation. The content of the scripts are exactly the example hellospecial from the documentation here Developing special agents

One interesting note maybe, if I actually delete/rename the file just to see what happens the error is slightly different without the relative path:

OMD[main]:~$ mv ~/local/lib/python3/cmk_addons/plugins/hellospecial/libexec/agent_hellospecial ~/local/lib/python3/cmk_addons/plugins/hellospecial/libexec/agent_hellospecial.bak
OMD[main]:~$ cmk --debug testhost
[special_hellospecial] Program 'agent_hellospecial' not found (exit code 127)(!!),
4

Does running it as a site user using a relative path work? This should have most of the environment the same as when run from Checkmk:

./local/lib/python3/cmk_addons/plugins/hellospecial/libexec/agent_hellospecial

5

Yes indeed running it manually from the OMD user works fine:
OMD[main]:~$ ./local/lib/python3/cmk_addons/plugins/hellospecial/libexec/agent_hellospecial
<<>>
0 “Hello special” - This static service is always OK

The issue is only when called from the special agent.

6

I tested it with a relatively fresh 2.5.0p3 on Ubuntu 24.04 where I have no issues with the examples I have written for the User Guide. I will not have time today to run a 26.04 cloud init image today. Besides being on a different OS: Has the site been updated all the way from 2.3 to 2.5 or was it already started at some late 2.4 or even 2.5?

7

It is better to look first with “cmk -D testhost”. Here the system should show you the complete path it tries to execute.

Agent mode:             Normal Checkmk agent, or special agent if configured
Type of agent:          
  Program: /omd/sites/cmk/lib/python3/cmk_addons/plugins/hellospecial/libexec/agent_hellospecial
8

Installed fresh on 2.5.0p3, this is a brand new setup. We’re using the ultimate version if that makes any difference.

@andreas-doehler I tried with cmk -D but I still don’t get the full path, which might be the issue

Agent mode:             Normal Checkmk agent, or special agent if configured
Type of agent:
  Program: local/lib/python3/cmk_addons/plugins/hellospecial/libexec/agent_hellospecial
9

OK, I spun up a fresh Ubuntu cloud init image, installed Checkmk 2.5.0p3 Ultimate on it and tested the example:

OMD[mysite]:~$ cmk -D localhost

localhost
Addresses:              127.0.0.1
Tags:                   [address_family:ip-v4-only], [agent:cmk-agent], [checkmk-agent:checkmk-agent], [criticality:prod], [ip-v4:ip-v4], [networking:lan], [piggyback:auto-piggyback], [site:mysite], [snmp_ds:no-snmp], [tcp:tcp]
Labels:                 [cmk/site:mysite]
Host groups:            check_mk
Contact groups:         all
Agent mode:             Normal Checkmk agent, or special agent if configured
Type of agent:
Program: /omd/sites/mysite/local/lib/python3/cmk_addons/plugins/hellospecial/libexec/agent_hellospecial
Process piggyback data
Services:
checktype item          params description   groups
--------- ------------- ------ ------------- ------
local     Hello special {}     Hello special

There is no difference whether I log in as site user via ssh or via omd su mysite. Even su mysite (not recommended!) works. What can be different in our setups?

10

This is my environment logging in via omd su mysite. Please compare with your environment.

root@cloud:/home/ubuntu# omd su mysite
OMD[mysite]:~$ env
SHELL=/bin/bash
OMD_ROOT=/omd/sites/mysite
SERVER_SIDE_PROGRAM_STORAGE_PATH=/omd/sites/mysite/var/check_mk/server_side_program_storage
NAGIOS_PLUGIN_STATE_DIRECTORY=/omd/sites/mysite/var/monitoring-plugins
RABBITMQ_NODENAME=rabbit-mysite@localhost
PWD=/omd/sites/mysite
LOGNAME=mysite
MANPATH=/omd/sites/mysite/share/man:
MODULEBUILDRC=/omd/sites/mysite/.modulebuildrc
HOME=/omd/sites/mysite
LANG=C.utf8
RABBITMQ_HOME=/omd/sites/mysite
PERL5LIB=/omd/sites/mysite/local/lib/perl5/lib/perl5:/omd/sites/mysite/lib/perl5/lib/perl5:
SERVER_SIDE_PROGRAM_CRASHES_PATH=/omd/sites/mysite/var/check_mk/crashes
PASSWORD_STORE_SECRET_FILE=/omd/sites/mysite/etc/password_store.secret
OMD_SITE=mysite
TERM=rxvt-unicode-256color
USER=mysite
SQLITE_TMPDIR=/omd/sites/mysite/tmp
PERL_MM_OPT=INSTALL_BASE=/omd/sites/mysite/local/lib/perl5/
SHLVL=1
LD_LIBRARY_PATH=/omd/sites/mysite/local/lib:/omd/sites/mysite/lib
SSL_CERT_FILE=/omd/sites/mysite/var/ssl/ca-certificates.crt
DEBUGINFOD_URLS=https://debuginfod.ubuntu.com 
REQUESTS_CA_BUNDLE=/omd/sites/mysite/var/ssl/ca-certificates.crt
LC_ALL=C.utf8
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
PATH=/omd/sites/mysite/lib/rabbitmq/sbin:/omd/sites/mysite/lib/perl5/bin:/omd/sites/mysite/local/bin:/omd/sites/mysite/bin:/omd/sites/mysite/local/lib/perl5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
MP_STATE_DIRECTORY=/omd/sites/mysite/var/monitoring-plugins
MAIL=/var/mail/mysite
MAILRC=/omd/sites/mysite/etc/mail.rc
_=/usr/bin/env
11

I have the exact same apart from obviously the site name and one additional variable “SGX_AESM_ADDR=1” which I doubt is related

OMD[main]:~$ env
SHELL=/bin/bash
OMD_ROOT=/omd/sites/main
SERVER_SIDE_PROGRAM_STORAGE_PATH=/omd/sites/main/var/check_mk/server_side_program_storage
NAGIOS_PLUGIN_STATE_DIRECTORY=/omd/sites/main/var/monitoring-plugins
RABBITMQ_NODENAME=rabbit-main@localhost
PWD=/omd/sites/main
LOGNAME=main
MANPATH=/omd/sites/main/share/man:
MODULEBUILDRC=/omd/sites/main/.modulebuildrc
HOME=/omd/sites/main
LANG=C.utf8
RABBITMQ_HOME=/omd/sites/main
PERL5LIB=/omd/sites/main/local/lib/perl5/lib/perl5:/omd/sites/main/lib/perl5/lib/perl5:
SERVER_SIDE_PROGRAM_CRASHES_PATH=/omd/sites/main/var/check_mk/crashes
PASSWORD_STORE_SECRET_FILE=/omd/sites/main/etc/password_store.secret
SGX_AESM_ADDR=1
OMD_SITE=main
TERM=xterm-256color
USER=main
SQLITE_TMPDIR=/omd/sites/main/tmp
PERL_MM_OPT=INSTALL_BASE=/omd/sites/main/local/lib/perl5/
SHLVL=1
LD_LIBRARY_PATH=/omd/sites/main/local/lib:/omd/sites/main/lib
SSL_CERT_FILE=/omd/sites/main/var/ssl/ca-certificates.crt
DEBUGINFOD_URLS=https://debuginfod.ubuntu.com
REQUESTS_CA_BUNDLE=/omd/sites/main/var/ssl/ca-certificates.crt
LC_ALL=C.utf8
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
PATH=/omd/sites/main/lib/rabbitmq/sbin:/omd/sites/main/lib/perl5/bin:/omd/sites/main/local/bin:/omd/sites/main/bin:/omd/sites/main/local/lib/perl5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
MP_STATE_DIRECTORY=/omd/sites/main/var/monitoring-plugins
MAIL=/var/mail/main
MAILRC=/omd/sites/main/etc/mail.rc
_=/usr/bin/env

The only other thing maybe, is that this server is a VM running on Azure, so using the provided Ubuntu image on Azure portal.

I am really not sure what else it could be? :frowning:

12

As I am using the Cloud Init image without conversion and directly on KVM, / is on ext4 for me. Can you check your mounts whether / is on a network file system and/or /opt is a separate mountpoint?

root@cloud:/home/ubuntu# cat /proc/mounts 
tmpfs /run tmpfs rw,nosuid,nodev,size=799696k,nr_inodes=819200,mode=755,inode64 0 0
/dev/vda1 / ext4 rw,relatime,discard,errors=remount-ro,commit=30 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=1542444k,nr_inodes=385611,mode=755,inode64 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64,usrquota 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=600,ptmxmode=000 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot,memory_hugetlb_accounting 0 0
none /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5726 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,nosuid,nodev,relatime,pagesize=2M 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,nr_inodes=1048576,inode64,usrquota 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
none /run/credentials/systemd-journald.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap 0 0
none /run/credentials/systemd-resolved.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap 0 0
/dev/vda13 /boot ext4 rw,relatime 0 0
/dev/vda15 /boot/efi vfat rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
none /run/credentials/systemd-networkd.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap 0 0
none /run/credentials/serial-getty@ttyS0.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap 0 0
none /run/credentials/getty@tty1.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=399844k,nr_inodes=99961,mode=700,uid=1000,gid=1000,inode64 0 0
tmpfs /opt/omd/sites/mysite/tmp tmpfs rw,nosuid,nodev,noexec,relatime,mode=751,uid=999,gid=982,inode64 0 0
tmpfs /run/user/999 tmpfs rw,nosuid,nodev,relatime,size=399844k,nr_inodes=99961,mode=700,uid=999,gid=982,inode64 0 0
13

It looks normal on ext4 and not a separate mountpoint:

sudo cat /proc/mounts
/dev/root / ext4 rw,relatime,discard,errors=remount-ro,commit=30 0 0
devtmpfs /dev devtmpfs rw,nosuid,noexec,relatime,size=3564252k,nr_inodes=891063,mode=755,inode64 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64,usrquota 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=600,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,size=1615048k,nr_inodes=819200,mode=755,inode64 0 0
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot,memory_hugetlb_accounting 0 0
none /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1893 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,nosuid,nodev,relatime,pagesize=2M 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,nr_inodes=1048576,inode64,usrquota 0 0
none /run/credentials/systemd-journald.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
none /run/credentials/systemd-resolved.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap 0 0
/dev/loop0 /snap/certbot/5603 squashfs ro,nodev,relatime,errors=continue,threads=single 0 0
/dev/loop1 /snap/snapd/26865 squashfs ro,nodev,relatime,errors=continue,threads=single 0 0
/dev/loop2 /snap/core24/1643 squashfs ro,nodev,relatime,errors=continue,threads=single 0 0
/dev/sda13 /boot ext4 rw,relatime,discard 0 0
/dev/sda15 /boot/efi vfat rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
none /run/credentials/systemd-networkd.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap 0 0
tmpfs /opt/omd/sites/main/tmp tmpfs rw,nosuid,nodev,noexec,relatime,mode=751,uid=997,gid=981,inode64 0 0
none /run/credentials/serial-getty@ttyS0.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap 0 0
none /run/credentials/getty@tty1.service tmpfs ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=807520k,nr_inodes=201880,mode=700,uid=1000,gid=1000,inode64 0 0
14

At that point, I am out of ideas. I will forward to our devs.

15

Thank you, I’m available for any more debugs / tests if it helps.

16

What would help is your complete code that is used in your test.
It think the problem lies somewhere in the written code..

17

As I explained I am NOT using my custom plugin for these tests, I am using the example plugin from the documentation Developing special agents

The code is also on github if you wish checkmk-docs/examples/devel_special_agents at 2.5.0 · Checkmk/checkmk-docs · GitHub but this is not the issue, @mschlenker has no problem with that same code on his side.

18

Also if you use the example code the real files on the system can be different from the example. If your used example is inside an MKP every one can test with exactly the same code as on your system.
One simple typo can be enough to produce an problem as on your system.
I also checked with the example from the documentation and got no problem.
That’s why i said without the real code used it will be very difficult to find the real cause.

19

I simply did a copy paste from the documentation, but to avoid any doubt on this here are the files:

OMD[main]:~/local/lib/python3/cmk_addons/plugins/hellospecial$ cat rulesets/special_agent.py
#!/usr/bin/env python3
# Shebang needed only for editors

from cmk.rulesets.v1.form_specs import Dictionary
from cmk.rulesets.v1.rule_specs import SpecialAgent, Topic, Help, Title

def _formspec():
    return Dictionary(
        title=Title("Hello special!"),
        help_text=Help("This rule is to demonstrate the minimum special agent."),
        elements={}
    )

rule_spec_hellospecial = SpecialAgent(
    topic=Topic.GENERAL,
    name="hellospecial",
    title=Title("Hello special!"),
    parameter_form=_formspec
)


OMD[main]:~/local/lib/python3/cmk_addons/plugins/hellospecial$ cat server_side_calls/special_agent.py
#!/usr/bin/env python3
# Shebang needed only for editors

from cmk.server_side_calls.v1 import noop_parser, SpecialAgentConfig, SpecialAgentCommand

def _agent_arguments(params, host_config):
    yield SpecialAgentCommand(command_arguments=[])

special_agent_hellospecial = SpecialAgentConfig(
    name="hellospecial",
    parameter_parser=noop_parser,
    commands_function=_agent_arguments
)


OMD[main]:~/local/lib/python3/cmk_addons/plugins/hellospecial$ cat libexec/agent_hellospecial
#!/bin/bash
echo '<<<local>>>'
echo '0 "Hello special" - This static service is always OK'
20

I also did copy and paste from the User Guide to rule out problems elsewhere. Broken syntax means the rule cannot be configured, so we can assume something like that is not the cause. Since the site was new, subtle differences between the Azure provided cloud image and the vanilla cloud init images might be a cause.

Another question: Did you setup Checkmk completely fresh on this 26.04 virtual machine or has it been updated before for example from Checkmk 2.4 on 24.04 to the current state? Could you maybe invest the one € or so to spin up a completely fresh 26.04 (throw-away) cloud image to do completely make sure no other or older configuration causes problems?