2.1.0p11: cmk-agent-ctl not working inside the official docker container

CMK version: 2.1.0p11
(image digest: 42f32e675ab8)

OS version:
Debian 10 (buster) - inside the container
Host is running Ubuntu 20.04.5 LTS, and Docker version 20.10.18, build b40c2f6

Error message:

ERROR [cmk_agent_ctl] Failed to run as user 'cmk-agent'. Please execute with sufficient permissions (maybe try 'sudo').

Caused by:
    Could not find dedicated Checkmk agent user cmk-agent

Output of “cmk --debug -vvn hostname”:

cmk debug output
OMD[cmk]:~$ cmk --debug -vvn localhost
Checkmk version 2.1.0p11
Try license usage history update.
Trying to acquire lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
Got lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Got lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Released lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
Released lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7f0749cf1b50]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(localhost, base_path=/omd/sites/cmk/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 53 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 127.0.0.1:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Reading data from agent
Write data to cache file /omd/sites/cmk/tmp/check_mk/cache/localhost
Trying to acquire lock on /omd/sites/cmk/tmp/check_mk/cache/localhost
Got lock on /omd/sites/cmk/tmp/check_mk/cache/localhost
Releasing lock on /omd/sites/cmk/tmp/check_mk/cache/localhost
Released lock on /omd/sites/cmk/tmp/check_mk/cache/localhost
Closing TCP connection to 127.0.0.1:6556
[cpu_tracking] Stop [7f0749cf1b50 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.47999999672174454))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f0749ce0fa0]
[PiggybackFetcher] Fetch with cache settings: NoCache(localhost, base_path=/omd/sites/cmk/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'localhost'. Skip processing.
No piggyback files for '127.0.0.1'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7f0749ce0fa0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<docker_container_mem>>> / Transition HostSectionParser -> HostSectionParser
<<<docker_container_cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<docker_container_diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_status:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<livestatus_ssl_certs:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<mkeventd_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cmk_site_statistics:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_status:cached(1663099775,60)>>> / Transition HostSectionParser -> HostSectionParser
<<<mknotifyd:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_apache:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<omd_info:sep(59)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'cifsmounts', 'cmk_agent_ctl_status', 'cmk_site_statistics', 'docker_container_cpu', 'docker_container_diskstat', 'docker_container_mem', 'job', 'labels', 'livestatus_ssl_certs', 'livestatus_status', 'lnx_if', 'local', 'md', 'mkeventd_status', 'mknotifyd', 'mounts', 'nfsmounts', 'omd_apache', 'omd_info', 'omd_status', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
  -> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7f0749ce0190]
value store: synchronizing
Trying to acquire lock on /omd/sites/cmk/tmp/check_mk/counters/localhost
Got lock on /omd/sites/cmk/tmp/check_mk/counters/localhost
value store: loading from disk
Releasing lock on /omd/sites/cmk/tmp/check_mk/counters/localhost
Released lock on /omd/sites/cmk/tmp/check_mk/counters/localhost
CPU utilization      Total CPU: 4.03%
Check_MK Agent       Version: 2.1.0p11, OS: linux, Agent plugins: 0, Local checks: 0
Disk IO SUMMARY      Read: 0.00 B/s, Write: 9.18 kB/s
Interface 2          [eth0], (up), MAC: 02:42:AC:12:00:02, Speed: 10 GBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
MD Softraid md0      Status: active, Spare: 0, Failed: 0, Active: 2, Status: 2/2, UU
MD Softraid md1      Status: active, Spare: 0, Failed: 0, Active: 2, Status: 2/2, UU
Memory               RAM: 1.17% - 751 MiB of 62.7 GiB
Mount options of /opt/omd/sites Mount options exactly as expected
OMD cmk Event Console Current events: 0, Virtual memory: 178.79 MB, Overall event limit inactive, No hosts event limit active, No rules event limit active, Received messages: 0.00/s, Rule hits: 0.00/s, Rule tries: 0.00/s, Message drops: 0.00/s, Created events: 0.00/s, Client connects: 0.02/s, Rule hit ratio: -, Processing time per message: -, Time per client request: 0.21 ms
OMD cmk apache       0.02 Requests/s, 0.00 Seconds serving/s, 0.06 B Sent/s
OMD cmk performance  Livestatus version: 2.1.0p11, Host checks: 0.0/s, Service checks: 0.3/s
OMD cmk status       running
Postfix Queue        Deferred queue length: 0, Active queue length: 0
Site cmk statistics  Total hosts: 1, Problem hosts: 0, Total services: 19, Problem services: 0
TCP Connections      Established: 2
Uptime               Up since Sep 13 2022 15:46:58, Uptime: 6 hours 24 minutes
+ EXECUTING INVENTORY PLUGINS
 aruba_wlc_aps: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 check_mk: ok
 check_mk: skipped (no data)
 checkmk_agent_plugins: ok
 checkmk_agent_plugins: skipped (no data)
 docker_node_info: skipped (no data)
 docker_node_info: skipped (no data)
 fritz: skipped (no data)
 fritz: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_mem: skipped (no data)
 hp_proliant_mem: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ipmi_firmware: skipped (no data)
 ipmi_firmware: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_if: ok
 inv_if: skipped (no data)
 aix_packages: skipped (no data)
 aix_packages: skipped (no data)
 aix_service_packs: skipped (no data)
 aix_service_packs: skipped (no data)
 inventory_checkmk: ok
 inventory_checkmk: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 dmidecode: skipped (no data)
 dmidecode: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_network: skipped (no data)
 docker_container_network: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_network: skipped (no data)
 docker_node_network: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortigate_ha: skipped (no data)
 fortigate_ha: skipped (no data)
 fortimail_system: skipped (no data)
 fortimail_system: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_system: skipped (no data)
 fortisandbox_system: skipped (no data)
 k8s_assigned_pods: skipped (no data)
 k8s_assigned_pods: skipped (no data)
 k8s_daemon_pod_containers: skipped (no data)
 k8s_daemon_pod_containers: skipped (no data)
 k8s_endpoint_info: skipped (no data)
 k8s_endpoint_info: skipped (no data)
 k8s_ingress_infos: skipped (no data)
 k8s_ingress_infos: skipped (no data)
 k8s_job_container: skipped (no data)
 k8s_job_container: skipped (no data)
 kube_cluster: skipped (no data)
 kube_cluster: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_deployment: skipped (no data)
 kube_deployment: skipped (no data)
 kube_namespace: skipped (no data)
 kube_namespace: skipped (no data)
 kube_node: skipped (no data)
 kube_node: skipped (no data)
 kube_pod: skipped (no data)
 kube_pod: skipped (no data)
 kube_statefulset: skipped (no data)
 kube_statefulset: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_video: skipped (no data)
 lnx_video: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mssql_clusters: skipped (no data)
 mssql_clusters: skipped (no data)
 oracle_systemparameter: skipped (no data)
 oracle_systemparameter: skipped (no data)
 snmp_extended_info: skipped (no data)
 snmp_extended_info: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_psrinfo: skipped (no data)
 solaris_psrinfo: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_prtpicl: skipped (no data)
 solaris_prtpicl: skipped (no data)
 solaris_routes: skipped (no data)
 solaris_routes: skipped (no data)
 statgrab_net: skipped (no data)
 statgrab_net: skipped (no data)
 win_disks: skipped (no data)
 win_disks: skipped (no data)
 win_exefiles: skipped (no data)
 win_exefiles: skipped (no data)
 win_ip_r: skipped (no data)
 win_ip_r: skipped (no data)
 win_networkadapter: skipped (no data)
 win_networkadapter: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_video: skipped (no data)
 win_video: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_updates: skipped (no data)
 win_wmi_updates: skipped (no data)
 juniper_info: skipped (no data)
 juniper_info: skipped (no data)
 k8s_nodes: skipped (no data)
 k8s_nodes: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_roles: skipped (no data)
 k8s_roles: skipped (no data)
 lnx_if: ok
 lnx_if: skipped (no data)
 mem: skipped (no data)
 mem: skipped (no data)
 mem_used: ok
 mem_used: skipped (no data)
 mssql_instance: skipped (no data)
 mssql_instance: skipped (no data)
 netapp_api_disk: skipped (no data)
 netapp_api_disk: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_tablespaces: skipped (no data)
 oracle_tablespaces: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_psmu: skipped (no data)
 perle_psmu: skipped (no data)
 snmp_info: skipped (no data)
 snmp_info: skipped (no data)
 winperf_if: skipped (no data)
 winperf_if: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_state: skipped (no data)
 citrix_state: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 dell_hw_info: skipped (no data)
 dell_hw_info: skipped (no data)
 docker_container_node_name: skipped (no data)
 docker_container_node_name: skipped (no data)
 esx_systeminfo: skipped (no data)
 esx_systeminfo: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fireeye_sys_status: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 k8s_pod_info: skipped (no data)
 k8s_pod_info: skipped (no data)
 k8s_selector: skipped (no data)
 k8s_selector: skipped (no data)
 k8s_service_info: skipped (no data)
 k8s_service_info: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_uname: skipped (no data)
 lnx_uname: skipped (no data)
 lparstat_aix: skipped (no data)
 lparstat_aix: skipped (no data)
 netapp_api_info: skipped (no data)
 netapp_api_info: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis: skipped (no data)
 prtconf: skipped (no data)
 prtconf: skipped (no data)
 snmp_os: skipped (no data)
 snmp_os: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_uname: skipped (no data)
 solaris_uname: skipped (no data)
 suseconnect: skipped (no data)
 suseconnect: skipped (no data)
 win_bios: skipped (no data)
 win_bios: skipped (no data)
 win_computersystem: skipped (no data)
 win_computersystem: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_os: skipped (no data)
 win_os: skipped (no data)
 win_system: skipped (no data)
 win_system: skipped (no data)

Trying to acquire lock on /omd/sites/cmk/tmp/check_mk/status_data/localhost
Got lock on /omd/sites/cmk/tmp/check_mk/status_data/localhost
Releasing lock on /omd/sites/cmk/tmp/check_mk/status_data/localhost
Released lock on /omd/sites/cmk/tmp/check_mk/status_data/localhost
Trying to acquire lock on /omd/sites/cmk/tmp/check_mk/status_data/localhost.gz
Got lock on /omd/sites/cmk/tmp/check_mk/status_data/localhost.gz
Releasing lock on /omd/sites/cmk/tmp/check_mk/status_data/localhost.gz
Released lock on /omd/sites/cmk/tmp/check_mk/status_data/localhost.gz
Trying to acquire lock on /omd/sites/cmk/tmp/check_mk/status_data/.last
Got lock on /omd/sites/cmk/tmp/check_mk/status_data/.last
Releasing lock on /omd/sites/cmk/tmp/check_mk/status_data/.last
Released lock on /omd/sites/cmk/tmp/check_mk/status_data/.last
No piggyback files for 'localhost'. Skip processing.
No piggyback files for '127.0.0.1'. Skip processing.
[cpu_tracking] Stop [7f0749ce0190 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.009999999999999995, children_user=0.0, children_system=0.0, elapsed=0.019999995827674866))]
[agent] Success, execution time 0.5 sec | execution_time=0.500 user_time=0.010 system_time=0.010 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.480

Hey guys!

I’m new to checkmk, just learned about this project yesterday, and it looks very interesting! Right now I’m playing around with the raw version, and running it as a docker container.

While going through this guide:

I got stuck at this command:
cmk-agent-ctl register --hostname localhost --server linux --site cmk --user cmkadmin

It fails with the error mentioned above, Failed to run as user 'cmk-agent'. And I can see that this user does not exist inside the container: cat /etc/passwd | grep cmk-agent outputs nothing.

Googling gave me this article, that explains how this user is created: Checkmk agent with systemd creates dedicated system user

I tried re-installing the check-mk-agent inside the container:
dpkg -i /opt/omd/versions/2.1.0p11.cre/share/check_mk/agents/check-mk-agent_2.1.0p11-1_all.deb

And the output indicates that the user should not be created:

Preparing to unpack .../check-mk-agent_2.1.0p11-1_all.deb ...

Unpacking check-mk-agent (2.1.0p11-1) over (2.1.0p11-1) ...
Setting up check-mk-agent (2.1.0p11-1) ...

systemd not found on this system
Deployed xinetd

Which makes sense, since there’s no systemd inside the container.

I also trying creating the user manualy:
adduser --disabled-password --disabled-login --home /var/lib/cmk-agent cmk-agent

But it doesn’t help as well:

ERROR [cmk_agent_ctl] Something seems wrong with the agent socket (/run/check-mk-agent.socket), aborting```

If I als run the daemon:
```cmk-agent-ctl daemon &```

It starts responding to status command, but still won't register the node:
```root@adfb306b5d58:/# cmk-agent-ctl status
Version: 2.1.0p11
Agent socket: inoperational (!!)
IP allowlist: any
No connections
root@adfb306b5d58:/# cmk-agent-ctl register --hostname localhost --server linux --site cmk --user cmkadmin
ERROR [cmk_agent_ctl] Something seems wrong with the agent socket (/run/check-mk-agent.socket), aborting

Is this a bug inside the docker image? Or am I doing something wrong? Thanks!

UPD: The issue with missing cmk-agent user does not reproduce with the latest checkmk/check-mk-raw:2022.09.15 docker image.

However, the socket issue is still there (/run/check-mk-agent.socket is missing). As a workaround, I tried to create symlink, and it seems to work fine:
ln -s /var/lib/cmk-agent/scripts/super-server/0_systemd/check-mk-agent.socket /run/check-mk-agent.socket

1 Like

I ran into the same issue as @AntonD with the checkmk/check-mk-raw:2.1.0-latest docker image.

Creating the symlink fixed things for me too.

Thanks @AntonD !

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.