Hello,
I have one monitored Linux server which over night decided to think he has two service descriptions double. This suddenly appeared and it now stuck. The monitored server has the current agent installed with plugins mk_docker.py, mk_logins and mk_sshd_config. The host has only an agent running, no SNMP and no API integrations.
One of the CPU (the lower one) and one of the Memory (the RAM one) services are complete fantasy values. I am also unable to disable those two services.
I have tried deleting the host and adding it again as well as reinstalling the agent. Both didn’t change anything. There is nothing that could cause this. Also there are multiple servers which are exactly the same and it occurs only on this single server.
Regards,
Felix
CMK version: 2.0.0p39 (CRE)
OS version: Ubuntu Server 22.04.3 LTS Minimal
Error message:
check_mk: ERROR: Duplicate service description (auto check) 'Memory' for host 'graylog'! - 1st occurrence: check plugin / item: mem_linux / None - 2nd occurrence: check plugin / item: mem_used / None
check_mk: ERROR: Duplicate service description (auto check) 'CPU utilization' for host 'graylog'! - 1st occurrence: check plugin / item: cpu_utilization_os / None - 2nd occurrence: check plugin / item: kernel_util / None
Output of “cmk --debug -vvn hostname”:
Checkmk version 2.0.0p39
Try license usage history update.
Trying to acquire lock on /omd/sites/hv2/var/check_mk/license_usage/next_run
Got lock on /omd/sites/hv2/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/hv2/var/check_mk/license_usage/history.json
Got lock on /omd/sites/hv2/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/hv2/var/check_mk/license_usage/history.json
Released lock on /omd/sites/hv2/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/hv2/var/check_mk/license_usage/next_run
Released lock on /omd/sites/hv2/var/check_mk/license_usage/next_run
Loading autochecks from /omd/sites/hv2/var/check_mk/autochecks/graylog.mk
+ FETCHING DATA
Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7f02ecd9d880]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(base_path=PosixPath('/omd/sites/hv2/tmp/check_mk/cache/graylog'), max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 11 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 172.16.240.31:6556 (5.0s timeout)
Reading data from agent
Output is not encrypted
Write data to cache file /omd/sites/hv2/tmp/check_mk/cache/graylog
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/cache/graylog
Got lock on /omd/sites/hv2/tmp/check_mk/cache/graylog
Releasing lock on /omd/sites/hv2/tmp/check_mk/cache/graylog
Released lock on /omd/sites/hv2/tmp/check_mk/cache/graylog
Closing TCP connection to 172.16.240.31:6556
[cpu_tracking] Stop [7f02ecd9d880 - Snapshot(process=posix.times_result(user=0.0, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=4.299999997019768))]
Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f02ecd9dcd0]
[PiggybackFetcher] Fetch with cache settings: NoCache(base_path=PosixPath('/omd/sites/hv2/tmp/check_mk/data_source_cache/piggyback/graylog'), max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
[PiggybackFetcher] Execute data source
No piggyback files for 'graylog'. Skip processing.
No piggyback files for '172.16.240.31'. Skip processing.
[cpu_tracking] Stop [7f02ecd9dcd0 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[cpu_tracking] Start [7f02ece08b80]
+ PARSE FETCHER RESULTS
Source: SourceType.HOST/FetcherType.TCP
No persisted sections loaded
-> Add sections: ['check_mk', 'cifsmounts', 'cpu', 'df', 'diskstat', 'docker_container_cpu', 'docker_container_diskstat', 'docker_container_labels', 'docker_container_mem', 'docker_container_network', 'docker_container_node_name', 'docker_container_status', 'docker_node_info', 'docker_node_network', 'job', 'kernel', 'labels', 'lnx_if', 'local', 'logins', 'md', 'mem', 'mounts', 'nfsmounts', 'ps_lnx', 'sshd_config', 'systemd_units', 'tcp_conn_stats', 'timesyncd', 'uptime', 'vbox_guest']
Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections loaded
-> Add sections: []
Storing piggyback data for: opensearch
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/piggyback/opensearch/graylog
Got lock on /omd/sites/hv2/tmp/check_mk/piggyback/opensearch/graylog
Releasing lock on /omd/sites/hv2/tmp/check_mk/piggyback/opensearch/graylog
Released lock on /omd/sites/hv2/tmp/check_mk/piggyback/opensearch/graylog
Storing piggyback data for: mongodb
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/piggyback/mongodb/graylog
Got lock on /omd/sites/hv2/tmp/check_mk/piggyback/mongodb/graylog
Releasing lock on /omd/sites/hv2/tmp/check_mk/piggyback/mongodb/graylog
Released lock on /omd/sites/hv2/tmp/check_mk/piggyback/mongodb/graylog
Received piggyback data for 2 hosts
Loading item states
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/counters/graylog
Got lock on /omd/sites/hv2/tmp/check_mk/counters/graylog
Releasing lock on /omd/sites/hv2/tmp/check_mk/counters/graylog
Released lock on /omd/sites/hv2/tmp/check_mk/counters/graylog
CPU load 15 min load: 2.14 at 3 cores (0.71 per core)
CPU utilization Total CPU: 2.67%
CPU utilization Total CPU: 43.52%
Docker container health Health status: Healthy, Last health report: {"cluster_id":"8ec024c1-6cc5-43b6-90be-3f6b028a62f2","node_id":"ca32a1e4-117b-4cc9-a321-97ad575ea564","version":"5.1.5+993cd0f","tagline":"Manage your logs in the dark and have lasers going and make it look like you're from space!"}, Health test: CMD-SHELL /health_check.sh
Docker container status Container running on node graylog
Docker containers Containers: 3, Running: 3, Paused: 0, Stopped: 0
Docker node info Daemon running on host graylog
Filesystem / 22.28% used (13.09 of 58.74 GB), trend: +72.43 MB / 24 hours
Filesystem /boot/efi 21.33% used (2.12 of 9.96 MB), trend: 0.00 B / 24 hours
Interface 2 [eth0], (up), MAC: 00:50:56:AB:43:27, Speed: 10 GBit/s, In: 1.69 kB/s (<0.01%), Out: 6.14 kB/s (<0.01%)
Kernel Performance Process Creations: 11.86/s, Context Switches: 1024.86/s, Major Page Faults: 0.00/s, Page Swap in: 0.00/s, Page Swap Out: 0.00/s
Logins On system: 1
Memory Total virtual memory: 40.6% - 3.16 GB of 7.78 GB
Memory RAM: 11.71% - 933 MiB of 7.78 GiB
Mount options of / Mount options exactly as expected
Mount options of /boot/efi Mount options exactly as expected
Number of threads Count: 717 threads, Usage: 1.13%
SSH daemon configuration Ports: 22, Use pluggable authentication module: yes, Permit root login: yes, Allow password authentication: yes, Allow keyboard-interactive authentication: yes, Permit X11 forwarding: no, Permit empty passwords: no, Ciphers: aes128-ctr, aes128-gcm@openssh.com, aes192-ctr, aes256-ctr, aes256-gcm@openssh.com, chacha20-poly1305@openssh.com
System Reboot required A system reboot is not required.
Systemd Service Summary Total: 107, Disabled: 3, Failed: 11
Systemd Service node_exporter Status: active, Prometheus Node Exporter Service
Systemd Timesyncd Time Offset: 24 microseconds, Time since last sync: 45 seconds, Stratum: 1.00, Jitter: 12.0 µs, synchronized on 216.239.35.0
TCP Connections Established: 3
Uptime Up since Sep 26 2023 06:33:14, Uptime: 36 days 7 hours
+ EXECUTING INVENTORY PLUGINS
hp_proliant_mem: skipped (no data)
hp_proliant_mem: skipped (no data)
ipmi_firmware: skipped (no data)
ipmi_firmware: skipped (no data)
inventory_checkmk: skipped (no data)
inventory_checkmk: skipped (no data)
inv_cisco_vlans: skipped (no data)
inv_cisco_vlans: skipped (no data)
inv_esx_vsphere_hostsystem: skipped (no data)
inv_esx_vsphere_hostsystem: skipped (no data)
inv_if: ok
inv_if: skipped (no data)
inventory_oracle_tablespaces: skipped (no data)
inventory_oracle_tablespaces: skipped (no data)
dmidecode: skipped (no data)
dmidecode: skipped (no data)
docker_node_network: ok
docker_node_network: skipped (no data)
inventory_esx_vsphere_clusters: skipped (no data)
inventory_esx_vsphere_clusters: skipped (no data)
inventory_esx_vsphere_virtual_machines: skipped (no data)
inventory_esx_vsphere_virtual_machines: skipped (no data)
k8s_endpoint_info: skipped (no data)
k8s_endpoint_info: skipped (no data)
k8s_ingress_infos: skipped (no data)
k8s_ingress_infos: skipped (no data)
juniper_info: skipped (no data)
juniper_info: skipped (no data)
mem: ok
mem: skipped (no data)
mem_used: ok
mem_used: skipped (no data)
snmp_info: skipped (no data)
snmp_info: skipped (no data)
aix_baselevel: skipped (no data)
aix_baselevel: skipped (no data)
aix_lparstat_inventory: skipped (no data)
aix_lparstat_inventory: skipped (no data)
aix_packages: skipped (no data)
aix_packages: skipped (no data)
aix_service_packs: skipped (no data)
aix_service_packs: skipped (no data)
allnet_ip_sensoric: skipped (no data)
allnet_ip_sensoric: skipped (no data)
aruba_wlc_aps: skipped (no data)
aruba_wlc_aps: skipped (no data)
check_mk: ok
check_mk: skipped (no data)
checkpoint_inv_tunnels: skipped (no data)
checkpoint_inv_tunnels: skipped (no data)
citrix_controller: skipped (no data)
citrix_controller: skipped (no data)
citrix_state: skipped (no data)
citrix_state: skipped (no data)
couchbase_nodes_ports: skipped (no data)
couchbase_nodes_ports: skipped (no data)
dell_hw_info: skipped (no data)
dell_hw_info: skipped (no data)
docker_container_labels: ok
docker_container_labels: skipped (no data)
docker_container_network: ok
docker_container_network: skipped (no data)
docker_container_node_name: ok
docker_container_node_name: skipped (no data)
docker_node_images: skipped (no data)
docker_node_images: skipped (no data)
docker_node_info: ok
docker_node_info: skipped (no data)
esx_systeminfo: skipped (no data)
esx_systeminfo: skipped (no data)
fireeye_sys_status: skipped (no data)
fireeye_sys_status: skipped (no data)
fritz: skipped (no data)
fritz: skipped (no data)
hp_proliant_da_phydrv: skipped (no data)
hp_proliant_da_phydrv: skipped (no data)
hp_proliant_systeminfo: skipped (no data)
hp_proliant_systeminfo: skipped (no data)
ibm_mq_channels: skipped (no data)
ibm_mq_channels: skipped (no data)
ibm_mq_managers: skipped (no data)
ibm_mq_managers: skipped (no data)
ibm_mq_queues: skipped (no data)
ibm_mq_queues: skipped (no data)
infoblox_osinfo: skipped (no data)
infoblox_osinfo: skipped (no data)
infoblox_systeminfo: skipped (no data)
infoblox_systeminfo: skipped (no data)
k8s_assigned_pods: skipped (no data)
k8s_assigned_pods: skipped (no data)
k8s_daemon_pod_containers: skipped (no data)
k8s_daemon_pod_containers: skipped (no data)
k8s_job_container: skipped (no data)
k8s_job_container: skipped (no data)
k8s_nodes: skipped (no data)
k8s_nodes: skipped (no data)
k8s_pod_container: skipped (no data)
k8s_pod_container: skipped (no data)
k8s_pod_info: skipped (no data)
k8s_pod_info: skipped (no data)
k8s_roles: skipped (no data)
k8s_roles: skipped (no data)
k8s_selector: skipped (no data)
k8s_selector: skipped (no data)
k8s_service_info: skipped (no data)
k8s_service_info: skipped (no data)
lnx_block_devices: skipped (no data)
lnx_block_devices: skipped (no data)
lnx_cpuinfo: skipped (no data)
lnx_cpuinfo: skipped (no data)
lnx_distro: skipped (no data)
lnx_distro: skipped (no data)
lnx_if: ok
lnx_if: skipped (no data)
lnx_ip_r: skipped (no data)
lnx_ip_r: skipped (no data)
lnx_packages: skipped (no data)
lnx_packages: skipped (no data)
lnx_sysctl: skipped (no data)
lnx_sysctl: skipped (no data)
lnx_uname: skipped (no data)
lnx_uname: skipped (no data)
lnx_video: skipped (no data)
lnx_video: skipped (no data)
lparstat_aix: skipped (no data)
lparstat_aix: skipped (no data)
mssql_clusters: skipped (no data)
mssql_clusters: skipped (no data)
mssql_versions: skipped (no data)
mssql_versions: skipped (no data)
netapp_api_disk: skipped (no data)
netapp_api_disk: skipped (no data)
netapp_api_info: skipped (no data)
netapp_api_info: skipped (no data)
oracle_dataguard_stats: skipped (no data)
oracle_dataguard_stats: skipped (no data)
oracle_instance: skipped (no data)
oracle_instance: skipped (no data)
oracle_performance: skipped (no data)
oracle_performance: skipped (no data)
oracle_recovery_area: skipped (no data)
oracle_recovery_area: skipped (no data)
oracle_systemparameter: skipped (no data)
oracle_systemparameter: skipped (no data)
perle_chassis: skipped (no data)
perle_chassis: skipped (no data)
perle_chassis_slots: skipped (no data)
perle_chassis_slots: skipped (no data)
perle_psmu: skipped (no data)
perle_psmu: skipped (no data)
prtconf: skipped (no data)
prtconf: skipped (no data)
snmp_extended_info: skipped (no data)
snmp_extended_info: skipped (no data)
snmp_os: skipped (no data)
snmp_os: skipped (no data)
snmp_quantum_storage_info: skipped (no data)
snmp_quantum_storage_info: skipped (no data)
solaris_addresses: skipped (no data)
solaris_addresses: skipped (no data)
solaris_pkginfo: skipped (no data)
solaris_pkginfo: skipped (no data)
solaris_prtdiag: skipped (no data)
solaris_prtdiag: skipped (no data)
solaris_prtpicl: skipped (no data)
solaris_prtpicl: skipped (no data)
solaris_psrinfo: skipped (no data)
solaris_psrinfo: skipped (no data)
solaris_routes: skipped (no data)
solaris_routes: skipped (no data)
solaris_uname: skipped (no data)
solaris_uname: skipped (no data)
statgrab_net: skipped (no data)
statgrab_net: skipped (no data)
suseconnect: skipped (no data)
suseconnect: skipped (no data)
win_bios: skipped (no data)
win_bios: skipped (no data)
win_computersystem: skipped (no data)
win_computersystem: skipped (no data)
win_cpuinfo: skipped (no data)
win_cpuinfo: skipped (no data)
win_disks: skipped (no data)
win_disks: skipped (no data)
win_exefiles: skipped (no data)
win_exefiles: skipped (no data)
win_ip_r: skipped (no data)
win_ip_r: skipped (no data)
win_networkadapter: skipped (no data)
win_networkadapter: skipped (no data)
win_os: skipped (no data)
win_os: skipped (no data)
win_reg_uninstall: skipped (no data)
win_reg_uninstall: skipped (no data)
win_system: skipped (no data)
win_system: skipped (no data)
win_video: skipped (no data)
win_video: skipped (no data)
win_wmi_software: skipped (no data)
win_wmi_software: skipped (no data)
win_wmi_updates: skipped (no data)
win_wmi_updates: skipped (no data)
winperf_if: skipped (no data)
winperf_if: skipped (no data)
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/status_data/graylog
Got lock on /omd/sites/hv2/tmp/check_mk/status_data/graylog
Releasing lock on /omd/sites/hv2/tmp/check_mk/status_data/graylog
Released lock on /omd/sites/hv2/tmp/check_mk/status_data/graylog
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/status_data/graylog.gz
Got lock on /omd/sites/hv2/tmp/check_mk/status_data/graylog.gz
Releasing lock on /omd/sites/hv2/tmp/check_mk/status_data/graylog.gz
Released lock on /omd/sites/hv2/tmp/check_mk/status_data/graylog.gz
Trying to acquire lock on /omd/sites/hv2/tmp/check_mk/status_data/.last
Got lock on /omd/sites/hv2/tmp/check_mk/status_data/.last
Releasing lock on /omd/sites/hv2/tmp/check_mk/status_data/.last
Released lock on /omd/sites/hv2/tmp/check_mk/status_data/.last
No piggyback files for 'graylog'. Skip processing.
No piggyback files for '172.16.240.31'. Skip processing.
[cpu_tracking] Stop [7f02ece08b80 - Snapshot(process=posix.times_result(user=0.19999999999999973, system=0.030000000000000027, children_user=0.0, children_system=0.0, elapsed=0.24000000208616257))]
[agent] Version: 2.0.0p39, OS: linux, execution time 4.5 sec | execution_time=4.540 user_time=0.210 system_time=0.040 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=4.280