CheckMK Self Monitoring

Troubleshooting
,
1

Hello Guys,

I hope this is the right format and everything. My issue is that I would like to monitor the monitoring server itself, which unfortunatly does not yield good results.
Other servers/hosts work fine. I reinstalled the agent, which also seemed to be fine.

The service discovery works, when I use it manually but no data is stored.
The Agent checks fails aswell.
The CheckMK Agents seems to only get empty output.

Overview

2025-02-12 11_25_54-Checkmk Local site monitoring - Services of host checkmk - Brave
2025-02-12 11_25_54-Checkmk Local site monitoring - Services of host checkmk - Brave1920×745 44.1 KB

Error Message:
[agent] Empty output from host X.X.X.115:6556CRIT , [piggyback] Successfully processed from source ‘Vcenter’, Missing monitoring data for plugins, checkmk_agentWARN , cpu_threadsWARN , dfWARN , diskstatWARN , kernel_utilWARN , livestatus_statusWARN , mem_linuxWARN , mknotifydWARN , omd_apacheWARN , omd_diskusageWARN , omd_statusWARN , postfix_mailqWARN , postfix_mailq_statusWARN , uptimeWARN , execution time 0.1 sec

I tried it over the server IP X.X.X.115 and with 127.0.0.1.

Made sure to check the port, looks fine.

adminiti@checkmk:~$ netstat -na | grep :6556
tcp 0 0 127.0.0.1:36360 127.0.0.1:6556 VERBUNDEN
tcp 0 0 X.X.X.115:39276 X.X.X.98:6556 VERBUNDEN
tcp 0 0 X.X.X.115:44570 X.X.X.115:6556 VERBUNDEN
tcp6 0 0 :::6556 :::* LISTEN
tcp6 1 0 127.0.0.1:6556 127.0.0.1:39980 CLOSE_WAIT
tcp6 0 0 X.X.X.115:6556 X.X.X.115:44570 VERBUNDEN
tcp6 1 0 X.X.X.115:6556 X.X.X.115:40836 CLOSE_WAIT
tcp6 1 0 X.X.X.115:6556 X.X.X.115:54022 CLOSE_WAIT
tcp6 0 0 127.0.0.1:6556 127.0.0.1:36360 VERBUNDEN
tcp6 0 0 X.X.X.115:6556 X.X.X.115:36028 TIME_WAIT

Any idea what I could check next?

cmk --debug -vvn Output

OMD[monitoring]:~$ cmk --debug -vvn checkmk
value store: synchronizing
Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/counters/checkmk
Got lock on /omd/sites/monitoring/tmp/check_mk/counters/checkmk
value store: loading from disk
Releasing lock on /omd/sites/monitoring/tmp/check_mk/counters/checkmk
Released lock on /omd/sites/monitoring/tmp/check_mk/counters/checkmk
Checkmk version 2.3.0p20

  • FETCHING DATA
    Source: SourceInfo(hostname=‘checkmk’, ipaddress=‘X.X.X.115’, ident=‘agent’, fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
    [cpu_tracking] Start [7f80b9878890]
    Read from cache: AgentFileCache(checkmk, path_template=/omd/sites/monitoring/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
    Not using cache (does not exist)
    Connecting via TCP to X.X.X.115:6556 (5.0s timeout)
    Closing TCP connection to X.X.X.115:6556
    [cpu_tracking] Stop [7f80b9878890 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
    Source: SourceInfo(hostname=‘checkmk’, ipaddress=‘X.X.X.115’, ident=‘piggyback’, fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
    [cpu_tracking] Start [7f80b96a7050]
    Read from cache: NoCache(checkmk, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
    Piggyback file ‘/omd/sites/monitoring/tmp/check_mk/piggyback/checkmk/Vcenter’: Successfully processed from source ‘Vcenter’
    No piggyback files for ‘X.X.X.115’. Skip processing.
    Get piggybacked data
    [cpu_tracking] Stop [7f80b96a7050 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.010000001639127731))]
    [cpu_tracking] Start [7f80ba020c80]
  • PARSE FETCHER RESULTS
    <<<esx_vsphere_vm:cached(1739354057,90)>>> / Transition NOOPParser → HostSectionParser
    <<<esx_vsphere_objects:cached(1739354057,90):sep(9)>>> / Transition HostSectionParser → HostSectionParser
    <<labels:sep(0)>> / Transition HostSectionParser → HostSectionParser
    HostKey(hostname=‘checkmk’, source_type=<SourceType.HOST: 1>) → Add sections: [‘esx_vsphere_objects’, ‘esx_vsphere_vm’, ‘labels’]
    Received no piggyback data
    ESX Snapshots Count: 0
  • EXECUTING INVENTORY PLUGINS
    allnet_ip_sensoric: skipped (no data)
    allnet_ip_sensoric: skipped (no data)
    aruba_wlc_aps: skipped (no data)
    aruba_wlc_aps: skipped (no data)
    check_mk: skipped (no data)
    check_mk: skipped (no data)
    checkmk_agent_plugins: skipped (no data)
    checkmk_agent_plugins: skipped (no data)
    cisco_meraki_org_device_info: skipped (no data)
    cisco_meraki_org_device_info: skipped (no data)
    cisco_meraki_power_supplies: skipped (no data)
    cisco_meraki_power_supplies: skipped (no data)
    dell_hw_info: skipped (no data)
    dell_hw_info: skipped (no data)
    docker_node_info: skipped (no data)
    docker_node_info: skipped (no data)
    esx_systeminfo: skipped (no data)
    esx_systeminfo: skipped (no data)
    fireeye_sys_status: skipped (no data)
    fireeye_sys_status: skipped (no data)
    fritz: skipped (no data)
    fritz: skipped (no data)
    hp_proliant_da_phydrv: skipped (no data)
    hp_proliant_da_phydrv: skipped (no data)
    hp_proliant_mem: skipped (no data)
    hp_proliant_mem: skipped (no data)
    hp_proliant_systeminfo: skipped (no data)
    hp_proliant_systeminfo: skipped (no data)
    ibm_mq_channels: skipped (no data)
    ibm_mq_channels: skipped (no data)
    ibm_mq_managers: skipped (no data)
    ibm_mq_managers: skipped (no data)
    ibm_mq_queues: skipped (no data)
    ibm_mq_queues: skipped (no data)
    infoblox_osinfo: skipped (no data)
    infoblox_osinfo: skipped (no data)
    infoblox_systeminfo: skipped (no data)
    infoblox_systeminfo: skipped (no data)
    inv_cisco_vlans: skipped (no data)
    inv_cisco_vlans: skipped (no data)
    inv_esx_vsphere_hostsystem: skipped (no data)
    inv_esx_vsphere_hostsystem: skipped (no data)
    inv_if: skipped (no data)
    inv_if: skipped (no data)
    aix_baselevel: skipped (no data)
    aix_baselevel: skipped (no data)
    aix_lparstat_inventory: skipped (no data)
    aix_lparstat_inventory: skipped (no data)
    aix_packages: skipped (no data)
    aix_packages: skipped (no data)
    aix_service_packs: skipped (no data)
    aix_service_packs: skipped (no data)
    azure_load_balancer: skipped (no data)
    azure_load_balancer: skipped (no data)
    inventory_checkmk: skipped (no data)
    inventory_checkmk: skipped (no data)
    checkpoint_vpn_tunnels: skipped (no data)
    checkpoint_vpn_tunnels: skipped (no data)
    citrix_controller: skipped (no data)
    citrix_controller: skipped (no data)
    citrix_state: skipped (no data)
    citrix_state: skipped (no data)
    couchbase_nodes_ports: skipped (no data)
    couchbase_nodes_ports: skipped (no data)
    dmidecode: skipped (no data)
    dmidecode: skipped (no data)
    docker_container_labels: skipped (no data)
    docker_container_labels: skipped (no data)
    docker_container_network: skipped (no data)
    docker_container_network: skipped (no data)
    inventory_docker_container_node_name: skipped (no data)
    inventory_docker_container_node_name: skipped (no data)
    docker_node_images: skipped (no data)
    docker_node_images: skipped (no data)
    docker_node_network: skipped (no data)
    docker_node_network: skipped (no data)
    enviromux_micro_information: skipped (no data)
    enviromux_micro_information: skipped (no data)
    inventory_esx_vsphere_clusters: skipped (no data)
    inventory_esx_vsphere_clusters: skipped (no data)
    inventory_esx_vsphere_virtual_machines: skipped (no data)
    inventory_esx_vsphere_virtual_machines: skipped (no data)
    fortiauthenticator_system: skipped (no data)
    fortiauthenticator_system: skipped (no data)
    fortigate_ha: skipped (no data)
    fortigate_ha: skipped (no data)
    fortimail_system: skipped (no data)
    fortimail_system: skipped (no data)
    fortinet_firewall_network_interfaces: skipped (no data)
    fortinet_firewall_network_interfaces: skipped (no data)
    fortisandbox_software: skipped (no data)
    fortisandbox_software: skipped (no data)
    fortisandbox_system: skipped (no data)
    fortisandbox_system: skipped (no data)
    ipmi_firmware: skipped (no data)
    ipmi_firmware: skipped (no data)
    kube_cluster: skipped (no data)
    kube_cluster: skipped (no data)
    kube_cronjob: skipped (no data)
    kube_cronjob: skipped (no data)
    kube_daemonset: skipped (no data)
    kube_daemonset: skipped (no data)
    kube_deployment: skipped (no data)
    kube_deployment: skipped (no data)
    kube_namespace: skipped (no data)
    kube_namespace: skipped (no data)
    kube_node: skipped (no data)
    kube_node: skipped (no data)
    kube_pod: skipped (no data)
    kube_pod: skipped (no data)
    kube_statefulset: skipped (no data)
    kube_statefulset: skipped (no data)
    kyocera_printer: skipped (no data)
    kyocera_printer: skipped (no data)
    lnx_block_devices: skipped (no data)
    lnx_block_devices: skipped (no data)
    lnx_ip_r: skipped (no data)
    lnx_ip_r: skipped (no data)
    lnx_packages: skipped (no data)
    lnx_packages: skipped (no data)
    lnx_sysctl: skipped (no data)
    lnx_sysctl: skipped (no data)
    lnx_video: skipped (no data)
    lnx_video: skipped (no data)
    mobileiron_inventory: skipped (no data)
    mobileiron_inventory: skipped (no data)
    mssql_clusters: skipped (no data)
    mssql_clusters: skipped (no data)
    oracle_systemparameter: skipped (no data)
    oracle_systemparameter: skipped (no data)
    primekey: skipped (no data)
    primekey: skipped (no data)
    snmp_extended_info: skipped (no data)
    snmp_extended_info: skipped (no data)
    solaris_addresses: skipped (no data)
    solaris_addresses: skipped (no data)
    solaris_pkginfo: skipped (no data)
    solaris_pkginfo: skipped (no data)
    solaris_cpus: skipped (no data)
    solaris_cpus: skipped (no data)
    solaris_routes: skipped (no data)
    solaris_routes: skipped (no data)
    win_bios: skipped (no data)
    win_bios: skipped (no data)
    win_disks: skipped (no data)
    win_disks: skipped (no data)
    win_exefiles: skipped (no data)
    win_exefiles: skipped (no data)
    win_ip_r: skipped (no data)
    win_ip_r: skipped (no data)
    win_networkadapter: skipped (no data)
    win_networkadapter: skipped (no data)
    win_reg_uninstall: skipped (no data)
    win_reg_uninstall: skipped (no data)
    win_video: skipped (no data)
    win_video: skipped (no data)
    win_wmi_software: skipped (no data)
    win_wmi_software: skipped (no data)
    win_wmi_updates: skipped (no data)
    win_wmi_updates: skipped (no data)
    juniper_info: skipped (no data)
    juniper_info: skipped (no data)
    lnx_cpuinfo: skipped (no data)
    lnx_cpuinfo: skipped (no data)
    lnx_distro: skipped (no data)
    lnx_distro: skipped (no data)
    lnx_if: skipped (no data)
    lnx_if: skipped (no data)
    lnx_uname: skipped (no data)
    lnx_uname: skipped (no data)
    lparstat_aix: skipped (no data)
    lparstat_aix: skipped (no data)
    mem: skipped (no data)
    mem: skipped (no data)
    mem_used: skipped (no data)
    mem_used: skipped (no data)
    mssql_instance: skipped (no data)
    mssql_instance: skipped (no data)
    netapp_api_disk: skipped (no data)
    netapp_api_disk: skipped (no data)
    netapp_api_info: skipped (no data)
    netapp_api_info: skipped (no data)
    oracle_dataguard_stats: skipped (no data)
    oracle_dataguard_stats: skipped (no data)
    oracle_instance: skipped (no data)
    oracle_instance: skipped (no data)
    oracle_performance: skipped (no data)
    oracle_performance: skipped (no data)
    oracle_recovery_area: skipped (no data)
    oracle_recovery_area: skipped (no data)
    oracle_tablespaces: skipped (no data)
    oracle_tablespaces: skipped (no data)
    perle_chassis: skipped (no data)
    perle_chassis: skipped (no data)
    perle_chassis_slots: skipped (no data)
    perle_chassis_slots: skipped (no data)
    perle_psmu: skipped (no data)
    perle_psmu: skipped (no data)
    prtconf: skipped (no data)
    prtconf: skipped (no data)
    snmp_quantum_storage_info: skipped (no data)
    snmp_quantum_storage_info: skipped (no data)
    snmp_info: skipped (no data)
    snmp_info: skipped (no data)
    solaris_prtdiag: skipped (no data)
    solaris_prtdiag: skipped (no data)
    solaris_uname: skipped (no data)
    solaris_uname: skipped (no data)
    suseconnect: skipped (no data)
    suseconnect: skipped (no data)
    win_computersystem: skipped (no data)
    win_computersystem: skipped (no data)
    win_computersystemproduct: skipped (no data)
    win_computersystemproduct: skipped (no data)
    win_cpuinfo: skipped (no data)
    win_cpuinfo: skipped (no data)
    win_os: skipped (no data)
    win_os: skipped (no data)
    win_system: skipped (no data)
    win_system: skipped (no data)
    winperf_if: skipped (no data)
    winperf_if: skipped (no data)
    netapp_ontap_info: skipped (no data)
    netapp_ontap_info: skipped (no data)
    netapp_ontap_disk: skipped (no data)
    netapp_ontap_disk: skipped (no data)
    robotmk: skipped (no data)
    robotmk: skipped (no data)
    Piggyback file ‘/omd/sites/monitoring/tmp/check_mk/piggyback/checkmk/Vcenter’: Successfully processed from source ‘Vcenter’
    No piggyback files for ‘X.X.X.115’. Skip processing.
    [cpu_tracking] Stop [7f80ba020c80 - Snapshot(process=posix.times_result(user=0.020000000000000018, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.019999999552965164))]
    [agent] Empty output from host X.X.X.115:6556(!!), [piggyback] Successfully processed from source ‘Vcenter’, execution time 0.0 sec | execution_time=0.030 user_time=0.020 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.010
cmk --debug -vvII Output

OMD[monitoring]:~$ cmk --debug -vvII checkmk
Discovering services and host labels on: checkmk
checkmk:

  • FETCHING DATA
    Source: SourceInfo(hostname=‘checkmk’, ipaddress=‘X.X.X.115’, ident=‘agent’, fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
    [cpu_tracking] Start [7f5d16212420]
    Read from cache: AgentFileCache(checkmk, path_template=/omd/sites/monitoring/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=1)
    Connecting via TCP to X.X.X.115:6556 (5.0s timeout)
    Closing TCP connection to X.X.X.115:6556
    [cpu_tracking] Stop [7f5d16212420 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
    Source: SourceInfo(hostname=‘checkmk’, ipaddress=‘X.X.X.115’, ident=‘piggyback’, fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
    [cpu_tracking] Start [7f5d16212a80]
    Read from cache: NoCache(checkmk, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
    Piggyback file ‘/omd/sites/monitoring/tmp/check_mk/piggyback/checkmk/Vcenter’: Successfully processed from source ‘Vcenter’
    No piggyback files for ‘X.X.X.115’. Skip processing.
    Get piggybacked data
    [cpu_tracking] Stop [7f5d16212a80 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
  • PARSE FETCHER RESULTS
    <<<esx_vsphere_vm:cached(1739354239,90)>>> / Transition NOOPParser → HostSectionParser
    <<<esx_vsphere_objects:cached(1739354239,90):sep(9)>>> / Transition HostSectionParser → HostSectionParser
    <<labels:sep(0)>> / Transition HostSectionParser → HostSectionParser
    HostKey(hostname=‘checkmk’, source_type=<SourceType.HOST: 1>) → Add sections: [‘esx_vsphere_objects’, ‘esx_vsphere_vm’, ‘labels’]
    Received no piggyback data
  • ANALYSE DISCOVERED HOST LABELS
    Trying host label discovery with: esx_vsphere_objects, esx_vsphere_vm, labels
    cmk/vsphere_object: vm (esx_vsphere_vm)
    cmk/piggyback_source_Vcenter: yes (labels)
    Trying host label discovery with:
    SUCCESS - Found 2 host labels
  • ANALYSE DISCOVERED SERVICES
  • EXECUTING DISCOVERY PLUGINS (11)
    Trying discovery with: esx_vsphere_vm_datastores, esx_vsphere_vm_snapshots, esx_vsphere_vm_cpu, esx_vsphere_vm_mounted_devices, esx_vsphere_vm_guest_tools, esx_vsphere_vm_running_on, esx_vsphere_objects, esx_vsphere_vm_name, esx_vsphere_vm_mem_usage, esx_vsphere_objects_count, esx_vsphere_vm_heartbeat
    1 esx_vsphere_objects
    1 esx_vsphere_objects_count
    1 esx_vsphere_vm_cpu
    1 esx_vsphere_vm_datastores
    1 esx_vsphere_vm_guest_tools
    1 esx_vsphere_vm_heartbeat
    1 esx_vsphere_vm_mem_usage
    1 esx_vsphere_vm_mounted_devices
    1 esx_vsphere_vm_name
    1 esx_vsphere_vm_running_on
    1 esx_vsphere_vm_snapshots
    SUCCESS - Found 11 services
CMK version

OMD - Open Monitoring Distribution Version 2.3.0p20.cee

OS version

Virtualization: vmware
Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.1.0-27-amd64
Architecture: x86-64
Hardware Vendor: VMware, Inc.
Hardware Model: VMware Virtual Platform
Firmware Version: 6.00

Thanks in advance.

2

What does “cmk-agent-ctl dump” and “cmk-agent-ctl status” give you?

3

The outputs:

I had to remove quite a bit of the log of the dump due to post length. Please let me know if there was a section which was needed. Also couldn’t post due to links limit, so the log ist logpaste.

cmk-agent-ctl status

Version: 2.3.0p20
Agent socket: operational
IP allowlist: any

Connection: checkmk.domain.com/monitoring
UUID: xxxx
Local:
Connection mode: pull-agent
Connecting to receiver port: 8000
Certificate issuer: Site ‘monitoring’ agent signing CA
Certificate validity: Wed, 12 Feb 2025 09:10:20 +0000 - Tue, 12 Feb 2030 09:10:20 +0000
Remote:
Connection mode: pull-agent
Hostname: checkmk

cmk-agent-ctl dump
https://logpaste.com/5yZNYy8x

Thank you :slight_smile:

4

Hi, this looks pretty normal :confused:
Since you’re on CEE, do you have a support partner or support contract with Checkmk itself to take a look at the live system?

I’d maybe also still check that all necessary systemd services are running (and though that’s rarely necessary, maybe restart them - something is weird on this client)

systemctl status 'cmk*'
systemctl status 'check-mk*'
5

Thanks for the reply and the help!

I don’t know why I haven’t thought of just opening a ticket, but will do :smiley:

6

@SenacorITI if you got this fixed, feel free to share :slight_smile: I’m interested and so might others for additional troubleshooting steps or the actual solution :wink: