Custom check plugin is not being discovered

jarckn · September 8, 2022, 11:32am

Hello,

I am suddenly having problems bringing my custom plugins to work.
Interestingly do they work sometimes, but not always and not on all involved systems.
They also worked before, but stopped working today after doing some changes.
Reverting these changes however did not make it work again.

What I did so far:

Checking my host plugin for errors (none)
Checking the python plugin on my CheckMK server for errors (none? at least not syntactical and cmk gave no errors either)
Rebooting the CheckMK services
Rebooting the whole server
Rebooting the client
Deleting everything in /var/lib/check_mk_agent/cache/

The host plugins are located in /usr/lib/check_mk_agent/plugins.
The server plugins in ~/local/lib/check_mk/base/plugins/agent_based.

These are the plugins:

Plugins

zeek_remote_connections.sh

#!/bin/sh
echo '<<<zeek_remote_connections>>>'
docker exec zeek zeekctl doctor check_local_connections | awk '/%/'

Python script: zeek_remote_connections.py · GitHub

zeek_packet_loss.sh

#!/bin/sh
echo '<<<zeek_packet_loss>>>'
docker exec zeek zeekctl doctor check_capture_loss_conn_pct | awk '/%/'

Python script: zeek_packet_loss.py · GitHub

zeek_halfduplex_connections.sh

#!/bin/sh
echo '<<<zeek_halfduplex_connections>>>'
docker exec zeek zeekctl doctor check_SAD_connections | awk '/%/'

Python script: zeek_halfduplex_connections.py · GitHub

The output of these scripts are included in the agent and I checked with cmk -d if they arrive at the server. They do.

I am out of ideas. Does anyone know what else I can check?

Best regards,

Niklas

CMK version: 2.0.0p22 (CRE)
OS version: Ubuntu 20.04 LTS (both, client and server)

Error message: No specific error message.

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

Output

OMD[REDACTED-SITE]:~/local/lib/check_mk/base/plugins/agent_based$ cmk --debug -vvn REDACTED-HOSTNAME


Checkmk version 2.0.0p22

Try license usage history update.

Trying to acquire lock on /omd/sites/REDACTED-SITE/var/check_mk/license_usage/next_run

Got lock on /omd/sites/REDACTED-SITE/var/check_mk/license_usage/next_run

Trying to acquire lock on /omd/sites/REDACTED-SITE/var/check_mk/license_usage/history.json

Got lock on /omd/sites/REDACTED-SITE/var/check_mk/license_usage/history.json

Next run time has not been reached yet. Abort.

Releasing lock on /omd/sites/REDACTED-SITE/var/check_mk/license_usage/history.json

Released lock on /omd/sites/REDACTED-SITE/var/check_mk/license_usage/history.json

Releasing lock on /omd/sites/REDACTED-SITE/var/check_mk/license_usage/next_run

Released lock on /omd/sites/REDACTED-SITE/var/check_mk/license_usage/next_run

+ FETCHING DATA

Source: SourceType.HOST/FetcherType.TCP

[cpu_tracking] Start [7f1695942dc0]

Connecting via TCP to REDACTED-HOSTNAME.REDACTED-DOMAIN:6556 (5.0s timeout)

[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(base_path=PosixPath('/omd/sites/REDACTED-SITE/tmp/check_mk/cache/REDACTED-HOSTNAME'), max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)

Not using cache (Too old. Age is 26 sec, allowed is 0 sec)

[TCPFetcher] Execute data source

Reading data from agent

Output is not encrypted

Write data to cache file /omd/sites/REDACTED-SITE/tmp/check_mk/cache/REDACTED-HOSTNAME

Trying to acquire lock on /omd/sites/REDACTED-SITE/tmp/check_mk/cache/REDACTED-HOSTNAME

Got lock on /omd/sites/REDACTED-SITE/tmp/check_mk/cache/REDACTED-HOSTNAME

Releasing lock on /omd/sites/REDACTED-SITE/tmp/check_mk/cache/REDACTED-HOSTNAME

Released lock on /omd/sites/REDACTED-SITE/tmp/check_mk/cache/REDACTED-HOSTNAME

Closing TCP connection to REDACTED-HOSTNAME.REDACTED-DOMAIN:6556

[cpu_tracking] Stop [7f1695942dc0 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=4.78999999910593))]

Source: SourceType.HOST/FetcherType.PIGGYBACK

[cpu_tracking] Start [7f1695942eb0]

No piggyback files for 'REDACTED-HOSTNAME'. Skip processing.

No piggyback files for 'REDACTED-HOSTNAME.REDACTED-DOMAIN'. Skip processing.

[PiggybackFetcher] Fetch with cache settings: NoCache(base_path=PosixPath('/omd/sites/REDACTED-SITE/tmp/check_mk/data_source_cache/piggyback/REDACTED-HOSTNAME'), max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)

[PiggybackFetcher] Execute data source

[cpu_tracking] Stop [7f1695942eb0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]

[cpu_tracking] Start [7f1695942eb0]

+ PARSE FETCHER RESULTS

Source: SourceType.HOST/FetcherType.TCP

No persisted sections loaded

-> Add sections: ['check_mk', 'cifsmounts', 'common_os_version', 'cpu', 'df', 'diskstat', 'docker_node_info', 'docker_node_network', 'job', 'kernel', 'lnx_if', 'lnx_thermal', 'local', 'md', 'mem', 'mounts', 'multipath', 'nfsmounts', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'timesyncd', 'uptime', 'vbox_guest', 'zeek_halfduplex_connections', 'zeek_packet_loss', 'zeek_remote_connections']

Source: SourceType.HOST/FetcherType.PIGGYBACK

No persisted sections loaded

-> Add sections: []

Storing piggyback data for: zeek

Trying to acquire lock on /omd/sites/REDACTED-SITE/tmp/check_mk/piggyback/zeek/REDACTED-HOSTNAME

Got lock on /omd/sites/REDACTED-SITE/tmp/check_mk/piggyback/zeek/REDACTED-HOSTNAME

Releasing lock on /omd/sites/REDACTED-SITE/tmp/check_mk/piggyback/zeek/REDACTED-HOSTNAME

Released lock on /omd/sites/REDACTED-SITE/tmp/check_mk/piggyback/zeek/REDACTED-HOSTNAME

Received piggyback data for 1 hosts

Loading item states

Trying to acquire lock on /omd/sites/REDACTED-SITE/tmp/check_mk/counters/REDACTED-HOSTNAME

Got lock on /omd/sites/REDACTED-SITE/tmp/check_mk/counters/REDACTED-HOSTNAME

Releasing lock on /omd/sites/REDACTED-SITE/tmp/check_mk/counters/REDACTED-HOSTNAME

Released lock on /omd/sites/REDACTED-SITE/tmp/check_mk/counters/REDACTED-HOSTNAME

+ EXECUTING INVENTORY PLUGINS

hp_proliant_mem: skipped (no data)

hp_proliant_mem: skipped (no data)

ipmi_firmware: skipped (no data)

ipmi_firmware: skipped (no data)

inventory_checkmk: skipped (no data)

inventory_checkmk: skipped (no data)

inv_cisco_vlans: skipped (no data)

inv_cisco_vlans: skipped (no data)

inv_esx_vsphere_hostsystem: skipped (no data)

inv_esx_vsphere_hostsystem: skipped (no data)

inv_if: ok

inv_if: skipped (no data)

inventory_oracle_tablespaces: skipped (no data)

inventory_oracle_tablespaces: skipped (no data)

dmidecode: skipped (no data)

dmidecode: skipped (no data)

docker_node_network: ok

docker_node_network: skipped (no data)

inventory_esx_vsphere_clusters: skipped (no data)

inventory_esx_vsphere_clusters: skipped (no data)

inventory_esx_vsphere_virtual_machines: skipped (no data)

inventory_esx_vsphere_virtual_machines: skipped (no data)

k8s_endpoint_info: skipped (no data)

k8s_endpoint_info: skipped (no data)

k8s_ingress_infos: skipped (no data)

k8s_ingress_infos: skipped (no data)

juniper_info: skipped (no data)

juniper_info: skipped (no data)

mem: ok

mem: skipped (no data)

mem_used: skipped (no data)

mem_used: skipped (no data)

snmp_info: skipped (no data)

snmp_info: skipped (no data)

aix_baselevel: skipped (no data)

aix_baselevel: skipped (no data)

aix_lparstat_inventory: skipped (no data)

aix_lparstat_inventory: skipped (no data)

aix_packages: skipped (no data)

aix_packages: skipped (no data)

aix_service_packs: skipped (no data)

aix_service_packs: skipped (no data)

allnet_ip_sensoric: skipped (no data)

allnet_ip_sensoric: skipped (no data)

aruba_wlc_aps: skipped (no data)

aruba_wlc_aps: skipped (no data)

check_mk: ok

check_mk: skipped (no data)

checkpoint_inv_tunnels: skipped (no data)

checkpoint_inv_tunnels: skipped (no data)

citrix_controller: skipped (no data)

citrix_controller: skipped (no data)

citrix_state: skipped (no data)

citrix_state: skipped (no data)

couchbase_nodes_ports: skipped (no data)

couchbase_nodes_ports: skipped (no data)

dell_hw_info: skipped (no data)

dell_hw_info: skipped (no data)

docker_container_labels: skipped (no data)

docker_container_labels: skipped (no data)

docker_container_network: skipped (no data)

docker_container_network: skipped (no data)

docker_container_node_name: skipped (no data)

docker_container_node_name: skipped (no data)

docker_node_images: skipped (no data)

docker_node_images: skipped (no data)

docker_node_info: ok

docker_node_info: skipped (no data)

esx_systeminfo: skipped (no data)

esx_systeminfo: skipped (no data)

fireeye_sys_status: skipped (no data)

fireeye_sys_status: skipped (no data)

fritz: skipped (no data)

fritz: skipped (no data)

hp_proliant_da_phydrv: skipped (no data)

hp_proliant_da_phydrv: skipped (no data)

hp_proliant_systeminfo: skipped (no data)

hp_proliant_systeminfo: skipped (no data)

ibm_mq_channels: skipped (no data)

ibm_mq_channels: skipped (no data)

ibm_mq_managers: skipped (no data)

ibm_mq_managers: skipped (no data)

ibm_mq_queues: skipped (no data)

ibm_mq_queues: skipped (no data)

infoblox_osinfo: skipped (no data)

infoblox_osinfo: skipped (no data)

infoblox_systeminfo: skipped (no data)

infoblox_systeminfo: skipped (no data)

k8s_assigned_pods: skipped (no data)

k8s_assigned_pods: skipped (no data)

k8s_daemon_pod_containers: skipped (no data)

k8s_daemon_pod_containers: skipped (no data)

k8s_job_container: skipped (no data)

k8s_job_container: skipped (no data)

k8s_nodes: skipped (no data)

k8s_nodes: skipped (no data)

k8s_pod_container: skipped (no data)

k8s_pod_container: skipped (no data)

k8s_pod_info: skipped (no data)

k8s_pod_info: skipped (no data)

k8s_roles: skipped (no data)

k8s_roles: skipped (no data)

k8s_selector: skipped (no data)

k8s_selector: skipped (no data)

k8s_service_info: skipped (no data)

k8s_service_info: skipped (no data)

lnx_block_devices: skipped (no data)

lnx_block_devices: skipped (no data)

lnx_cpuinfo: skipped (no data)

lnx_cpuinfo: skipped (no data)

lnx_distro: skipped (no data)

lnx_distro: skipped (no data)

lnx_if: ok

lnx_if: skipped (no data)

lnx_ip_r: skipped (no data)

lnx_ip_r: skipped (no data)

lnx_packages: skipped (no data)

lnx_packages: skipped (no data)

lnx_sysctl: skipped (no data)

lnx_sysctl: skipped (no data)

lnx_uname: skipped (no data)

lnx_uname: skipped (no data)

lnx_video: skipped (no data)

lnx_video: skipped (no data)

lparstat_aix: skipped (no data)

lparstat_aix: skipped (no data)

mssql_clusters: skipped (no data)

mssql_clusters: skipped (no data)

mssql_versions: skipped (no data)

mssql_versions: skipped (no data)

netapp_api_disk: skipped (no data)

netapp_api_disk: skipped (no data)

netapp_api_info: skipped (no data)

netapp_api_info: skipped (no data)

oracle_dataguard_stats: skipped (no data)

oracle_dataguard_stats: skipped (no data)

oracle_instance: skipped (no data)

oracle_instance: skipped (no data)

oracle_performance: skipped (no data)

oracle_performance: skipped (no data)

oracle_recovery_area: skipped (no data)

oracle_recovery_area: skipped (no data)

oracle_systemparameter: skipped (no data)

oracle_systemparameter: skipped (no data)

perle_chassis: skipped (no data)

perle_chassis: skipped (no data)

perle_chassis_slots: skipped (no data)

perle_chassis_slots: skipped (no data)

perle_psmu: skipped (no data)

perle_psmu: skipped (no data)

prtconf: skipped (no data)

prtconf: skipped (no data)

snmp_extended_info: skipped (no data)

snmp_extended_info: skipped (no data)

snmp_os: skipped (no data)

snmp_os: skipped (no data)

snmp_quantum_storage_info: skipped (no data)

snmp_quantum_storage_info: skipped (no data)

solaris_addresses: skipped (no data)

solaris_addresses: skipped (no data)

solaris_pkginfo: skipped (no data)

solaris_pkginfo: skipped (no data)

solaris_prtdiag: skipped (no data)

solaris_prtdiag: skipped (no data)

solaris_prtpicl: skipped (no data)

solaris_prtpicl: skipped (no data)

solaris_psrinfo: skipped (no data)

solaris_psrinfo: skipped (no data)

solaris_routes: skipped (no data)

solaris_routes: skipped (no data)

solaris_uname: skipped (no data)

solaris_uname: skipped (no data)

statgrab_net: skipped (no data)

statgrab_net: skipped (no data)

suseconnect: skipped (no data)

suseconnect: skipped (no data)

win_bios: skipped (no data)

win_bios: skipped (no data)

win_computersystem: skipped (no data)

win_computersystem: skipped (no data)

win_cpuinfo: skipped (no data)

win_cpuinfo: skipped (no data)

win_disks: skipped (no data)

win_disks: skipped (no data)

win_exefiles: skipped (no data)

win_exefiles: skipped (no data)

win_ip_r: skipped (no data)

win_ip_r: skipped (no data)

win_networkadapter: skipped (no data)

win_networkadapter: skipped (no data)

win_os: skipped (no data)

win_os: skipped (no data)

win_reg_uninstall: skipped (no data)

win_reg_uninstall: skipped (no data)

win_system: skipped (no data)

win_system: skipped (no data)

win_video: skipped (no data)

win_video: skipped (no data)

win_wmi_software: skipped (no data)

win_wmi_software: skipped (no data)

win_wmi_updates: skipped (no data)

win_wmi_updates: skipped (no data)

winperf_if: skipped (no data)

winperf_if: skipped (no data)

Trying to acquire lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/REDACTED-HOSTNAME

Got lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/REDACTED-HOSTNAME

Releasing lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/REDACTED-HOSTNAME

Released lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/REDACTED-HOSTNAME

Trying to acquire lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/REDACTED-HOSTNAME.gz

Got lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/REDACTED-HOSTNAME.gz

Releasing lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/REDACTED-HOSTNAME.gz

Released lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/REDACTED-HOSTNAME.gz

Trying to acquire lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/.last

Got lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/.last

Releasing lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/.last

Released lock on /omd/sites/REDACTED-SITE/tmp/check_mk/status_data/.last

No piggyback files for 'REDACTED-HOSTNAME'. Skip processing.

No piggyback files for 'REDACTED-HOSTNAME.REDACTED-DOMAIN'. Skip processing.

[cpu_tracking] Stop [7f1695942eb0 - Snapshot(process=posix.times_result(user=0.07000000000000006, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=0.07999999821186066))]

[agent] Version: 2.0.0p22, OS: linux, execution time 4.9 sec | execution_time=4.870 user_time=0.080 system_time=0.010 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=4.780

mike1098 · September 8, 2022, 5:07pm

Hello Niklas,

I didnt had a so close look at your code but I can give some recommendations.
Go with a simulated host which contain just your agent output. This makes in general debug output easier to read.
Add pprint statements to your code to show variables on console.
Start building the parse function and then the discover function.
Debug your code with:

cmk --debug -Ivv <hostname>

Then go on with the check function.
debug your code with:

cmk --debug -vvv --detect-plugins <check plugin name> <hostname>

Start with something simple and then elaborate

I hope that helps

Michael

jarckn · September 9, 2022, 9:16am

Hi Michael,

you brought me right on track there with your tip on using print in basically all functions.
I was using print before to see any output, but it didn’t show anything so I thought the python script gets executed silently.

After seeing that I was using .match for regex instead of .search, I quickly switched it and it works now.
There wasn’t an regex match so the object basically was None, causing the whole script to “soft fail”.

Works as intended now and I finally know how to troubleshoot these scripts a bit easier.

Thank you!

Best regards,

Niklas

mike1098 · September 9, 2022, 9:26am

I am happy to be able to help here.
Would be nice to understand the root cause and learn from it.
Please also allow me to do some recommendations to your code.
Python has a rich feature set for str objects which makes in most cases regular expressions obsolete.
Regular expression’s are resource intense and dangerous.

May think about using check_levels() instead of the “old” way.

To make your code save it is also recommended to write a short function like this to make type conversion’s safe:

def save_int(value):
    try:
        ret_val=int(value)
    except ValueError:
        ret_val=0
    return ret_val

Have fun with coding

Michael

system · September 9, 2023, 9:26am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.