Local checks distributed with agent bakery not functioning

jorgejuan · December 19, 2023, 11:11am

CMK version: 2.1.0p37 for docker
**OS version: Red Hat 8.5.0-16 **

Error message:
We deployed a custom .ps1 script using the Agent Bakery but there is no <<<local>>> agent section in any of the agentes (we tried windows server 2012 and 2019). We started facing this with check_mk 2.1.0p30 and after upgrading to p37 still the same. One more thing to mention is that sometimes the Agent Bakery process (baske and sign agents) shows an error message in the web console for just couple of seconds, then the message disappear (giving not enough time to read it) and the bake process finish successfully

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

Checkmk version 2.1.0p37
Try license usage history update.
Trying to acquire lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
Got lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Got lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Released lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
Released lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7f27a6bc6220]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(<hostname>, base_path=/omd/sites/cmk/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=90, inventory=90), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 20 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 10.10.10.10:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Reading data from agent
Write data to cache file /omd/sites/cmk/tmp/check_mk/cache/<hostname>
Trying to acquire lock on /omd/sites/cmk/tmp/check_mk/cache/<hostname>
Got lock on /omd/sites/cmk/tmp/check_mk/cache/<hostname>
Releasing lock on /omd/sites/cmk/tmp/check_mk/cache/<hostname>
Released lock on /omd/sites/cmk/tmp/check_mk/cache/<hostname>
Closing TCP connection to 10.10.10.10:6556
[cpu_tracking] Stop [7f27a6bc6220 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=20.87999999988824))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f27a6bc6ca0]
[PiggybackFetcher] Fetch with cache settings: NoCache(<hostname>, base_path=/omd/sites/cmk/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=90, inventory=90), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
Piggyback file '/omd/sites/cmk/tmp/check_mk/piggyback/<hostname>/mad-m01-vc01.servidores.ine.es': Successfully processed from source 'mad-m01-vc01.servidores.ine.es'
No piggyback files for '10.10.10.10'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7f27a6bc6ca0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<wmi_cpuload:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<fileinfo:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<winperf_phydisk>>> / Transition HostSectionParser -> HostSectionParser
<<<winperf_if>>> / Transition HostSectionParser -> HostSectionParser
<<<winperf_processor>>> / Transition HostSectionParser -> HostSectionParser
<<<df:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
<<<services>>> / Transition HostSectionParser -> HostSectionParser
<<<wmi_webservices:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<ps:sep(9)>>> / Transition HostSectionParser -> HostSectionParser
Transition HostSectionParser -> NOOPParser
Transition NOOPParser -> NOOPParser
<<<systemtime>>> / Transition NOOPParser -> HostSectionParser
Trying to acquire lock on /omd/sites/cmk/var/check_mk/persisted/<hostname>
Got lock on /omd/sites/cmk/var/check_mk/persisted/<hostname>
Releasing lock on /omd/sites/cmk/var/check_mk/persisted/<hostname>
Released lock on /omd/sites/cmk/var/check_mk/persisted/<hostname>
Stored persisted sections: win_cpuinfo, win_os, win_bios, win_system, win_computersystem, win_disks, win_video, win_networkadapter, win_ip_r, win_wmi_software, win_wmi_updates, win_reg_uninstall, win_exefiles
Using persisted section SectionName('win_cpuinfo')
Using persisted section SectionName('win_os')
Using persisted section SectionName('win_bios')
Using persisted section SectionName('win_system')
Using persisted section SectionName('win_computersystem')
Using persisted section SectionName('win_disks')
Using persisted section SectionName('win_video')
Using persisted section SectionName('win_networkadapter')
Using persisted section SectionName('win_ip_r')
Using persisted section SectionName('win_wmi_software')
Using persisted section SectionName('win_wmi_updates')
Using persisted section SectionName('win_reg_uninstall')
Using persisted section SectionName('win_exefiles')
  -> Add sections: ['check_mk', 'cmk_agent_ctl_status', 'df', 'fileinfo', 'mem', 'ps', 'services', 'systemtime', 'uptime', 'win_bios', 'win_computersystem', 'win_cpuinfo', 'win_disks', 'win_exefiles', 'win_ip_r', 'win_networkadapter', 'win_os', 'win_reg_uninstall', 'win_system', 'win_video', 'win_wmi_software', 'win_wmi_updates', 'winperf_if', 'winperf_phydisk', 'winperf_processor', 'wmi_cpuload', 'wmi_webservices']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
<<<esx_vsphere_vm:cached(1702983662,90)>>> / Transition NOOPParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: ['esx_vsphere_vm', 'labels']
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7f27a67876d0]
value store: synchronizing
Trying to acquire lock on /omd/sites/cmk/tmp/check_mk/counters/<hostname>
Got lock on /omd/sites/cmk/tmp/check_mk/counters/<hostname>
value store: loading from disk
Releasing lock on /omd/sites/cmk/tmp/check_mk/counters/<hostname>
Released lock on /omd/sites/cmk/tmp/check_mk/counters/<hostname>
CPU utilization      Total CPU: 4.03%
Check_MK Agent       Version: 2.1.0p37, OS: windows, TLS is not activated on monitored host (see details)
Disk IO SUMMARY      Read: 33.1 kB/s, Write: 305 kB/s, Latency: 175 milliseconds
ESX Guest Tools      VMware Tools are installed, but the version is not current
ESX Snapshots        Count: 0
Filesystem C:/       90.3% used (503.89 of 558.03 GB), trend: +34.63 MB / 24 hours
Interface Conexión de red Gigabit Intel[R] 82574L [1], (Connected), Speed: 1 GBit/s, In: 128 B/s (<0.01%), Out: 9.39 B/s (<0.01%)
Interface Conexión de red Gigabit Intel[R] 82574L 2 [2], (Connected), Speed: 1 GBit/s, In: 127 B/s (<0.01%), Out: 4.69 B/s (<0.01%)
Interface Conexión de red Gigabit Intel[R] 82574L 3 [3], (Connected), Speed: 1 GBit/s, In: 6.14 kB/s (<0.01%), Out: 7.58 kB/s (<0.01%)
Interface Conexión de red Gigabit Intel[R] 82574L 4 [4], (Connected), Speed: 1 GBit/s, In: 241 B/s (<0.01%), Out: 1.47 B/s (<0.01%)
Interface isatap.ine.es [7], (Connected), Speed: 100 kBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface isatap.{55608C67-4548-4661-8072-9F78E8765F6F} [6], (Connected), Speed: 100 kBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface isatap.{A5969D73-C5BA-4810-AC3D-CBC86E27DDAA} [5], (Connected), Speed: 100 kBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Interface isatap.{DFEA0D50-34E6-4167-BF82-4DA004779A78} [8], (Connected), Speed: 100 kBit/s, In: 0.00 B/s (0%), Out: 0.00 B/s (0%)
Memory               RAM: 34.55% - 44.22 GB of 128.00 GB, Commit charge: 67.57% - 97.97 GB of 145.00 GB
Processor Queue      15 min load: 0.40, 15 min load per core: 0.01 (32 logical cores)
Service Summary      Autostart services: 74, Stopped services: 10
System Time          Offset: -16.7 s
Uptime               Up since Dec 02 2023 19:32:30, Uptime: 16 days 16 hours
Web Service Default Web Site Connections: 0.00
+ EXECUTING INVENTORY PLUGINS
 aruba_wlc_aps: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 check_mk: ok
 check_mk: skipped (no data)
 checkmk_agent_plugins: skipped (no data)
 checkmk_agent_plugins: skipped (no data)
 docker_node_info: skipped (no data)
 docker_node_info: skipped (no data)
 fritz: skipped (no data)
 fritz: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_mem: skipped (no data)
 hp_proliant_mem: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ipmi_firmware: skipped (no data)
 ipmi_firmware: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_if: ok
 inv_if: skipped (no data)
 aix_packages: skipped (no data)
 aix_packages: skipped (no data)
 aix_service_packs: skipped (no data)
 aix_service_packs: skipped (no data)
 inventory_checkmk: skipped (no data)
 inventory_checkmk: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 dmidecode: skipped (no data)
 dmidecode: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_network: skipped (no data)
 docker_container_network: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_network: skipped (no data)
 docker_node_network: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortigate_ha: skipped (no data)
 fortigate_ha: skipped (no data)
 fortimail_system: skipped (no data)
 fortimail_system: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_system: skipped (no data)
 fortisandbox_system: skipped (no data)
 k8s_assigned_pods: skipped (no data)
 k8s_assigned_pods: skipped (no data)
 k8s_daemon_pod_containers: skipped (no data)
 k8s_daemon_pod_containers: skipped (no data)
 k8s_endpoint_info: skipped (no data)
 k8s_endpoint_info: skipped (no data)
 k8s_ingress_infos: skipped (no data)
 k8s_ingress_infos: skipped (no data)
 k8s_job_container: skipped (no data)
 k8s_job_container: skipped (no data)
 kube_cluster: skipped (no data)
 kube_cluster: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_deployment: skipped (no data)
 kube_deployment: skipped (no data)
 kube_namespace: skipped (no data)
 kube_namespace: skipped (no data)
 kube_node: skipped (no data)
 kube_node: skipped (no data)
 kube_pod: skipped (no data)
 kube_pod: skipped (no data)
 kube_statefulset: skipped (no data)
 kube_statefulset: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_video: skipped (no data)
 lnx_video: skipped (no data)
 mobileiron_inventory: ok
 mobileiron_inventory: skipped (no data)
 mssql_clusters: skipped (no data)
 mssql_clusters: skipped (no data)
 oracle_systemparameter: skipped (no data)
 oracle_systemparameter: skipped (no data)
 snmp_extended_info: skipped (no data)
 snmp_extended_info: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_psrinfo: skipped (no data)
 solaris_psrinfo: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_prtpicl: skipped (no data)
 solaris_prtpicl: skipped (no data)
 solaris_routes: skipped (no data)
 solaris_routes: skipped (no data)
 statgrab_net: skipped (no data)
 statgrab_net: skipped (no data)
 win_disks: ok
 win_disks: skipped (no data)
 win_exefiles: ok
 win_exefiles: skipped (no data)
 win_ip_r: ok
 win_ip_r: skipped (no data)
 win_networkadapter: ok
 win_networkadapter: skipped (no data)
 win_reg_uninstall: ok
 win_reg_uninstall: skipped (no data)
 win_video: ok
 win_video: skipped (no data)
 win_wmi_software: ok
 win_wmi_software: skipped (no data)
 win_wmi_updates: ok
 win_wmi_updates: skipped (no data)
 juniper_info: skipped (no data)
 juniper_info: skipped (no data)
 k8s_nodes: skipped (no data)
 k8s_nodes: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_roles: skipped (no data)
 k8s_roles: skipped (no data)
 lnx_if: skipped (no data)
 lnx_if: skipped (no data)
 mem: ok
 mem: skipped (no data)
 mem_used: skipped (no data)
 mem_used: skipped (no data)
 mssql_instance: skipped (no data)
 mssql_instance: skipped (no data)
 netapp_api_disk: skipped (no data)
 netapp_api_disk: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_tablespaces: skipped (no data)
 oracle_tablespaces: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_psmu: skipped (no data)
 perle_psmu: skipped (no data)
 snmp_info: skipped (no data)
 snmp_info: skipped (no data)
 winperf_if: ok
 winperf_if: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_state: skipped (no data)
 citrix_state: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 dell_hw_info: skipped (no data)
 dell_hw_info: skipped (no data)
 docker_container_node_name: skipped (no data)
 docker_container_node_name: skipped (no data)
 esx_systeminfo: skipped (no data)
 esx_systeminfo: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fireeye_sys_status: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 k8s_pod_info: skipped (no data)
 k8s_pod_info: skipped (no data)
 k8s_selector: skipped (no data)
 k8s_selector: skipped (no data)
 k8s_service_info: skipped (no data)
 k8s_service_info: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_uname: skipped (no data)
 lnx_uname: skipped (no data)
 lparstat_aix: skipped (no data)
 lparstat_aix: skipped (no data)
 netapp_api_info: skipped (no data)
 netapp_api_info: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis: skipped (no data)
 prtconf: skipped (no data)
 prtconf: skipped (no data)
 snmp_os: skipped (no data)
 snmp_os: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_uname: skipped (no data)
 solaris_uname: skipped (no data)
 suseconnect: skipped (no data)
 suseconnect: skipped (no data)
 win_bios: ok
 win_bios: skipped (no data)
 win_computersystem: ok
 win_computersystem: skipped (no data)
 win_cpuinfo: ok
 win_cpuinfo: skipped (no data)
 win_os: ok
 win_os: skipped (no data)
 win_system: ok
 win_system: skipped (no data)

Piggyback file '/omd/sites/cmk/tmp/check_mk/piggyback/<hostname>/mad-m01-vc01.servidores.ine.es': Successfully processed from source 'mad-m01-vc01.servidores.ine.es'
No piggyback files for '10.10.10.10'. Skip processing.
[cpu_tracking] Stop [7f27a67876d0 - Snapshot(process=posix.times_result(user=0.050000000000000044, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.049999999813735485))]
[agent] Success, [piggyback] Successfully processed from source 'mad-m01-vc01.servidores.ine.es', execution time 20.9 sec | execution_time=20.930 user_time=0.050 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=20.880

Please, ¿any help?

BR

andreas-doehler · December 19, 2023, 11:29am

After you baked the agent and the new agent is distributed to the systems do you see anywhere on this system inside the “C:\ProgramData\checkmk\agent\local” or “\plugins” your script?

Another test if your script is really baked into the agent is on the command line inside your CMK site.

cmk --debug -vvA -f hostname

Here you should see if your script is included inside the agent bake process.

jorgejuan · December 19, 2023, 4:26pm

Hi,

The output of the command passed (yes, the local check is there)

cmk --debug -vvA -f
Trying to acquire lock on /omd/sites/cmk/etc/check_mk/multisite.mk
Got lock on /omd/sites/cmk/etc/check_mk/multisite.mk
Releasing lock on /omd/sites/cmk/etc/check_mk/multisite.mk
Released lock on /omd/sites/cmk/etc/check_mk/multisite.mk
BakeryTargetHost(‘’)…Found no Python 2 version for cmk_update_agent.py, deploying with Python 3 support only.linux_deb:baking…linux_rpm:baking…(fast repackage)…solaris_pkg:baking…windows_msi:baking…plugins/cmk_update_agent.checkmk.py 78.65 kB
plugins/cmk-update-agent.exe.README.txt 458.00 B
plugins/mk_inventory.vbs 13.73 kB
plugins/mk_logwatch.checkmk.py 47.61 kB
config/logwatch.cfg 482.00 B
config/cmk-update-agent.cfg 1.80 kB
local/sym_link_current_log.ps1 679.00 B
linux_tgz:baking…(fast repackage)…solaris_tgz:baking…(fast repackage)…aix_tgz:baking…OK
Trying to acquire lock on /omd/sites/cmk/var/check_mk/agents/.bake_revision
Got lock on /omd/sites/cmk/var/check_mk/agents/.bake_revision
Releasing lock on /omd/sites/cmk/var/check_mk/agents/.bake_revision
Released lock on /omd/sites/cmk/var/check_mk/agents/.bake_revision

I catched the screen shot with the baking error check_mk returns. I realized that it is happening with all Windows Server 2012, if you do a local agent test there is no <<<local>>> section, eventhough it is enabled in the yml file. In 2019 the same settings works fine, the local check executes correctly.

BR

Anders · December 19, 2023, 5:21pm

what happens if you run the ps1 on that windows hosts, in that folder? preferably you should run it as SYSTEM as well, there are methods of this can be done.

perhaps your script does not output anything? needs special permissions to read a folder?

jorgejuan · December 20, 2023, 7:56am

Hello Anders,

The same script distributed to windows server 2019 works correctly. This only happens with windows server 2012. It is very weird behaviour:

a) Script distributed with Agent Bakery:
– The script is send to the host and appears in the /local folder, but for the agent is like the folder is empty and no <<<local>>> section is generated when invoking the agent

b) Script distributed manually:
– Works correctly. <<<local>>> section is generated and the output of the local script is appended.

BR.

Anders · December 20, 2023, 5:06pm

so you don’t want to test to run the script manually? ok

andreas-doehler · December 20, 2023, 5:11pm

I would also had said it like @Anders. Test the distributed powershell manually and see what it does on 2012.
If your distributed script has an encoding not correctly handled by 2012 then it will not run.
Your manually installed script can have a complete different encoding then the distributed one.

jorgejuan · January 8, 2024, 10:02am

Hi,

My apologies for not responding, I was out of the office.

Yes, of course, I did basic troubleshooting: check PS execution policy, run the script manually, reinstall the basic agent manually, and so on… I don’t know what is going on.

Any way, I can not spend more time on this. I wrote same script in VBS and it worked.

Thank you so much

BR