Weird errors when going 1.4.0p38.cre to 1.5.0 -- "Internal automation error"

[cdnvballer]

after distributed monitoring upgrade,
the following errors come up when I try to apply changes:

Failed: Error running check_mk --automation restart --. Invalid output from webservice (invalid syntax (, line 1)):

Error in plugin file /omd/sites/testinstance/share/check_mk/checks/chrony: name ‘discover_single’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/cisco_ip_sla: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/citrix_licenses: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/docker_node_info: name ‘parse_docker_node_info’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/f5_bigip_vserver: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_fans: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_sc2_cpu_status: name ‘is_fsc_sc2’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_sc2_fans: name ‘is_fsc_sc2’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_sc2_info: name ‘is_fsc_sc2’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_sc2_mem_status: name ‘is_fsc_sc2’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_sc2_power_consumption: name ‘is_fsc_sc2’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_sc2_psu: name ‘is_fsc_sc2’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_sc2_temp: name ‘is_fsc_sc2’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_sc2_voltage: name ‘is_fsc_sc2’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_subsystems: name ‘is_fsc’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/fsc_temp: name ‘is_fsc_temp_prefer_sc2’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/genua_carp: name ‘scan_genua’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/genua_fan: name ‘scan_genua’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/genua_pfstate: name ‘scan_genua’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/genua_state_correlation: name ‘scan_genua’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/genua_vpn: name ‘scan_genua’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_bat_temp: name ‘discover’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_compressor: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_cooling: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_cooling_position: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_cooling_status: name ‘discover’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_fans: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_fans_condenser: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_pump: name ‘discover’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_system_events: name ‘discover_single’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_temp_fluid: name ‘discover’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/liebert_temp_general: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/lnx_thermal: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/mongodb_collections: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/mongodb_mem: name ‘discover_single’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/nginx_status: name ‘get_parsed_item_data’ is not defined
Error in check include file /omd/sites/testinstance/share/check_mk/checks/firewall_if.include: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/pfsense_if: name ‘check_firewall_if’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/uptime: name ‘discover_single’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/veeam_tapejobs: name ‘get_parsed_item_data’ is not defined
Error in plugin file /omd/sites/testinstance/share/check_mk/checks/winperf_tcp_conn: name ‘check_tcp_connections’ is not defined
[]

Any ideas on how to troubleshoot this? I’ll try to revert and see if there is something in the existing configuration that’s causing this…

[cdnvballer]

when I revert back to 1.4.0p38, I get this error when trying to apply changes:
Failed: Got invalid data:

Internal automation error: The domain “apache” does not exist
Traceback (most recent call last):
File “/omd/sites/testslave/share/check_mk/web/htdocs/wato.py”, line 10261, in page_automation
html.write(repr(automation_commandscommand))
File “/omd/sites/testslave/share/check_mk/web/htdocs/wato.py”, line 5625, in do_activate_changes_automation
return execute_activate_changes(domains)
File “/omd/sites/testslave/share/check_mk/web/htdocs/watolib.py”, line 5243, in execute_activate_changes
domain_class = ConfigDomain.get_class(domain)
File “/omd/sites/testslave/share/check_mk/web/htdocs/watolib.py”, line 292, in get_class
raise NotImplementedError(_(“The domain “%s” does not exist”) % ident)
NotImplementedError: The domain “apache” does not exist

[cdnvballer]

issue resolved =) turns out the 1.4.0p38 setup was fine but had some outstanding monitoring alerts related to discoveries… once I cleared these, the upgrade went fine… =)