CMK 2.1 - "Activating changes" significantly slower than in all previous releases

[2022-08-08 22:12:58] Starting full compilation for all hosts Creating global helper config...OK
[...]
[2022-08-08 22:13:28] Released lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2420/helper_cache_info
[2022-08-08 22:13:28] OK
[2022-08-08 22:13:28] /omd/sites/MYSITE/var/check_mk/core/config.pb written.
[2022-08-08 22:13:28] Baking agents...Trying to acquire lock on /omd/sites/MYSITE/etc/check_mk/multisite.mk
[...]
[2022-08-08 22:13:36] OK
[2022-08-08 22:13:51] Restarting monitoring core...OK

Moving the non-resolvable hosts out of the CMK config brings the protobuf delta down from 49 seconds to 30 seconds, so it makes an impact - but we’re still far from the original 14 seconds, and even further from any “improvements” touted as significant during the Checkmk conference.

This message “Cannot lookup IP address of” began to be logged once we’d updated to 2.0 and only concerns hosts that we inject into the site via cmcdump of another site. Our site is indeed unable to resolve any of those hosts, and we wouldn’t expect it to. We expect that Checkmk would not need to worry about that at all, after all it passively consumes the host and service metadata and check results as-is.

Also, all of those hosts dutifully get their IP addresses imported into our CEE as part of the cmcdump and neither the original instance nor our instance need to look up anything at all, given that all of our hosts on all of our sites have their primary IP address statically defined as host property - we never do DNS lookups for any of our hosts, save 2 or 3 such as “www.google.com” as reference ICMP targets for WAN connectivity reports.