CMK 2.1 - "Activating changes" significantly slower than in all previous releases

[2022-10-11 08:40:54]  Creating cmc protobuf configuration...Trying to acquire lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/core_host_config/Telekom-Router
[2022-10-11 08:40:54] Got lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/core_host_config/Telekom-Router
[2022-10-11 08:40:54] Releasing lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/core_host_config/Telekom-Router
[2022-10-11 08:40:54] Released lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/core_host_config/Telekom-Router
...
[2022-10-11 08:41:21] Trying to acquire lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/core_host_config/WAN-HOST1
[2022-10-11 08:41:21] Got lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/core_host_config/WAN-HOST1
[2022-10-11 08:41:21] Releasing lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/core_host_config/WAN-HOST1
[2022-10-11 08:41:21] Released lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/core_host_config/WAN-HOST1
[2022-10-11 08:41:21] Trying to acquire lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/helper_cache_info
[2022-10-11 08:41:21] Got lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/helper_cache_info
[2022-10-11 08:41:21] Releasing lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/helper_cache_info
[2022-10-11 08:41:21] Released lock on /omd/sites/MYSITE/var/check_mk/core/helper_config/2760/helper_cache_info
[2022-10-11 08:41:21] OK
[2022-10-11 08:41:22] /omd/sites/MYSITE/var/check_mk/core/config.pb written.
[2022-10-11 08:41:22] Reloading monitoring core...OK
[2022-10-11 08:41:22] Releasing lock on /omd/sites/MYSITE/etc/check_mk/main.mk
[2022-10-11 08:41:22] Released lock on /omd/sites/MYSITE/etc/check_mk/main.mk

This is for cmk -vvO, pretty much the entire duration is spent iterating through the helper_config files for every host. 27 seconds in this current example.

Now one could assume this might be bottlenecked by disk IO of the server, but we’re writing onto SSDs, not HDD (and not virtualized but installed on the hardware), so random disk access should be more than fast enough.

And “incremental”, the new innovation with 2.1 is nowhere to be seen in any case, if it needs to touch all of the files?

The “parallelize core config creation” is still at default, so “on”.

As mentioned, agent baking is not the issue for us, we’ve set up the Agent OS rule in order to bake Windows and Linux agents only and it takes a few seconds on top and is not the issue here. We’ve always had the auto-baking setting enabled in the past (and intend to re-enable it) because we do not need to sign the agents anyway as the agent bakery does not work for us (because we do not allow access from less trusted networks to trusted networks, therefore the agents cannot reach the Checkmk server, only the other way around)