Update 2.2 -> 2.3: High CPU Peaks central Site

[CFriedrich (Christian Friedrich)]

CMK version: 2.3.0p33 cee
OS version: RHEL 8.10

Good Morning,
yesterday we’ve updated our prod environment from 2.2 to 2.3.0p33 cee. Everything works fine so far. But we notice a high cpu load peak every hour. Is there maybe a new job, which runs every hour? Or any other idea? We have one central site and 10 remote sites.

regards
Christian

[mbunkus (Moritz Bunkus)]

One thing that happened to more than one person (to me, too) is that when you upgrade your agents to 2.3.0 & enable the new way of connecting to the server, you might not have deactivated the agent encryption that was present before 2.3.0 already — ending up with two layers of encryption. Yes, that costs quite a bit of processing power. See the following:

Significant CPU Increase After Upgrading to Checkmk 2.3.0p24

We have now successfully resolved the issue on our side. The following measures helped: Changing the Proxmox CPU type to x86-64-v2-AES (available since Proxmox version 8). We implemented this change yesterday morning, which roughly halved the load (see graphs). Disabling the “Symmetric encryption” rule. This encryption has already been active by default with agent registration since CMK Agent version 2.2. After disabling it, the load is now even lower than before the upgrade to 2.3. We made t…

That being said, this higher load would not be an hourly spike but higher load all of the time. Nevertheless, worth looking into.

[CFriedrich (Christian Friedrich)]

Hi, thanks for your feedback. We haven’t updated the agents yet; they’re still on version 2.2. But thanks for the tip; I’ve definitely made a note of it for the agent upgrade. Best regards

[Heavy]

@mbunkus

One thing that happened to more than one person (to me, too) is that when you upgrade your agents to 2.3.0 & enable the new way of connecting to the server, you might not have deactivated the agent encryption that was present before 2.3.0 already — ending up with two layers of encryption.

It is not only the double layer of encryption, but also a change in the key derivation function of the symmetric encryption between Checkmk Agent 2.2 and 2.3 that eats up much CPU time. See this forum thread:

Significant CPU Increase After Upgrading to Checkmk 2.3.0p24

Preliminary results seem to indicate that it drops CPU usage by about half (meaning down from 60% total to 30% total), which is incredibly nice to see. Very good catch, & thanks for posting about it.