Loading increases every two hours (RRDCached tuning) since a check_mk crash

Hi,

I am using version check-mk-raw-2.2.0p47-el9-38.x86_64
OS : Redhat 9.x
RAM : 16G
CPU : 8

I wanted to get your opinion after a Nagios/check-mk crash, I have attached the load graph from my check_mk server.

(0) Before the crash, the load is normal.

(1) During the crash, there is no data.

(2) After restarting the check_mk service, the load spikes to 50 every two hours. Every two hours, the following command is executed: RRDCached tuning → Disk flush interval of old data: 2 hours

(3) We increased the number of CPUs on the server—from 8 to 12—and the load decreased.

(4) Then the load increased on its own.

(5) I scheduled the checks to run every 20 minutes instead of every 2 hours, and the load decreased again. It spikes less frequently, but consequently, it occurs more often.

Do you have any idea why I didn’t have any loading problems before, but I’ve had problems since the crash?

Regards

Stan

Maybe changing max_concurrent_checks in ~/etc/nagios/nagios.d/timing.cfg helps.
See:

In my situation it dropped the mem usage and cpu load a lot.

1 Like

Yes the graph looks like the problem @Yggy linked from my earlier post.

2 Likes

Hello,

Thank you for your reply.

If I understand correctly, my current value is 0.

max_concurrent_checks=0

So if I have 8 CPUs, I set the value to 16?

How do you modify this value in the command line for this file?

However, what’s strange is that I didn’t have this problem before the crash.

Thanks, Stan

After you edited the file you only need to do a “cmk -R”
The reason for the problem is not so easy to describe. After your crash the core detected some check the he needed to check and tried to check all at the same time (caused by the 0 seeing) this leads to an overload state on the machine and you get the very high CPU load. This again leads to timeouts and so on. Sometimes the problems gets worse and you get higher spikes.
With the limit of concurrent checks you prevent the overload state and the timeout problematic.
If 12, 16 or 20 is a good value you need to test for your system. I had only the experience that it should not be much more than the double number of cores available.

Ah, okay, I see about cmk -R. I thought there was another, more complex command.

I’ll test it with different values ​​and let you know how it goes.

Thanks.