check_httpv2 seems to consume significantly more CPU resources than the legacy check_http plugin.
My small benchmark script runs curl, check_http, and check_httpv2 sequentially against the same local HTTPS URL and measures wall time, user CPU, and sys CPU using time. Here are the results from 10 cycles:
Usage of --no-body shows barely any improvement, the overhead of check_httpv2 is not caused by body parsing.
Impact at scale: check_httpv2 is still fast per individual execution, no doubt. A small overhead of maybe 5% compared to check_http would be perfectly understandable given the additional features. But a 15Ă— factor is significant.
Small things add up! If you’re running 30-40 active HTTP checks and other active checks every minute, this additional CPU cost accumulates and adds noticeable load to the system.
And on top of that, Checkmk adds its own overhead per execution (scheduling, queuing, perfdata parsing, state handling, notification evaluation), which adds a significant layer of CPU cost.
Have any of you observed similar behavior? Would be interested to hear your experiences.
Questions:
Is the significantly higher CPU overhead of check_httpv2 a known issue?
Nach dem Update auf die Version 2.4.0p8 hatte ich massive Probleme mit dem check_httpv2 bekommen. Diese aktiven Checks bremsen das ganze System aus. Der CPU Load ist heftig nach oben gegangen (s. Screenshot). Dadurch wurde das System tlw. so langsam das Benachrichtigungen nicht mehr zugestellt wurden und das die Aktivierung von Änderungen fehl schlug. An den check_httpv2 Checks hat sich nichts geändert. Das sind einige (1402 Stück), die aber unter der Version 2.3.0.p35 problemlos liefen. Der Support konnte mir da leider auch nicht weiter helfen oder eine Lösung anbieten. Ich habe die check_httpv2 auf eine zweite Maschine bzw. Site verlagert damit ich wieder arbeiten konnte.
After updating to version 2.4.0p8, I encountered severe issues with check_httpv2. These active checks are slowing down the entire system. The CPU load skyrocketed (see screenshot). As a result, the system became so sluggish—at times—that notifications were no longer delivered, and the activation of changes failed. No changes were made to the check_httpv2 checks themselves; while there are quite a few of them (1,402 instances), they ran without any issues under version 2.3.0p35. Unfortunately, Support was unable to provide any further assistance or offer a solution. I have since offloaded the check_httpv2 checks to a second machine—or site—so that I could resume my work.
I also ran some benchmarks and can confirm that check_httpv2 has an increased CPU load in Checkmk 2.4.0 compared to 2.3.0.
This seems to be due to some changes in the dependencies used that introduced more overhead. This overhead shouldn’t have such much impact, but check_httpv2 used multi-threading, which means the overhead occured for every thread.
I cannot confirm the ~15x CPU factor with my setup, but the number of threads scale with the number of CPUs, so it would actually be more inefficient on a system with more CPUs.
