the new stable release 2.5.0p11 of Checkmk is ready for download.
This stable release ships with 42 changes affecting all editions of Checkmk,
3 changes for all commercial editions, 0 Ultimate specific,
0 Ultimate with multi-tenancy specific and 0 Cloud specific changes.
Attempting to install check-mk-ultimate-2.5.0p11-el10-38.x86_64.rpm on Red Hat Enterprise Linux release 9.8 (Plow) gives these errors:
pre-compile bytecode (.pyc) for Python files..
/omd/versions/2.5.0p11.ultimate/bin/python3: /lib64/libm.so.6: version GLIBC_2.38’ not found (required by /opt/omd/versions/2.5.0p11.ultimate/bin/../lib/libpython3.13.so.1.0) /omd/versions/2.5.0p11.ultimate/bin/python3: /lib64/libm.so.6: version GLIBC_2.35’ not found (required by /opt/omd/versions/2.5.0p11.ultimate/bin/../lib/libpython3.13.so.1.0)
/omd/versions/2.5.0p11.ultimate/bin/python3: /lib64/libc.so.6: version `GLIBC_2.38’ not found (required by /opt/omd/versions/2.5.0p11.ultimate/bin/../lib/libpython3.13.so.1.0)
I know there have been a lot of changes around this recently, is this expected behavior?
After upgrading from Latest 2.4.0pX → 2.5.0p11 on a testbox i kept getting ‘crit’ on amount of processes ‘automation-helpers’
omd status reported green all across the board
logfile on automation helper showed a clean startup (master+ helpers) , no errors at all.
In the end i decided to turn the automation helpers in omd config.
As to continue my investigation i flipped the switch on automation helpers again next day, only to find out that the observed service was not going back to crit.
Tonight i upgraded a 2nd testbox in the same way (this one is multisite), and again for both sites this specific helper processes went to crit after the upgrade.
Now, taking it as a shot in the dark, i flipped the automation-helper switch to off one site.
Restarted the site, and then again enabled it, and miraculously for both sites the issue went away.
I do not know as to why i had to deliberately/force-flip this switch in omd config to get this running, and its even more weird that on the multi-site box flipping the switch on → off and again off → on solved it for both sites…
But it worked
So i have no clue as to why the UI is reporting this at all after an upgrade.
Enabling it after it was already disabled did not solve the service’s Crit state.
So i disabled it again, then did a rediscover services, and the service went away.
Then again enabled te option in omd config
As expected (having seen same behaviour on 2 other test-boxes), the service did not return as ‘crit’, but as OK.
Personal note: (may be experienced as shocking):
i have a bad feeling that since CMK exploded its employee numbers after the take-over its releases, choices and future aims are far from what the using community is able to take in.
Also from my personal, and above experience update quality control seems to be ‘less-then-optimal’.
From my personal standing, and using the ‘community’ -edition i am more then welcome to run the extra mile, but some basic things - like my issue posted above will need to get sorted at CMK itself.
One of those other items really botherig me is package building, like @LaSoe mentioned.
IMHO this is a basic requirement, and its been ignored for far too long.
The issue should not have appeared, thus would be curious to understand it better.
From 2.4.0p26 we added a forward-compatible mechanism as the process name changed to ensure that on upgrade this service actually doesn’t go CRIT. It captures processes with by matching ~(.*cmk-automation-helper.*|gunicorn:.*automation-helper). I have a hypothesis with regards to that: the autochecks still have the old matching logic, which is not migrated. Will look into that. That’s why Andreas’ proposal also works I think.
Regarding your meta feedback:
The change in the process discovery rules follows the wish to have better out-of-the-box monitoring for the Checkmk server in place. Something a lot of users have wished for a long time.
Previously, you couldn’t spot how a Checkmk instance performed pre-/post update with regards to key components. Now, for each process which plays in important role, you get CPU and Memory metrics.
What I hear from Checkmk partners as well as from users is that many of the changes in the last years have improved their experience. Stand-still is the death of software. Ask Ethan and Nagios back in 200x.
Automation helpers were introduced to increase the speed of Checkmk, i.e. when adding/removing/changing services. See for yourself: https://youtu.be/K0R1tGx2rxY?si=08i7pfJTsrYfaqwN&t=325
And with regards to package building. The issue mentioned exactly here has nothing to do with a wrongly built package or alike. The user just accidentally downloaded the wrong package. He downloaded the RHEL 10 package while his system is running 9.8. Happened to everyone and Lars’ change would not solve it, but helps highlighting the root-cause. But fortunately we have a helpful community here!
Thanks all for reporting this! I am currently investigating this issue and I think I can shine some light on why this suddenly appears:
With 2.4.0p26 we changed the way the automation-helper process is discovered in a forward compatible manner, since we intended to change how the automation-helper is run and the old rule would not have worked with this.
In 2.5 we implemented the change to the automation-helper process, changing its name.
However, the match pattern from the discovery rule is copied into the discovered service at discovery time and never changed afterwards. So if you happened to discover the process on a pre-2.4.0p26 site, the service still carries the old pattern, no matter that the rule itself was updated. The 2.5.0 site then tries to match the old pattern against the new process name → this leads to a critical state. Unfortunately, the 2.4.0p26 pattern also turned out to be incomplete, so services discovered on 2.4.0p26 or later are equally affected after the update.
This also explains why a re-discovery fixes the issue: the current rule is applied and its pattern are persisted again, so everything works as expected. This is clearly an oversight on our side and should not have slipped through our QA process. I will try to raise this further with my colleagues, so we can avoid this kind of disruption in the future.
I am working on an update action, migrating the stored service parameters for both cases. This hopefully makes it into the next 2.5.0 release, so future updates to the latest version are safe.
