Kubernetes monitoring on CRE 2.0.0p34: CPU resources includes evicted pods

Troubleshooting
, ,
1

Dear Forum,

I just had an issue on a site where we are using Kubernetes Monitoring
as per Monitoring Kubernetes.

I have to use version 2.0 for now, because the clusters I’m monitoring
with Checkmk are on OpenShift 3.x, and I don’t know if the “newer”
implementation starting with 2.1, works for OpenShift 3.x.

A worker was reporting 134% CPU requests:

image
image1206×697 78.2 KB

Having checked the worker via oc, it’s “real” requests, were
about 11%:

image
image1358×976 143 KB

Then I noticed, that this particular worker, had many “Evicted” pods,
due to a “DiskPressure” condition it had experienced earlier that day:

image
image1425×888 179 KB

As soon as I deleted all “Evicted” pods, the check went OK again, now
reflecting the “real” values:

image
image1495×54 2.11 KB

So it looks like, Checkmk did not exclude “Evicted” pods in its calculation.

Again: The instance in question is a “RAW” edition in version 2.0.0p34. I’m
aware of the fact that there is a newer release in the “2.0” branch, so
I haven’t checked werks, but if there aren’t any, I believe this is a bug.

Anyone ever came across such a situation?

Regards,
Thomas

2

Your Checkmk version is very old. We put a lot of work into Kubernetes monitoring in recent versions. I suggest you update as soon as possible.

3

Hi Robin!

“Very old” ? There was a release in the 2.0 branch yesterday, Aug 10th 2023:
Checkmk 2.0.0p38

Anyway… Does Checkmk 2.1 or 2.2 support OpenShift 3.x in the RAW edition?
If yes then I’m happy to embark on this adventure. If not, I cannot do this, and simply “lose monitoring”.

Furthermore the - quite specific potential bug - I’m pointing out was more FYI, in case it affects
other versions, too…

Regards,
Thomas

4

You are right, 2.0.0 still receives security updates and some bug fixes until the beginning of September. Then the passive support ends, putting the version into EOL.

What I mean with “very old”: There were two major releases with tons of features and improvements after 2.0.0. Updating is inevitable. Or as the Borg say: “Resistance is futile”. :slight_smile:

1 Like
5

Hi Robin,

:slight_smile: OK, perfect. I want to upgrade to 2.2 as soon as possible, I’m already
using it on other sites, and I’m happy with it.

Again: Does 2.1 or 2.2 support OpenShift 3.x in the RAW edition?
If yes, I’m gonna try it. If not, I can’t upgrade.

Thomas

6

Partial monitoring of openshift clusters is possible in 2.1 and 2.2
Full monitoring in 2.2 with the cloud edition
In 2.1, via the right scc and selinux changes, full monitoring is possible but not recommended.
There is enough material in the docs explaining this, please check it out. We also shared updates on recent Checkmk conferences on this topic

1 Like
7

Thank you very much Martin, I appreciate it!

Regards,
Thomas