Long outstanding issue (growing), PEND - No time difference

General
,
1

So we have a check that we wrote that gathers microservices information. New services get added. But randomly, a service will configure and get locked into a forever PEND - No time difference state. It never leaves this state. You can manually run the check, you can see the data, the data changes for the service, you can debug… all works… but if you throw it to the microcore, you get PEND - No time difference. Is there anything I can do to bring these back to some sort of original state? Ideally I don’t want to nuke all the data I’ve collected over the years for the host.

Btw, variants of this same question have been made in the past without satisfactory resolution, IMHO. But, maybe this time… ???

2

Hi Chrisotpher,

So we have a check that we wrote that gathers microservices information.
Is this a local check or an agent plugin being parsed by a check plugin ?

Is it possible to reproduce the problem easily with an example ?

You can manually run the check, you can see the data, the data changes for the service, you can debug… all works… but if you throw it to the microcore, you get PEND - No time difference.

Do you see the data when running “cmk --debug -vII or cmk --debug -vn” as well , but the UI doesn’t display it?

3

Again, debugs, running manually, all work, even works with -nv, but as soon as microcore in involved… presto.

It “stays”, but the check returns lots of service data, and it could happen to any of the ones returned, no pattern.

That’s why I’m seeking a way to “reset” whatever it is that causes the issue on the microcore side. There’s got to be a way to see the service again as “brand new” without destroying the whole host. And, disabling or removing the service from the UI side isn’t clearing what I’m talking about.

4

That’s why I’m seeking a way to “reset” whatever it is that causes the issue on the microcore side. There’s got to be a way to see the service again as “brand new” without destroying the whole host. And, disabling or removing the service from the UI side isn’t clearing what I’m talking about.

Regarding the state “Pending”:CMC has no such state for hosts/services. Its a GUI notion.
I can recommend to try to set the “Logging of the Core” / “Core” to “debug” . This will show the scheduling of checks in the CMC.

So we have a check that we wrote that gathers microservices information.

I would like to reproduce the problem in my K8 environment. Is it possible to use your check ?

5

It’s custom. Makes springboot rest calls to produce stats for rest methods.