Distributed monitoring with docker image + livestatus not working

Hello,

I’m struggling now with pretty weird situation. I would like to know which all ports is Check_MK using.
We have Master instance (let’s call it master1 and multiple satelites (let’s call one of them satelite1. That’s okay, everything works many years.

Now I have to connect another satelite (satelite2), but we don’t have direct network connection between master1 and satelite2. But satelite2 has direct connect to satelite1. So we made iptables DNAT on satelite1, so satelite1 is kind of jump server between master1 and satelite2:

master1 -> satelite1
master1 -> satelite1 DNAT -> satelite2

On satelite1 we are DNATing ports:
80 --> 8080
443 --> 8443
6557 --> 9000

On satelite2 we are running official Check_MK Docker image 1.6.0p11 on Kubernetes. There is no 100% compatibility with Kubernetes, we had to do some workarounds with tmpfs to run in properly.

On Kubernetes LoadBalancer we are redirecting:
9000 --> 6557
8080 --> 5001 (I made new site inside container, that’s why it is not 5000)

On Docker image, via omd config I enabled livestatus on port 6557, enabled distributed monitoring.
With these crazy port forwarding I was able to Login Docker image into master1 instance in Distributed Monitoring section.

However master1 instance can’t get Livestatus info.
I get error:
“Unhandled exception: 400: Site connection not initiated (Heartbeat timeout after 2.0 sec).”

Am I missing some other port in forwarding?

This is output from Docker container. Why it is listening also on 6556? Why only this port has PID, and 5001 doesn’t have PID of running Apache?

netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5001 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:6556 0.0.0.0:* LISTEN 55/xinetd
tcp 0 0 0.0.0.0:6557 0.0.0.0:* LISTEN -

Only one idea from my side, can your loadbalancer handle non HTTP/S connection types.
I had problems with Ingress as i played a little bit with Kubernetes and CMK containers.
It was not possible to connect to livestatus over Ingress as it is only supporting HTTP/S connections.

What kind of Non HTTP/S connection it needs?
I have no idea if LoadBalancer can handle it, have to ask colleagues who installed it.

Port 6557 with the livestatus is a TCP connection in newer versions it is also secured with TLS.

We do not have TLS enabled, using plain text. I installed check-mk rpm directly to the OS, on another server in network, so no docker container.

Now we have master1 --> satelite1 DNAT --> satelite2 with bare metal installation.

And Livestatus also doesn’t work. Ports 80, 443, 6557 are telnet-able from master1 to satelite2. So what could be wrong, that 6557 is allowed, but Livestatus can’t access over it? Are we missing some other unknown port in iptables DNAT?

I found solution. From Check_MK 1.6.xx default value was changed to “on” for Livestatus TLS in omd config. I didn’t know that and doesn’t care of it when looking in omd config.

I thought that it is enough to use plain text in Portal.

So after switching Livestatus TLS to off in omd config, satelite started to work correctly.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.