Check_MK running behind LB on port 80 redirects to URL including port :5000

Hi folks, I am trying to run Check_MK in k8s behind a cert-manager, and I almost succeeded. Basically I have these statefulset and ingress definitions (dumbed down to the bare minimum), and check_MK starts up just fine.

BUT. When I connect to our load balancer, I get the following result:

  • the container answers with a redirect (which is fine)
  • but the redirect has a :5000 attached to the redirection URL.

that - of course - does not work (see bottom for curl trace). can I somehow convince the check_MK container to stop doing that?

thanks for any help!
/axel

All right, this is a severe bug if you ask me. Check_MK redirects differently depending on which Host: header is passed, which is just wrong.

Test preparation

$ CMK="checkmk/check-mk-raw:1.6.0-latest"
$ docker run -dit -p 80:5000 --name cmk80 $CMK
$ docker run -dit -p 8080:5000 --name cmk8080 $CMK

Test

I have removed all non-essential output lines to illustrate the behavior. The crucial part is the location: header in the redirect respose, which is different depending on which Host: header you send, which is just bad.

#   B A D  ! ! !   will never work, and this is a typical load balancer use case.
$ curl -sv localhost
> GET / HTTP/1.1
> Host: localhost
< HTTP/1.1 302 Found
< Location: http://localhost:5000/cmk/


# you add the port "80" on curl, which - fully understandibly - removes it
# so back to   B A D  ! ! !   ...
$ curl -sv localhost:80
> GET / HTTP/1.1
> Host: localhost
< HTTP/1.1 302 Found
< Location: http://localhost:5000/cmk/


# let's *explicitly* add the Host: header
# (spoiler: then it works, but now *Check_MK* omits the port)
$ curl -sv localhost -H "Host: localhost:80"
> GET / HTTP/1.1
> Host: localhost:80
< HTTP/1.1 302 Found
< Location: http://localhost/cmk/

With the non-port-80-container all seems harmless, but isn’t.

# variant 1, works
$ curl -sv localhost:8080
> GET / HTTP/1.1
> Host: localhost:8080
< HTTP/1.1 302 Found
< Location: http://localhost:8080/cmk/


# variant 2 - same as variant 1.
curl -sv localhost:8080 -H "Host: localhost:8080"
> GET / HTTP/1.1
> Host: localhost:8080
< HTTP/1.1 302 Found
< Location: http://localhost:8080/cmk/


# BUT: variant 3 is fucked again. (and again this is a typical LB use case)
curl -sv localhost:8080 -H "Host: localhost"
> GET / HTTP/1.1
> Host: localhost
< HTTP/1.1 302 Found
< Location: http://localhost:5000/cmk/

I dug myself into the apache docs, and by trial and error I found a solution: Delete the ServerName config setting from /opt/omd/sites/cmk/etc/apache/listen-port.conf. Then it behaves “nicely”.

Running behind a k8s load balancer (and this is MY use case) you only get to the backend if the load balancer has already routed you by interpreting the Host: header, so currently see no disadvantages.

For anything else … I’m neither a security nor an apache / website hosting person, so probably I can’t really judge if this is a good idea. But solving my problem - it can’t be too bad :wink: .

so for completeness sake I have solved my problem.

I THINK THIS IS STILL A BUG cause I cannot determine if what I did has any negative impacts.

1 Like

Hi,
thanks for pointing this out. We will discuss that internally with our developers and take a look, if we can patch the official version, too.

2 Likes