Hi folks, I am trying to run Check_MK in k8s behind a cert-manager, and I almost succeeded. Basically I have these statefulset and ingress definitions (dumbed down to the bare minimum), and check_MK starts up just fine.
BUT. When I connect to our load balancer, I get the following result:
the container answers with a redirect (which is fine)
but the redirect has a :5000 attached to the redirection URL.
that - of course - does not work (see bottom for curl trace). can I somehow convince the check_MK container to stop doing that?
All right, this is a severe bug if you ask me. Check_MK redirects differently depending on which Host: header is passed, which is just wrong.
Test preparation
$ CMK="checkmk/check-mk-raw:1.6.0-latest"
$ docker run -dit -p 80:5000 --name cmk80 $CMK
$ docker run -dit -p 8080:5000 --name cmk8080 $CMK
Test
I have removed all non-essential output lines to illustrate the behavior. The crucial part is the location: header in the redirect respose, which is different depending on which Host: header you send, which is just bad.
# B A D ! ! ! will never work, and this is a typical load balancer use case.
$ curl -sv localhost
> GET / HTTP/1.1
> Host: localhost
< HTTP/1.1 302 Found
< Location: http://localhost:5000/cmk/
# you add the port "80" on curl, which - fully understandibly - removes it
# so back to B A D ! ! ! ...
$ curl -sv localhost:80
> GET / HTTP/1.1
> Host: localhost
< HTTP/1.1 302 Found
< Location: http://localhost:5000/cmk/
# let's *explicitly* add the Host: header
# (spoiler: then it works, but now *Check_MK* omits the port)
$ curl -sv localhost -H "Host: localhost:80"
> GET / HTTP/1.1
> Host: localhost:80
< HTTP/1.1 302 Found
< Location: http://localhost/cmk/
With the non-port-80-container all seems harmless, but isn’t.
# variant 1, works
$ curl -sv localhost:8080
> GET / HTTP/1.1
> Host: localhost:8080
< HTTP/1.1 302 Found
< Location: http://localhost:8080/cmk/
# variant 2 - same as variant 1.
curl -sv localhost:8080 -H "Host: localhost:8080"
> GET / HTTP/1.1
> Host: localhost:8080
< HTTP/1.1 302 Found
< Location: http://localhost:8080/cmk/
# BUT: variant 3 is fucked again. (and again this is a typical LB use case)
curl -sv localhost:8080 -H "Host: localhost"
> GET / HTTP/1.1
> Host: localhost
< HTTP/1.1 302 Found
< Location: http://localhost:5000/cmk/
I dug myself into the apache docs, and by trial and error I found a solution: Delete the ServerName config setting from /opt/omd/sites/cmk/etc/apache/listen-port.conf. Then it behaves “nicely”.
Running behind a k8s load balancer (and this is MY use case) you only get to the backend if the load balancer has already routed you by interpreting the Host: header, so currently see no disadvantages.
For anything else … I’m neither a security nor an apache / website hosting person, so probably I can’t really judge if this is a good idea. But solving my problem - it can’t be too bad .
After this upgrade I noticed that my docker instance is redirecting my https traffic back to http instead of serving the portal.
I use Traefik as reverse proxy and SSLoffloader. Serving check_mk on 443
.