Check_mk kubernetes monitoring - 401, Reason: Unauthorized

Troubleshooting
,
1

CMK version:2.2.0p18 CEE
OS version:CentOS

Error message: [special_kube] Agent exited with code 1: 401, Reason: Unauthorized, Message: Unauthorized 401: must authenticateCRIT**, execution time 4.4 sec**

I have set check_mk monitoring according to Monitoring Kubernetes
In my case I have used ingress.
Access to the monitoring collector is working from command line, as is described into the documentation, but when I set Kubernetes agent, the result is 401, Reason: Unauthorized

I have checked twice, step by step the configuration, but without success. I don’t know what I have missed. My guess is the Kubernetes permission.
Probably I need to deploy some RBAC rules. During installation I did not see any RBAC yaml config to use. I have installed via helm chart.
Another thing is that I have added API server connection identically how it shows on “kubectl config view” command.(https://site_name:443/k8s/clusters/cluster_id)

Any help will be appreciated. Thank you!

2

Did you even put the right URL or the right token ?
Did a simple curl against the Control plane API URL or your cluster collector API endpoint even works ?

3

Thank you!
Using cli, I have made a request with token to check_mk collector it works.
curl -H “Authorization: Bearer $TOKEN” https://my-check_mk-collector.mydomain.com/metadata | jq

Not sure if is working also interrogating cluster API. I did not understand if that Token is also working on cluster API level. It should work because on Check_mk User interface, under kubernetes rule is only one token to complete for both cluster API and cluster collector(different address than API).

4

Just a curl without a token against your control plane endpoint should yield:

{
“kind”: “Status”,
“apiVersion”: “v1”,
“metadata”: {},
“status”: “Failure”,
“message”: “forbidden: User "system:anonymous" cannot get path "/"”,
“reason”: “Forbidden”,
“details”: {},
“code”: 403
}

Do you even get the above output? If yes, then the API-server endpoint is reachable.

Using cli, I have made a request with token to check_mk collector it works.
curl -H “Authorization: Bearer $TOKEN” https://my-check_mk-collector.mydomain.com/metadata | jq

Sounds good.

5

curl https://my-cluster-endpoint/k8s/clusters/my-cluster
I get:
{“type”:“error”,“status”:“401”,“message”:“Unauthorized 401: must authenticate”}

6

If I’m using a token that I usually use it when I am setting kubernetes(my user token) communication is working against ckuster kubernetes API, but not to cluster-collector.
Or on that check_mk UI kubernetes interface I only have one line where I could add only one token.
In check-mk documentation it show to add token generated during check_mk kubernetes installation.
Looks like I need 2 token to use on that special_kube agent, but I have only one line. Either add cluster API token either check_mk collector token.

Thank you!

7

You only need one as described in the documentation which will authenticate against both endpoints.

8

Then, I think token from Check_mk doesn’t have rights to make requests to cluster API.
I’m looking to this…