Monitoring kubernetes, Rancher and cluster-collector error

Troubleshooting
,
1

CMK version: Checkmk Free Edition 2.1.0p28
I installed cluster collector from helm.
Part about NodePort in values is:

service:
    # if required specify "NodePort" here to expose the cluster-collector via the "nodePort" specified below
    type: NodePort
    port: 8080
    nodePort: 30035
    annotations: {}

Kubernetes is managed by Rancher, version Provider: RKE2
Kubernetes Version: v1.22.11.
I have configured:

scr1
scr11313×508 31.6 KB

Configuration for Kubernetes rule:
scr2
scr2851×939 32.4 KB

Address for NodePort is from this command:

echo http://$NODE_IP:$NODE_PORT
http://10.210.95.164:30035

API Address for server is server name from command (server name). It is the same name as in KubeConfig downloaded from GUI in Rancher.

kubectl config view

Access token was generated on Rancher side with all scope (it is not the same as token generated during installing collector with using helm). With token generated during creation by helm - connection to cluster doesn’t work.
When I check services, I have:

scr3
scr3939×338 11.2 KB

NodePort ip is different than ip from command echo http://$NODE_IP:$NODE_PORT

In browser, with address 10.210.95.164:30035/metadata I have info:
{"detail":"Not authenticated"}
but when I used this command:

scr4
scr4886×244 12.3 KB

I need to user password for collector service but where is place to add it? I cloned Kubernetes rule but it not works too.

All advice welcome.

2

This is expected when you don’t use the token.

With the token it looks like this:

export TOKEN=$(kubectl get secret $(kubectl get serviceaccount checkmk-checkmk -o=jsonpath='{.secrets[*].name}' -n checkmk-monitoring) -n checkmk-monitoring -o=jsonpath='{.data.token}' | base64 --decode);

Please make sure to update your namespace or just use the command from the output of the helm chart.

and then:

curl -H "Authorization: Bearer $TOKEN" http://IP:30035/metadata |jq

3

I checked in this way:

export TOKEN=$(kubectl get secret $(kubectl get serviceaccount checkmk-checkmk -o=jsonpath='{.secrets[*].name}' -n checkmk-test) -n checkmk-test -o=jsonpath='{
.data.token}' | base64 --decode);

curl -H "Authorization: Bearer $TOKEN" http://10.210.95.164:30035/metadata |jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2737  100  2737    0     0  53666      0 --:--:-- --:--:-- --:--:-- 54740
{
  "cluster_collector_metadata": {
    "node": "xxxxx",
    "host_name": "checkmk-cluster-collector-65b485879-xjrr5",
    "container_platform": {
      "os_name": "alpine",
      "os_version": "3.15.0",

I added token from Rancher, not from checkmk collector service.
When I added from checkmk collector service, I don’t have connection to cluster managed by Rancher.

4

When you use this token in the Kubernetes agent rule, then it is possible to fetch the data via the K8 API + cluster collector endpoint ?

5

I can fetch data from K8 API but not from CheckMK Collector - like in screnshoot #1

6

Please see this comment, which explains how you can monitor Rancher: Kubernetes / Rancher 1.21 error - #5 by martin.hirschvogel

For Checkmk 2.3, we plan to officially support monitoring downstream RKE clusters via the Rancher API.

7

I checked this comment before.
I used command:
kubectl get nodes -o wide
and I have:

scr5
scr51301×101 8.78 KB

I added name or internal ip address from node and password from service generated by checkmk helm. I tried with http/https and port 443 on the end but without any result.
Maybe I did something wrong?

8

The internal IP can only be reached internally, and not via an external service.

a) API server connection you get via your kubeconfig (use the control plane nodes tough)
b) Most reliable for any internal service (here: cluster collector) to be reached by the outside is an ingress in Kubernetes. Create one and use that one.

9

I created ingress for checkmk-collector. It is reached by dns name in browser and in browser I have {“detail”:“Not authenticated”}. What I need to add in field API server connection? I added API server from kubeconfig, internal ip of node, ingress endpoint but in each case I have cluster collector Status: Connection Error (Failed attempting to communicate with cluster collector at URL. I added password from token from checkmk service.

10
  1. Enter a control plane node URL in the API server field
  2. Enter the URL of the cluster connector ingress in the checkmk-cluster connection.
11

I configured in this way:

scr8
scr8875×719 23 KB

After this I have cluster collector green but no data from cluster about metrics
scr7
scr71202×380 22.7 KB

nad main dashboard looks like below
scr6
scr61780×949 81.3 KB

Maybe there is something wrong with node address. When I added port 6443, cluster collector service changed to green but still no metrics available.

12

After this I have cluster collector green but no data
Can you try to modify the node-collector-container-metrics daemon-set with the configuration as decribed in the below link?
Kubernetes Cluster collector doesn't show CPU, memory usage or container metrics - #11 by MartinM