[Check_mk (english)] Monitoring Kubernetes

Archive checkmk-en
1

Hello,

I’m trying to monitor a Kubernetes cluster with check mk, but when I follow these instructions

https://mathias-kettner.com/cms_monitoring_kubernetes.html

I get down to the point where I create the cluster, but I keep getting the error

The node (ip address) does not exist (must be a host that is configured with WATO)

This is true, it’s not currently being monitored by check mk, but the instructions don’t seem to suggest that it already should be.

Should I add the node at that IP as a check mk host? Do I need to install the agent on it?

Any ideas or tutorials would be appreciated. If there’s not one, I’d be happy to write up a walkthough myself.

Thanks,

Joe

···

“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

2

Hi

Yes, think agent need to be installed and running. That is in any case how we configured it

Nar

···

On Sun, May 5, 2019, 18:27 Bothari bothari@gmail.com wrote:

Hello,

I’m trying to monitor a Kubernetes cluster with check mk, but when I follow these instructions

https://mathias-kettner.com/cms_monitoring_kubernetes.html

I get down to the point where I create the cluster, but I keep getting the error

The node (ip address) does not exist (must be a host that is configured with WATO)

This is true, it’s not currently being monitored by check mk, but the instructions don’t seem to suggest that it already should be.

Should I add the node at that IP as a check mk host? Do I need to install the agent on it?

Any ideas or tutorials would be appreciated. If there’s not one, I’d be happy to write up a walkthough myself.

Thanks,

Joe

–
“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

checkmk-en mailing list

checkmk-en@lists.mathias-kettner.de

Manage your subscription or unsubscribe

https://lists.mathias-kettner.de/cgi-bin/mailman/listinfo/checkmk-en

3

Hi Joe,

I don’t understand your problem. Chapter 2.4 of the article clearly says it: “the cluster is monitored as host” (not “as cluster host”).

…so you have to configure your k8s cluster as normal host. Data for the nodes is provided as piggyback data - so if you want the clusters nodes in monitoring, too, you’ve to configure them. For the nodes it’s enough to configure them with “no agent”, because they get the data as piggyback from the cluster.

HTH,

Marcel

···

Bothari bothari@gmail.com schrieb am So., 5. Mai 2019, 18:27:

Hello,

I’m trying to monitor a Kubernetes cluster with check mk, but when I follow these instructions

https://mathias-kettner.com/cms_monitoring_kubernetes.html

I get down to the point where I create the cluster, but I keep getting the error

The node (ip address) does not exist (must be a host that is configured with WATO)

This is true, it’s not currently being monitored by check mk, but the instructions don’t seem to suggest that it already should be.

Should I add the node at that IP as a check mk host? Do I need to install the agent on it?

Any ideas or tutorials would be appreciated. If there’s not one, I’d be happy to write up a walkthough myself.

Thanks,

Joe

–
“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

checkmk-en mailing list

checkmk-en@lists.mathias-kettner.de

Manage your subscription or unsubscribe

https://lists.mathias-kettner.de/cgi-bin/mailman/listinfo/checkmk-en

4

Thank you for your quick response. I see that the cluster should be monitored as a host, but I’m having trouble with the steps.

I’ve deleted the hosts and I’ve seen these combinations:
New Cluster -> Host 64.102.189.120 -> save -> Error: The cluster must have at least one node.

New Cluster -> Host 64.102.189.120 -> Node 64.102.189.120 -> save -> Error: The cluster can not be a node of its own.

New Cluster -> Host kube1 -> Node 64.102.189.120 -> save -> Error: The node 64.102.189.120 does not exist (must be a host that is configured with WATO).

Then, I add a node for 64.102.189.120. This has 2 problems: It can’t be pinged because of OpenStack, and it doesn’t have check mk installed.

However, I think that’s ok. As the instructions state in 2.5,

…So that the nodes are also monitored, you must also create them as hosts in WATO…

…Unless you have a Check_MK agent installed on the nodes themselves (which would generally be rather unusual), you will need to set the Check_MK Agent to No agent.

Once I do that and set it to No agent, I can see the 64.102.189.120 host in check mk. It has one service (PING) which fails.

This allows me to successfully do

New Cluster -> Host kube1 -> Node 64.102.189.120 -> save

Now we have a cluster, but no services.

At this point, I follow the instructions for creating the Datasource Programs -> Kubernetes. This time around I explicitly add the kube1 server, so this rule knows where to apply. I left the hosts blank before, which was probably my error.

My cluster now has 2 services, Ping which is Critical, and check mk discovery which is green with the message OK - no unmonitored services found, no vanished services found

I’ll admit, I’m still learning Check MK, so if there’s something obvious I’m not doing let me know. I can connect via kubectl and get nodes and pods, so I’m pretty sure it’s up.

Any ideas?

Joe

···

“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

5

Hi Joe,

You didn’t read exactly: create the cluster host as a normal host, not(!) as cluster host.

For this host create the data source rule for k8s.

The nodes must also be configured as normal hosts, with the exact name like they’re reported/provided by the k8s cluster. You can set the node’s IP address family attribute to “no IP”.

HTH,

Marcel

···

Bothari bothari@gmail.com schrieb am So., 5. Mai 2019, 20:30:

Thank you for your quick response. I see that the cluster should be monitored as a host, but I’m having trouble with the steps.

I’ve deleted the hosts and I’ve seen these combinations:
New Cluster → Host 64.102.189.120 → save → Error: The cluster must have at least one node.

New Cluster → Host 64.102.189.120 → Node 64.102.189.120 → save → Error: The cluster can not be a node of its own.

New Cluster → Host kube1 → Node 64.102.189.120 → save → Error: The node 64.102.189.120 does not exist (must be a host that is configured with WATO).

Then, I add a node for 64.102.189.120. This has 2 problems: It can’t be pinged because of OpenStack, and it doesn’t have check mk installed.

However, I think that’s ok. As the instructions state in 2.5,

…So that the nodes are also monitored, you must also create them as hosts in WATO…

…Unless you have a Check_MK agent installed on the nodes themselves (which would generally be rather unusual), you will need to set the Check_MK Agent to No agent.

Once I do that and set it to No agent, I can see the 64.102.189.120 host in check mk. It has one service (PING) which fails.

This allows me to successfully do

New Cluster → Host kube1 → Node 64.102.189.120 → save

Now we have a cluster, but no services.

At this point, I follow the instructions for creating the Datasource Programs → Kubernetes. This time around I explicitly add the kube1 server, so this rule knows where to apply. I left the hosts blank before, which was probably my error.

My cluster now has 2 services, Ping which is Critical, and check mk discovery which is green with the message OK - no unmonitored services found, no vanished services found

I’ll admit, I’m still learning Check MK, so if there’s something obvious I’m not doing let me know. I can connect via kubectl and get nodes and pods, so I’m pretty sure it’s up.

Any ideas?

Joe

On Sun, May 5, 2019 at 1:14 PM Marcel Schulte schulte.marcel@gmail.com wrote:

Hi Joe,

I don’t understand your problem. Chapter 2.4 of the article clearly says it: “the cluster is monitored as host” (not “as cluster host”).

…so you have to configure your k8s cluster as normal host. Data for the nodes is provided as piggyback data - so if you want the clusters nodes in monitoring, too, you’ve to configure them. For the nodes it’s enough to configure them with “no agent”, because they get the data as piggyback from the cluster.

HTH,

Marcel

Bothari bothari@gmail.com schrieb am So., 5. Mai 2019, 18:27:

Hello,

I’m trying to monitor a Kubernetes cluster with check mk, but when I follow these instructions

https://mathias-kettner.com/cms_monitoring_kubernetes.html

I get down to the point where I create the cluster, but I keep getting the error

The node (ip address) does not exist (must be a host that is configured with WATO)

This is true, it’s not currently being monitored by check mk, but the instructions don’t seem to suggest that it already should be.

Should I add the node at that IP as a check mk host? Do I need to install the agent on it?

Any ideas or tutorials would be appreciated. If there’s not one, I’d be happy to write up a walkthough myself.

Thanks,

Joe

–
“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

checkmk-en mailing list

checkmk-en@lists.mathias-kettner.de

Manage your subscription or unsubscribe

https://lists.mathias-kettner.de/cgi-bin/mailman/listinfo/checkmk-en

–
“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

6

Marcel,

That was very helpful! I’ve gotten further, but now I’m getting

Reason: SSLError
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
To troubleshoot, I’ve

  • Disabled the verify certs in the Datasource rule

  • Looked at the cert. The common name is kube-ca

  • Edited /etc/hosts so that kube-ca points to the right IP address.

  • Restarted the omd site.

The log part of the response look like this

/omd/sites/master/lib/python/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)

(404)

Reason: Not Found

HTTP response headers: HTTPHeaderDict({‘Content-Length’: ‘21’, ‘Strict-Transport-Security’: ‘max-age=15724800; includeSubDomains’, ‘Server’: ‘nginx/1.15.6’, ‘Connection’: ‘keep-alive’, ‘Date’: ‘Sun, 05 May 2019 19:34:19 GMT’, ‘Content-Type’: ‘text/plain; charset=utf-8’})

HTTP response body: default backend - 404

What do you think?

Joe

···

“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

7

Hi Joe,

I think nothing - never had my own k8s nor have seen any. Just brought you to the right host configuration… :wink:

Regards,

Marcel

···

Bothari bothari@gmail.com schrieb am So., 5. Mai 2019, 21:36:

Marcel,

That was very helpful! I’ve gotten further, but now I’m getting

Reason: SSLError
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
To troubleshoot, I’ve

  • Disabled the verify certs in the Datasource rule
  • Looked at the cert. The common name is kube-ca
  • Edited /etc/hosts so that kube-ca points to the right IP address.
  • Restarted the omd site.

The log part of the response look like this

/omd/sites/master/lib/python/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)

(404)

Reason: Not Found

HTTP response headers: HTTPHeaderDict({‘Content-Length’: ‘21’, ‘Strict-Transport-Security’: ‘max-age=15724800; includeSubDomains’, ‘Server’: ‘nginx/1.15.6’, ‘Connection’: ‘keep-alive’, ‘Date’: ‘Sun, 05 May 2019 19:34:19 GMT’, ‘Content-Type’: ‘text/plain; charset=utf-8’})

HTTP response body: default backend - 404

What do you think?

Joe

On Sun, May 5, 2019 at 2:55 PM Marcel Schulte schulte.marcel@gmail.com wrote:

Hi Joe,

You didn’t read exactly: create the cluster host as a normal host, not(!) as cluster host.

For this host create the data source rule for k8s.

The nodes must also be configured as normal hosts, with the exact name like they’re reported/provided by the k8s cluster. You can set the node’s IP address family attribute to “no IP”.

HTH,

Marcel

Bothari bothari@gmail.com schrieb am So., 5. Mai 2019, 20:30:

Thank you for your quick response. I see that the cluster should be monitored as a host, but I’m having trouble with the steps.

I’ve deleted the hosts and I’ve seen these combinations:
New Cluster → Host 64.102.189.120 → save → Error: The cluster must have at least one node.

New Cluster → Host 64.102.189.120 → Node 64.102.189.120 → save → Error: The cluster can not be a node of its own.

New Cluster → Host kube1 → Node 64.102.189.120 → save → Error: The node 64.102.189.120 does not exist (must be a host that is configured with WATO).

Then, I add a node for 64.102.189.120. This has 2 problems: It can’t be pinged because of OpenStack, and it doesn’t have check mk installed.

However, I think that’s ok. As the instructions state in 2.5,

…So that the nodes are also monitored, you must also create them as hosts in WATO…

…Unless you have a Check_MK agent installed on the nodes themselves (which would generally be rather unusual), you will need to set the Check_MK Agent to No agent.

Once I do that and set it to No agent, I can see the 64.102.189.120 host in check mk. It has one service (PING) which fails.

This allows me to successfully do

New Cluster → Host kube1 → Node 64.102.189.120 → save

Now we have a cluster, but no services.

At this point, I follow the instructions for creating the Datasource Programs → Kubernetes. This time around I explicitly add the kube1 server, so this rule knows where to apply. I left the hosts blank before, which was probably my error.

My cluster now has 2 services, Ping which is Critical, and check mk discovery which is green with the message OK - no unmonitored services found, no vanished services found

I’ll admit, I’m still learning Check MK, so if there’s something obvious I’m not doing let me know. I can connect via kubectl and get nodes and pods, so I’m pretty sure it’s up.

Any ideas?

Joe

On Sun, May 5, 2019 at 1:14 PM Marcel Schulte schulte.marcel@gmail.com wrote:

Hi Joe,

I don’t understand your problem. Chapter 2.4 of the article clearly says it: “the cluster is monitored as host” (not “as cluster host”).

…so you have to configure your k8s cluster as normal host. Data for the nodes is provided as piggyback data - so if you want the clusters nodes in monitoring, too, you’ve to configure them. For the nodes it’s enough to configure them with “no agent”, because they get the data as piggyback from the cluster.

HTH,

Marcel

Bothari bothari@gmail.com schrieb am So., 5. Mai 2019, 18:27:

Hello,

I’m trying to monitor a Kubernetes cluster with check mk, but when I follow these instructions

https://mathias-kettner.com/cms_monitoring_kubernetes.html

I get down to the point where I create the cluster, but I keep getting the error

The node (ip address) does not exist (must be a host that is configured with WATO)

This is true, it’s not currently being monitored by check mk, but the instructions don’t seem to suggest that it already should be.

Should I add the node at that IP as a check mk host? Do I need to install the agent on it?

Any ideas or tutorials would be appreciated. If there’s not one, I’d be happy to write up a walkthough myself.

Thanks,

Joe

–
“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

checkmk-en mailing list

checkmk-en@lists.mathias-kettner.de

Manage your subscription or unsubscribe

https://lists.mathias-kettner.de/cgi-bin/mailman/listinfo/checkmk-en

–
“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

–
“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

8

Ok, thanks.

···

“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

9

Joe - it seems you are still trying to add the Kubernetes host as a cluster host. That is incorrect. Add it as a ‘normal’ host. (New Host vs New cluster)

···

On Wed, May 8, 2019 at 7:46 AM Bothari bothari@gmail.com wrote:

Thank you for your quick response. I see that the cluster should be monitored as a host, but I’m having trouble with the steps.

I’ve deleted the hosts and I’ve seen these combinations:
New Cluster → Host 64.102.189.120 → save → Error: The cluster must have at least one node.

New Cluster → Host 64.102.189.120 → Node 64.102.189.120 → save → Error: The cluster can not be a node of its own.

New Cluster → Host kube1 → Node 64.102.189.120 → save → Error: The node 64.102.189.120 does not exist (must be a host that is configured with WATO).

Then, I add a node for 64.102.189.120. This has 2 problems: It can’t be pinged because of OpenStack, and it doesn’t have check mk installed.

However, I think that’s ok. As the instructions state in 2.5,

…So that the nodes are also monitored, you must also create them as hosts in WATO…

…Unless you have a Check_MK agent installed on the nodes themselves (which would generally be rather unusual), you will need to set the Check_MK Agent to No agent.

Once I do that and set it to No agent, I can see the 64.102.189.120 host in check mk. It has one service (PING) which fails.

This allows me to successfully do

New Cluster → Host kube1 → Node 64.102.189.120 → save

Now we have a cluster, but no services.

At this point, I follow the instructions for creating the Datasource Programs → Kubernetes. This time around I explicitly add the kube1 server, so this rule knows where to apply. I left the hosts blank before, which was probably my error.

My cluster now has 2 services, Ping which is Critical, and check mk discovery which is green with the message OK - no unmonitored services found, no vanished services found

I’ll admit, I’m still learning Check MK, so if there’s something obvious I’m not doing let me know. I can connect via kubectl and get nodes and pods, so I’m pretty sure it’s up.

Any ideas?

Joe

On Sun, May 5, 2019 at 1:14 PM Marcel Schulte schulte.marcel@gmail.com wrote:

Hi Joe,

I don’t understand your problem. Chapter 2.4 of the article clearly says it: “the cluster is monitored as host” (not “as cluster host”).

…so you have to configure your k8s cluster as normal host. Data for the nodes is provided as piggyback data - so if you want the clusters nodes in monitoring, too, you’ve to configure them. For the nodes it’s enough to configure them with “no agent”, because they get the data as piggyback from the cluster.

HTH,

Marcel

Bothari bothari@gmail.com schrieb am So., 5. Mai 2019, 18:27:

Hello,

I’m trying to monitor a Kubernetes cluster with check mk, but when I follow these instructions

https://mathias-kettner.com/cms_monitoring_kubernetes.html

I get down to the point where I create the cluster, but I keep getting the error

The node (ip address) does not exist (must be a host that is configured with WATO)

This is true, it’s not currently being monitored by check mk, but the instructions don’t seem to suggest that it already should be.

Should I add the node at that IP as a check mk host? Do I need to install the agent on it?

Any ideas or tutorials would be appreciated. If there’s not one, I’d be happy to write up a walkthough myself.

Thanks,

Joe

–
“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

checkmk-en mailing list

checkmk-en@lists.mathias-kettner.de

Manage your subscription or unsubscribe

https://lists.mathias-kettner.de/cgi-bin/mailman/listinfo/checkmk-en

–
“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

checkmk-en mailing list

checkmk-en@lists.mathias-kettner.de

Manage your subscription or unsubscribe

https://lists.mathias-kettner.de/cgi-bin/mailman/listinfo/checkmk-en

10

Solved!

  1. In the instructions, it shows the top of the rbac file. I didn’t realize that file was just the beginning, so I hadn’t set up all the security.

  2. In my environment, I needed to bypass certificates and set the port. This may not be common, but a short explanation might help.

On my team all 3 of use first tried to add a cluster host, AND mixed up the rbac file. Might want to either take out the example, or put the whole file on that page.

I there a way to contribute to the documentation?

Thanks,

Joe

···

“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte

11

Also, debugging was GREATLY simplified by knowing how to run the special agent. That would be good to add to the docs, too.

Thanks for all your help!

Joe

···

“There are only two industries that refer to their customers as ‘users’.” - Edward Tufte