Datasource agent for prometheus node exporter

Checkmk Exchange
1

Hello community,

I am currently working on a monitoring for our new EDI System, BISv6.7 for Seeburger.
I was told by the Seeburger consultant that they implemented an exporter for Prometheus and this would be their preferred way of monitoring.
I don’t want to install a Prometheus instance just for this.
As the node exporter is reachable via HTTP/HTTPS protocol and the output seems to be in a structure that could be parsed, I am wondering if not already someone tried to develop a Datasource Agent to acquire the data from a node exporter.
At least at the know places I found nothing useful.

Thanks for your hints/thoughts on this.

regards

Michael

2

Am i wrong, but the documentation explicit talks about a connection to a node exporter?
As from your description and the documentation it seems exactly like what are you looking for.
(linked the german version :slight_smile:)

3

Thanks tosch, as far as I understood this, it connects to the Prometheus server and not to the node itself. The config option just define the initial data source.
Nevertheless I will have a look at the code if I find something useful.

4

Maybe @martin.hirschvogel could give you some hints here. He seems to be the expert on this topic.

1 Like
5

@mike1098 mentioned it already correctly. There is no native connection to a node exporter.
However, if you look into the new Kubernetes special agent, you will see methods how to parse Prometheus metrics. Because that’s what is happening there massively.

2 Likes
6

Isn’t the OpenMetrics spec used by Prometheus fundamentally different to things checkmk needs?

AFAIK you only get metrics but no semantics. You would need to implement a check logic but the metrics presented may be too dynamic.

7

May you can explain this for a poor schoolboy like me :wink:
If I do a curl to the node exporter I get mainly key/value pairs

# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.562352e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.562352e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.445247e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 570

For some we see also items in the key:

# TYPE node_cpu_scaling_frequency_max_hertz gauge
node_cpu_scaling_frequency_max_hertz{cpu="0"} 2.8e+09
node_cpu_scaling_frequency_max_hertz{cpu="1"} 2.8e+09
node_cpu_scaling_frequency_max_hertz{cpu="2"} 2.8e+09
node_cpu_scaling_frequency_max_hertz{cpu="3"} 2.8e+09
# HELP node_cpu_scaling_frequency_min_hertz Minimum scaled CPU thread frequency in hertz.
# TYPE node_cpu_scaling_frequency_min_hertz gauge
node_cpu_scaling_frequency_min_hertz{cpu="0"} 1.2e+09
node_cpu_scaling_frequency_min_hertz{cpu="1"} 1.2e+09
node_cpu_scaling_frequency_min_hertz{cpu="2"} 1.2e+09
node_cpu_scaling_frequency_min_hertz{cpu="3"} 1.2e+09

Looks to me as pretty good parsable but maybe I overlocked something

8

Correct, these are just metrics. There are no semantics like thresholds for WARN or CRIT or any other correlations between the single metric values.

You would need to write a check plugin that implements the specific check logic for the collection of metrics you get from this specific exporter.

I do not think that there can be a generic solution to just connect to any exporter and get anything useful out of it.

In Prometheus you also have to form queries to get alarms, am I right?

9

@r.sander Yes, examples for alerts are Awesome Prometheus alerts | Collection of alerting rules

By the way, I think the metrics you get from the Prometheus exporters are pretty much identical how Prometheus stores them. So actually the Prometheus special agent is the better source.
Example: checkmk/agent_prometheus.py at master · tribe29/checkmk · GitHub

10

Did you create a custom datasource program for the prometheus output?

I have a similar case where we have a Sonatype Nexus application which has an API for metics. They serve a JSON format as well as a prometheus format, both are not supported right now bij CheckMK.
Reference:

The prometheus format seems easier since CheckMK would already know how to handle this data format by the node export functionality, except that it needs some customization to handle the data directly.

11

Hi, This project was recently restarted, so stay tuned.

12

I have encountered even more applicable applications that would benefit from being able to handle prometheus exporter data:

  • Sonatype Nexus
  • Jenkins with Prometheus plugin
  • SonarQube
  • GitLab
  • RabbitMQ with rabbitmq_prometheus plugin
  • KeyCloak

This is useful when there is no possibility to install a CheckMK agent, but another benefit of using the prometheus data is that there are also many application specific checks in the prometheus data, for instance a license expiry check in SonarQube.

13

There is already a feature request for Prometheus integration, but that may take some time, so I will definitely stay stuned on any update from your side.

Also make sure you upvote the feature request:

1 Like