Checkmk Prometheus agent: cAdvisor fails with code 1, Node Exporter only returns status info

Troubleshooting
, ,
1

CMK version:
2.3.0p30

OS version:
AlmaLinux 9.4

Error message:

Note that I have switched to trying the Kubernetes setup, and I have an open forum post about that issue here.

When using the Prometheus integration in Checkmk, selecting cAdvisor as the scrape target causes the agent to fail with:

[special_prometheus]: Agent exited with code 1: 'prometheus-kube-state-metrics-84cdc5bd67-bswq7'

When I switch the target to Node Exporter, the integration is “successful” — but it only reports the following:

OK  Prometheus Build  Version: 3.2.1, Config reload: Success, Storage retention: 15d, Scrape Targets in up state: 22 out of 22

This doesn’t yield any host- or pod-level metrics — just the Prometheus server health, which isn’t useful for actual monitoring.

Output of cmk --debug -vvn prometheus-kube-state-metrics-84cdc5bd67-bswq7:

Checkmk version 2.3.0p30
Failed to lookup IPv4 address of prometheus-kube-state-metrics-84cdc5bd67-bswq7 via DNS: [Errno -3] Temporary failure in name resolution

Extra context:
Prometheus is deployed via Helm with NodePort access on 31090. Kube State Metrics is also deployed via Helm and is scraping successfully — I can confirm the endpoint http://<ksm pod IP>:8080/metrics returns Prometheus metrics.

Prometheus UI confirms KSM is up:

http://10.244.0.49:8080/metrics
job="kubernetes-service-endpoints"
service="kube-state-metrics"
state=up

The Checkmk Prometheus integration UI is configured like this:

  • Server address: za-cpt-lmkb01.eqh.local:9090
  • Protocol: HTTP
  • Prometheus Scrape Targets: cAdvisor (when error occurs)
  • Entity level: Both (pods and containers)
  • Host name for containers: Name
  • Namespace filter: .*
  • Metrics retrieved: CPU, Memory, Disk IO, Filesystem, Network

I’ve also tried using relabeling in extraScrapeConfigs to give kube-state-metrics an instance label that matches a known host — but it didn’t change the outcome.

Request:
Any ideas why cAdvisor fails and Node Exporter provides no useful data? Are there additional steps required to make this work in a Minikube environment? Or is this a known issue with 2.3.0p30?

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.