Getting error: no unmonitored services found, no vanished services found [special_kubernetes] Agent exited with code 1: (404)

version: 2.0.0p23
OS kubernetes worker nodes with ubuntu OS

Error:
no unmonitored services found, no vanished services found, no new host labels, [special_kubernetes] Agent exited with code 1: (404)

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)
nigoyal7@worker-2:/omd/sites/checkmk/tmp/check_mk/piggyback$ cmk --debug -vvn worker-2

Command ‘cmk’ not found, but can be installed with:

snap install cloudmonkey

Hi,
welcome to community. When you are using cmk commands you need to do as omd user so:
su - [sitename]. if I see correctly you tried this commands as normal user.

Best regards,
JF

Hi I am able to run commands as mentioned below inside checkmk pod as mentioned below

root@checkmk-74d9f9fd79-scbqk:/# cd /omd/sites/
root@checkmk-74d9f9fd79-scbqk:/omd/sites# ls
checkmk
root@checkmk-74d9f9fd79-scbqk:/omd/sites# omd su checkmk
OMD[checkmk]:~$ omd config checkmk

However i am getting below error in checkmk GUI…after adding host and then save and go to service configuration section
Error:
no unmonitored services found, no vanished services found, no new host labels, [special_kubernetes] Agent exited with code 1: (404)

I have configured kubernetes special agent in below path
Go to Setup > Agents > VM, Cloud, Container > Kubernetes:

still getting above error, please help to solve this issue

Hi,
we spend some time on creating a debuging manual for the new k8s agent.
https://kb.checkmk.com/display/KB/Debugging+the+kubernetes+-+k8s+special+agent

Could you go through that and see if that fixes the issue? Or at least if you figure that out?

Edit: I miss understood that. You’re using Checkmk 2.0 and there we’re using the “old” k8s monitoring. Please go through that manual:
https://kb.checkmk.com/display/KB/Debugging+Special+Agents

BR
Anastasios

Hi

Below links are not working, giving 503 Service unavailable error while opening

https://kb.checkmk.com/display/KB/Debugging+the+kubernetes+-+k8s+special+agent
https://kb.checkmk.com/display/KB/Debugging+Special+Agents

I am using checkmk free edition stable version 2.0.0p23 to test and following below documentation to implement in kubernetes cluster
Deploying Checkmk in Kubernetes | Checkmk

Thanks & Regards,
Nitin Goyal

Hi, our Knowledge base is reachable now. Please try to open the provided links again.

1 Like

Hi,

I am deploying checkmk version:2.0.0p23 following steps mentioned in below link:

I have configured kubernetes agent configuration as mentioned in above link in below path: *Go to Setup > Agents > VM, Cloud, Container > Kubernetes and done other configurations as well…however getting below error, when host is discovering services:
no unmonitored services found, no vanished services found, no new host labels, [special_kubernetes] Agent exited with code 1: (404)

Steps mentioned in below links are related to version: 2.1, i am looking to solve issue in checkmk version: 2.0.0p23
https://kb.checkmk.com/display/KB/Debugging+the+kubernetes+-+k8s+special+agent
https://kb.checkmk.com/display/KB/Debugging+Special+Agents

Thanks & Regards,
Nitin Goyal

This documentation is to deploy Checkmk monitoring instance in Kubernetes. Is this waht you want ?

To debug the special agent in 2.0.x, you can follow the below:

  1. Execute the cmk -D and then look for “Type of agent >> Program”
$ cmk -D test-kube

test-kube                                                                      
Addresses:              1.2.3.4
Tags:                   [address_family:ip-v4-only], [agent:cmk-agent], [criticality:prod], [ip-v4:ip-v4], [networking:lan], [piggyback:auto-piggyback], [site:v200p24], [snmp_ds:no-snmp], [tcp:tcp]
Labels:                 
Host groups:            check_mk
Contact groups:         all
Agent mode:             Normal Checkmk agent, or special agent if configured
Type of agent:          
  Program: /omd/sites/yoursite/share/check_mk/agents/special/agent_kubernetes '--token' 'test-token' '--infos' 'nodes,services,ingresses,endpoints' '--no-cert-check' '--api-server-endpoint' 'https://1.2.3.4' '--port' '443'
  1. Now, look for the long command under “Type of agent >> Program” and run the following command on console as site user:

$ /omd/sites/yoursite/share/check_mk/agents/special/agent_kubernetes '--token' 'test-token' '--infos' 'nodes,services,ingresses,endpoints' '--no-cert-check' '--api-server-endpoint' 'https://1.2.3.4' '--port' '443' --debug --verbose

This should give you bit more information.

As Sudhir already said, you have followed the guide to run a Checkmk server in Kubernetes. Not monitoring Kubernetes.

I recommend:

Hi Martin,

Is monitoring of kubernetes cluster works with checkmk version 2.1.0 only, and not with checkmk version 2.0.0?

Can you please confirm if below is correct documentation link for deploying and configuring monitoring in kubernetes or i need to follow any other documentation as well, please guide:

Also please help to know if there is any way i can set monitoring of kubernetes cluster with checkmk version 2.0.0, please provide any documentation links for this

Thanks
Nitin Goyal

For 2.0.x, just follow Monitoring Kubernetes

Hi Chauhan,

Yes i have followed these configurations as well in checkmk version: 2.0.0, however i am getting below error after adding host and discovering service…
no unmonitored services found, no vanished services found, no new host labels, [special_kubernetes] Agent exited with code 1: (404)

Let me share the debug logs in a while…thankyou

Thanks
Nitin Goyal

Debug logs will be interesting as I stated in my first comment.

Hi,

I am getting below output in debug logs, please check…thankyou

nigoyal7@master:~$ kubectl get pods -n checkmk
NAME                       READY   STATUS    RESTARTS       AGE
checkmk-74d9f9fd79-scbqk   1/1     Running   8 (2m5s ago)   3d19h
nigoyal7@master:~$ kubectl exec -it checkmk-74d9f9fd79-scbqk -n checkmk -- bash
root@checkmk-74d9f9fd79-scbqk:/# omd su checkmk
OMD[checkmk]:~$ cmk -D test-kube

test-kube                                                                      
Addresses:              0.0.0.0
Tags:                   [address_family:ip-v4-only], [agent:cmk-agent], [criticality:prod], [networking:lan], [piggyback:auto-piggyback], [site:checkmk], [snmp_ds:no-snmp]
Labels:                 
Host groups:            check_mk
Contact groups:         
Agent mode:             No agent
Type of agent:          
  Process piggyback data from /omd/sites/checkmk/tmp/check_mk/piggyback/test-kube
  PING only
Services:
  checktype item params description groups
  --------- ---- ------ ----------- ------
OMD[checkmk]:~$

Thanks
Nitin Goyal

Agent mode: No agent

This is your problem. Change it to API integration.

Hi, i tried with API integrations option also in host settings…let me check again and check the logs in debug…thankyou


I have already selected API integration option under host settings…please check in attached screenshot

Thanks
Nitin Goyal

Please share now the debug logs + the screenshot of the Kubernetes Datasource rule


Below is debug output…

nigoyal7@master:~$ kubectl exec -it checkmk-74d9f9fd79-scbqk -n checkmk -- bash
root@checkmk-74d9f9fd79-scbqk:/# omd su checkmk
OMD[checkmk]:~$ cmk -D test-kube

test-kube                                                                      
Addresses:              0.0.0.0
Tags:                   [address_family:ip-v4-only], [agent:cmk-agent], [criticality:prod], [networking:lan], [piggyback:auto-piggyback], [site:checkmk], [snmp_ds:no-snmp]
Labels:                 
Host groups:            check_mk
Contact groups:         
Agent mode:             No agent
Type of agent:          
  Process piggyback data from /omd/sites/checkmk/tmp/check_mk/piggyback/test-kube
  PING only
Services:
  checktype item params description groups
  --------- ---- ------ ----------- ------
OMD[checkmk]:~$

Thanks
Nitin Goyal

If you speciy the IP in the host properties then you don’t need to specify the second time again in "API server endpoint:. Just use IP address here and change the port to 6443 like in my screenshot I have it set as 443.

As an initial test, disable certificate verification.