CMK version:
2.4.0p3 (Enterprise Edition inside Docker)
OS version:
CheckMk Host:
Debian 12
Kubernetes Cluster (k3s):
Debian 12 & Ubuntu 22.04 w/ k3s v1.31.9+k3s1
Error message:
[special_kube]: Agent exited with code 1: Invalid control character at: line 1 column 3 (char 2)
Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)
value store: loading from disk
Checkmk version 2.4.0p3
+ FETCHING DATA
Source: SourceInfo(hostname='mycluster-3', ipaddress=None, ident='special_kube', fetcher_type=<FetcherType.SPECIAL_AGENT: 6>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f6a600e1b50]
Read from cache: AgentFileCache(path_template=/omd/sites/cmk/tmp/check_mk/data_source_cache/special_kube/mycluster-3, max_age=MaxAge(checking=0, discovery=90.0, inventory=90.0), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (does not exist)
Calling: /omd/sites/cmk/share/check_mk/agents/special/agent_kube --pwstore=6@0@/omd/sites/cmk/var/check_mk/passwords_merged@uuid9d19447d-b9bf-4a6c-851a-0006e6ae2a9a --cluster mycluster-3 --kubernetes-cluster-hostname mycluster-3 --token '*****' --monitored-objects deployments daemonsets statefulsets namespaces nodes pods pvcs cronjobs --cluster-aggregation-exclude-node-roles control-plane infra --include-annotations-as-host-labels --api-server-endpoint https://192.168.1.200:6443 --api-server-proxy FROM_ENVIRONMENT --cluster-collector-endpoint https://mycluster-checkmk-collector.k8s.homelab.local --usage-proxy FROM_ENVIRONMENT --usage-verify-cert
Get data from program
[cpu_tracking] Stop [7f6a600e1b50 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=10.46, children_system=0.58, elapsed=14.820000000298023))]
Source: SourceInfo(hostname='mycluster-3', ipaddress=None, ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f6a600af800]
Read from cache: NoCache(path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
0 piggyback files for 'mycluster-3'.
Get piggybacked data
[cpu_tracking] Stop [7f6a600af800 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[cpu_tracking] Start [7f6a60ab5e20]
+ PARSE FETCHER RESULTS
HostKey(hostname='mycluster-3', source_type=<SourceType.HOST: 1>) -> Add sections: []
Received no piggyback data
0 piggyback files for 'mycluster-3'.
[cpu_tracking] Stop [7f6a60ab5e20 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[special_kube] Agent exited with code 1: Invalid control character at: line 1 column 3 (char 2)(!!), [piggyback] Success (but no data found for this host), execution time 14.8 sec | execution_time=14.820 user_time=0.000 system_time=0.000 children_user_time=10.460 children_system_time=0.580 cmk_time_ds=3.780 cmk_time_agent=0.000
What I already tried:
I already ensured that I can reach both the Kuberenetes API on port 6443 and the collector on the cluster from inside the CheckMk container.
I have also disabled certificate validation to make sure this is not the problem.