Kubernetes 401 after one day

brightdroid · September 29, 2022, 10:56am

Initial the Connection was working, but now I get a 401, so I think it should be the token, but I created a new one which also did not work.

CMK version: 2.1.0p13
OS version: K3s v1.24.4+k3s1 on Ubuntu 22.04

Error message:

[special_kube] Agent exited with code 1: 401, Reason: Unauthorized, Message: Unauthorized**CRIT**, Got no information from host**CRIT**, execution time 1.1 sec

Output of “cmk --debug -vvn hostname”:

OMD[cmk]:~$ cmk --debug -vvn kube-internal
Checkmk version 2.1.0p13
Try license usage history update.
Trying to acquire lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
Got lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Got lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Released lock on /omd/sites/cmk/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
Released lock on /omd/sites/cmk/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.PROGRAM
[cpu_tracking] Start [7f7cefc669d0]
[ProgramFetcher] Fetch with cache settings: DefaultAgentFileCache(kube-internal, base_path=/omd/sites/cmk/tmp/check_mk/data_source_cache/special_kube, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 58551 sec, allowed is 0 sec)
[ProgramFetcher] Execute data source
Calling: /omd/sites/cmk/share/check_mk/agents/special/agent_kube --pwstore=4@0@k3s_token '--cluster' 'local' '--token' '******************************************************************' '--monitored-objects' 'deployments' 'daemonsets' 'statefulsets' 'nodes' '--namespace-include-patterns' 'default' '--namespace-include-patterns' 'pihole' '--namespace-include-patterns' 'harbor' '--namespace-include-patterns' 'authelia' '--namespace-include-patterns' 'traefik' '--cluster-aggregation-include-all-nodes' '--api-server-endpoint' 'https://kubernetes.default.svc.cluster.local:443' '--api-server-proxy' 'FROM_ENVIRONMENT' '--cluster-collector-endpoint' 'http://checkmk-cluster-collector.checkmk-monitoring.svc.cluster.local:8080' '--cluster-collector-proxy' 'FROM_ENVIRONMENT'
[cpu_tracking] Stop [7f7cefc669d0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.83, children_system=0.22, elapsed=1.1099999994039536))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f7cefc66670]
[PiggybackFetcher] Fetch with cache settings: NoCache(kube-internal, base_path=/omd/sites/cmk/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'kube-internal'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7f7cefc66670 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.PROGRAM
  -> Not adding sections: Agent exited with code 1: 401, Reason: Unauthorized, Message: Unauthorized
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
  -> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7f7cefc667f0]
value store: synchronizing
Trying to acquire lock on /omd/sites/cmk/tmp/check_mk/counters/kube-internal
Got lock on /omd/sites/cmk/tmp/check_mk/counters/kube-internal
value store: loading from disk
Releasing lock on /omd/sites/cmk/tmp/check_mk/counters/kube-internal
Released lock on /omd/sites/cmk/tmp/check_mk/counters/kube-internal
CPU resources        PEND - Check plugin received no monitoring data
Cluster collector    PEND - Check plugin received no monitoring data
Info                 PEND - Check plugin received no monitoring data
Kubernetes API       PEND - Check plugin received no monitoring data
Memory resources     PEND - Check plugin received no monitoring data
Nodes                PEND - Check plugin received no monitoring data
Pod resources        PEND - Check plugin received no monitoring data
+ EXECUTING INVENTORY PLUGINS
 aruba_wlc_aps: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 check_mk: skipped (no data)
 check_mk: skipped (no data)
 checkmk_agent_plugins: skipped (no data)
 checkmk_agent_plugins: skipped (no data)
 docker_node_info: skipped (no data)
 docker_node_info: skipped (no data)
 fritz: skipped (no data)
 fritz: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_mem: skipped (no data)
 hp_proliant_mem: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ipmi_firmware: skipped (no data)
 ipmi_firmware: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_if: skipped (no data)
 inv_if: skipped (no data)
 aix_packages: skipped (no data)
 aix_packages: skipped (no data)
 aix_service_packs: skipped (no data)
 aix_service_packs: skipped (no data)
 inventory_checkmk: skipped (no data)
 inventory_checkmk: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 checkpoint_vpn_tunnels: skipped (no data)
 dmidecode: skipped (no data)
 dmidecode: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_network: skipped (no data)
 docker_container_network: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_network: skipped (no data)
 docker_node_network: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortiauthenticator_system: skipped (no data)
 fortigate_ha: skipped (no data)
 fortigate_ha: skipped (no data)
 fortimail_system: skipped (no data)
 fortimail_system: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_software: skipped (no data)
 fortisandbox_system: skipped (no data)
 fortisandbox_system: skipped (no data)
 k8s_assigned_pods: skipped (no data)
 k8s_assigned_pods: skipped (no data)
 k8s_daemon_pod_containers: skipped (no data)
 k8s_daemon_pod_containers: skipped (no data)
 k8s_endpoint_info: skipped (no data)
 k8s_endpoint_info: skipped (no data)
 k8s_ingress_infos: skipped (no data)
 k8s_ingress_infos: skipped (no data)
 k8s_job_container: skipped (no data)
 k8s_job_container: skipped (no data)
 kube_cluster: skipped (no data)
 kube_cluster: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_daemonset: skipped (no data)
 kube_deployment: skipped (no data)
 kube_deployment: skipped (no data)
 kube_namespace: skipped (no data)
 kube_namespace: skipped (no data)
 kube_node: skipped (no data)
 kube_node: skipped (no data)
 kube_pod: skipped (no data)
 kube_pod: skipped (no data)
 kube_statefulset: skipped (no data)
 kube_statefulset: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_video: skipped (no data)
 lnx_video: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mobileiron_inventory: skipped (no data)
 mssql_clusters: skipped (no data)
 mssql_clusters: skipped (no data)
 oracle_systemparameter: skipped (no data)
 oracle_systemparameter: skipped (no data)
 snmp_extended_info: skipped (no data)
 snmp_extended_info: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_psrinfo: skipped (no data)
 solaris_psrinfo: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_cpus: skipped (no data)
 solaris_prtpicl: skipped (no data)
 solaris_prtpicl: skipped (no data)
 solaris_routes: skipped (no data)
 solaris_routes: skipped (no data)
 statgrab_net: skipped (no data)
 statgrab_net: skipped (no data)
 win_disks: skipped (no data)
 win_disks: skipped (no data)
 win_exefiles: skipped (no data)
 win_exefiles: skipped (no data)
 win_ip_r: skipped (no data)
 win_ip_r: skipped (no data)
 win_networkadapter: skipped (no data)
 win_networkadapter: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_video: skipped (no data)
 win_video: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_updates: skipped (no data)
 win_wmi_updates: skipped (no data)
 juniper_info: skipped (no data)
 juniper_info: skipped (no data)
 k8s_nodes: skipped (no data)
 k8s_nodes: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_roles: skipped (no data)
 k8s_roles: skipped (no data)
 lnx_if: skipped (no data)
 lnx_if: skipped (no data)
 mem: skipped (no data)
 mem: skipped (no data)
 mem_used: skipped (no data)
 mem_used: skipped (no data)
 mssql_instance: skipped (no data)
 mssql_instance: skipped (no data)
 netapp_api_disk: skipped (no data)
 netapp_api_disk: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_tablespaces: skipped (no data)
 oracle_tablespaces: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_psmu: skipped (no data)
 perle_psmu: skipped (no data)
 snmp_info: skipped (no data)
 snmp_info: skipped (no data)
 winperf_if: skipped (no data)
 winperf_if: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_state: skipped (no data)
 citrix_state: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 dell_hw_info: skipped (no data)
 dell_hw_info: skipped (no data)
 docker_container_node_name: skipped (no data)
 docker_container_node_name: skipped (no data)
 esx_systeminfo: skipped (no data)
 esx_systeminfo: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fireeye_sys_status: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 k8s_pod_info: skipped (no data)
 k8s_pod_info: skipped (no data)
 k8s_selector: skipped (no data)
 k8s_selector: skipped (no data)
 k8s_service_info: skipped (no data)
 k8s_service_info: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_uname: skipped (no data)
 lnx_uname: skipped (no data)
 lparstat_aix: skipped (no data)
 lparstat_aix: skipped (no data)
 netapp_api_info: skipped (no data)
 netapp_api_info: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis: skipped (no data)
 prtconf: skipped (no data)
 prtconf: skipped (no data)
 snmp_os: skipped (no data)
 snmp_os: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_uname: skipped (no data)
 solaris_uname: skipped (no data)
 suseconnect: skipped (no data)
 suseconnect: skipped (no data)
 win_bios: skipped (no data)
 win_bios: skipped (no data)
 win_computersystem: skipped (no data)
 win_computersystem: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_os: skipped (no data)
 win_os: skipped (no data)
 win_system: skipped (no data)
 win_system: skipped (no data)

No piggyback files for 'kube-internal'. Skip processing.
[cpu_tracking] Stop [7f7cefc667f0 - Snapshot(process=posix.times_result(user=0.030000000000000027, system=0.020000000000000018, children_user=0.0, children_system=0.0, elapsed=0.07000000029802322))]
[special_kube] Agent exited with code 1: 401, Reason: Unauthorized, Message: Unauthorized(!!), Got no information from host(!!), execution time 1.2 sec | execution_time=1.180 user_time=0.040 system_time=0.020 children_user_time=0.830 children_system_time=0.220 cmk_time_ds=0.060 cmk_time_agent=0.000

dabrain · October 26, 2022, 12:41pm

I’m observing the same issue, at my setup it is more worse because the default duration for the token is 1hour only, so after generating the token and configure checkMK with the correct token everything works for one hour only.

You can check the default token duration, check expirationSeconds. So even you request no expiration the server can overwrite this with the default value.

So most likely your token will expire after 24hour, you might would like to check.

# kubectl create token default --output yaml
apiVersion: authentication.k8s.io/v1
kind: TokenRequest
metadata:
  creationTimestamp: "2022-10-26T11:51:50Z"
  name: default
  namespace: default
spec:
  audiences:
  - https://kubernetes.default.svc.cluster.local
  - k3s
  boundObjectRef: null
  expirationSeconds: 3600
status:
  expirationTimestamp: "2022-10-26T12:51:50Z"

NOTE: Still searching for a solution here, once found will update this thread

martin.hirschvogel · November 14, 2022, 9:18am

kubectl create token --duration=0s -n [namespace] [serviceaccountname]

If they made it consistent with kubeadm, then this would set it to never expire. However, I don’t believe in any consistency anymore of Kubernetes
Because I found this here as well…

–duration=0s Requested lifetime of the issued token. The server may return a token with a longer or shorter lifetime.

Thus, best to set it to some timeframe you feel comfortable with. E.g. 1y via the --duration flag

martin.hirschvogel · November 14, 2022, 12:22pm

Yeah, how I love K8s…
--duration=0s seems to be using the default setting of the cluster. That’s 3600s as @dabrain already figured out.
Thus, set the duration to something you feel comfortable. --duration=8760h would be one year.

martin.hirschvogel · November 16, 2022, 7:26am

We are testing now to replace the kubectl create token with a different method for K8s >= 1.24

If anyone wants to try that out as well, please feel free and let us know if this change works

martin.hirschvogel · December 12, 2022, 2:22pm

This is now included in the current helm chart with the release of v1.2.0 of the K8s collectors.

morbloe · December 16, 2022, 2:33pm

But only for “K8s >= 1.24” ?

martin.hirschvogel · December 19, 2022, 7:27am

afaik you don’t need it 1.23 and lower as they only introduced the breaking change in 1.24

system · December 19, 2023, 7:28am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.