Cmc does not start, config missing, Configuration Error: Error creating configuration: '24X7'

christophe.marbacher · March 15, 2022, 10:54am

**CMK version:**2.0.0p19
**OS version:**Rocky Linux 8.5

Error message:

OMD[xxx]:~$ omd start
Temporary filesystem already mounted
Starting mkeventd...Already running.
Starting liveproxyd...Already running.
Starting mknotifyd...OK
Starting rrdcached...Already running.
Starting cmc...Failed (Config /omd/sites/xxx/var/check_mk/core/config missing, run "cmk -U" and try again)
Starting apache...(already running: 3853363)...OK
Starting dcd...Already running.
Starting redis...Already running.
Initializing Crontab...OK

OMD[xxx]:~$ cmk -vvU --debug
Trying to acquire lock on /omd/sites/xxx/etc/check_mk/main.mk
Got lock on /omd/sites/xxx/etc/check_mk/main.mk
Generating configuration for core (type cmc)...Trying to acquire lock on /omd/sites/xxx/var/check_mk/core/helper_config/serial.mk
Got lock on /omd/sites/xxx/var/check_mk/core/helper_config/serial.mk
Releasing lock on /omd/sites/xxx/var/check_mk/core/helper_config/serial.mk
Released lock on /omd/sites/xxx/var/check_mk/core/helper_config/serial.mk
Creating helper config...OK

Time needed [<lambda>]: 0.00 sec
Releasing lock on /omd/sites/xxx/etc/check_mk/main.mk
Released lock on /omd/sites/xxx/etc/check_mk/main.mk
Configuration Error: '24X7'
Trying to acquire lock on /omd/sites/xxx/var/check_mk/crashes/base/5f279270-a44d-11ec-bd2a-005056b5330e/crash.info
Got lock on /omd/sites/xxx/var/check_mk/crashes/base/5f279270-a44d-11ec-bd2a-005056b5330e/crash.info
Releasing lock on /omd/sites/xxx/var/check_mk/crashes/base/5f279270-a44d-11ec-bd2a-005056b5330e/crash.info
Released lock on /omd/sites/xxx/var/check_mk/crashes/base/5f279270-a44d-11ec-bd2a-005056b5330e/crash.info
Traceback (most recent call last):
  File "/omd/sites/xxx/bin/cmk", line 92, in <module>
    exit_status = modes.call(mode_name, mode_args, opts, args)
  File "/omd/sites/xxx/lib/python3/cmk/base/modes/__init__.py", line 69, in call
    return handler(*handler_args)
  File "/omd/sites/xxx/lib/python3/cmk/base/modes/check_mk.py", line 1116, in mode_update
    do_create_config(create_core(config.monitoring_core))
  File "/omd/sites/xxx/lib/python3/cmk/base/core_config.py", line 336, in do_create_config
    _create_core_config(core)
  File "/omd/sites/xxx/lib/python3/cmk/base/core_config.py", line 401, in _create_core_config
    core.create_config(helper_config.serial)
  File "/omd/sites/xxx/lib/python3/cmk/base/cee/core_cmc.py", line 87, in create_config
    _create_core_config(serial)
  File "/omd/sites/xxx/lib/python3/cmk/base/cee/core_cmc.py", line 148, in _create_core_config
    timeperiods_config = _measure_time(cmc_timeperiods)
  File "/omd/sites/xxx/lib/python3/cmk/base/cee/core_cmc.py", line 186, in _measure_time
    result = func(*args, **kwargs)
  File "/omd/sites/xxx/lib/python3/cmk/base/cee/core_cmc.py", line 376, in cmc_timeperiods
    serialize_timeperiod(tps_to_do[0], sequences)
  File "/omd/sites/xxx/lib/python3/cmk/base/cee/core_cmc.py", line 2006, in serialize_timeperiod
    serialize_timeperiod(ex, serialized)  # infinite recursion not regarded here
  File "/omd/sites/xxx/lib/python3/cmk/base/cee/core_cmc.py", line 2001, in serialize_timeperiod
    tp = config.timeperiods[tpname]
KeyError: '24X7'

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

OMD[xxx]:~$ cmk --debug -vvn vmsrvchkmk01t
Checkmk version 2.0.0p19
Try license usage history update.
Trying to acquire lock on /omd/sites/xxx/var/check_mk/license_usage/next_run
Got lock on /omd/sites/xxx/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/xxx/var/check_mk/license_usage/history.json
Got lock on /omd/sites/xxx/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/xxx/var/check_mk/license_usage/history.json
Released lock on /omd/sites/xxx/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/xxx/var/check_mk/license_usage/next_run
Released lock on /omd/sites/xxx/var/check_mk/license_usage/next_run
Loading autochecks from /omd/sites/xxx/var/check_mk/autochecks/vmsrvchkmk01t.mk
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7fcaed337f40]
Connecting via TCP to x.x.x.23:6556 (5.0s timeout)
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(base_path=PosixPath('/omd/sites/xxx/tmp/check_mk/cache/vmsrvchkmk01t'), max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Does not exist)
[TCPFetcher] Execute data source
Reading data from agent
Output is not encrypted
Write data to cache file /omd/sites/xxx/tmp/check_mk/cache/vmsrvchkmk01t
Trying to acquire lock on /omd/sites/xxx/tmp/check_mk/cache/vmsrvchkmk01t
Got lock on /omd/sites/xxx/tmp/check_mk/cache/vmsrvchkmk01t
Releasing lock on /omd/sites/xxx/tmp/check_mk/cache/vmsrvchkmk01t
Released lock on /omd/sites/xxx/tmp/check_mk/cache/vmsrvchkmk01t
Closing TCP connection to x.x.x.23:6556
[cpu_tracking] Stop [7fcaed337f40 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=3.6200000001117587))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7fcaed348220]
Piggyback file '/omd/sites/xxx/tmp/check_mk/piggyback/vmsrvchkmk01t/vmsrvvc01p.hw.nets.lan' is outdated (Piggyback file too old: 21 d). Skip processing.
No piggyback files for 'x.x.x.23'. Skip processing.
[PiggybackFetcher] Fetch with cache settings: NoCache(base_path=PosixPath('/omd/sites/xxx/tmp/check_mk/data_source_cache/piggyback/vmsrvchkmk01t'), max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
[PiggybackFetcher] Execute data source
[cpu_tracking] Stop [7fcaed348220 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
[cpu_tracking] Start [7fcaed337e50]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.TCP
No persisted sections loaded
  -> Add sections: ['check_mk', 'chrony', 'cifsmounts', 'cmk_site_statistics', 'cpu', 'df', 'diskstat', 'job', 'kernel', 'labels', 'livestatus_ssl_certs', 'livestatus_status', 'lnx_if', 'local', 'md', 'mem', 'mkeventd_status', 'mknotifyd', 'mounts', 'nfsmounts', 'omd_apache', 'omd_info', 'omd_status', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections loaded
  -> Add sections: ['labels']
Received no piggyback data
Loading item states
Trying to acquire lock on /omd/sites/xxx/tmp/check_mk/counters/vmsrvchkmk01t
Got lock on /omd/sites/xxx/tmp/check_mk/counters/vmsrvchkmk01t
Releasing lock on /omd/sites/xxx/tmp/check_mk/counters/vmsrvchkmk01t
Released lock on /omd/sites/xxx/tmp/check_mk/counters/vmsrvchkmk01t
CPU load             15 min load: 0.34 at 2 cores (0.17 per core)
CPU utilization      Total CPU: 7.15%
Disk IO SUMMARY      PEND - Initializing counters
Filesystem /         28.6% used (13.25 of 46.33 GB), trend: 0.00 B / 24 hours
Filesystem /boot     29.56% used (299.73 of 1014.00 MB), trend: 0.00 B / 24 hours
Filesystem /boot/efi 0.96% used (5.76 of 598.81 MB), trend: 0.00 B / 24 hours
Interface 2          [ens192], (up), MAC: 00:50:56:B5:33:0E, Speed: 10 GBit/s
Kernel Performance   Process Creations: 0.00/s, Context Switches: 0.00/s, Major Page Faults: 0.00/s, Page Swap in: 0.00/s, Page Swap Out: 0.00/s
Memory               Total virtual memory: 32.85% - 1.87 GB of 5.71 GB
Mount options of /   Mount options exactly as expected
Mount options of /boot Mount options exactly as expected
Mount options of /boot/efi Mount options exactly as expected
NTP Time             Offset: 0.0137 ms, Stratum: 2, Time since last sync: 10 minutes 22 seconds
Number of threads    Count: 643 threads, Usage: 2.17%
OMD xxx Event Console Current events: 0, Virtual memory: 411.48 MB, Overall event limit inactive, No hosts event limit active, No rules event limit active, Received messages: 0.00/s, Rule hits: 0.00/s, Rule tries: 0.00/s, Message drops: 0.00/s, Created events: 0.00/s, Client connects: 0.00/s, Rule hit ratio: -, Processing time per message: -, Time per client request: -
OMD xxx Notification Spooler Version: 2.0.0p19, Spooler running
OMD xxx apache   0.00 Requests/s, 0.00 Seconds serving/s, 0.00 B Sent/s
OMD xxx performance Item not found in monitoring data
OMD xxx status   partially running, stopped services: cmc
OMD SCPMS Event Console Current events: 0, Virtual memory: 411.44 MB, Overall event limit inactive, No hosts event limit active, No rules event limit active, Received messages: 0.00/s, Rule hits: 0.00/s, Rule tries: 0.00/s, Message drops: 0.00/s, Created events: 0.00/s, Client connects: 0.00/s, Rule hit ratio: -, Processing time per message: -, Time per client request: 0.29 ms
OMD SCPMS Notification Spooler Version: 2.0.0p21, Spooler running
OMD SCPMS apache     0.00 Requests/s, 0.00 Seconds serving/s, 0.00 B Sent/s
OMD SCPMS performance PEND - Initialized: 'host_checks'
OMD SCPMS status     running
Site xxx statistics Item not found in monitoring data
Site SCPMS statistics Total hosts: 0, Problem hosts: 0, Total services: 0, Problem services: 0
Systemd Service Summary Total: 93, Disabled: 6, Failed: 0
TCP Connections      Established: 7
Uptime               Up since Mar 01 2022 07:17:13, Uptime: 14 days 4 hours
+ EXECUTING INVENTORY PLUGINS
 hp_proliant_mem: skipped (no data)
 hp_proliant_mem: skipped (no data)
 ipmi_firmware: skipped (no data)
 ipmi_firmware: skipped (no data)
 inventory_checkmk: ok
 inventory_checkmk: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_cisco_vlans: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_esx_vsphere_hostsystem: skipped (no data)
 inv_if: ok
 inv_if: skipped (no data)
 inventory_oracle_tablespaces: skipped (no data)
 inventory_oracle_tablespaces: skipped (no data)
 dmidecode: skipped (no data)
 dmidecode: skipped (no data)
 docker_node_network: skipped (no data)
 docker_node_network: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_clusters: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 inventory_esx_vsphere_virtual_machines: skipped (no data)
 k8s_endpoint_info: skipped (no data)
 k8s_endpoint_info: skipped (no data)
 k8s_ingress_infos: skipped (no data)
 k8s_ingress_infos: skipped (no data)
 juniper_info: skipped (no data)
 juniper_info: skipped (no data)
 mem: ok
 mem: skipped (no data)
 mem_used: skipped (no data)
 mem_used: skipped (no data)
 snmp_info: skipped (no data)
 snmp_info: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_baselevel: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_lparstat_inventory: skipped (no data)
 aix_packages: skipped (no data)
 aix_packages: skipped (no data)
 aix_service_packs: skipped (no data)
 aix_service_packs: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 allnet_ip_sensoric: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 aruba_wlc_aps: skipped (no data)
 check_mk: ok
 check_mk: skipped (no data)
 checkpoint_inv_tunnels: skipped (no data)
 checkpoint_inv_tunnels: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_controller: skipped (no data)
 citrix_state: skipped (no data)
 citrix_state: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 couchbase_nodes_ports: skipped (no data)
 dell_hw_info: skipped (no data)
 dell_hw_info: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_labels: skipped (no data)
 docker_container_network: skipped (no data)
 docker_container_network: skipped (no data)
 docker_container_node_name: skipped (no data)
 docker_container_node_name: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_images: skipped (no data)
 docker_node_info: skipped (no data)
 docker_node_info: skipped (no data)
 esx_systeminfo: skipped (no data)
 esx_systeminfo: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fireeye_sys_status: skipped (no data)
 fritz: skipped (no data)
 fritz: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_da_phydrv: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 hp_proliant_systeminfo: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_channels: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_managers: skipped (no data)
 ibm_mq_queues: skipped (no data)
 ibm_mq_queues: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_osinfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 infoblox_systeminfo: skipped (no data)
 k8s_assigned_pods: skipped (no data)
 k8s_assigned_pods: skipped (no data)
 k8s_daemon_pod_containers: skipped (no data)
 k8s_daemon_pod_containers: skipped (no data)
 k8s_job_container: skipped (no data)
 k8s_job_container: skipped (no data)
 k8s_nodes: skipped (no data)
 k8s_nodes: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_pod_container: skipped (no data)
 k8s_pod_info: skipped (no data)
 k8s_pod_info: skipped (no data)
 k8s_roles: skipped (no data)
 k8s_roles: skipped (no data)
 k8s_selector: skipped (no data)
 k8s_selector: skipped (no data)
 k8s_service_info: skipped (no data)
 k8s_service_info: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_block_devices: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_cpuinfo: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_distro: skipped (no data)
 lnx_if: ok
 lnx_if: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_ip_r: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_packages: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_sysctl: skipped (no data)
 lnx_uname: skipped (no data)
 lnx_uname: skipped (no data)
 lnx_video: skipped (no data)
 lnx_video: skipped (no data)
 lparstat_aix: skipped (no data)
 lparstat_aix: skipped (no data)
 mssql_clusters: skipped (no data)
 mssql_clusters: skipped (no data)
 mssql_versions: skipped (no data)
 mssql_versions: skipped (no data)
 netapp_api_disk: skipped (no data)
 netapp_api_disk: skipped (no data)
 netapp_api_info: skipped (no data)
 netapp_api_info: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_dataguard_stats: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_instance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_performance: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_recovery_area: skipped (no data)
 oracle_systemparameter: skipped (no data)
 oracle_systemparameter: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_chassis_slots: skipped (no data)
 perle_psmu: skipped (no data)
 perle_psmu: skipped (no data)
 prtconf: skipped (no data)
 prtconf: skipped (no data)
 snmp_extended_info: skipped (no data)
 snmp_extended_info: skipped (no data)
 snmp_os: skipped (no data)
 snmp_os: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 snmp_quantum_storage_info: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_addresses: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_pkginfo: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_prtdiag: skipped (no data)
 solaris_prtpicl: skipped (no data)
 solaris_prtpicl: skipped (no data)
 solaris_psrinfo: skipped (no data)
 solaris_psrinfo: skipped (no data)
 solaris_routes: skipped (no data)
 solaris_routes: skipped (no data)
 solaris_uname: skipped (no data)
 solaris_uname: skipped (no data)
 statgrab_net: skipped (no data)
 statgrab_net: skipped (no data)
 suseconnect: skipped (no data)
 suseconnect: skipped (no data)
 win_bios: skipped (no data)
 win_bios: skipped (no data)
 win_computersystem: skipped (no data)
 win_computersystem: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_cpuinfo: skipped (no data)
 win_disks: skipped (no data)
 win_disks: skipped (no data)
 win_exefiles: skipped (no data)
 win_exefiles: skipped (no data)
 win_ip_r: skipped (no data)
 win_ip_r: skipped (no data)
 win_networkadapter: skipped (no data)
 win_networkadapter: skipped (no data)
 win_os: skipped (no data)
 win_os: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_reg_uninstall: skipped (no data)
 win_system: skipped (no data)
 win_system: skipped (no data)
 win_video: skipped (no data)
 win_video: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_software: skipped (no data)
 win_wmi_updates: skipped (no data)
 win_wmi_updates: skipped (no data)
 winperf_if: skipped (no data)
 winperf_if: skipped (no data)

Trying to acquire lock on /omd/sites/xxx/tmp/check_mk/status_data/vmsrvchkmk01t
Got lock on /omd/sites/xxx/tmp/check_mk/status_data/vmsrvchkmk01t
Releasing lock on /omd/sites/xxx/tmp/check_mk/status_data/vmsrvchkmk01t
Released lock on /omd/sites/xxx/tmp/check_mk/status_data/vmsrvchkmk01t
Trying to acquire lock on /omd/sites/xxx/tmp/check_mk/status_data/vmsrvchkmk01t.gz
Got lock on /omd/sites/xxx/tmp/check_mk/status_data/vmsrvchkmk01t.gz
Releasing lock on /omd/sites/xxx/tmp/check_mk/status_data/vmsrvchkmk01t.gz
Released lock on /omd/sites/xxx/tmp/check_mk/status_data/vmsrvchkmk01t.gz
Trying to acquire lock on /omd/sites/xxx/tmp/check_mk/status_data/.last
Got lock on /omd/sites/xxx/tmp/check_mk/status_data/.last
Releasing lock on /omd/sites/xxx/tmp/check_mk/status_data/.last
Released lock on /omd/sites/xxx/tmp/check_mk/status_data/.last
Piggyback file '/omd/sites/xxx/tmp/check_mk/piggyback/vmsrvchkmk01t/vmsrvvc01p.hw.nets.lan' is outdated (Piggyback file too old: 21 d). Skip processing.
No piggyback files for 'x.x.x.23'. Skip processing.
[cpu_tracking] Stop [7fcaed337e50 - Snapshot(process=posix.times_result(user=0.050000000000000044, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=0.11999999918043613))]
[agent] Version: 2.0.0p19, OS: linux, [piggyback] Failed sources: Piggyback file too old: 21 d, Missing monitoring data for check plugins: esx_vsphere_vm_cpu, esx_vsphere_vm_datastores, esx_vsphere_vm_guest_tools, esx_vsphere_vm_heartbeat, esx_vsphere_vm_mem_usage, esx_vsphere_vm_mounted_devices, esx_vsphere_vm_name, esx_vsphere_vm_running_on, esx_vsphere_vm_snapshots(!), execution time 3.7 sec | execution_time=3.740 user_time=0.050 system_time=0.010 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=3.620

christophe.marbacher · March 15, 2022, 1:05pm

I fixed the issue by temporarily returning an empty timeperiod exclusion list.

Edit ~/lib/python3/cmk/base/cee/core_cmc.py and go to function serialize_timeperiod.

Comment line 2001 and add and empty excludes array.

def serialize_timeperiod(tpname: TimeperiodName, serialized: TimeperiodSequences) -> None:
    tp = config.timeperiods[tpname]
    #excludes = tp.get("exclude", [])
    excludes = []
    ex_sequences = []
    for ex in excludes:
        if ex not in serialized:
            serialize_timeperiod(ex, serialized)  # infinite recursion not regarded here
        ex_sequences.append(serialized[ex])
    sequence = timeperiod_sequence_bare(tpname)
    for ex_seq in ex_sequences:
        sequence = timeperiod_substract(sequence, ex_seq)
    serialized[tpname] = sequence

Then generate the core configuration (cmk -U) and start the site. I could apply modifications. I brought back the original python script after unchecking “Always” in one of the added time period.

To reproduce the issue with the original file, create a Time period and select “Always” in the exclude list. Try applying your modification, you may get the error message.

athomaidis · February 22, 2023, 5:16pm

Hi guys,

first, this is hack and not update resistant.

It’s a misunderstanding, and I’ll show you why:

Checkmk is complaining about the 24x7 Keyword, because you want to exclude something from this timeperiod.

You should do the other way around:

create a custom always time period with 00:00 - 24:00 (custom_always)
create a time period for the exceptions (exceptions)
open custom_always and exclude exceptions

Otherwise, your exceptions won’t match at all
Please see also this example: Time periods - Restricting actions to specific times of the day

In the raw edition we use the nagios core and nagios is not considering, if this is valid or not.

Best
Anastasios

system · February 22, 2024, 5:16pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.