Local checks not discovered on Check_MK Server

Troubleshooting
,
1

** check-mk-raw-2.1.0p18-el9-38.x86_64.rpm**
Alma Linux 9.1

I am not able to discover the local checks on the Check_MK Server with the example script from the wiki. Locally running check_mk_agent the local checks are found, but then are not discovered in check_mk Server:

<<>>
0 “My 1st service” - This static service is always OK
2 “My 2st service” - This static service is always CRITICAL

ls -ltra /usr/lib/check_mk_agent/local/
total 16
drwxr-xr-x 4 root root 4096 Jan 19  2015 ..
-rwxr-xr-x 1 root root   75 Jan  4 16:02 mylocalcheck.sh
drwxr-xr-x 2 root root 4096 Jan  4 16:02 .

OMD[monitoring]:~$ cmk --debug -vvn [Hostname]
Checkmk version 2.1.0p18
Try license usage history update.
Trying to acquire lock on /omd/sites/monitoring/var/check_mk/license_usage/next_run
Got lock on /omd/sites/monitoring/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/monitoring/var/check_mk/license_usage/history.json
Got lock on /omd/sites/monitoring/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/monitoring/var/check_mk/license_usage/history.json
Released lock on /omd/sites/monitoring/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/monitoring/var/check_mk/license_usage/next_run
Released lock on /omd/sites/monitoring/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.PROGRAM
[cpu_tracking] Start [7fb1dd53c490]
[ProgramFetcher] Fetch with cache settings: DefaultAgentFileCache([Hostname], base_path=/omd/sites/monitoring/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 1 sec, allowed is 0 sec)
[ProgramFetcher] Execute data source
Calling: ssh -i /opt/omd/sites/monitoring/.ssh/id_rsa_dev -l root IP_OF_HOST check_mk_agent2
Write data to cache file /omd/sites/monitoring/tmp/check_mk/cache/[Hostname]
Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/cache/[Hostname]
Got lock on /omd/sites/monitoring/tmp/check_mk/cache/[Hostname]
Releasing lock on /omd/sites/monitoring/tmp/check_mk/cache/[Hostname]
Released lock on /omd/sites/monitoring/tmp/check_mk/cache/[Hostname]
[cpu_tracking] Stop [7fb1dd53c490 - Snapshot(process=posix.times_result(user=0.0, system=0.010000000000000009, children_user=0.05, children_system=0.01, elapsed=12.660000000149012))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7fb1dd481040]
[PiggybackFetcher] Fetch with cache settings: NoCache([Hostname], base_path=/omd/sites/monitoring/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for '[Hostname]'. Skip processing.
No piggyback files for '[IP]'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7fb1dd481040 - Snapshot(process=posix.times_result(user=0.009999999999999787, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.PROGRAM
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_bonding:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_thermal:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<ntp:cached(1672844111,120)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<hpsa_controller>>> / Transition HostSectionParser -> HostSectionParser
<<<hpsa_array>>> / Transition HostSectionParser -> HostSectionParser
<<<hpsa_drive>>> / Transition HostSectionParser -> HostSectionParser
<<<fred>>> / Transition HostSectionParser -> HostSectionParser
<<<hpsa>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'cifsmounts', 'cpu', 'df', 'diskstat', 'fred', 'hpsa', 'hpsa_array', 'hpsa_controller', 'hpsa_drive', 'job', 'kernel', 'labels', 'lnx_bonding', 'lnx_if', 'lnx_thermal', 'local', 'md', 'mem', 'mounts', 'nfsmounts', 'ntp', 'ps_lnx', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
  -> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7fb1dd4a2640]
value store: synchronizing
Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/counters/[Hostname]
Got lock on /omd/sites/monitoring/tmp/check_mk/counters/[Hostname]
value store: loading from disk
Releasing lock on /omd/sites/monitoring/tmp/check_mk/counters/[Hostname]
Released lock on /omd/sites/monitoring/tmp/check_mk/counters/[Hostname]
CPU load             15 min load: 3.32, 15 min load per core: 0.21 (16 cores)
CPU utilization      Total CPU: 2.33%
Check_MK Agent       Version: 2.1.0p18, OS: linux, Transport via SSH, Agent plugins: 1, Local checks: 2
Disk IO SUMMARY      Read: 440 kB/s, Write: 3.69 MB/s, Latency: 1 millisecond
HP RAID Smart Array P212 in Slot 2 Array A SAS, Unused Space: 0 MB
HP RAID Smart Array P212 in Slot 2 Logical Drive 1 18.2 TB RAID 5 OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:1 port 2E:box 1:bay 1 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:10 port 2E:box 1:bay 10 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:11 port 2E:box 1:bay 11 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:12 port 2E:box 1:bay 12 SAS 2 TB OK spare, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:2 port 2E:box 1:bay 2 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:3 port 2E:box 1:bay 3 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:4 port 2E:box 1:bay 4 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:5 port 2E:box 1:bay 5 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:6 port 2E:box 1:bay 6 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:7 port 2E:box 1:bay 7 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:8 port 2E:box 1:bay 8 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P212 in Slot 2 Physical Drive 2E:1:9 port 2E:box 1:bay 9 SAS 2 TB OK, Array Smart Array P212 in Slot 2 Array A
HP RAID Smart Array P410i in Slot 0 Array A SAS, Unused Space: 0 MB
HP RAID Smart Array P410i in Slot 0 Logical Drive 1 136.7 GB RAID 1 OK, Array Smart Array P410i in Slot 0 Array A
HP RAID Smart Array P410i in Slot 0 Physical Drive 1I:1:1 port 1I:box 1:bay 1 SAS 146 GB OK, Array Smart Array P410i in Slot 0 Array A
HP RAID Smart Array P410i in Slot 0 Physical Drive 1I:1:2 port 1I:box 1:bay 2 SAS 146 GB OK, Array Smart Array P410i in Slot 0 Array A
Interface 2          [eth0], (up), MAC: 68:B5:99:C5:B9:EC, Speed: 1 GBit/s, In: 2.15 kB/s (<0.01%), Out: 6.42 kB/s (<0.01%)
Interface 4          [eth2], (up), MAC: 68:B5:99:C5:49:D8, Speed: 1 GBit/s, In: 2.42 MB/s (1.93%), Out: 132 kB/s (0.11%)
Interface 5          [eth3], (up), MAC: 68:B5:99:C5:49:DA, Speed: 1 GBit/s, In: 1.23 MB/s (0.98%), Out: 137 kB/s (0.11%)
Kernel Performance   Process Creations: 66.29/s, Context Switches: 2999.14/s, Major Page Faults: 0.50/s, Page Swap in: 0.00/s, Page Swap Out: 0.00/s
Memory               Total virtual memory: 1.09% - 170.24 MB of 15.30 GB, 9 additional details available
Mount options of /   Mount options exactly as expected
Mount options of /app Mount options exactly as expected
Mount options of /boot Mount options exactly as expected
Mount options of /home Mount options exactly as expected
Mount options of /tmp Mount options exactly as expected
Mount options of /usr Mount options exactly as expected
Mount options of /var Mount options exactly as expected
NTP Time             Offset: 0.6270 ms, Stratum: 3, Jitter: 0.1010 ms, Time since last sync: 19 seconds
Number of threads    337, Usage: 0.26%
TCP Connections      Established: 15
Temperature Zone 0   8.3 °C
Uptime               Up since Apr 17 2022 08:55:32, Uptime: 262 days 8 hours
fs_/                 33.95% used (513.35 MB of 1.48 GB), trend: 0.00 B / 24 hours
fs_/app              48.89% used (50.38 of 103.05 GB), trend: +78.46 kB / 24 hours
fs_/boot             9.06% used (43.87 of 484.25 MB), trend: 0.00 B / 24 hours
fs_/home             8.54% used (86.03 of 1007.90 MB), trend: 0.00 B / 24 hours
fs_/tmp              5.34% used (51.22 of 959.70 MB), trend: -92.37 B / 24 hours
fs_/usr              25.71% used (777.54 MB of 2.95 GB), trend: +50.85 kB / 24 hours
fs_/var              53.29% used (2.10 of 3.94 GB), trend: +501.15 kB / 24 hours
No piggyback files for '[Hostname]'. Skip processing.
No piggyback files for '[IP]'. Skip processing.
[cpu_tracking] Stop [7fb1dd4a2640 - Snapshot(process=posix.times_result(user=0.1299999999999999, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=0.13999999966472387))]
[agent] Success, execution time 12.8 sec | execution_time=12.800 user_time=0.140 system_time=0.020 children_user_time=0.050 children_system_time=0.010 cmk_time_ds=12.590 cmk_time_agent=0.000
2

Hi,
you should get a line like <<<local:sep(0)>>> before 0 "My 1st ..."

What’s before <<>>?

Karl

3

Hi Karl,
Changed the local script to be exactly like in the wiki, this is the output if i call the check_mk_agent remotly:

<<<local:sep(0)>>>
<<<hpsa_controller>>>
2 OK P212
0 OK P410i
<<<hpsa_array>>>
2:A OK SAS 0 MB
0:A OK SAS 0 MB
<<<hpsa_drive>>>
2:A:2E:1:1 OK SAS 2 TB assigned
2:A:2E:1:2 OK SAS 2 TB assigned
2:A:2E:1:3 OK SAS 2 TB assigned
2:A:2E:1:4 OK SAS 2 TB assigned
2:A:2E:1:5 OK SAS 2 TB assigned
2:A:2E:1:6 OK SAS 2 TB assigned
2:A:2E:1:7 OK SAS 2 TB assigned
2:A:2E:1:8 OK SAS 2 TB assigned
2:A:2E:1:9 OK SAS 2 TB assigned
2:A:2E:1:10 OK SAS 2 TB assigned
2:A:2E:1:11 OK SAS 2 TB assigned
2:A:2E:1:12 OK SAS 2 TB spare
0:A:1I:1:1 OK SAS 146 GB assigned
0:A:1I:1:2 OK SAS 146 GB assigned
0 "My 1st service" - This static service is always OK
<<<hpsa>>>

Smart Array P212 in Slot 2                (sn: PACCPID10371945)

   array A (SAS, Unused Space: 0 MB)


      logicaldrive 1 (18.2 TB, RAID 5, OK)

      physicaldrive 2E:1:1 (port 2E:box 1:bay 1, SAS, 2 TB, OK)
      physicaldrive 2E:1:2 (port 2E:box 1:bay 2, SAS, 2 TB, OK)
      physicaldrive 2E:1:3 (port 2E:box 1:bay 3, SAS, 2 TB, OK)
      physicaldrive 2E:1:4 (port 2E:box 1:bay 4, SAS, 2 TB, OK)
      physicaldrive 2E:1:5 (port 2E:box 1:bay 5, SAS, 2 TB, OK)
      physicaldrive 2E:1:6 (port 2E:box 1:bay 6, SAS, 2 TB, OK)
      physicaldrive 2E:1:7 (port 2E:box 1:bay 7, SAS, 2 TB, OK)
      physicaldrive 2E:1:8 (port 2E:box 1:bay 8, SAS, 2 TB, OK)
      physicaldrive 2E:1:9 (port 2E:box 1:bay 9, SAS, 2 TB, OK)
      physicaldrive 2E:1:10 (port 2E:box 1:bay 10, SAS, 2 TB, OK)
      physicaldrive 2E:1:11 (port 2E:box 1:bay 11, SAS, 2 TB, OK)
      physicaldrive 2E:1:12 (port 2E:box 1:bay 12, SAS, 2 TB, OK, spare)

   Expander 249 (WWID: 5001438006CD70BF, Port: 2E, Box: 1)

   Enclosure SEP (Vendor ID HP, Model D2600 SAS AJ940A) 248 (WWID: 5001438006CD70BE, Port: 2E, Box: 1)

   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 250 (WWID: 5001438010D0BBCF)


Smart Array P410i in Slot 0 (Embedded)    (sn: 50014380123C6400)

   array A (SAS, Unused Space: 0 MB)


      logicaldrive 1 (136.7 GB, RAID 1, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)

   SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 250 (WWID: 50014380123C640F)

Br,
Fred

4

Hi,
do you run the checkmk agent as root?
How much time does the local check need? (-> you could cache it)
Best regards
Dennis

5

There is something broken here. The 0 My 1st service... line must appear directly below the <<<local>>> section start, not in the section of another check plugin. This looks like if a Local Check was placed under plugins, or an agent plugin under local.

If you experimented with different setups earlier, it helps to remove the cache files under /var/lib/check_mk_agent/cache.

1 Like
6

Hi,
@Heavy has an important tip for you: The result should be directly under local and not as part of hpsa_drive.

Karl

7

Hi,
Thanks guys for the responses, i tried it in a lab environment and it works as expected.
Need to figure out now what is wrong on the production agent server. I will post my findings.

8

There is a script that produces the “hpsa…” output. This script also is inside the local folder. But it should be inside the plugin folder.

1 Like
9

Hi all,
Yes, that was the problem after all as pointed out by heavy. The hpsa plugin was under the local folder. Change it to the plugins folder. Now the local checks are discovered. Topic can be closed.
thank you very much.

Best regards,
Fred

10

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.