OpenWrt Agent not writing cache for .../local/<seconds>/scripts

Troubleshooting
1

CMK version: 2.3.0p17
OS version: Alpine Linux v3.19

Error message: none

Output of “cmk --debug -vvn hostname”: (I’ll give parts of it when asked for)

Hello community,

I have a tiny Intel N100 server(chen) which runs 15 LXC containers with proxmox.

They do “single things” like being a mosquitto broker, a NextCloud instance, my SearXNG, etc.

I’d like to monitor them and currently testing out checkmk.

I like to keep things small and tiny, therefore I use Alpine Linux for my containers.
I decided to go with OpenWrt agents, in order not to install some *inetd and even no bash.

I access the agents via “indiviudual ssh command”.

I want a check for “get latest apk updates, count number of upgradable packages and list them”, if more than 3, get yellow, if more than 10, get red. This check might possibly take a while, but needs to be run only every hour or more.

That’s the point I came to the documentation:
6.2 Agent extensions - last point “Local checks”
3.7. Executing asynchronously and caching output
Configuring Linux

Of course I read all the above, regarding the output format, regarding the metrics, followed the link for the cache directories. I decided to go for an output like
'P "Alpine Updates" alpine_packets=$count;3;10 <depending on the outcome the userfriendly text, perhaps containig the list of packets with \n's to make it readable in the checkmk WebGUI (and a final empty line not to smear into the next local output)>'

and put my script into /usr/lib/check_mk_agent/local/600/mk_alpine_update.

Since the OpenWrt agent is just the script to put in /usr/bin/check_mk_agent,
I manually created all the directories from check_mk_agent | head -n 9
and watched e.g. the cache with tree /var/lib/check_mk_agent/cache.

But there’s no caching information written…

I followed the output of check_mk_agent -d and saw those lines:

[...]
+ run_local_checks
+ cd /usr/lib/check_mk_agent/local
+ true
+ echo '<<<local:sep(0)>>>'
+ for script in ./*
+ is_valid_plugin ./600
+ case "${1:?No plugin defined}" in
+ '[' -f ./600 ']'
+ for script in [1-9]*/*
+ is_valid_plugin 600/mk_alpine_update
+ case "${1:?No plugin defined}" in
+ '[' -f 600/mk_alpine_update ']'
+ '[' -x 600/mk_alpine_update ']'
+ interval=600
+ _run_cached_internal local_mk_alpine_update 600 600 1800 1200 '_log_section_time '\''600/mk_alpine_update'\'''
+ NAME=local_mk_alpine_update
+ REFRESH_INTERVAL=600
+ MAXAGE=600
+ OUTPUT_TIMEOUT=1800
+ CREATION_TIMEOUT=1200
+ shift 5
+ false
+ '[' -d /var/lib/check_mk_agent/cache ']'
+ CACHEFILE=/var/lib/check_mk_agent/cache/local_mk_alpine_update.cache
+ FAIL_REPORT_FILE=/var/lib/check_mk_agent/spool/local_mk_alpine_update.cachefail
<<<local:sep(0)>>>
++ get_epoch
++ date +%s
+ NOW=1727724171
++ get_file_mtime /var/lib/check_mk_agent/cache/local_mk_alpine_update.cache
+ MTIME=
+ MTIME=0
+ true
+ '[' -s /var/lib/check_mk_agent/cache/local_mk_alpine_update.cache ']'
+ true
+ for cfile in "${CACHEFILE}.new."*
+ '[' -e '/var/lib/check_mk_agent/cache/local_mk_alpine_update.cache.new.*' ']'
+ break
+ '[' 1727724171 -gt 600 ']'
++ _cfile_in_use
++ for cfile in "${CACHEFILE}.new."*
++ printf '%s\n' '/var/lib/check_mk_agent/cache/local_mk_alpine_update.cache.new.*'
++ break
+ '[' '!' -e '/var/lib/check_mk_agent/cache/local_mk_alpine_update.cache.new.*' ']'
+ unset NAME + cat
+ nohup ''
MAXAGE CREATION_TIMEOUT REFRESH_INTERVAL CACHEFILE NOW MTIME CACHE_INFO TRYING_SINCE OUTPUT_TIMEOUT


+ run_plugins
+ cd /usr/lib/check_mk_agent/plugins
[...]

I set up a test-Debian, downloaded and installed the *.deb of my baked agents, which works fine. I see nearly immediately the created files in the cache directory.

Is there a optimization potential on the OpenWrt agent?
Is this feature only working with plugins? (I didn’t try this with the different output for plugins)

What could I try to get the asynchronously executed checks running on Alpine/OpenWrt?
Is there a better approach to fit my tiny setup with not-so-often-run checks?

Thanks for wrapping your head around this and finding more ideas,
since I tried now for three days on this. And just integrating this check into the normal ...local/ slows down the monitoring of my 15 containers, specially when they all are doing nearly the same, asking for new Alpine packages on the internet…, every minute…

Have a nice day
Frank

2

Hi all
I solved it.
I talked to some experienced users and I learned quite a bit.
Even when the OpenWrt agent seems not do support relayed and cached local checks,
you are free to create your own logic and use the features of the spool directory:

want a local check (not a plugin) to run only every hour (or less)?
use crontab:

03 * * * * /usr/bin/check_for_alpine_updates.sh

code what needs to be done:
“get latest apk updates, count number of upgradable packages and list them”

echo this outcome in a “local check” manner:

apk update > /dev/null

count=$(apk -u list | wc -l)
list=$(apk -u list | sed 's/$/\\n/g')

echo -n 'P "Alpine Updates" ' >> /tmp/spool.tmp
echo -n 'alpine_packages='$count >> /tmp/spool.tmp

if [ "$count" -gt "0" ]; then
  echo -n ';3;10 These packages are available for update:\n'$list >> /tmp/spool.tmp
else
  echo -n ';3;10 No packages are available for update.' >> /tmp/spool.tmp
fi

echo >> /tmp/spool.tmp 
mv /tmp/spool.tmp /var/lib/check_mk_agent/spool/4200_alpine_update

put your output into the /var/lib/check_mk_agent/spool directory with a name starting with the seconds it should wait for… one hour (60*60) plus ten minutes = 4200_alpine_updates

don’t forget to add an echo "<<<local>>>"
before, to be recognized as local check,

and (even) the OpenWrt agent will add this output to it’s minute-checks.

Have a nice day
Frank

1 Like