Weird local check issue with RRD not being updated

Troubleshooting
, ,
1

Using cre 2.2.0p23, I’ve recently been having some issues with local checks not logging rrd anymore. Everything looks fine in the interface… except the charts are not updated.

Full exemple of what is in the UI with emphasis:

image
image1080×1920 158 KB

The script output:

P os_ha_addons started=4|failed=0;1;1|stopped=0;1 All add-ons are running

I can say it does update if I remove the warn/crit levels in the above check. I also tried with the complete syntax (started=4;;;;|failed=0;1;1;;|stopped=0;1;;), and with different metric names in case some names were reserved for whatever reason. No luck.

I can see the rrd files are not getting updated:

OMD[home]:~$ ls -l /opt/omd/sites/home/var/pnp4nagios/perfdata/vm-automation/os_ha_addons*
-rw-rw-r-- 1 home home   3117 Mar  8 23:05 /opt/omd/sites/home/var/pnp4nagios/perfdata/vm-automation/os_ha_addons.xml
-rw-r--r-- 1 home home 384952 Mar  8 23:05 /opt/omd/sites/home/var/pnp4nagios/perfdata/vm-automation/os_ha_addons_failed.rrd
-rw-r--r-- 1 home home 384952 Mar  8 23:05 /opt/omd/sites/home/var/pnp4nagios/perfdata/vm-automation/os_ha_addons_started.rrd
-rw-r--r-- 1 home home 384952 Mar  8 23:05 /opt/omd/sites/home/var/pnp4nagios/perfdata/vm-automation/os_ha_addons_stopped.rrd

It worked in the past, I’m unsure when it stopped working exactly and I honestly don’t see what is wrong exactly with the syntax (if anything). The warn/crit lines are correctly added to the charts, everything seems fine? Any help would be appreciated.

Here is the complete local script just in case:

#!/usr/bin/sh
json=$(ha addons --raw-json)
started_nb=$(echo "$json" | jq -r '[.data.addons[] | select(.state=="started")] | length')
stopped_nb=$(echo "$json" | jq -r '[.data.addons[] | select(.state=="stopped")] | length')
error_nb=$(echo "$json" | jq -r '[.data.addons[] | select(.state=="error")] | length')
stopped=$(echo "$json" | jq -r '.data.addons[] | select(.state=="stopped") | .name' | sed ':a;N;$!ba;s/\n/, /g')
error=$(echo "$json" | jq -r '.data.addons[] | select(.state=="error") | .name' | sed ':a;N;$!ba;s/\n/, /g')

echo -n "P os_ha_addons started=$started_nb|failed=$error_nb;1;1|stopped=$stopped_nb;1"

if [ $stopped_nb -gt 0 ] && [ $error_nb -gt 0 ]
then
        if [ $stopped_nb -gt 1 ]
        then
                echo " $stopped are stopped,"
        else
                echo " $stopped is stopped,"
        fi
        if [ $error_nb -gt 1 ]
        then
                echo " $error are in error"
        else
                echo " $error is in error"
        fi
elif [ $stopped_nb -gt 0 ]
then
        if [ $stopped_nb -gt 1 ]
        then
                echo " $stopped are stopped"
        else
                echo " $stopped is stopped"
        fi
elif [ $error_nb -gt 0 ]
then
        if [ $error_nb -gt 1 ]
        then
                echo " $error are in error"
        else
                echo " $error is in error"
        fi
else
        echo " All add-ons are running"
fi

And finally the local section from the agent output, taken from the diagnostics page in WATO:

<<<local:sep(0)>>>
P os_ha_addons started=4|failed=0;1;1|stopped=0;1 All add-ons are running
0 os_release - Home Assistant OS 12.0
0 os_kernel - 6.6.16-haos
2

I still have no luck debugging this and don’t know what/where to look for. Any insight would be appreciated.

3

As a temporary workaround, with additional logic within the code, this gets the rrd working:

1 os_ha_addons started=3|failed=0|stopped=1

This doesn’t:

P os_ha_addons started=3|failed=0;1;1|stopped=1;1

By hardcoding the status, I don’t get the warn/crit lines showing in the charts. When it works, I do see the writings/updates in /opt/omd/sites/home/var/rrdcached/rrd.journal.*.

4

Fixed by adding a ridiculously high critical value for my last variable.

P os_ha_addons started=3;;|failed=0;1;1|stopped=1;1;99999

This is (sort of) a bug, when there is no value defined for the critical level, checkmk automatically translates it to inf and this is when no data is written to rrd.