Stale nfs mounts on aix leads to timeout of the agent

[kohly.de (kohly)]

Checkmk version:
2.3.0p24 cee

OS version of monitored system:
AIX 7.2 - Technology Level 5 - Service Pack 8 (7200-05-08-2420)

Description of the problem :
Every night, when the backup software named rubrik does its work it does not unmount nfs-mounts on the AIX system before removing or disabling access to the share on the backup server. Rubrik creates many stale shares this way at the same time. This leads into timeouts in the section_df and section_nfs_mounts from the agent.

Steps to reproduce:

1. Create many nfs shares
2. Mount them on AIX
3. Prevent the access to the nfs server (e.g. on osi layer 3 by droping nfs related traffic by a firewall rule on the nfs server) to get a stale nfs mounts on AIX
4. Run the agent

If profiling is enabled you will find that the sections of df and nfs_mounts needs too long so the agent will cause a timeout on the server. This leads to a critical ‘Check_MK’ service.
As soon you enable access to the nfs share again, the agent comes back.

Question:
Is it possible to run internal sections of the agent asynchronous without modifying the agents code?
(as already described in AIX agent and hard mounted NFS shares)
In other words, is there a chance to create a config-file e.g. in /etc/check_mk/… to do so?

Best Regards
Chrsitian

[andreas-doehler (Andreas)]

@kohly.de

Is it possible to run internal sections of the agent asynchronous without modifying the agents code?

At the moment not.

For the NFS problem the comment inside the agent should show a valid approach.

section_df() {
    echo '<<<df>>>'
    if [ -x /usr/opt/freeware/bin/df ]; then
        excludefs="-x smbfs -x cifs -x iso9660 -x udf -x nfsv4 -x nfs -x mvfs -x zfs -x cdrfs"
        # shellcheck disable=SC2086
        /usr/opt/freeware/bin/df -PTlk ${excludefs} | sed 1d

        # df inodes information
        echo '<<<df>>>'
        echo '[df_inodes_start]'
        # shellcheck disable=SC2086
        /usr/opt/freeware/bin/df -PTli ${excludefs} | sed 1d
        echo '[df_inodes_end]'
    else
        df -kP | sed 's/ / - /' | grep -v ^/proc | grep -v ^Filesystem | grep -v :
    fi
}

If you have only the internal df available you will have no chance to avoid the timeout problem. With the extra df command there are the same excludes defined as on Linux.

[kohly.de (kohly)]

We modified the agent actually, but this is not update save.

1. The internal df knows the parameter ‘-T local’
   The code is now the following:

section_df() {
    echo '<<<df>>>'
    if [ -x /usr/opt/freeware/bin/df_unused ]; then
        excludefs="-x smbfs -x cifs -x iso9660 -x udf -x nfsv4 -x nfs -x mvfs -x zfs -x cdrfs"
        # shellcheck disable=SC2086
        /usr/opt/freeware/bin/df -PTlk ${excludefs} | sed 1d

        # df inodes information
        echo '<<<df>>>'
        echo '[df_inodes_start]'
        # shellcheck disable=SC2086
        /usr/opt/freeware/bin/df -PTli ${excludefs} | sed 1d
        echo '[df_inodes_end]'
    else
        df -kP -T local | sed 's/ / - /' | grep -v ^/proc | grep -v ^Filesystem | grep -v :
    fi
}

The ‘_unused’ modification prevents the agent to use the /opt/freeware/bin/df which is some kind of ‘unteachable’ in this way, so the ‘else’ is used instead.
The ‘-T local’ is exactly what we need at this place. The ‘grep -v :’ at the end is most probably way of unnecessary.

2. The nfs section i modified this way:

section_nfs_mounts() {
    # Check for hanging NFS mounts. This needs a GNU stat installed in the PATH
    json_templ() {
        echo '{"mountpoint": "'"${1}"'", "source": "'"${2}"'", "state": "ok", "mount_seems_okay": "true"}'
    }
    json_templ_empty() {
        echo '{"mountpoint": "'"${1}"'", "source": "'"${2}"'", "state": "hanging", "mount_seems_okay": "false"}'
    }
    if inpath stat; then
        echo '<<<nfsmounts_v2:sep(0)>>>'
        mount | grep ' nfs' | grep -v -e rubrik -e unwanted_2 -e unwanted_3 -e etc_etc_etc | while read -r HN MD MP _; do
            waitmax 2 stat -f -c "'$(json_templ "${MP}" "${HN}:${MD}")'" "${MP}" || json_templ_empty "${MP}" "${HN}:${MD}"
        done
        echo '<<<cifsmounts>>>'
        mount | grep ' cifs' | while read -r _ _ MP _; do
            if [ ! -r "${MP}" ]; then
                echo "${MP} Permission denied"
            else
                waitmax 2 stat -f -c '"'"${MP}"' ok - - - -"' "${MP}" ||
                    echo "${MP} hanging 0 0 0 0"
            fi
        done
    fi
}

As you can see i have simply inserted a ‘| grep -v -e unwanted_mountpoint’ statement

These two modification should be enough, but - again - this is not update save.
I wish there would be an option to configure sections run asyncronously in the background instead of disabling them in /etc/check_mk/exclude_sections.cfg or modifying the agent every time a new one is created by the cmk devs.

Perhaps this find a way to a further version of cmk

br
christian

[mike1098 (Michael FRANK)]

Unfortunately there is not so much interest in AIX because ‘rarely’ used. We have only about 200 Hosts in our active monitoring.

Here is how we fixed the issue with stale hard mounts:

echo '<<<df>>>'
if [ -x /usr/opt/freeware/bin/df ] ; then
    excludefs="-x smbfs -x cifs -x iso9660 -x udf -x nfsv4 -x nfs -x mvfs -x zfs -x cdrfs"
    # shellcheck disable=SC2086
    /usr/opt/freeware/bin/df -PTlk $excludefs | sed 1d

    # df inodes information
    echo '<<<df>>>'
    echo '[df_inodes_start]'
    # shellcheck disable=SC2086
    /usr/opt/freeware/bin/df -PTli $excludefs | sed 1d
    echo '[df_inodes_end]'
else
    #MF, Jan 2022, Fixed issue with stale NFS hard mounts
    df -kP -T local | sed 's/ / - /' | grep -v ^/proc | grep -v ^Filesystem | grep -v :
fi

# Check for hanging NFS mounts.
# Modified version using df instead stat because stat is not
# available in AIX
# Januar 2022
# 
echo '<<<nfsmounts>>>'
mount | grep ' nfs' | awk '{print $3;}' | \
while read MP
do
    waitmax 1 df -k ${MP} > /dev/null 2>&1
        if [ "$?" -eq 0 ]; then
            df -k ${MP} | sed 1d | awk '{print "'$MP' ok",$2,$3,$3,"1024";}'
        else
            echo "${MP} hanging 0 0 0 0"
        fi
done

Might be dirty hack but works since couple of years. I am on the way to review AIX agent to see the differences made in official version.

regards

Michael