CheckMK on TrueNAS Scale

ColinP · November 12, 2022, 12:44pm

CMK version: 2.1.0p15
OS version: TrueNAS SCALE 22.02.4

I installed the CheckMK Agent on a TrueNAS SCALE host. After the installation everything worked. After a reboot the cmk-agent-ctl-daemon and the check-mk-agent.socket failed.
I had to add the checkmk user again.
After that the service and the socket both seemed to work again but the daemon did not bind port 6556 so the agent did not work.

Any ideas how to get this working?

Doc · November 13, 2022, 8:24pm

Hi, I use ssh with the authorized_keys file to connect to the agent instead of systemd.
You can configure the authorized_keys file in the root user settings via the WebGUI.

ColinP · November 14, 2022, 6:45am

Thanks, great idea. Had to disable agent output encryption for this though. (Just wanted to note this here if anyone stumbles across this.)

Overlord · June 23, 2023, 1:02pm

How have you realized that the CheckMK is still installed after reboot?

ColinP · June 23, 2023, 1:45pm

it should still be installed, the user is missing though.

I know it’s a hacky solution and there’s probably a better one, but I added the following command as a pre init script: useradd -r -s /bin/false cmk-agent

After a TrueNAS update you also have to reinstall the agent and reregister for updates and TLS I think.

Maybe I’ll switch to just reinstalling the deb package in the future…

Overlord · August 31, 2023, 8:47am

Ok since an update or something it’s not really working anymore. After a reboot of the system CheckMK isn’t able anymore to get access to the system.

I tried with “sudo apt install --reinstall ./check-mk*” but the same - no connection.

Has anybody a good solution?

-Install is not possible / only reinstall without any effect
-check-mk-agent.socket fails - after creating the user it’s possible to start the service
-no open socket for 6556
-not able to register the agent (cmk_agent_ctl - No such file or directory)*

*it’s there with +x

Any ideas how to the the CheckMK service running again?

Doc · September 12, 2023, 11:14am

I use this script for installing and registering the agent. It is placed in my home folder “/mnt/zfs_pool_01/myhomedir/install_checkmk_agent_from_bakery_true_nas_scale.sh”

The script is executed as user Root. (login as user; then sudo su -; then execute the install script)

This:

apt -y install python3-pip
pip3 install docker

is possibly not working in the current version of TrueNAS Scale. The Debian repo is missing. If you want to see the docker values, you have to add the repo tempoarily, install the package and add the docker plugin to the agent.


install_checkmk_agent_from_bakery_true_nas_scale.sh
#!/bin/bash

#set -x

AUTOMATION_USER="automation"
AUTOMATION_PASS="xxxxxxx"

CMK_SERVER="xxxxxx"
CMK_SITE="xxxxxx"
CMK_PROTO="https"

REGISTRATION_USER="cmkadmin"
REGISTRATION_PASS='xxxxxxx'
REGISTER_PROTO="https"

HOSTNAME=`hostname -s`

CMK_URL="$CMK_PROTO://$CMK_SERVER/$CMK_SITE"

chmod +x /bin/apt*
chmod +x /usr/bin/dpkg

apt -y install curl

# for docker plugin
apt -y install python3-pip
pip3 install docker
# pip3 install --upgrade docker==5

rm -rf /tmp/downloaded_agent.deb

# host = hostname
HOST="$HOSTNAME"
curl --insecure -o /tmp/downloaded_agent.deb $CMK_URL/check_mk/download_agent.py -d "_username=$AUTOMATION_USER&_secret=$AUTOMATION_PASS&host=$HOST&os=linux_deb"

apt install /tmp/downloaded_agent.deb

# register agent
CMK_UPDATE_AGENT=`which cmk-update-agent`
$CMK_UPDATE_AGENT register -s $CMK_SERVER -i $CMK_SITE -H $HOSTNAME -p $REGISTER_PROTO -U $REGISTRATION_USER -P $REGISTRATION_PASS -v

# tls (optional - if I don't have to, I will not use tls)
#CMK_AGENT_CTL=`which cmk-agent-ctl`
#$CMK_AGENT_CTL register -s $CMK_SERVER -i $CMK_SITE -H $HOSTNAME -U $REGISTRATION_USER -P $REGISTRATION_PASS -v --trust-cert

$CMK_UPDATE_AGENT -v

And I have some Cronjobs in the TrueNAS Scale GUI defined. I’m unsure if they are still needed. They are running every minute.

a)
if [ ! -d /usr/lib/check_mk_agent ]; then ln -s /mnt/zfs_pool_01/checkmk/check_mk_agent /usr/lib/ > /dev/null 2> /dev/null; else echo ‘/usr/lib/check_mk_agent does exist’; fi

b)
chmod +x /bin/apt* ; chmod +x /usr/bin/dpkg

Overlord · September 13, 2023, 11:37am

Thank you, I will check this out. The issue I had where to activate CheckMK again after the reboot

Btw. do you use TrueNAS Plugins? I used some plugins but they are only compatible with Core - So I have to translate them.

Edit:
I made it - CheckMK access via SSH. And I wrote a script which Checks for TrueNAS Scale Updates.

Doc · October 6, 2023, 10:30pm

Sounds good. Would you post it here?

mschlenker · October 7, 2023, 8:24am

There is also the possibility to pipe the agent to the shell that is opened in the SSH connection. This works as long as no plugins are required:

Overlord · November 19, 2023, 12:10pm

github.com

47k/checkmk/blob/master/truenas/check_truenas_scale_version.sh

#!/bin/bash

# Version 0.1 - Manuel Michalski
# Date: 15. September 2023
# Description: Check TrueNAS Version (Scale)

### Variables ###
UPCHECK=$(midclt call update.check_available)
STATUS=$(echo "$UPCHECK" | jq -r '.status')
DEBUG=NO
################

# Run with sudo if we're not root
if (( EUID ))
then
    SUDO='/usr/local/bin/sudo'
fi

# Run the command and:
#  - Dump all output

This file has been truncated. show original

Overlord · May 30, 2024, 12:22pm

Hey guys

Since Dragonfish and the “read-only” problematic - any workaround to get the CheckMK agent running?

Greetz

ColinP · May 30, 2024, 12:45pm

I’m still on Cobia as 24.04.1 was just released a few days ago. Which read only problem?

Edit: I found what you mean, the boot pool is readonly. I think we’ll either have to use developer mode or try to set it to rw otherwise.

Another option would be to not use the standard Debian package but rather create a custom agent…that would disable agent updates though

ColinP · May 31, 2024, 6:36am

I took a closer look at the install-dev-tools command.

The interesting part for us is /usr/local/libexec/disable-rootfs-protection $FORCE_ARG.

/usr/local/libexec/disable-rootfs-protection is a python script which does a few things:

checks the license (if an enterprise license is used, the above-mentioned “FORCE_ARG” (–force) is needed)
looks up the dataset configuration (/conf/truenas_root_ds.json)
flags the root dataset as developer mode (truenas:developer=on)
sets all datasets listed in /conf/truenas_root_ds.jsonto rw
chmods apt and dpkg as executable

To install the checkmk agent we already chmod apt and dpkg so /usr/local/libexec/disable-rootfs-protection is exactly what we want for non-enterprise configurations.

I am personally using checkmk to monitor my home NAS so running this script is no problem for me.

For enterprise users who might need support for their TrueNAS installation this is no option of course…
In this case a minimal agent installed to some writable dataset and called via SSH would probably be a better solution.

system · May 31, 2025, 6:36am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.