Multiple btrfs-subvolumes from same partition not recognized properly in monitoring data

KomtGoed · January 1, 2026, 4:39pm

Hi,

(Is this the right subforum for this kind of questions?)

I’m using the raw version of 2.4.0p18.

The agent is installed on my file server which has the following fstab for an HDD.

UUID=d4f0cb29-ca51-4188-aab9-3884d480e7f1 /var/samba/share1 btrfs defaults,subvol=@share1 0 0
UUID=d4f0cb29-ca51-4188-aab9-3884d480e7f1 /var/samba/share2 btrfs defaults,subvol=@share2 0 0
UUID=d4f0cb29-ca51-4188-aab9-3884d480e7f1 /var/samba/share3 btrfs defaults,subvol=@share3 0 0
UUID=d4f0cb29-ca51-4188-aab9-3884d480e7f1 /var/samba/share4 btrfs defaults,subvol=@share4 0 0

Share4 was recognized during a previous service discovery, all good.

However, I am getting the following error now:

Mount options of /var/samba/share4 –> Item not found in monitoring data

However, I do see share4 included with the same data points as e.g. share1 when checking the output of check_mk_agent.

Similar behavior for another HDD with more than two btrfs-subvolumes.

Interestingly,

Two other HDD with just one respetively two subvolumes and

b) the main SSD with multiple subvolumes

Are properly included in the monitoring data.

Any ideas on what’s happening?

Thanks a lot in advance!

KomtGoed · January 2, 2026, 11:20am

Hi,

To add to this topic: similar issue with check_mk_agent (through SSH) on a HAOS (Home Assistant) instance.

Yesterday, mount options of /homeassistant were discovered.

After a reboot tonight, I am getting the same “Item not found in monitoring data” error.

Output of df on that machine is as follows. Difference with yesterday could be (don’t fully remember though) that /homeassistant was displayed on a earlier row when invoking df).

Filesystem 1K-blocks Used Available Use% Mounted on
overlay 32284132 8687140 22246040 28% /
/dev/xvda8 32284132 8687140 22246040 28% /ssl
/dev/xvda8 32284132 8687140 22246040 28% /media
/dev/xvda8 32284132 8687140 22246040 28% /share
/dev/xvda8 32284132 8687140 22246040 28% /backup
devtmpfs 1514888 0 1514888 0% /dev
tmpfs 1512788 0 1512788 0% /dev/shm
/dev/xvda8 32284132 8687140 22246040 28% /homeassistant
/dev/xvda8 32284132 8687140 22246040 28% /addon_configs
/dev/xvda8 32284132 8687140 22246040 28% /data
/dev/xvda8 32284132 8687140 22246040 28% /addons
/dev/xvda8 32284132 8687140 22246040 28% /run/audio
tmpfs 1512788 0 1512788 0% /dev/shm
tmpfs 605116 1256 603860 0% /run/dbus
/dev/xvda8 32284132 8687140 22246040 28% /etc/asound.conf
/dev/xvda8 32284132 8687140 22246040 28% /run/cid
/dev/xvda8 32284132 8687140 22246040 28% /etc/resolv.conf
/dev/xvda8 32284132 8687140 22246040 28% /etc/hostname
/dev/xvda8 32284132 8687140 22246040 28% /etc/hosts
/dev/xvda8 32284132 8687140 22246040 28% /var/log/journal
tmpfs 605116 1256 603860 0% /run/log/journal
/dev/xvda8 32284132 8687140 22246040 28% /etc/pulse/client.conf
tmpfs 1512788 0 1512788 0% /proc/asound
devtmpfs 1514888 0 1514888 0% /proc/interrupts
devtmpfs 1514888 0 1514888 0% /proc/kcore
devtmpfs 1514888 0 1514888 0% /proc/keys
devtmpfs 1514888 0 1514888 0% /proc/timer_list
tmpfs 1512788 0 1512788 0% /proc/scsi
tmpfs 1512788 0 1512788 0% /sys/firmware

KomtGoed · January 2, 2026, 11:25am

Output of check_mk_agent on the HAOS host:

<<<check_mk>>>
Version: 2.4.0p18
AgentOS: linux
Hostname: a0d7b954-ssh
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
OSType: linux
OSName: Alpine Linux
OSPlatform: alpine
OSVersion: 3.22.2
FailedPythonReason:
SSHClient:
<<<checkmk_agent_plugins_lnx:sep(0)>>>
pluginsdir /usr/lib/check_mk_agent/plugins
localdir /usr/lib/check_mk_agent/local
<<<labels:sep(0)>>>
{"cmk/device_type":"container"}
<<<nfsmounts_v2:sep(0)>>>
<<<cifsmounts>>>
<<<mounts>>>
/dev/xvda8 /ssl ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /media ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /share ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /backup ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /homeassistant ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /addon_configs ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /data ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /addons ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /run/audio ext4 ro,relatime,commit=30 0 0
/dev/xvda8 /etc/asound.conf ext4 ro,relatime,commit=30 0 0
/dev/xvda8 /run/cid ext4 ro,relatime,commit=30 0 0
/dev/xvda8 /etc/resolv.conf ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /etc/hostname ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /etc/hosts ext4 rw,relatime,commit=30 0 0
/dev/xvda8 /var/log/journal ext4 ro,relatime,commit=30 0 0
/dev/xvda8 /etc/pulse/client.conf ext4 ro,relatime,commit=30 0 0
<<<ps_lnx>>>
[time]
1767352878
[processes]
[header] CGROUP USER VSZ RSS TIME ELAPSED PID COMMAND
- root 440 100 00:00:00 09:16:33 1 /package/admin/s6/command/s6-svscan -d4 -- /run/service
- root 220 80 00:00:00 09:16:32 19 s6-supervise s6-linux-init-shutdownd
- root 208 60 00:00:00 09:16:32 21 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -d3 -c /run/s6/basedir -g 3000 -C -B
- root 220 76 00:00:00 09:16:32 30 s6-supervise ttyd
- root 220 72 00:00:00 09:16:32 31 s6-supervise s6rc-fdholder
- root 220 76 00:00:00 09:16:32 32 s6-supervise sshd
- root 220 76 00:00:00 09:16:32 33 s6-supervise s6rc-oneshot-runner
- root 208 68 00:00:00 09:16:32 41 /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --
- root 6704 4696 00:00:01 09:16:31 305 sshd: /usr/sbin/sshd -D -e [listener] 0 of 10-100 startups
- root 7884 2844 00:00:00 09:16:31 308 ttyd -d1 -i hassio --writable -p 64851 tmux -u new -A -s homeassistant zsh -l
- root 7168 5112 00:00:00 08:33 73224 sshd-session: hassio [priv]
- hassio 7468 4032 00:00:00 08:33 73226 sshd-session: hassio@pts/0
- root 1936 1504 00:00:00 08:32 73227 sudo -i
- root 1936 852 00:00:00 08:32 73229 sudo -i
- root 4612 4336 00:00:02 08:32 73230 -zsh
- root 3480 2836 00:00:00 00:00 75605 /bin/bash ./check_mk_agent
- root 3496 2460 00:00:00 00:00 75622 /bin/bash ./check_mk_agent
- root 3452 2388 00:00:00 00:00 75624 /bin/bash ./check_mk_agent
- root 1688 928 00:00:00 00:00 75626 cat
- root 3496 2340 00:00:00 00:00 75641 /bin/bash ./check_mk_agent
- root 2844 1908 00:00:00 00:00 75642 ps ax -ww -o cgroup:512,user:32,vsz,rss,cputime,etime,pid,command
- root 1628 844 00:00:00 00:00 75643 tr -s
<<<docker_container_mem_cgroupv2>>>
anon 8572928
file 41893888
kernel 16797696
kernel_stack 344064
pagetables 864256
sec_pagetables 0
percpu 960
sock 0
vmalloc 0
shmem 0
zswap 0
zswapped 0
file_mapped 5906432
file_dirty 4096
file_writeback 0
swapcached 0
inactive_anon 0
active_anon 8548352
inactive_file 9936896
active_file 31956992
unevictable 0
slab_reclaimable 14833896
slab_unreclaimable 519960
slab 15353856
workingset_refault_anon 0
workingset_refault_file 0
workingset_activate_anon 0
workingset_activate_file 0
workingset_restore_anon 0
workingset_restore_file 0
workingset_nodereclaim 0
pgdemote_kswapd 0
pgdemote_direct 0
pgdemote_khugepaged 0
pgscan 0
pgsteal 0
pgscan_kswapd 0
pgscan_direct 0
pgscan_khugepaged 0
pgsteal_kswapd 0
pgsteal_direct 0
pgsteal_khugepaged 0
pgfault 13460159
pgmajfault 287
pgrefill 0
pgactivate 0
pgdeactivate 0
pglazyfree 0
pglazyfreed 0
swpin_zero 0
swpout_zero 0
zswpin 0
zswpout 0
zswpwb 0
memory.current 67325952
memory.max max
MemTotal:        3025580 kB
<<<docker_container_cpu_cgroupv2>>>
uptime 33406.87 31688.80
num_cpus 1
usage_usec 110325155
user_usec 57559752
system_usec 52765403
nice_usec 4000
nr_periods 0
nr_throttled 0
throttled_usec 0
nr_bursts 0
burst_usec 0
<<<uptime>>>
33404
<<<lnx_if>>>
[start_iplink]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enX0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    link/ether 00:16:3e:f2:93:d2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.2/24 brd 192.168.200.255 scope global dynamic noprefixroute enX0
       valid_lft 571403sec preferred_lft 571403sec
    inet6 fe80::188f:a3de:9a13:e9cc/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 0e:2d:40:0c:28:ed brd ff:ff:ff:ff:ff:ff
    inet 172.30.232.1/23 brd 172.30.233.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::c2d:40ff:fe0c:28ed/64 scope link
       valid_lft forever preferred_lft forever
4: hassio: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 66:03:e9:e6:e3:d0 brd ff:ff:ff:ff:ff:ff
    inet 172.30.32.1/23 brd 172.30.33.255 scope global hassio
       valid_lft forever preferred_lft forever
    inet6 fe80::6403:e9ff:fee6:e3d0/64 scope link
       valid_lft forever preferred_lft forever
5: veth9f14b13@enX0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master hassio state UP
    link/ether 9e:b5:c6:4a:a6:07 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9cb5:c6ff:fe4a:a607/64 scope link
       valid_lft forever preferred_lft forever
6: vethdccf4dd@enX0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP
    link/ether 16:d4:ab:8b:3b:da brd ff:ff:ff:ff:ff:ff
    inet6 fe80::14d4:abff:fe8b:3bda/64 scope link
       valid_lft forever preferred_lft forever
7: veth1192fe3@docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master hassio state UP
    link/ether e2:5b:14:39:b6:5e brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e05b:14ff:fe39:b65e/64 scope link
       valid_lft forever preferred_lft forever
8: veth26f9732@enX0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master hassio state UP
    link/ether c2:32:1e:ab:93:38 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c032:1eff:feab:9338/64 scope link
       valid_lft forever preferred_lft forever
9: veth2f9876b@enX0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master hassio state UP
    link/ether 62:87:91:32:8b:21 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::6087:91ff:fe32:8b21/64 scope link
       valid_lft forever preferred_lft forever
10: veth7f10837@enX0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master hassio state UP
    link/ether ae:41:62:62:7d:a4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ac41:62ff:fe62:7da4/64 scope link
       valid_lft forever preferred_lft forever
11: veth6c16f62@enX0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master hassio state UP
    link/ether 9a:3a:2a:0a:3b:d0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::983a:2aff:fe0a:3bd0/64 scope link
       valid_lft forever preferred_lft forever
12: veth0788af3@enX0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master hassio state UP
    link/ether de:79:da:c4:3c:8b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::dc79:daff:fec4:3c8b/64 scope link
       valid_lft forever preferred_lft forever
13: veth027ce25@enX0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master hassio state UP
    link/ether da:66:f9:5d:82:62 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::d866:f9ff:fe5d:8262/64 scope link
       valid_lft forever preferred_lft forever
14: vethe64e8f5@enX0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master hassio state UP
    link/ether b2:25:63:a3:88:6a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b025:63ff:fea3:886a/64 scope link
       valid_lft forever preferred_lft forever
[end_iplink]
<<<lnx_if:sep(58)>>>
    lo: 2984551   12424    0    0    0     0          0         0  2984551   12424    0    0    0     0       0          0
  enX0: 680135059 1342789    0    0    0     0          0         0 118169041 1316766    0    0    0     0       0          0
docker0:  200916     917    0    0    0     0          0         0   473451     861    0    2    0     0       0          0
hassio: 67669145  289636    0    0    0     0          0         0 227576053  272916    0    2    0     0       0          0
veth9f14b13: 1661580    9177    0    0    0     0          0         0  3983615   19155    0    0    0     0       0          0
vethdccf4dd:  213754     917    0    0    0     0          0         0   476106     876    0    0    0     0       0          0
veth1192fe3: 497466963  268976    0    0    0     0          0         0 228316350  282805    0    0    0     0       0          0
veth26f9732:    2022      14    0    0    0     0          0         0  1499251   10151    0    0    0     0       0          0
veth2f9876b: 3640307   34960    0    0    0     0          0         0  4360914   34495    0    0    0     0       0          0
veth7f10837:     126       3    0    0    0     0          0         0  1495791   10141    0    0    0     0       0          0
veth6c16f62:    4258      40    0    0    0     0          0         0  1501626   10150    0    0    0     0       0          0
veth0788af3: 3791125   19915    0    0    0     0          0         0  1527561   10891    0    0    0     0       0          0
veth027ce25: 3162834   33280    0    0    0     0          0         0 433349740   42817    0    0    0     0       0          0
vethe64e8f5:    2897      28    0    0    0     0          0         0  1481946   10083    0    0    0     0       0          0
[lo]
Address: 00:00:00:00:00:00
[enX0]
Address: 00:16:3e:f2:93:d2
[docker0]
Speed: 10000Mb/s
Address: 0e:2d:40:0c:28:ed
[hassio]
Speed: 10000Mb/s
Address: 66:03:e9:e6:e3:d0
[veth027ce25]
Speed: 10000Mb/s
Address: da:66:f9:5d:82:62
[veth0788af3]
Speed: 10000Mb/s
Address: de:79:da:c4:3c:8b
[veth1192fe3]
Speed: 10000Mb/s
Address: e2:5b:14:39:b6:5e
[veth26f9732]
Speed: 10000Mb/s
Address: c2:32:1e:ab:93:38
[veth2f9876b]
Speed: 10000Mb/s
Address: 62:87:91:32:8b:21
[veth6c16f62]
Speed: 10000Mb/s
Address: 9a:3a:2a:0a:3b:d0
[veth7f10837]
Speed: 10000Mb/s
Address: ae:41:62:62:7d:a4
[veth9f14b13]
Speed: 10000Mb/s
Address: 9e:b5:c6:4a:a6:07
[vethdccf4dd]
Speed: 10000Mb/s
Address: 16:d4:ab:8b:3b:da
[vethe64e8f5]
Speed: 10000Mb/s
Address: b2:25:63:a3:88:6a
<<<tcp_conn_stats>>>
01 19
06 58
0A 17
<<<docker_container_diskstat_cgroupv2>>>
[time]
1767352878
[io.stat]
202:0 rbytes=41234432 wbytes=851968 rios=1983 wios=154 dbytes=0 dios=0
[names]
loop0 7:0
loop1 7:1
loop2 7:2
loop3 7:3
loop4 7:4
loop5 7:5
loop6 7:6
loop7 7:7
xvda 202:0
zram0 252:0
zram1 252:1
zram2 252:2
<<<md>>>
Personalities :
unused devices: <none>
<<<vbox_guest>>>

BH2005 · March 2, 2026, 9:15am

Hi @KomtGoed

Both issues you describe have the same root cause: the mounts check in Checkmk identifies mount entries primarily by the device name/UUID, and when the same device appears multiple times in the agent output (same UUID for btrfs subvolumes, same /dev/xvda8 for HAOS), only the first occurrence is reliably tracked. Additional mountpoints from the same device are either dropped or become unstable because their position in the output can change between reboots.

For the btrfs subvolumes case:
Btrfs subvolumes all share the same underlying device UUID. The agent reports them correctly, but the check deduplicates by device — so share4 (or whichever appears last) loses its entry. This is a known limitation of how the df/mounts check handles btrfs multi-subvolume setups.

Workarounds:

Use the df check instead of mounts — in some cases the disk usage check (df) handles multiple mountpoints from the same device better than the mount options check.
Ignore the “mount options” check, monitor via disk usage — if you only need to know that the shares are present and accessible, monitoring free space via df is more reliable than mount option checks for btrfs subvolumes.
File a bug report with Checkmk — this is arguably a bug. The check should use the mountpoint path as the primary identifier, not the device. Since the agent output clearly contains all subvolumes, the check should be able to handle them. But even if something looks like a bug, there is no effective way for us to handle incoming emails from feedback@checkmk.com in our development workflow (in contrast to our support ticket system).

For the HAOS case:
The position change of /homeassistant in df output after reboot confirms the same issue — the check tracks by device, and when the row order shifts, a different mountpoint “wins”. Same workaround applies: rely on disk usage monitoring rather than mount options for this host.

Hope that helps clarify what’s happening!

Greetz Bernd

KomtGoed · March 3, 2026, 9:26pm

Hi @BH2005 ,

Thanks for the elaborate response!

Two follow-up questions:

Regarding 1 and 2, to double check: This means in practice, disable the “mount options of XYZ” services and only rely e.g. on service “Filesystem /boot/efi”? (as in: this is what you mean with “use df”?)

As a regular mortal, I think I cannot open a bug report on github, the “issues” option I see at other projects is missing. Looking at this post, it seems that the forum is monitored for bug reports. Is that also your understanding?

Thanks again and cheers, KG

BH2005 · March 4, 2026, 5:47am

this was in the passt soory … too fast an d not good thooth …

But even if something looks like a bug, there is no effective way for us to handle incoming emails from feedback@checkmk.com in our development workflow (in contrast to our support ticket system).

this is the rigth way …

This means in practice, disable the “mount options of XYZ” services and only rely e.g. on service “Filesystem /boot/efi”? (as in: this is what you mean with “use df”?)

yes

Sara · March 4, 2026, 1:39pm

Hi @BH2005 and @KomtGoed !

To clarify, the forum is definitely still the place for users without a support contract to report bugs (I’d recommend using the Troubleshooting category for best visibility though).

I keep a close eye on the forum and for the bugs we can confirm (for example, things that we officially support that reproducibly do not work as intended) I create internal tickets, as long as there are clear steps to reproduce and strong confirmation that it is indeed a bug. There is a process though – for example, multiple reports from different users on the same issue would increase the priority. And another point from that post stands – unfortunately, we cannot give any promises on this.

In this case, I will create a ticket for the BTRFS issue, as it was confirmed. The case with HAOS might be different though, as it could be due to the structure that is not supported.

In short, please keep sharing potential bugs in the forum. Clear steps to reproduce and confirmation replies from others facing the same issue are helpful for getting things moving

BH2005 · March 4, 2026, 1:48pm

I have found also a other solution:

Werk #12383: df: The UUID can be used as part of the service item

this could be the way

Filesystem discovery rule: Go to Setup > Services > Discovery rules > Filesystem discovery and set “Mountpoint for block devices” to “UUID”. This ensures stable service naming regardless of mount order, and avoids services disappearing after a reboot. This option was introduced in Werk #12383.
Understand the root cause: Since Werk #2671, Checkmk intentionally creates only one filesystem service per btrfs device, because all subvolumes on the same device share the same storage pool and report identical usage statistics. This is why only the first mount point gets picked up and the others result in “Item not found in monitoring data” — especially after a reboot when the mount order may change.
Fix the “Mount options” services: Run a fresh service discovery (Setup > Hosts > your host > Service discovery) to pick up the currently visible mount points. For mount points that keep fluctuating, you can disable the affected “Mount options” services via Setup > Services > Disabled services to avoid recurring false alerts.

Greetz Bernd

Sara · March 4, 2026, 4:06pm

Well, I guess a solution came before the ticket this time

KomtGoed · March 4, 2026, 8:58pm

Thanks a lot @BH2005, great detective work!

Performed the changes. See what happens after the next reboots, I’ll report back over a few weeks.