Hi everyone,
I am running Checkmk Raw Edition 2.4.0p32 in Docker on an Ubuntu server.
The monitored host is the Docker host itself, reachable in Checkmk as host.docker.internal. I have an ext4 mount at /mnt/volume2. After deleting ca. 9 GB of data below a directory on that filesystem, the OS clearly shows that disk space was freed, but the Checkmk filesystem service still shows the old usage.
What Checkmk currently shows in the UI / service output:
Service: Filesystem /mnt/volume2, host.docker.internal
Used: 61.02% - 119 GiB of 196 GiB
trend per 1 day 0 hours: +10.6 GiB
trend per 1 day 0 hours: +5.42%
Time left until disk full: 7 days 4 hours
Inodes used: 0.01%, Inodes available: 13,105,299 (99.99%)
On the host itself, df shows the freed space correctly:
df -hT /mnt/volume2
Output:
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdd ext4 196G 110G 77G 59% /mnt/volume2
The local Checkmk agent output on the host also reports the correct value:
sudo check_mk_agent | grep -A20 -E '^<<<df>>>|^<<<filesystem>>>|/mnt/volume2'
Relevant output:
/dev/sdd ext4 205314024 114787624 80024256 59% /mnt/volume2
...
/dev/sdd ext4 13107200 1901 13105299 1% /mnt/volume2
From inside the Checkmk container, direct agent access also returns the correct value:
docker exec -it check-mk-raw bash -lc 'su - cmk -c "cmk -d host.docker.internal | grep -E \"/mnt/volume2|^<<<df>>>|^<<<filesystem>>>\" -A2"'
Relevant output:
/dev/sdd ext4 205314024 114787636 80024244 59% /mnt/volume2
...
/dev/sdd ext4 13107200 1901 13105299 1% /mnt/volume2
However, when I execute the check manually, Checkmk still evaluates the old value:
docker exec -it check-mk-raw bash -lc 'su - cmk -c "cmk -nv host.docker.internal | grep -i -E \"Filesystem /mnt/volume2|/mnt/volume2|volume2\""'
Output:
Filesystem /mnt/volume2 Used: 61.02% - 119 GiB of 196 GiB, trend per 1 day 0 hours: +10.6 GiB, trend per 1 day 0 hours: +5.39%, Time left until disk full: 7 days 5 hours
Mount options of /mnt/volume2 Mount options exactly as expected
I also tried bypassing the agent cache:
docker exec -it check-mk-raw bash -lc 'su - cmk -c "cmk --no-cache -nv host.docker.internal | grep -i -E \"Filesystem /mnt/volume2|/mnt/volume2|volume2\""'
But the result was still the stale value:
Filesystem /mnt/volume2 Used: 61.02% - 119 GiB of 196 GiB, trend per 1 day 0 hours: +10.5 GiB, trend per 1 day 0 hours: +5.39%, Time left until disk full: 7 days 5 hours
Mount options of /mnt/volume2 Mount options exactly as expected
I then searched the Checkmk site files for /mnt/volume2:
docker exec -it check-mk-raw bash -lc 'su - cmk -c "find ~/var/check_mk ~/tmp/check_mk -type f 2>/dev/null | xargs grep -l \"/mnt/volume2\" 2>/dev/null | head -30"'
Relevant files found:
/omd/sites/cmk/tmp/check_mk/cache/host.docker.internal
/omd/sites/cmk/tmp/check_mk/counters/host.docker.internal
The raw cache contains the correct current value:
/omd/sites/cmk/tmp/check_mk/cache/host.docker.internal:34:/dev/sdd ext4 205314024 114787648 80024232 59% /mnt/volume2
But the counters file still contains this for /mnt/volume2:
[["host.docker.internal", "df", "/mnt/volume2"], {
"/mnt/volume2.delta": "(1782202073.0171146, 122353.3125)",
"/mnt/volume2.trend": "(1782157779.913484, 1782202073.0171146, 0.12490796558671166)"
}]
122353.3125 MiB is about 119 GiB, which matches the stale value shown by the service.
So it looks to me as if the agent data is correct, but the filesystem check is still using persisted counter/trend state for the used-space value.
My questions:
-
Is this expected behavior for the filesystem check in Checkmk?
-
Why would the service still show
119 GiBused when the current agent output reports about110 GiB(2 hours after deleting the data)? -
Is it safe to delete or reset the specific counter entry for this filesystem, or should this be handled differently?
-
Is there a recommended Checkmk command to reset only the stored state/trend/counter data for one filesystem service?
-
Could this be related to the filesystem trend calculation, or is it more likely stale service state?
I would like to avoid deleting too much state manually if there is a proper Checkmk way to reset this.
Thanks in advance for any guidance.