i’m getting memory warn/critical messages on systems with ZFS because checkmk does not take zfs arc cache into consideration, i.e. it does not subtract memory occupied by arc cache from memory usage.
half of the memory is by default being used by zfs arc cache and can be reclaimed like ordinary buffers/caches
how can i solve this in a proper way ?
arcstat
time read miss miss% dmis dm% pmis pm% mmis mm% size c avail
10:45:48 20 0 0 0 0 0 0 0 0 118G 118G 12G
ARC cache is used RAM. The Memory check is reporting correctly here.
If ZFS would use filesystem cache the kernel would be able to report it separatedly.
but like filesystem/pagecache it can be freed with comand like echo 3 >/proc/sys/vm/drop_caches
so my question how i can avoid getting warn/critical messages on system where half of the ram is being used for processes/daemons and the other half being used for arc cache. with that, there is no need to get alarmed, because demanding processes can reclaim memory from arc like they can reclaim memory from ordinary buffers/caches.
indeed this is not checkmk’s fault that zfs arc is doing wrong - but at least i consider zfs important enough technology for which special case handling would be helpful, until this is resolved with zfs
really curious to see that in 2025 we still need to solve things people already struggled 15 years ago…even with major/well known monitoring platforms
I saw several oom problems using the memory with arch cache to the limit. This seems to be fixed now (Fix some signedness issues in arc_evict(). by amotin · Pull Request #14692 · openzfs/zfs · GitHub). Anyway, you cannot be sure as long as the cache is not properly marked as cache. Example: echo 3 >/proc/sys/vm/drop_caches may take a long time depending on your current arc cache size. As a consequence, the kernel may not be able to free up enough memory in time to avoid oom events if much memory is needed in a short amount of time.
Additionally, you can see that the arc cache is mainly allocated as “unreclaimable slab”.
Long story short: It’s not that easy. People struggeling 15 years with that problem may be a hint that the problem are not the monitoring tools but the monitored system.
As a consequence, the kernel may not be able to free up enough memory in time
to avoid oom events if much memory is needed in a short amount of time.
i have proxmox servers running for years and at least for 2 years for many of those the memory is always full >90% and i never have seen OOM conditions with VMs , as long most of the free memoy is being occupied by arc.
that’s what arc is being made for - to use the free memory for caching.
i’m not really happy with reducing arc to a “minimum reserved amount” to make monitoring happy.
I use this local service on my proxmox, I’m not 100% sure the logic is not flawed, but the results seem to line up with the sum of the “ram usage” (as reported by the proxmox gui) of all running VMs/LXCs. It calculates the total reclaimable memory inclusing the arc one, and then calculate the ram used by all apps as total memory minus reclaimable memory.
#!/bin/bash
# This local check provides an accurate memory usage reading for ZFS systems
# by MANUALLY calculating application memory usage. It reads the total memory
# and subtracts all known reclaimable caches, including the ZFS ARC.
# This avoids the known issue where the kernel's 'MemAvailable' metric
# does not correctly account for the ZFS ARC size.
# --- SCRIPT CONFIGURATION ---
WARN_LEVEL=80
CRIT_LEVEL=90
SERVICE_NAME="Memory_excluding_ZFS_Cache"
# --- SCRIPT LOGIC ---
MEMINFO_FILE="/proc/meminfo"
ZFS_STATS_FILE="/proc/spl/kstat/zfs/arcstats"
# Check if required files exist
if [[ ! -r "${MEMINFO_FILE}" ]]; then
echo "3 ${SERVICE_NAME} - UNKNOWN - Cannot read ${MEMINFO_FILE}"
exit 0
fi
# Read necessary values from /proc/meminfo in KB
MEM_TOTAL_KB=$(grep '^MemTotal:' ${MEMINFO_FILE} | awk '{print $2}')
MEM_FREE_KB=$(grep '^MemFree:' ${MEMINFO_FILE} | awk '{print $2}')
BUFFERS_KB=$(grep '^Buffers:' ${MEMINFO_FILE} | awk '{print $2}')
CACHED_KB=$(grep '^Cached:' ${MEMINFO_FILE} | awk '{print $2}')
SRECLAIMABLE_KB=$(grep '^SReclaimable:' ${MEMINFO_FILE} | awk '{print $2}')
# Initialize ARC size to 0. It will be updated only if ZFS stats are available.
ARC_KB=0
if [[ -r "${ZFS_STATS_FILE}" ]]; then
ARC_B=$(grep '^size ' "${ZFS_STATS_FILE}" | awk '{print $NF}')
ARC_KB=$((ARC_B / 1024))
fi
# Calculate the total amount of memory that is reclaimable (cache or free)
# This includes Free memory, Buffers, the Page Cache (Cached), Slab Reclaimable, AND the ZFS ARC.
RECLAIMABLE_KB=$((MEM_FREE_KB + BUFFERS_KB + CACHED_KB + SRECLAIMABLE_KB + ARC_KB))
# Calculate memory used by applications: Total Memory - All Reclaimable Memory
USED_BY_APPS_KB=$((MEM_TOTAL_KB - RECLAIMABLE_KB))
# Safety check: if calculation is weird, used memory can't be negative.
if (( USED_BY_APPS_KB < 0 )); then
USED_BY_APPS_KB=0
fi
# Calculate percentage
USED_PERCENT=$((USED_BY_APPS_KB * 100 / MEM_TOTAL_KB))
# Convert KB to Bytes for Checkmk performance data.
MEM_TOTAL_B=$((MEM_TOTAL_KB * 1024))
USED_B=$((USED_BY_APPS_KB * 1024))
WARN_B=$((MEM_TOTAL_B * WARN_LEVEL / 100))
CRIT_B=$((MEM_TOTAL_B * CRIT_LEVEL / 100))
# Determine the status code based on the thresholds.
STATUS_CODE=0 # OK
STATUS_TEXT="OK"
if (( USED_PERCENT >= CRIT_LEVEL )); then
STATUS_CODE=2 # CRITICAL
STATUS_TEXT="CRITICAL"
elif (( USED_PERCENT >= WARN_LEVEL )); then
STATUS_CODE=1 # WARNING
STATUS_TEXT="WARNING"
fi
# Format human-readable output.
USED_GIB=$(echo "scale=2; ${USED_BY_APPS_KB} / (1024*1024)" | bc)
TOTAL_GIB=$(echo "scale=2; ${MEM_TOTAL_KB} / (1024*1024)" | bc)
SUMMARY="${STATUS_TEXT} - App Memory Usage: ${USED_PERCENT}% (${USED_GIB}GiB of ${TOTAL_GIB}GiB)"
# Format performance data for graphs.
PERFDATA="usage=${USED_B};${WARN_B};${CRIT_B};0;${MEM_TOTAL_B}"
echo "${STATUS_CODE} ${SERVICE_NAME} ${PERFDATA} ${SUMMARY}"
exit 0
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.
