Proxmox VE special agent fails with JSONDecodeError after host system ran out of disk space

Hi everyone,

Here is a final update with the definitive, verified solution for the issue with the Proxmox Special Agent.

The root cause was a specific bug in the Checkmk (2.4.0p10) agent caching framework.

The exact chain of events:

  1. Missing Cache File: For some reason (e.g., after a restart or on the initial run), the agent could not find the expected cache file at ~/tmp/check_mk/json_cache/.
  2. Faulty Read Attempt: The agent’s caching logic first tries to read a potential cache file before querying the API.
  3. The Bug: Since no file was found, this read attempt fails. Due to a bug, the responsible function in misc.py incorrectly returns an empty string ("") instead of a valid JSON default ("{}").
  4. The Crash: The attempt to parse this empty string (json.loads("")) inevitably leads to the known JSONDecodeError. The agent’s write routine, which would have correctly created the directory and file at the end, is never reached.

The Solution: Fixing the Bug with a Patch

The only required action is to patch the faulty function in the Checkmk library. Manually creating directories is not necessary.

File: /omd/versions/default/lib/python3/cmk/special_agents/v0_unstable/misc.py

Procedure: Find the original line (around line 369):

cache = json.loads(storage.read(key=storage_key, default="{}"))

And replace this single line with the following three lines, which handle the faulty empty string:

# --- BEGIN PATCH ---
raw_from_storage = storage.read(key=storage_key, default="{}")
if not raw_from_storage:
    raw_from_storage = "{}"
cache = json.loads(raw_from_storage)
# --- END PATCH ---

This patch prevents the read attempt from crashing. This allows the agent to complete its work and, at the end, call the correct write routine, which then creates the cache directory and files as intended.

Everything is running stable after applying the patch. It would be great if a developer could look into this and have the problem fixed in p11.

1 Like