Special_proxmox_ve JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Troubleshooting
, ,
1

CMK version: RAW 2.4.0p14
OS version: Debian 12.12

Error message:

[agent] Success, [special_proxmox_ve] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Output of “./share/check_mk/agents/special/agent_proxmox_ve --debug --verbose”:

OMD[poller1]:~/share/check_mk/agents/special$ cd
OMD[poller1]:~$ ./share/check_mk/agents/special/agent_proxmox_ve --debug --verbose  --username 'checkmk@pve' --password hunter2 --port 8006 --no-cert-check 'pveclus-node01'
INFO 2025-10-29 14:40:11 root: running file /omd/sites/poller1/lib/python3/cmk/special_agents/v0_unstable/agent_common.py
INFO 2025-10-29 14:40:11 root: using Python interpreter v3.12.11.final.0 at /omd/sites/poller1/bin/python3
INFO 2025-10-29 14:40:11 agent_proxmox_ve: Establish connection to Proxmox VE host 'pveclus1-node01.infra.xantion.net'
INFO 2025-10-29 14:40:11 agent_proxmox_ve: Fetch general cluster and node information..
INFO 2025-10-29 14:40:35 agent_proxmox_ve: Fetch and process backup logs..
Traceback (most recent call last):
  File "/opt/omd/sites/poller1/./share/check_mk/agents/special/agent_proxmox_ve", line 11, in <module>
    sys.exit(main())
             ^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 901, in main
    return special_agent_main(parse_arguments, agent_proxmox_ve_main)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 180, in special_agent_main
    return _special_agent_main_core(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 151, in _special_agent_main_core
    return main_fn(args)
           ^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 480, in agent_proxmox_ve_main
    logged_backup_data = fetch_backup_data(args, session, data["nodes"])
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 401, in fetch_backup_data
    with JsonCachedData(
         ^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/v0_unstable/misc.py", line 369, in JsonCachedData
    cache = json.loads(storage.read(key=storage_key, default="{}"))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3.12/json/decoder.py", line 338, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3.12/json/decoder.py", line 356, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This is a three node Proxmox Cluster, node 2 and 3 have no issues. Is there a way to just dump the json to validate / troubleshoot?

edit: i moved the VMs in batches and the error does not seem to related to a specific VM

2

Log in to your CheckMK machine as the site’s user (poller1 going by your pasted output) & run the special agent manually:

~/lib/python3/cmk/special_agents/agent_proxmox_ve.py --debug --verbose --port 8006 --username THE-USERNAME --password THE-PASSWORD --no-cert-check THE-HOST-IP

Replace all the THE-… thingies with your actual values. See what that outputs.

3

the same as above:

OMD[poller1]:~$ ~/lib/python3/cmk/special_agents/agent_proxmox_ve.py  --debug --verbose  --username 'checkmk@pve' --hunter2 --port 8006 --no-cert-check --timeout 60 'pveclus-node01'
INFO 2025-10-29 15:45:24 root: running file /omd/sites/poller1/lib/python3/cmk/special_agents/v0_unstable/agent_common.py
INFO 2025-10-29 15:45:24 root: using Python interpreter v3.12.11.final.0 at /omd/sites/poller1/bin/python3
INFO 2025-10-29 15:45:24 agent_proxmox_ve: Establish connection to Proxmox VE host 'pveclus1-node01.infra.xantion.net'
INFO 2025-10-29 15:45:24 agent_proxmox_ve: Fetch general cluster and node information..
INFO 2025-10-29 15:45:48 agent_proxmox_ve: Fetch and process backup logs..
Traceback (most recent call last):
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 905, in <module>
    sys.exit(main())
             ^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 901, in main
    return special_agent_main(parse_arguments, agent_proxmox_ve_main)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 180, in special_agent_main
    return _special_agent_main_core(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 151, in _special_agent_main_core
    return main_fn(args)
           ^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 480, in agent_proxmox_ve_main
    logged_backup_data = fetch_backup_data(args, session, data["nodes"])
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/agent_proxmox_ve.py", line 401, in fetch_backup_data
    with JsonCachedData(
         ^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3/cmk/special_agents/v0_unstable/misc.py", line 369, in JsonCachedData
    cache = json.loads(storage.read(key=storage_key, default="{}"))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3.12/json/decoder.py", line 338, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omd/sites/poller1/lib/python3.12/json/decoder.py", line 356, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
4

i might have found a clue;

OMD[poller1]:~$ ls -al ~/tmp/check_mk/data_source_cache/special_proxmox_ve/
total 296
drwxrwxr-x 2 poller1 poller1    120 Oct 29 15:50 ./
drwxrwxr-x 5 poller1 poller1    100 Oct 27 09:46 ../
-rw-rw---- 1 poller1 poller1 145643 Oct 29 15:49 pveclus-node02
-rw-rw---- 1 poller1 poller1 145641 Oct 29 15:49 pveclus-node03
OMD[poller1]:~$

Not sure if this is a result of, or the cause.

5

The special agent uses the PVE API to fetch information & converts it into an intermediate format. The CheckMK core is the one calling the special agent, taking its output & if the special agent call succeeds writing said output into the data source cache file. The special agent itself doesn’t write to the file.

What isn’t working is the API calls themselves, at least that’s what it looks like. The special agent sends a request & expects the API to reply in JSON, which the special agent then tries to decode. That the error occurs in the first column of the first line indicates that the answer is in fact not JSON.

Luckily the special agent has something for debugging these types of issues: the tracefile. Add the following args to your command line: --tracefile ~/tmp/debug/pve_debug1 Afterwards take a look at that file. You can paste it here, too, for more input, but be aware that it’ll include credentials & potentially other sensitive data which you should remove before posting.

1 Like
6

That’s some serious debugging. 20k lines. I guess the API is responding. Here is tail of the trace file

...
- request:
    body: null
    headers:
      Accept-Encoding:
      - gzip, deflate
      CSRFPreventionToken:
      - 6902334F:hfnRQ+2WKB9p4ym5sbMdP2H8uWr/N+fOZxYx8ypHj8A
      Connection:
      - keep-alive
      Cookie:
      - PVEAuthCookie=PVE:checkmk@pve:6902334F::<SNIP>
      User-Agent:
      - python-requests/2.32.4
      accept:
      - application/json, application/x-javascript, text/javascript, text/x-javascript,
        text/x-json
    method: GET
    uri: https://pveclus-node01:8006/api2/json/nodes/pveclus-node02/version
  response:
    body:
      string: '{"data":{"release":"9.0","version":"9.0.10","repoid":"deb1ca707ec72a89"}}'
    headers:
      Cache-Control:
      - max-age=0
      Connection:
      - Keep-Alive
      - Keep-Alive
      Content-Length:
      - '73'
      Content-Type:
      - application/json;charset=UTF-8
      Date:
      - Wed, 29 Oct 2025 15:32:14 GMT
      Expires:
      - Wed, 29 Oct 2025 15:32:14 GMT
      Pragma:
      - no-cache
      Server:
      - pve-api-daemon/3.0
    status:
      code: 200
      message: OK
version: 1
7

This is the real problem - your backup data.
Normally your Proxmox should return valid JSON, but here in your case it looks like the JSON is broken or has some other parsing problems.

Inside the trace file you should search for the backup data.

1 Like
8

Hi!
Has there been a solution to your problem? I started having the same randomly a few days ago.

Cheers!

9

Seems similar to

…which has been fixed by @MarcelIT but seem the fix was never acknowledged, as it persists in p17. Unluckily, changes in misc.py do not survive the container restart, so i can’t test that one.

Btw. all the data needed is saved in “~/tmp/check_mk/data_source_cache/special_proxmox_ve”, so it is not a problem of the API or its response, but parsing that one.

@andreas-doehler i wanted to ask you last week, but i forgot :confused:

10

After the second sleepless night, i backed up my data, recreated the container from scratch and recovered the data. Working now, but i feel lucky that my customer isn’t yet using proxmox :slight_smile:

11

Where can i find this fix then?

12

Just follow the link i provided exactly above the quoted sentence.

13

Just had the same problem (appeared on 2025-12-28 but.. holidays) and applied the linked fix - which worked.

After updating to 2.4.0.p18 (from p17) the error also didn’t reappear although the file didn’t include the patch.

I don’t want to have to apply this after every update though. Any chance this will be included in 2.4.0p19? It seems the error could just occur again at any time?

14

This is fixed in 2.4.0p20

https://checkmk.com/werk/19100

15

Just as nice as the veeam v13 fix in the same version :+1:t2:

1 Like