Hi,
during the work on the Proxmox VE Agent Plugin i reviewed the code of the Proxmox VE Special agent very closely.
The Backup information is determined from the logs and so first the tasks list is requested and vzdump logs afterwards. Task list is requested via get_tree already before processing backup logs. Getting logs is done in fetch_backup_log in libbackups.py. The function get_tree is used to retrieve the logs. But this will result in 5 unnecessary API requests FOR EACH LOG.
Due to caching this will not be done on every run and only if uncached logs are found.
Example: Requesting a log via API-Request to “/api2/json/nodes/NODENAME/tasks/UPID:NODENAME:001952F5:05783C58:6A752003:vzdump::root@pam:/log?start=0&limit=5000” will result in API requests to
- “/api2/json/”
- “/api2/json/nodes”
- “/api2/json/nodes/NODENAME”
- “/api2/json/nodes/NODENAME/tasks?start=0&limit=5000”
- “/api2/json/nodes/NODENAME/tasks/UPID:NODENAME:001952F5:05783C58:6A752003:vzdump::root@pam:”
- “/api2/json/nodes/NODENAME/tasks/UPID:NODENAME:001952F5:05783C58:6A752003:vzdump::root@pam:/log?start=0&limit=5000”
Especially the API request ending in “tasks/” returns the last 5000 tasks and can potentially be large and slow.
As fetch_backup_log only needs to fetch the logs you can use session.get instead of session.get_tree to just request the last API url in the list above without missing anything.
diff '--color=auto' -Naur a/libbackups.py b/libbackups.py
--- libbackups.py 2026-01-07 01:00:00.000000000 +0100
+++ libbackups.py 2026-08-07 15:55:41.647369718 +0200
@@ -429,7 +429,7 @@
) as cached:
def fetch_backup_log(task: TaskInfo, node: str) -> tuple[str, LogData]:
- """Make a call to session.get_tree() to get a log only if it's not cached
+ """Make a call to session.get() to get a log only if it's not cached
Note: this is just a closure to make the call below less complicated - it could
also be part of the generator"""
# todo: specify type, date in request
@@ -437,9 +437,7 @@
task["upid"],
lambda t=task, n=node: (
t["starttime"],
- session.get_tree({"nodes": {n: {"tasks": {t["upid"]: {"log": []}}}}})["nodes"][
- n
- ]["tasks"][t["upid"]]["log"],
+ session.get(["nodes", n, "tasks", t["upid"], "log"])
),
)
return timestamp, logs
Log before the change
"GET /api2/json/nodes/NODENAME/tasks?start=0&limit=5000 HTTP/1.1" 200 28397
"GET /api2/json/nodes/NODENAME/time HTTP/1.1" 200 78
"GET /api2/json/nodes/NODENAME/version HTTP/1.1" 200 72
"GET /api2/json/ HTTP/1.1" 200 131
"GET /api2/json/nodes HTTP/1.1" 200 340
"GET /api2/json/nodes/NODENAME HTTP/1.1" 200 806
"GET /api2/json/nodes/NODENAME/tasks?start=0&limit=5000 HTTP/1.1" 200 28419
"GET /api2/json/nodes/NODENAME/tasks/UPID:NODENAME:001952F5:05783C58:6A752003:vzdump::root@pam: HTTP/1.1" 200 43
"GET /api2/json/nodes/NODENAME/tasks/UPID:NODENAME:001952F5:05783C58:6A752003:vzdump::root@pam:/log?start=0&limit=5000 HTTP/1.1" 200 6070
"GET /api2/json/ HTTP/1.1" 200 131
"GET /api2/json/nodes HTTP/1.1" 200 340
"GET /api2/json/nodes/NODENAME HTTP/1.1" 200 806
"GET /api2/json/nodes/NODENAME/tasks?start=0&limit=5000 HTTP/1.1" 200 28489
"GET /api2/json/nodes/NODENAME/tasks/UPID:NODENAME:0033C8BC:04F4680D:6A73CE87:vzdump::root@pam: HTTP/1.1" 200 43
"GET /api2/json/nodes/NODENAME/tasks/UPID:NODENAME:0033C8BC:04F4680D:6A73CE87:vzdump::root@pam:/log?start=0&limit=5000 HTTP/1.1" 200 5341
Log after the change
"GET /api2/json/nodes/NODENAME/tasks?start=0&limit=5000 HTTP/1.1" 200 28452
"GET /api2/json/nodes/NODENAME/time HTTP/1.1" 200 78
"GET /api2/json/nodes/NODENAME/version HTTP/1.1" 200 72
"GET /api2/json/nodes/NODENAME/tasks/UPID:NODENAME:001952F5:05783C58:6A752003:vzdump::root@pam:/log?start=0&limit=5000 HTTP/1.1" 200 6093
"GET /api2/json/nodes/NODENAME/tasks/UPID:NODENAME:0033C8BC:04F4680D:6A73CE87:vzdump::root@pam:/log?start=0&limit=5000 HTTP/1.1" 200 5315