What I can see is that this happens only on Linux VMs in Proxmox with installed Agent.
But not on all, only on some.
Windows VM (with installed Agent) and HW (with installed Agent) do not have these issues.
For example the PVE hosts itself are not having this issue but the VMs running inside having it (all are Debian 13) and it is mostly just a little bit over the threshholds (mostly Jitter of 3s for example or sometimes an offset of 2-3 seconds).