r.sander, you were right in everything 
The slowness came not from the vbox_guest section, but the one following it - timesyncd.
My case was also Debian 10 with ntp package installed. I noticed Debian 10 does not disable the systemd-timesyncd.service, but rather lets its start condition fail with ConditionFileIsExecutable=!/usr/sbin/ntpd was not met
Since I don’t need the “ntp server” part of ntpd, my solution was to uninstall ntp and let systemd-timesyncd.service do the clock synchronization.
Thanks for your help.