Using proxmox special agent without having access to port 8006

Yes, or OpenVPN or Wireguard.