I would stay with a single host in your case if possible. Splitting into several sites and multisite-architecture brings new problems (Piggyback, Eventconsole, …) and your hardware should be capable to handle the load.
Right now we have around 2000 devices per VM before we add another one. One could add more CPU/memory but activating changes just takes too long with big sites.
Have a look at the used helper-processes to tune them if neccessary.
EDIT: Sorry, just reread your post and saw that you want to use only one site anyway. But if your hosts go stale it may be the helper-processes are exhausted.