I’m missing things here.
What service and host rules did you create? I assume you placed the hosts the user should see in a folder? what settings did you use?
You dont need rules for this, this is wrong.
Create a folder for “group a” where you set the contact group that group a should use, the same for folder b
Set the permissions accordingly.
You only need rules when you want to override things, mostly on a service level (that is overkill for most implementation)