I would say it is expected as the host still has this label.
You may be better off with setting a service level in the ruleset “Service Level of services” resp. “Service Level of hosts” based on the labels you assigned to the hosts and services.
This way you only have one attribute that can be matched in the notification rule.
Service levels can be defined in the global settings and do not have to be silver, gold and platinum. They take a numeric value and the notification rule can match on a range of these values.