Hi,
We are using special agent (in this case Prometheus and PromQL) for some services, once the services have been created we setup notifications.
However as the state of the service have already changed (OK->CRIT for example) we do not see any service notifications.
Is the best option here to create a Periodic notifications during service problems? Or should this essentially be picked up in the next check interval (even when state does not change)
If the service stays in the same (warn/crit) state then only periodic notification will trigger a new notification.
Other way to get one notification would be a “ok” fake check result.
What is better depends on your user case in the future.
we cannot of course create fake results…
not sure what you mean by that statement, I guess if the state have never trigged a notification this rule won’t help us.
our only way might be to create the notification rule before we create the hosts, but im still doubtful.
if you make a service discovery on a host that have problems, should not these get notifications oob?
Yes it generates a notification for the state transition from “Unknown” to “Crit”.
If there exists a rule that is respecting this transition a notification will be sent.
That is what I’m not sure about, sure for an OS agent that is how it works, the service discover will create unknown state until a scheduled check is performed (manual or automatically)
But for Prometheus the only way was to create the notification first, before creating the host, and running the service discovery. For us that’s a but awkward due to the way our automation works, but I think I need to spend more time on this 
I have no0 experience with the Prometheus integration, but the way I understand it, it behaves like every other special and OS agent when it comes to services. There should be no difference, hence what Andreas said should be true.