CMK version: 2.4.0p29 (RAW) (likely all 2.4.x versions affected)**
OS version:** Ubuntu 24.04 (docker
Error message: “Unable to retrieve password from passwordstore”
Output of “cmk --debug -vvn hostname”: N/A — this is a notification dispatcher bug, not a check/plugin issue.
Summary
PagerDuty notification rules created via the REST API silently produce broken on-disk configuration. When the rule fires, the notification dispatcher exits with:
Unable to retrieve password from passwordstore
Rules created via the web UI work correctly. Rules created via the REST API fail every time.
Root cause
The on-disk format written by the REST API does not match what the notification dispatcher expects.
APIPagerDutyKeyOption.to_mk_file_format() in cmk/gui/rest_api_types/notifications_rule_types.py (line ~2524) writes the integration key as:
("routing_key", "<the-actual-key>")
This is the legacy 2.3 format. Elsewhere in the same file (e.g. APIWebhookURLOption, lines ~2465-2472), the equivalent to_mk_file_format() method uses the new 2.4 form:
("cmk_postprocessed", "explicit_password", ("<uuid>", "<the-actual-value>"))
add_to_event_context() in cmk/base/events.py (line ~1099) serializes the tuple into environment variables. For tuples of all strings it sets both a tab-joined base variable and numbered variables:
PARAMETER_ROUTING_KEY = routing_key\t<key>
PARAMETER_ROUTING_KEY_1 = routing_key
PARAMETER_ROUTING_KEY_2 = <key>
The PagerDuty plugin (cmk/notification_plugins/pagerduty.py) calls:
get_password_from_env_or_context("PARAMETER_ROUTING_KEY", context)
That function collects every PARAMETER_ROUTING_KEY* value into a list and hands it to retrieve_from_passwordstore() in cmk/notification_plugins/utils.py (line ~263):
def retrieve_from_passwordstore(parameter: str | list[str]) -> str:
if isinstance(parameter, list):
if "explicit_password" in parameter:
value: str | None = parameter[-1]
else:
value = cmk.utils.password_store.extract(parameter[-2])
if value is None:
sys.stderr.write("Unable to retrieve password from passwordstore")
sys.exit(2)
...
The resulting list is:
["routing_key\t<key>", "routing_key", "<key>"]
"explicit_password" is not present, so it falls through and tries:
password_store.extract("routing_key")
"routing_key" is not a valid password store ID, so the dispatcher exits.
For comparison, when the rule is created via the UI and migrated, the on-disk tuple is:
("cmk_postprocessed", "explicit_password", ("<uuid>", "<key>"))
Because the outer tuple contains a non-string element, add_to_event_context() skips the tab-joined base variable and produces only numbered vars, yielding:
["cmk_postprocessed", "explicit_password", ...]
"explicit_password" is present, so:
parameter[-1]
returns the actual key successfully.
Steps to reproduce
-
POST a notification rule with:
plugin_type: "pagerduty"- an explicit integration key
to:
/api/1.0/domain-types/notification_rule/collections/all -
Inspect:
etc/check_mk/conf.d/wato/notification_parameter.mkThe parameter entry contains:
'routing_key': ('routing_key', '<key>') -
Trigger a matching alert
-
Tail:
var/log/notify.logResult:
Unable to retrieve password from passwordstore
Expected behaviour
The notification is delivered to PagerDuty. The on-disk tuple matches the format produced by the UI migration path.
Suggested fix
In APIPagerDutyKeyOption.to_mk_file_format(), produce the same cmk_postprocessed / explicit_password form as APIWebhookURLOption.to_mk_file_format() in the same file.
Update from_mk_file_format() accordingly to keep round-trip compatibility, and add a migration for already-written rules.
A minimal stop-gap (without fixing the writer) is to extend retrieve_from_passwordstore() to recognise the "routing_key" marker as equivalent to "explicit_password", but this only papers over the format mismatch.
Workaround
After creating PagerDuty rules via the REST API, edit notification_parameter.mk and replace the first tuple element:
'routing_key': ('routing_key', '<key>')
with:
'routing_key': ('explicit_password', '<key>')
This puts "explicit_password" into the env var list and the dispatcher returns the key directly.