After upgrade from 2.3.x to 2.4.x raw sporadically failing notifications with "jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'"

Troubleshooting
1

CMK version:
2.4.0p5 raw

OS version:
debian 12

Description
Since the upgrade from 2.3.x to 2.4.x we sporadically encounter failing notifications with the error postet below.
Host and service notifications are affected. All users are affected.
All are bulk mail and HTML notifications.
As our notification rules have been quite old and automatically migrated from upgrade to upgrade, my first try to fix this was completely rewriting them from scratch, as I assumed some parameters where missing in the old rules.
Unfortunately this did not help, so for now I am simply out of ideas.

Error message:
Traceback (most recent call last): -- File "/omd/sites/SPK8/share/check_mk/notifications/mail", line 12, in <module> -- mail.main() -- File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 747, in main -- BulkEmailContent(utils.read_bulk_contexts) -- File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 642, in __init__ -- txt, html, att = construct_content( -- ^^^^^^^^^^^^^^^^^^ -- File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 482, in construct_content -- TemplateRenderer().render_template( -- File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 280, in render_template -- return template.render(data) -- ^^^^^^^^^^^^^^^^^^^^^ -- File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render -- self.environment.handle_exception() -- File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception -- raise rewrite_traceback_stack(source=source) -- File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/base.html", line 91, in top-level template code -- {% include 'bulk.html' %} -- File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/bulk.html", line 45, in top-level template code -- {{ entry.PREVIOUSSERVICEHARDSTATE[:4] }} -- -- jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'

2025: The Year in Review
2

I don’t know if the log exists on Raw, but check your site var/log/ directory for logs about notifications. In CEE and CCE, the files are mknotify.log

Hopefully there will be some additional details there.

Also, have you completely deleted the old notifications rules after your recreated them in 2.4? I don’t mean just disable. If not, that may be worth while to try, but agree not ideal.

1 Like
3

Hello @briand
Thank you for your reply

The file in raw is called var/log/notify.log afaics
And sorry for not posting more context, but from my point of view there is not much more enlightening … I post some more context from the log below.

If I can provide more information, just let me know - my problem is that I have no real idea what I can do - especially as I can not trigger the failed notifications, the just occur randomly.
One obscure idea of mine is: as not just the notification rules, but the complete setup has been migrated over years (as far as I remember this setup started at version 1.6.0 ), at some point some setting host/service (PREVIOUSxxxHARDSTATE) has been introduced, but old hosts and their services have got no default for this, and now, after 2.4, each first notification for these will fail, but the next notification will work as during the first failed the missing var is set… but this is just a crude and vague idea …

And yes, as I had the same concerns like you, I really deleted the old rules, not just deactivated them.
But as the error reoccurred, I rechecked everything and realized that the old “Parameters for HTML Email” where still around (but the notification rules used the new parameters). Now I removed the old (auto-migrated) parameters too.

After the last failure with the new rules I set the “Notification log level” to “Full dump of all variables and command” … maybe this will give me more information if the error reoccurs … paradox expectation: in fact I hope the error will not show up again, but I want the error to show up again so I can get more information …

Anyhow, here a log excerpt of a failed notification (some values obfuscated).
There is a marker set by me: XXX: this seems to be notification for the same host, targeting the same users, but a different service, and this succeeds.

If I can provide more information, juts let me know. Thank you for your assistance and help.

2025-08-15 17:37:19,159 [20] [cmk.base.notify] ----------------------------------------------------------------------
2025-08-15 17:37:19,160 [20] [cmk.base.notify] Got raw notification (XXXhoszXXX;HTTP HTTP) context with 54 variables
2025-08-15 17:37:19,160 [15] [cmk.base.notify] Global rule 'Notify all contacts of a host/service by mail for AB - new'...
2025-08-15 17:37:19,160 [15] [cmk.base.notify]  -> does not match: The host is only in the groups notAB, but AB is required
2025-08-15 17:37:19,161 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service by mail - new'...
2025-08-15 17:37:19,161 [20] [cmk.base.notify]  -> matches!
2025-08-15 17:37:19,161 [20] [cmk.base.notify]    - adding notification of YYYanotheruser, XXXuserXXX via mail
2025-08-15 17:37:19,161 [20] [cmk.base.notify] Executing 1 notifications:
2025-08-15 17:37:19,161 [20] [cmk.base.notify]   * notifying YYYanotheruser, XXXuserXXX via mail, parameters: bulk_sort_order, host_subject, service_subject, url_prefix, graphs_per_notification, notifications
_with_graphs, matching_rule_nr, matching_rule_text, bulk: yes
2025-08-15 17:37:19,161 [20] [cmk.base.notify]     --> storing for bulk notification YYYanotheruser|mail|600|1000|host|XXXhoszXXX
2025-08-15 17:37:19,161 [20] [cmk.base.notify]         - stored in /omd/sites/SPK8/var/check_mk/notify/bulk/YYYanotheruser/mail/600,1000,host,XXXhoszXXX/7407a652-8a2f-4a8d-9e8d-36bc453fddfc
2025-08-15 17:37:19,161 [20] [cmk.base.notify]     --> storing for bulk notification XXXuserXXX|mail|600|1000|host|XXXhoszXXX
2025-08-15 17:37:19,162 [20] [cmk.base.notify]         - stored in /omd/sites/SPK8/var/check_mk/notify/bulk/XXXuserXXX/mail/600,1000,host,XXXhoszXXX/e9125196-b560-476b-9fc0-60e3bdff2e18
2025-08-15 17:38:02,857 [20] [cmk.base.notify] Sending out 2 ripe bulk notifications
2025-08-15 17:38:02,857 [20] [cmk.base.notify]    -> XXXuserXXX/mail /omd/sites/SPK8/var/check_mk/notify/bulk/XXXuserXXX/mail/600,1000,host,XXXhoszXXX
2025-08-15 17:38:02,859 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION: XXXuserXXX;XXXhoszXXX;HTTP HTTP;WARNING;bulk mail;HTTP WARNING: HTTP/1.1 403 Forbidden - 129 
bytes in 0.004 second response time
2025-08-15 17:38:02,859 [20] [cmk.events.log_to_history] sending command LOG;HOST NOTIFICATION: XXXuserXXX;XXXhoszXXX;UP;bulk mail;OK - XXXipXXX rta 0.469ms lost 0%
2025-08-15 17:38:02,859 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION: XXXuserXXX;XXXhoszXXX;HTTP HTTP;CRITICAL;bulk mail;CRITICAL - Socket timeout after 10 seconds
2025-08-15 17:38:04,185 [20] [cmk.base.notify] ERROR: script /omd/sites/SPK8/share/check_mk/notifications/mail --bulk returned with exit code 1
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail: Traceback (most recent call last):
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/share/check_mk/notifications/mail", line 12, in <module>
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:     mail.main()
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 747, in main
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:     BulkEmailContent(utils.read_bulk_contexts)
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 642, in __init__
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:     txt, html, att = construct_content(
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:                      ^^^^^^^^^^^^^^^^^^
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 482, in construct_content
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:     TemplateRenderer().render_template(
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 280, in render_template
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:     return template.render(data)
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:            ^^^^^^^^^^^^^^^^^^^^^
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:     self.environment.handle_exception()
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:     raise rewrite_traceback_stack(source=source)
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/base.html", line 91, in top-level template code
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:     {% include 'bulk.html' %}
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/bulk.html", line 45, in top-level template code
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:     {{ entry.PREVIOUSSERVICEHARDSTATE[:4] }}
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail:
2025-08-15 17:38:04,185 [20] [cmk.base.notify] mail: jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'
2025-08-15 17:38:04,186 [20] [cmk.base.notify]    -> YYYanotheruser/mail /omd/sites/SPK8/var/check_mk/notify/bulk/YYYanotheruser/mail/600,1000,host,XXXhoszXXX
2025-08-15 17:38:04,188 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION: YYYanotheruser;XXXhoszXXX;HTTP HTTP;WARNING;bulk mail;HTTP WARNING: HTTP/1.1 403 Forbidden - 129
 bytes in 0.004 second response time
2025-08-15 17:38:04,188 [20] [cmk.events.log_to_history] sending command LOG;HOST NOTIFICATION: YYYanotheruser;XXXhoszXXX;UP;bulk mail;OK - XXXipXXX rta 0.469ms lost 0%
2025-08-15 17:38:04,188 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION: YYYanotheruser;XXXhoszXXX;HTTP HTTP;CRITICAL;bulk mail;CRITICAL - Socket timeout after 10 second
s
2025-08-15 17:38:05,462 [20] [cmk.base.notify] ERROR: script /omd/sites/SPK8/share/check_mk/notifications/mail --bulk returned with exit code 1
2025-08-15 17:38:05,462 [20] [cmk.base.notify] mail: Traceback (most recent call last):
2025-08-15 17:38:05,462 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/share/check_mk/notifications/mail", line 12, in <module>
2025-08-15 17:38:05,462 [20] [cmk.base.notify] mail:     mail.main()
2025-08-15 17:38:05,462 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 747, in main
2025-08-15 17:38:05,462 [20] [cmk.base.notify] mail:     BulkEmailContent(utils.read_bulk_contexts)
2025-08-15 17:38:05,462 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 642, in __init__
2025-08-15 17:38:05,462 [20] [cmk.base.notify] mail:     txt, html, att = construct_content(
2025-08-15 17:38:05,462 [20] [cmk.base.notify] mail:                      ^^^^^^^^^^^^^^^^^^
2025-08-15 17:38:05,462 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 482, in construct_content
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:     TemplateRenderer().render_template(
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 280, in render_template
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:     return template.render(data)
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:            ^^^^^^^^^^^^^^^^^^^^^
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:     self.environment.handle_exception()
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:     raise rewrite_traceback_stack(source=source)
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/base.html", line 91, in top-level template code
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:     {% include 'bulk.html' %}
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/bulk.html", line 45, in top-level template code
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:     {{ entry.PREVIOUSSERVICEHARDSTATE[:4] }}
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail:
2025-08-15 17:38:05,463 [20] [cmk.base.notify] mail: jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'
2025-08-15 17:38:05,463 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: YYYanotheruser;XXXhoszXXX;HTTP HTTP;WARNING;bulk mail;jinja2.exceptions.UndefinedError: '
dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE';Traceback (most recent call last): --   File "/omd/sites/SPK8/share/check_mk/notifications/mail", line 12, in <module> --     mail.main() --   File "/omd/
sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 747, in main --     BulkEmailContent(utils.read_bulk_contexts) --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 642, i
n __init__ --     txt, html, att = construct_content( --                      ^^^^^^^^^^^^^^^^^^ --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 482, in construct_content --     Te
mplateRenderer().render_template( --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 280, in render_template --     return template.render(data) --            ^^^^^^^^^^^^^^^^^^^^^ --
   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render --     self.environment.handle_exception() --   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/enviro
nment.py", line 942, in handle_exception --     raise rewrite_traceback_stack(source=source) --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/base.html", line 91, in top-level template code
 --     {% include 'bulk.html' %} --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/bulk.html", line 45, in top-level template code --     {{ entry.PREVIOUSSERVICEHARDSTATE[:4] }} --  -- jin
ja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'
2025-08-15 17:38:05,463 [20] [cmk.events.log_to_history] sending command LOG;HOST NOTIFICATION RESULT: YYYanotheruser;XXXhoszXXX;WARNING;bulk mail;jinja2.exceptions.UndefinedError: 'dict object' 
has no attribute 'PREVIOUSSERVICEHARDSTATE';Traceback (most recent call last): --   File "/omd/sites/SPK8/share/check_mk/notifications/mail", line 12, in <module> --     mail.main() --   File "/omd/sites/SPK8/li
b/python3/cmk/notification_plugins/mail.py", line 747, in main --     BulkEmailContent(utils.read_bulk_contexts) --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 642, in __init__ --
     txt, html, att = construct_content( --                      ^^^^^^^^^^^^^^^^^^ --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 482, in construct_content --     TemplateRendere
r().render_template( --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 280, in render_template --     return template.render(data) --            ^^^^^^^^^^^^^^^^^^^^^ --   File "/omd
/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render --     self.environment.handle_exception() --   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", li
ne 942, in handle_exception --     raise rewrite_traceback_stack(source=source) --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/base.html", line 91, in top-level template code --     {% include 'bulk.html' %} --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/bulk.html", line 45, in top-level template code --     {{ entry.PREVIOUSSERVICEHARDSTATE[:4] }} --  -- jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'
2025-08-15 17:38:05,463 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: YYYanotheruser;XXXhoszXXX;HTTP HTTP;WARNING;bulk mail;jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE';Traceback (most recent call last): --   File "/omd/sites/SPK8/share/check_mk/notifications/mail", line 12, in <module> --     mail.main() --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 747, in main --     BulkEmailContent(utils.read_bulk_contexts) --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 642, in __init__ --     txt, html, att = construct_content( --                      ^^^^^^^^^^^^^^^^^^ --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 482, in construct_content --     TemplateRenderer().render_template( --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 280, in render_template --     return template.render(data) --            ^^^^^^^^^^^^^^^^^^^^^ --   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render --     self.environment.handle_exception() --   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception --     raise rewrite_traceback_stack(source=source) --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/base.html", line 91, in top-level template code --     {% include 'bulk.html' %} --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/bulk.html", line 45, in top-level template code --     {{ entry.PREVIOUSSERVICEHARDSTATE[:4] }} --  -- jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'
2025-08-15 17:38:24,163 [20] [cmk.base.notify] ----------------------------------------------------------------------
2025-08-15 17:38:24,163 [20] [cmk.base.notify] Got raw notification (XXXhoszXXX;ExchangeHealth) context with 54 variables
2025-08-15 17:38:24,163 [15] [cmk.base.notify] Global rule 'Notify all contacts of a host/service by mail for AB - new'...
2025-08-15 17:38:24,163 [15] [cmk.base.notify]  -> does not match: The host is only in the groups notAB, but AB is required
2025-08-15 17:38:24,164 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service by mail - new'...
2025-08-15 17:38:24,164 [20] [cmk.base.notify]  -> matches!
2025-08-15 17:38:24,164 [20] [cmk.base.notify]    - adding notification of XXXuserXXX, YYYanotheruser via mail
2025-08-15 17:38:24,164 [20] [cmk.base.notify] Executing 1 notifications:
2025-08-15 17:38:24,165 [20] [cmk.base.notify]   * notifying XXXuserXXX, YYYanotheruser via mail, parameters: bulk_sort_order, host_subject, service_subject, url_prefix, graphs_per_notification, notifications_with_graphs, matching_rule_nr, matching_rule_text, bulk: yes
2025-08-15 17:38:24,165 [20] [cmk.base.notify]     --> storing for bulk notification XXXuserXXX|mail|600|1000|host|XXXhoszXXX
2025-08-15 17:38:24,165 [20] [cmk.base.notify]         - created bulk directory /omd/sites/SPK8/var/check_mk/notify/bulk/XXXuserXXX/mail/600,1000,host,XXXhoszXXX
2025-08-15 17:38:24,165 [20] [cmk.base.notify]         - stored in /omd/sites/SPK8/var/check_mk/notify/bulk/XXXuserXXX/mail/600,1000,host,XXXhoszXXX/f39185a9-72b3-474b-a826-e5bfc354e732
2025-08-15 17:38:24,165 [20] [cmk.base.notify]     --> storing for bulk notification YYYanotheruser|mail|600|1000|host|XXXhoszXXX
2025-08-15 17:38:24,165 [20] [cmk.base.notify]         - created bulk directory /omd/sites/SPK8/var/check_mk/notify/bulk/YYYanotheruser/mail/600,1000,host,XXXhoszXXX
2025-08-15 17:38:24,166 [20] [cmk.base.notify]         - stored in /omd/sites/SPK8/var/check_mk/notify/bulk/YYYanotheruser/mail/600,1000,host,XXXhoszXXX/60ca5f2b-02eb-4389-bf6c-4a7fdf6d6e7f

2025-08-15 17:38:04,186 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: XXXuserXXX;XXXhoszXXX;HTTP HTTP;WARNING;bulk mail;jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE';Traceback (most recent call last): --   File "/omd/sites/SPK8/share/check_mk/notifications/mail", line 12, in <module> --     mail.main() --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 747, in main --     BulkEmailContent(utils.read_bulk_contexts) --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 642, in __init__ --     txt, html, att = construct_content( --                      ^^^^^^^^^^^^^^^^^^ --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 482, in construct_content --     TemplateRenderer().render_template( --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 280, in render_template --     return template.render(data) --            ^^^^^^^^^^^^^^^^^^^^^ --   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render --     self.environment.handle_exception() --   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception --     raise rewrite_traceback_stack(source=source) --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/base.html", line 91, in top-level template code --     {% include 'bulk.html' %} --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/bulk.html", line 45, in top-level template code --     {{ entry.PREVIOUSSERVICEHARDSTATE[:4] }} --  -- jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'
2025-08-15 17:38:04,186 [20] [cmk.events.log_to_history] sending command LOG;HOST NOTIFICATION RESULT: XXXuserXXX;XXXhoszXXX;WARNING;bulk mail;jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE';Traceback (most recent call last): --   File "/omd/sites/SPK8/share/check_mk/notifications/mail", line 12, in <module> --     mail.main() --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 747, in main --     BulkEmailContent(utils.read_bulk_contexts) --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 642, in __init__ --     txt, html, att = construct_content( --                      ^^^^^^^^^^^^^^^^^^ --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 482, in construct_content --     TemplateRenderer().render_template( --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 280, in render_template --     return template.render(data) --            ^^^^^^^^^^^^^^^^^^^^^ --   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render --     self.environment.handle_exception() --   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception --     raise rewrite_traceback_stack(source=source) --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/base.html", line 91, in top-level template code --     {% include 'bulk.html' %} --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/bulk.html", line 45, in top-level template code --     {{ entry.PREVIOUSSERVICEHARDSTATE[:4] }} --  -- jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'
2025-08-15 17:38:04,186 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: XXXuserXXX;XXXhoszXXX;HTTP HTTP;WARNING;bulk mail;jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE';Traceback (most recent call last): --   File "/omd/sites/SPK8/share/check_mk/notifications/mail", line 12, in <module> --     mail.main() --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 747, in main --     BulkEmailContent(utils.read_bulk_contexts) --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 642, in __init__ --     txt, html, att = construct_content( --                      ^^^^^^^^^^^^^^^^^^ --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 482, in construct_content --     TemplateRenderer().render_template( --   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 280, in render_template --     return template.render(data) --            ^^^^^^^^^^^^^^^^^^^^^ --   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render --     self.environment.handle_exception() --   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception --     raise rewrite_traceback_stack(source=source) --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/base.html", line 91, in top-level template code --     {% include 'bulk.html' %} --   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/bulk.html", line 45, in top-level template code --     {{ entry.PREVIOUSSERVICEHARDSTATE[:4] }} --  -- jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'
# XXX: this seems to be notification for the same host, but a different service, and this succeeds
2025-08-15 17:38:24,163 [20] [cmk.base.notify] ----------------------------------------------------------------------
2025-08-15 17:38:24,163 [20] [cmk.base.notify] Got raw notification (XXXhoszXXX;ExchangeHealth) context with 54 variables
2025-08-15 17:38:24,163 [15] [cmk.base.notify] Global rule 'Notify all contacts of a host/service by mail for AB - new'...
2025-08-15 17:38:24,163 [15] [cmk.base.notify]  -> does not match: The host is only in the groups notAB, but AB is required
2025-08-15 17:38:24,164 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service by mail - new'...
2025-08-15 17:38:24,164 [20] [cmk.base.notify]  -> matches!
2025-08-15 17:38:24,164 [20] [cmk.base.notify]    - adding notification of XXXuserXXX, YYYanotheruser via mail
2025-08-15 17:38:24,164 [20] [cmk.base.notify] Executing 1 notifications:
2025-08-15 17:38:24,165 [20] [cmk.base.notify]   * notifying XXXuserXXX, YYYanotheruser via mail, parameters: bulk_sort_order, host_subject, service_subject, url_prefix, graphs_per_notification, notifications
_with_graphs, matching_rule_nr, matching_rule_text, bulk: yes
2025-08-15 17:38:24,165 [20] [cmk.base.notify]     --> storing for bulk notification XXXuserXXX|mail|600|1000|host|XXXhoszXXX
2025-08-15 17:38:24,165 [20] [cmk.base.notify]         - created bulk directory /omd/sites/SPK8/var/check_mk/notify/bulk/XXXuserXXX/mail/600,1000,host,XXXhoszXXX
2025-08-15 17:38:24,165 [20] [cmk.base.notify]         - stored in /omd/sites/SPK8/var/check_mk/notify/bulk/XXXuserXXX/mail/600,1000,host,XXXhoszXXX/f39185a9-72b3-474b-a826-e5bfc354e732
2025-08-15 17:38:24,165 [20] [cmk.base.notify]     --> storing for bulk notification YYYanotheruser|mail|600|1000|host|XXXhoszXXX
2025-08-15 17:38:24,165 [20] [cmk.base.notify]         - created bulk directory /omd/sites/SPK8/var/check_mk/notify/bulk/YYYanotheruser/mail/600,1000,host,XXXhoszXXX
2025-08-15 17:38:24,166 [20] [cmk.base.notify]         - stored in /omd/sites/SPK8/var/check_mk/notify/bulk/YYYanotheruser/mail/600,1000,host,XXXhoszXXX/60ca5f2b-02eb-4389-bf6c-4a7fdf6d6e7f
4

OK, while greping the log for similar parameters I found another bulk notification, same host, same service, same users, but with the state: OK and the timestamp is before the failed event, so my crude theory from the last post is obsolete.

2025-08-15 08:57:27,148 [20] [cmk.base.notify] ----------------------------------------------------------------------
2025-08-15 08:57:27,148 [20] [cmk.base.notify] Got raw notification (XXXhoszXXX;HTTP HTTP) context with 54 variables
2025-08-15 08:57:27,149 [15] [cmk.base.notify] Global rule 'Notify all contacts of a host/service by mail for AB - new'...
2025-08-15 08:57:27,149 [15] [cmk.base.notify]  -> does not match: The host is only in the groups notAB, but AB is required
2025-08-15 08:57:27,149 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service by mail - new'...
2025-08-15 08:57:27,149 [20] [cmk.base.notify]  -> matches!
2025-08-15 08:57:27,149 [20] [cmk.base.notify]    - adding notification of YYYanotheruser, XXXuserXXX via mail
2025-08-15 08:57:27,150 [20] [cmk.base.notify] Executing 1 notifications:
2025-08-15 08:57:27,150 [20] [cmk.base.notify]   * notifying YYYanotheruser, XXXuserXXX via mail, parameters: bulk_sort_order, host_subject, service_subject, url_prefix, graphs_per_notification, notifications_with_graphs, matching_rule_nr, matching_rule_text, bulk: yes
2025-08-15 08:57:27,150 [20] [cmk.base.notify]     --> storing for bulk notification YYYanotheruser|mail|600|1000|host|XXXhoszXXX
2025-08-15 08:57:27,150 [20] [cmk.base.notify]         - created bulk directory /omd/sites/SPK8/var/check_mk/notify/bulk/YYYanotheruser/mail/600,1000,host,XXXhoszXXX
2025-08-15 08:57:27,150 [20] [cmk.base.notify]         - stored in /omd/sites/SPK8/var/check_mk/notify/bulk/YYYanotheruser/mail/600,1000,host,XXXhoszXXX/42e3b8fa-ff20-423c-8ce3-8001e8e0c03b
2025-08-15 08:57:27,150 [20] [cmk.base.notify]     --> storing for bulk notification XXXuserXXX|mail|600|1000|host|XXXhoszXXX
2025-08-15 08:57:27,150 [20] [cmk.base.notify]         - created bulk directory /omd/sites/SPK8/var/check_mk/notify/bulk/XXXuserXXX/mail/600,1000,host,XXXhoszXXX
2025-08-15 08:57:27,151 [20] [cmk.base.notify]         - stored in /omd/sites/SPK8/var/check_mk/notify/bulk/XXXuserXXX/mail/600,1000,host,XXXhoszXXX/d42c597a-13be-4d1b-b384-b5ab4abba955
2025-08-15 08:57:29,171 [20] [cmk.base.notify] ----------------------------------------------------------------------
...
2025-08-15 09:08:02,176 [20] [cmk.base.notify] Sending out 6 ripe bulk notifications
2025-08-15 09:08:02,176 [20] [cmk.base.notify]    -> XXXuserXXX/mail /omd/sites/SPK8/var/check_mk/notify/bulk/XXXuserXXX/mail/600,1000,host,XXXhoszXXX
2025-08-15 09:08:02,178 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION: XXXuserXXX;XXXhoszXXX;Msg_Queue_SmtpDeliveryToMailbox;OK;bulk mail;0 Mails in der Warteschlan
ge.
2025-08-15 09:08:02,178 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION: XXXuserXXX;XXXhoszXXX;ExchangeHealth;WARNING;bulk mail;WARNING: Issues found on server. Check
 Get-ServerHealth
2025-08-15 09:08:02,179 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION: XXXuserXXX;XXXhoszXXX;HTTP HTTP;WARNING;bulk mail;HTTP WARNING: HTTP/1.1 403 Forbidden - 129 bytes in 0.001 second response time
2025-08-15 09:08:03,911 [20] [cmk.base.notify] mail: Spooled mail to local mail transmission agent
2025-08-15 09:08:03,911 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: XXXuserXXX;XXXhoszXXX;Msg_Queue_SmtpDeliveryToMailbox;OK;bulk mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2025-08-15 09:08:03,911 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: XXXuserXXX;XXXhoszXXX;ExchangeHealth;OK;bulk mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2025-08-15 09:08:03,912 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: XXXuserXXX;XXXhoszXXX;HTTP HTTP;OK;bulk mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
...
2025-08-15 09:08:05,743 [20] [cmk.base.notify]    -> YYYanotheruser/mail /omd/sites/SPK8/var/check_mk/notify/bulk/YYYanotheruser/mail/600,1000,host,XXXhoszXXX
2025-08-15 09:08:05,745 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION: YYYanotheruser;XXXhoszXXX;Msg_Queue_SmtpDeliveryToMailbox;OK;bulk mail;0 Mails in der Warteschlange.
2025-08-15 09:08:05,745 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION: YYYanotheruser;XXXhoszXXX;ExchangeHealth;WARNING;bulk mail;WARNING: Issues found on server. Check Get-ServerHealth
2025-08-15 09:08:05,745 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION: YYYanotheruser;XXXhoszXXX;HTTP HTTP;WARNING;bulk mail;HTTP WARNING: HTTP/1.1 403 Forbidden - 129 bytes in 0.001 second response time
2025-08-15 09:08:07,431 [20] [cmk.base.notify] mail: Spooled mail to local mail transmission agent
2025-08-15 09:08:07,431 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: YYYanotheruser;XXXhoszXXX;Msg_Queue_SmtpDeliveryToMailbox;OK;bulk mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2025-08-15 09:08:07,431 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: YYYanotheruser;XXXhoszXXX;ExchangeHealth;OK;bulk mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
2025-08-15 09:08:07,431 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: YYYanotheruser;XXXhoszXXX;HTTP HTTP;OK;bulk mail;Spooled mail to local mail transmission agent;Spooled mail to local mail transmission agent
5

Have you tried the test notification tool? You can tell it to send the actual alert email (so you can hopefully get as close to a real notification as possible).

image
image763×507 19.6 KB

(Taken from fresh 2.4.0p9 RAW site)

Also, you can force a fake check result as another way to test. A bit more combersome, but if the test notification tool doesn’t trigger the error, you can try that.

6

Hello @briand - yes, this was one of the first things I have tested.
And like in my log excerpt above, the same notification passed without problems …
I will update to the latest version …

7

well, after a update to 2.4.0p11 this reoccurred

the error

2025-09-09 15:52:03,003 [20] [cmk.base.notify] ERROR: script /omd/sites/SPK8/share/check_mk/notifications/mail --bulk returned with exit code 1
2025-09-09 15:52:03,003 [20] [cmk.base.notify] mail: Traceback (most recent call last):
2025-09-09 15:52:03,003 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/share/check_mk/notifications/mail", line 12, in <module>
2025-09-09 15:52:03,003 [20] [cmk.base.notify] mail:     mail.main()
2025-09-09 15:52:03,003 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 747, in main
2025-09-09 15:52:03,003 [20] [cmk.base.notify] mail:     BulkEmailContent(utils.read_bulk_contexts)
2025-09-09 15:52:03,003 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 642, in __init__
2025-09-09 15:52:03,003 [20] [cmk.base.notify] mail:     txt, html, att = construct_content(
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:                      ^^^^^^^^^^^^^^^^^^
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 482, in construct_content
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:     TemplateRenderer().render_template(
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3/cmk/notification_plugins/mail.py", line 280, in render_template
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:     return template.render(data)
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:            ^^^^^^^^^^^^^^^^^^^^^
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:     self.environment.handle_exception()
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:     raise rewrite_traceback_stack(source=source)
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/base.html", line 91, in top-level template code
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:     {% include 'bulk.html' %}
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:   File "/omd/sites/SPK8/share/check_mk/notifications/templates/mail/bulk.html", line 36, in top-level template code
2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail:     {{ entry.PREVIOUSSERVICEHARDSTATE[:4] }} 

2025-09-09 15:52:03,004 [20] [cmk.base.notify] mail: jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'
2025-09-09 15:52:03,004 [20] [cmk.events.log_to_history] sending command LOG;SERVICE NOTIFICATION RESULT: $USER.NAME$;$HOSTNAME$;Task MicrosoftEdgeUpdateTaskMachineUA{FD58543;WARNING;bulk mail;jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'PREVIOUSSERVICEHARDSTATE'; ...

and as far as I can see the corresponding VARs in the debug output are:

025-09-09 15:42:53,083 [20] [cmk.base.notify] ----------------------------------------------------------------------
2025-09-09 15:42:53,083 [20] [cmk.base.notify] Got raw notification ($HOSTNAME$;Task MicrosoftEdgeUpdateTaskMachineUA{FD58543) context with 54 variables
2025-09-09 15:42:53,083 [10] [cmk.base.notify] Raw context:
                    CONTACTEMAIL=
                    CONTACTNAME=check-mk-notify
                    CONTACTPAGER=
                    CONTACTS=$ANOTHERUSER.NAME$,$USER.NAME$
                    DATE=2025-09-09
                    HOSTACKAUTHOR=
                    HOSTACKCOMMENT=
                    HOSTADDRESS=$HOSTIP$
                    HOSTALIAS=$HOSTNAME$
                    HOSTATTEMPT=1
                    HOSTCHECKCOMMAND=check-mk-host-ping!-w 200.00,80.00% -c 500.00,100.00%
                    HOSTDOWNTIME=0
                    HOSTFORURL=$HOSTNAME$
                    HOSTGROUPNAMES=xxx
                    HOSTLABEL_cmk/os_family=windows
                    HOSTLABEL_cmk/os_name=Microsoft Windows Server 2019 Standard
                    HOSTLABEL_cmk/os_platform=windows
                    HOSTLABEL_cmk/os_type=windows
                    HOSTLABEL_cmk/os_version=10.0.17763
                    HOSTLABEL_cmk/site=xxx
                    HOSTLABEL_location=xxx
                    HOSTLABEL_os=windows
                    HOSTNAME=$HOSTNAME$
                    HOSTNOTIFICATIONNUMBER=1
                    HOSTOUTPUT=OK - 172.26.4.67 rta 0.342ms lost 0%
                    HOSTPERFDATA=rta=0.342ms;200.000;500.000;0; rtmax=0.512ms;;;; rtmin=0.000ms;;;; pl=0%;80;100;0;100
                    HOSTPROBLEMID=0
                    HOSTSHORTSTATE=UP
                    HOSTSTATE=UP
                    HOSTSTATEID=0
                    HOSTTAGS=/wato/spk8/windows_server/ auto-piggyback checkmk-agent cmk-agent if_alias ip-v4 ip-v4-only lan no-snmp prod site:SPK8 tcp win2019 windows
                    HOSTTAG_address_family=ip-v4-only
                    HOSTTAG_agent=cmk-agent
                    HOSTTAG_checkmk-agent=checkmk-agent
                    HOSTTAG_criticality=prod
                    HOSTTAG_if_by_alias_by_desc=if_alias
                    HOSTTAG_ip-v4=ip-v4
                    HOSTTAG_networking=lan
                    HOSTTAG_opsys=win2019
                    HOSTTAG_piggyback=auto-piggyback
                    HOSTTAG_site=xxx
                    HOSTTAG_snmp_ds=no-snmp
                    HOSTTAG_tcp=tcp
                    HOSTTAG_windows=windows
                    HOSTURL=/check_mk/index.py?start_url=view.py?view_name%3Dhoststatus%26host%3D$HOSTNAME$%26site%3Dxxx
                    HOST_ADDRESS_4=$HOSTIP$
                    HOST_ADDRESS_6=
                    HOST_ADDRESS_FAMILY=4
                    LASTHOSTPROBLEMID=5680357
                    LASTHOSTSHORTSTATE=DOWN
                    LASTHOSTSTATE=DOWN
                    LASTHOSTSTATECHANGE=1757425269
                    LASTHOSTSTATECHANGE_REL=0d 00:01:44
                    LASTHOSTSTATEID=1
                    LASTHOSTUP=1757425269
                    LASTHOSTUP_REL=0d 00:01:44
                    LASTSERVICEOK=1757422925
                    LASTSERVICEOK_REL=0d 00:40:48
                    LASTSERVICEPROBLEMID=5648515
                    LASTSERVICESHORTSTATE=CRIT
                    LASTSERVICESTATE=CRITICAL
                    LASTSERVICESTATECHANGE=1757423165
                    LASTSERVICESTATECHANGE_REL=0d 00:36:48
                    LASTSERVICESTATEID=2
                    LONGDATETIME=Tue Sept 9 15:42:52 CEST 2025
                    LONGHOSTOUTPUT=
                    LONGSERVICEOUTPUT=MicrosoftEdgeUpdateTaskMachineUA{FD58543 wurde nicht erfolgreich ausgefuehrt.
                    MICROTIME=1757425373077630
                    MONITORING_HOST=$CHECKMHOSTNAME$
                    NOTIFICATIONAUTHOR=
                    NOTIFICATIONAUTHORALIAS=
                    NOTIFICATIONAUTHORNAME=
                    NOTIFICATIONCOMMENT=
                    NOTIFICATIONTYPE=PROBLEM
                    OMD_ROOT=/omd/sites/xxx
                    OMD_SITE=xxx
                    PREVIOUSHOSTHARDSHORTSTATE=DOWN
                    PREVIOUSHOSTHARDSTATE=DOWN
                    PREVIOUSSERVICEHARDSHORTSTATE=OK
                    PREVIOUSSERVICEHARDSTATE=OK
                    SERVICEACKAUTHOR=
                    SERVICEACKCOMMENT=
                    SERVICEATTEMPT=1
                    SERVICECHECKCOMMAND=check_mk-local
                    SERVICEDESC=Task MicrosoftEdgeUpdateTaskMachineUA{FD58543
                    SERVICEFORURL=Task%20MicrosoftEdgeUpdateTaskMachineUA%7BFD58543
                    SERVICEGROUPNAMES=
                    SERVICENOTIFICATIONNUMBER=1
                    SERVICEOUTPUT=MicrosoftEdgeUpdateTaskMachineUA{FD58543 wurde nicht erfolgreich ausgefuehrt.
                    SERVICEPERFDATA=
                    SERVICEPROBLEMID=5680346
                    SERVICESHORTSTATE=CRIT
                    SERVICESTATE=CRITICAL
                    SERVICESTATEID=2
                    SERVICEURL=/check_mk/index.py?start_url=view.py?view_name%3Dservice%26host%3D$HOSTNAME$%26service%3DTask%2BMicrosoftEdgeUpdateTaskMachineUA%257BFD58543%26site%3Dxxx
                    SHORTDATETIME=2025-09-09 15:42:52
                    WHAT=SERVICE

and here the PREVIOUSSERVICEHARDSTATE=OK is present …

I have no idea where the VAR gets lost on the way from the event to the bulk notification …

8

I can say for RAW edition it is very common that there is no “PREVIOUSSERVICEHARDSTATE” available.
If you restart the Nagios core, it can happen that there is some state available for a service, but not the last hard state. In the old notifications you saw then a “from ??? to CRIT” or so inside the mail.

In the end it is a bug that should be addressed by the team as it affects all RAW edition users.

9

Thank you @andreas-doehler for your reply.
I could really live better with a notification that has a undefined PREVIOUS*HARDSTATEthen with failing notification - in this case the notification was not that critical, but especially for monitoring a unreliable notification system is not the desired state.

Bugreports are closed on github for the checkmk repo, so I have no chance to open a werk …

But I can open a PR, as the fix can be quite simple in the jinja2 template

>>> from jinja2 import Environment, FileSystemLoader, Template
>>>
>>> class entry: 
...     def __init__(self):         
...         self.HOSTNAME='hostname'
>>> entry = entry()
   
>>> template = Template('hostname = {{ entry.HOSTNAME | default("UNDEFINED") }}')
>>> print(template.render(entry=entry))
hostname = hostname
>>>
>>> template = Template('prev hard state = {{ entry.PREVIOUSSERVICEHARDSTATE | default("UNDEFINED") }}')
>>> print(template.render(entry=entry))
prev hard state = UNDEFINED
>>>
>>> # or to mimic the syntax used in the bulk template
>>> template = Template('prev hard state = {{ (entry.PREVIOUSSERVICEHARDSTATE | default("NULL"))[:4] }}')
>>> print(template.render(entry=entry))
prev hard state = NULL

Or do you know a better way to spot some attention of the dev team to this?

10

for the record: Fix for missing previous xhardstate by zerwes · Pull Request #852 · Checkmk/checkmk · GitHub

11

I also found this problem in a system with enterpise edition today. It looks like a general problem with the new notification template.

Made a comment for your PR.

12

@andreas-doehler thank you for figuring that out.
This might elevate the chance for a fix for this issue.

13

for the record: new PR opened, as the original was closed while trying to bring the PR up to date with force pushes on master)

1 Like
14

well, they do not really make your feel like PR are welcome @ checkmk (and it is not the first time I have that impression) - first the clutter up master with forced pushes and then the PR is marked stale due to some broken test setup …

15

Same issue here. Please fix or accept the merge request :slight_smile:

16

Hi @zerwes,

I will have a look into the change today. Since the GitHub Actions has been running red for quite sometime, a ticket in our internal ticketing system wasn’t created. Our CI team is working hard to resolve this issue at the moment. Hopefully it will be fixed soon - sorry for the frustration.

If I have questions regarding the change, I will communicate it within your open PR.

Best,
Logan

1 Like
17

Re:

for the record: new PR opened, as the original was closed while trying to bring the PR up to date with force pushes on master)

To give some more context here. Internally, we work with a different remote repository (Gerrit) and every few hours, that remote syncs changes to the public GitHub remote repository. So don’t worry, we are not force pushing to master internally. Each change is reviewed before integrated into the master branch.

In the event of a conflict after one of these syncs, there is no need to open a new PR. Alternatively, you can fetch the new upstream changes from the master branch of Checkmk, rebase on top of it, and push your change(s) to the remote branch where you created the PR.

How this looks with an example:

  1. set upstream remote to checkmk (only do once):
git remote add upstream git@github.com:Checkmk/checkmk
  1. fetch upstream changes:
git fetch upstream
  1. rebase on top of upstream master with your local branch:
git rebase upstream/master
  1. resolve merge conflicts, stage resolved changes, and continue rebase.
  2. push changes to remote branch (fork remote is usually named origin):
# if it's the first time pushing local branch to remote
git push --set-upstream origin <branch-name>

# if branch already exists on remote.
git push --force-with-lease

Hope this helps a bit for future contributions. Let me know if something was unclear here. We do really appreciate the contributions that the community makes. Thank you for bringing this issue to our attention and the proposed patch :slight_smile:

1 Like
18

Thank you @logan-connolly for the explanation - my problem was not to use the command line, but the github “push the button” solution, that then closed the PR :frowning: .(and there is no way to reopen a closed PR as normal user)
Next time I am warned and will use the command line …

1 Like
19

Ahh, I see! That is definitely strange behavior. The issue you’ve described has been forwarded to the CI team. They are looking into it :+1:

20

Thanks again to @zerwes for publishing a fix to this problem! The code is now merged in and will be made available in the next release.

Have a great weekend everybody :slight_smile:

3 Likes