Notifications broken after upgrade to 2.4

CMK version: 2.4.0p2
OS version: Debian 12

It appears that something broke during my upgrade to 2.4, and I was unable to view/edit my previously configured notification rules. The rules run as expected when the conditions for the notification are met, but I cannot modify the rules. The error that comes up when I attempt to modify one is as follows:

Internal error: ‘rule_id’

An internal error occurred while processing your request (Crash ID: 8cc248b6-3747-11f0-96e3-00155d961f20). You can report this issue to the Checkmk team to help fixing this issue. Please open the crash report page and use the form for reporting the problem.

I have already submitted the crash report to the team.

Workaround

I looked in ~/etc/checkmk/conf.d/wato/notifications.mk and noticed that the pre-existing rules do not have a rule_id field. Rules created after the upgrade DO have a rule_id field, but are still unable to be edited.

I made a copy of my notifications.mk file stored in ~/etc/check_mk/conf.d/wato and started playing.

I found that if I clone each existing notification rule from the Web UI (dealing with the errors that appear each time I clone, I can manually edit the file to delete out the old rules. Once the cloned copies of the rules are the only ones remaining, I saved the file and then I could edit the notifications from the UI as expected.

I’m not sure if this experience is unique to me due to some lingering issue, but I did not notice any issues in the output when I upgraded the site from 2.3 to 2.4. Perhaps this was overlooked and will be fixed in an upcoming patch release.

1 Like

Good Morning @CarterScottM,

thank you for reporting the issue.

It’s good, that you were able to find a workaround.
Still we will have a look - I have found the crash report and created an internal ticket.

Could you also supply the following information?

  1. What was the original Checkmk version before updating?
  2. Your original notifications.mk file

Sunny Greetings and thank you
Hartmut

Attached is the notifications_backup.mk that was auto-created during the upgrade. We were running 2.3.0p23, but I upgraded the site to 2.3.0p29 just prior to bumping up to 2.4.0p1.

# Written by Checkmk store

notification_rules += [{'description': 'Downtime Alerts', 'comment': '', 'docu_url': '', 'disabled': False, 'allow_disable': True, 'match_timeperiod': 'WorkingHours', 'match_host_event': ['s'], 'contact_object': False, 'contact_all': False, 'contact_all_with_email': False, 'contact_groups': ['all'], 'notify_plugin': ('asciimail', {'common_body': 'Host:     $HOSTNAME$\nAddress:  $HOSTADDRESS$\n', 'host_body': 'Event:    $EVENT_TXT$\n$LONGHOSTOUTPUT$\n'}), 'bulk': ('always', {'interval': 120, 'count': 1000, 'groupby': ['folder'], 'groupby_custom': [], 'bulk_subject': 'Check_MK: Scheduled downtime for $COUNT_HOSTS$ hosts: $NOTIFICATIONCOMMENT})}, {'description': 'Time Drift', 'comment': 'Bulk notifications sent to all Check_MK users for NTP time drift issues.\n', 'docu_url': '', 'disabled': False, 'allow_disable': False, 'match_services': ['System Time'], 'match_timeperiod': 'WorkingHours', 'match_service_event': ['rw', 'rc', 'wr', 'wc', 'cr', 'cw', 'x'], 'contact_object': False, 'contact_all': False, 'contact_all_with_email': False, 'contact_groups': ['all'], 'notify_plugin': ('mail', {'elements': ['address', 'abstime', 'longoutput', 'graph'], 'notifications_with_graphs': 5}), 'bulk': ('always', {'interval': 180, 'count': 1000, 'groupby': ['state'], 'groupby_custom': []})}, {'description': 'Standard Host Notifications', 'comment': 'Standard set of host notifications sent to all Check_MK users.\n', 'docu_url': '', 'disabled': False, 'allow_disable': False, 'match_host_event': ['rd', 'dr', 'x'], 'contact_object': False, 'contact_all': False, 'contact_all_with_email': False, 'contact_groups': ['all'], 'notify_plugin': ('mail', {})}, {'description': 'Filesystem Free Space Notifications - SCCM DP Servers', 'comment': 'Free space notifications sent to Scott Carter.\n\nONLY includes free space notifications for SCCM DP servers\n', 'docu_url': '', 'disabled': False, 'allow_disable': False, 'match_hostgroups': ['sccmservers'], 'match_services': ['Filesystem', 'fs_'], 'match_timeperiod': 'WorkingHours', 'match_service_event': ['rw', 'rc', 'wr', 'wc', 'cr', 'cw', 'x'], 'contact_object': False, 'contact_all': False, 'contact_all_with_email': False, 'contact_groups': ['ScottCarter'], 'notify_plugin': ('mail', {'elements': ['address', 'abstime', 'longoutput', 'graph']})}, {'description': 'Filesystem Free Space Notifications', 'comment': 'Free space notifications sent to all Check_MK users.\n\nDoes NOT include free space notifications for SCCM DP servers\n', 'docu_url': '', 'disabled': False, 'allow_disable': False, 'match_hostgroups': ['adservers', 'appservers', 'exchangeservers', 'fileservers', 'firewalls', 'hypervservers', 'licenseservers', 'linnuxservers', 'miscdevs', 'miscservers', 'printers', 'revitservers', 'routers', 'No_OMSA', 'sqlservers', 'storage', 'switches', 'unassigned', 'vidcon', 'vmware', 'wap'], 'match_services': ['Filesystem', 'fs_'], 'match_timeperiod': 'WorkingHours', 'match_service_event': ['rw', 'rc', 'wr', 'wc', 'cr', 'cw', 'x'], 'contact_object': False, 'contact_all': False, 'contact_all_with_email': False, 'contact_groups': ['all'], 'notify_plugin': ('mail', {'elements': ['address', 'abstime', 'longoutput', 'graph']})}, {'description': 'Standard Service Notifications', 'comment': 'Standard set of notifications sent to all Check_MK users.\n\nThis includes:\nDisk failures for Promise and QNAP boxes\nAEC License availability\n', 'docu_url': '', 'disabled': False, 'allow_disable': False, 'match_services': ['AEC Licenses', 'BI_License_Servers', 'QNAP Disks'], 'match_timeperiod': 'WorkingHours', 'match_service_event': ['rw', 'rc', 'wr', 'wc', 'cr', 'cw', 'x'], 'contact_object': False, 'contact_all': False, 'contact_all_with_email': False, 'contact_groups': ['all'], 'notify_plugin': ('mail', {'elements': ['address', 'abstime', 'longoutput', 'graph']})}, {'description': 'BST10 Services', 'comment': '', 'docu_url': '', 'disabled': False, 'allow_disable': False, 'match_hosts': ['BST10', 'BSTDB10'], 'match_services': ['service_BSTGlobal_Transaction_Processor_Service', 'service_BSTGlobal_Reporting_Service', 'service_MSSQLSERVER', 'service_SQLServerReportingServices', 'service_BSTGlobal_Mobile_Private_Cache_(Redis)'], 'match_timeperiod': 'WorkingHours', 'contact_object': False, 'contact_all': False, 'contact_all_with_email': False, 'contact_groups': ['BST'], 'notify_plugin': ('mail', {})}, {'description': 'BSTVIR', 'comment': '', 'docu_url': '', 'disabled': False, 'allow_disable': False, 'match_services': ['service_BSTVIR'], 'match_timeperiod': 'WorkingHours', 'contact_object': False, 'contact_all': False, 'contact_all_with_email': False, 'contact_groups': ['all'], 'notify_plugin': ('mail', {})}, {'description': 'Dell OMSA Service Notifications', 'comment': 'Standard set of notifications sent to all Check_MK users.\n\nThis includes:\nDell OMSA issues (RAID/RAM/CPU failures, high temps, etc)\n\n', 'docu_url': '', 'disabled': False, 'allow_disable': False, 'match_services': ['Dell', 'Management Interface:'], 'match_timeperiod': 'WorkingHours', 'match_service_event': ['rw', 'rc', 'wr', 'wc', 'cr', 'cw', 'x'], 'contact_object': False, 'contact_all': False, 'contact_all_with_email': False, 'contact_groups': ['all'], 'notify_plugin': ('mail', {'elements': ['address', 'abstime', 'longoutput']}), 'bulk': ('always', {'interval': 60, 'count': 1000, 'groupby': ['host'], 'groupby_custom': [], 'bulk_subject': 'Check_MK: $COUNT_NOTIFICATIONS$ notifications for $COUNT_HOSTS$ hosts'})}, {'comment': '', 'match_service_event': ['rw', 'rr', 'rc', 'ru', 'wr', 'wc', 'cr', 'cw', 'ur', 'x'], 'description': 'Exchange Notifications', 'contact_groups': ['ScottCarter'], 'disabled': False, 'notify_plugin': ('mail', {'no_floating_graphs': True, 'elements': ['address', 'abstime', 'longoutput', 'graph']}), 'match_hostgroups': ['exchangeservers'], 'docu_url': '', 'contact_all': False, 'contact_object': False, 'allow_disable': False, 'contact_all_with_email': False, 'match_services': ['Exchange']}, {'description': 'Veeam Backup Jobs', 'comment': '', 'docu_url': '', 'disabled': True, 'allow_disable': True, 'match_services': ['VEEAM Job', 'Dell'], 'contact_object': False, 'contact_all': False, 'contact_all_with_email': False, 'contact_users': ['Teams'], 'notify_plugin': ('teams.py', ['https://<removed for privacy>'])}]

Good Morning @CarterScottM,

thank you for supplying the information, I forwarded it to our developers.
The internal ticket is scheduled for development.

Sunny Greetings
Hartmut

Good Morning @CarterScottM,

the issue has been fixed - it will be released with 2.3.0p34

So you would have to update your 2.3 instance first and then upgrade to 2.4

Sunny Greetings
Hartmut

2 Likes

What to do if I already upgraded to 2.4, before the fix was released?

Hi @guglez,

I see. I reached out internally to the developer.

Sunny Greetings
Hartmut

Good Morning @guglez,

the fix has also been picked to the 2.4 branch now as well - available with patch version 2.4.0p5
So you should be covered without going back to 2.3

Sunny Greetings
Hartmut

1 Like

I got same problem even I upgraded from 2.4.0p11 to 2.4.0p16 - is there any permanent fix?