Webhook producing 404 error

giliredbaron · June 25, 2023, 3:36pm

CMK version: 2.1.0p24.cre
OS version: Ubuntu 22.04.1 LTS

Hi All

I followed this manual: Notifications via Slack

When I send a message using checkmk (monitor → all hosts → my_host → commands → fake check results)
It failed with the error below.

When using the curl from slack website (curl -X POST -H ‘Content-type: application/json’ --data ‘{“text”:“Hello, World!”}’ Unlock your productivity potential with Slack Platform | Slack[my tocken]

It works ok…

What I’m doing wrong? What did I miss?

This is the error I’m getting:
Failed to send notification. – Response: no_service – 404: Not Found

Also available from this log: /opt/omd/sites/monitoring/var/log/notify.log

2023-06-25 15:21:25,081 [20] [cmk.base.notify] Got raw notification (my_server ;Check_MK) context with 52 variables
2023-06-25 15:21:25,082 [20] [cmk.base.notify] Global rule 'Test Script'...
2023-06-25 15:21:25,083 [20] [cmk.base.notify]  -> matches!
2023-06-25 15:21:25,083 [20] [cmk.base.notify] Global rule 'Notify all contacts of a host/service via HTML email'...
2023-06-25 15:21:25,083 [20] [cmk.base.notify]  -> matches!
2023-06-25 15:21:25,083 [20] [cmk.base.notify] User cmkadmin's rule ''...
2023-06-25 15:21:25,083 [20] [cmk.base.notify]  -> matches!
2023-06-25 15:21:25,083 [20] [cmk.base.notify]    - adding notification of cmkadmin via slack
2023-06-25 15:21:25,084 [15] [cmk.base.notify] User cmkadmin's rule 'test'...
2023-06-25 15:21:25,084 [15] [cmk.base.notify]  -> does not match: The site 'monitoring' is not in the required sites list:
2023-06-25 15:21:25,084 [20] [cmk.base.notify] Executing 1 notifications:
2023-06-25 15:21:25,084 [20] [cmk.base.notify]   * notifying cmkadmin via slack, parameters: webhook_url, proxy_url, bulk: no
2023-06-25 15:21:25,085 [20] [cmk.base.notify]      executing /omd/sites/monitoring/share/check_mk/notifications/slack
2023-06-25 15:21:25,876 [20] [cmk.base.notify]      Output: Failed to send notification.
2023-06-25 15:21:25,877 [20] [cmk.base.notify]      Output: Response: no_service
2023-06-25 15:21:25,938 [20] [cmk.base.notify]      Output: 404: Not Found
2023-06-25 15:21:25,938 [20] [cmk.base.notify]      Plugin exited with code 2

Error message:

**Output of “cmk --debug -vvn hostname”:
Checkmk version 2.1.0p24
Try license usage history update.
Trying to acquire lock on /omd/sites/monitoring/var/check_mk/license_usage/next_run
Got lock on /omd/sites/monitoring/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/monitoring/var/check_mk/license_usage/history.json
Got lock on /omd/sites/monitoring/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/monitoring/var/check_mk/license_usage/history.json
Released lock on /omd/sites/monitoring/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/monitoring/var/check_mk/license_usage/next_run
Released lock on /omd/sites/monitoring/var/check_mk/license_usage/next_run

FETCHING DATA
Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7ffaaf3a43d0]
[PiggybackFetcher] Fetch with cache settings: NoCache(monitoring, base_path=/omd/sites/monitoring/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for ‘monitoring’. Skip processing.
No piggyback files for ‘127.0.1.1’. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7ffaaf3a43d0 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
PARSE FETCHER RESULTS
Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
→ Add sections: []
Received no piggyback data
[cpu_tracking] Start [7ffaaf347580]
value store: synchronizing
Trying to acquire lock on /omd/sites/monitoring/tmp/check_mk/counters/monitoring
Got lock on /omd/sites/monitoring/tmp/check_mk/counters/monitoring
value store: loading from disk
Releasing lock on /omd/sites/monitoring/tmp/check_mk/counters/monitoring
Released lock on /omd/sites/monitoring/tmp/check_mk/counters/monitoring
No piggyback files for ‘monitoring’. Skip processing.
No piggyback files for ‘127.0.1.1’. Skip processing.
[cpu_tracking] Stop [7ffaaf347580 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.010000001639127731))]
execution time 0.0 sec | execution_time=0.010 user_time=0.000 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=0.000** (If it is a problem with checks or plugins)
**

masamon · June 25, 2023, 4:58pm

Hi @giliredbaron,

I’m not using the slack notification Plugin, so just a guess.
For the Slack API no_service means the webhook is either disabled, removed, or invalid. So maybe there is still something wrong with the configuration of the Plugin in the checkmk GUI?
If you tested the API call with curl I guess it can’t be disabled.

Anders · June 25, 2023, 6:11pm

What URL did you use in checkmk?

giliredbaron · June 26, 2023, 7:44am

I believe that this is something wrong with my configuration…
although, my CURL test works OK.

giliredbaron · June 26, 2023, 7:45am

I used this URL:
Screenshot 2023-06-26 at 10.39.55

masamon · June 26, 2023, 7:48pm

Maybe you can also share the configuration of the slack plugin and the rule which needs to get matched to send the slack mesage

giliredbaron · June 27, 2023, 6:57am

This is my config (screenshot)
Can I pull the configuration in a text format also?

giliredbaron · June 27, 2023, 7:04am

Also, is there a way to run this script manually?

/omd/sites/monitoring/share/check_mk/notifications/slack

OMD[monitoring]:~$ /omd/sites/monitoring/share/check_mk/notifications/slack
Traceback (most recent call last):
  File "/omd/sites/monitoring/share/check_mk/notifications/slack", line 13, in <module>
    response = post_request(slack_msg)
  File "/omd/sites/monitoring/lib/python3/cmk/notification_plugins/utils.py", line 317, in post_request
    url = retrieve_from_passwordstore(context.get("PARAMETER_WEBHOOK_URL"))
  File "/omd/sites/monitoring/lib/python3/cmk/notification_plugins/utils.py", line 300, in retrieve_from_passwordstore
    value = parameter.split()
AttributeError: 'NoneType' object has no attribute 'split'

masamon · June 28, 2023, 6:24pm

Config seems fine to me, besides any possible typos in the URL (including ending spaces).

I’m not quit sure if its easily possible to run the plugin on the command line.
Maybe a look to the codebase helps you

github.com

Checkmk/checkmk/blob/master/notifications/slack

#!/usr/bin/env python3
# Slack or Mattermost

# Copyright (C) 2019 Checkmk GmbH - License: GNU General Public License v2
# This file is part of Checkmk (https://checkmk.com). It is subject to the terms and
# conditions defined in the file COPYING, which is part of this source code package.

import sys

from cmk.notification_plugins.slack import main

if __name__ == "__main__":
    sys.exit(main())

github.com

Checkmk/checkmk/blob/master/cmk/notification_plugins/slack.py

#!/usr/bin/env python3
# Copyright (C) 2019 Checkmk GmbH - License: GNU General Public License v2
# This file is part of Checkmk (https://checkmk.com). It is subject to the terms and
# conditions defined in the file COPYING, which is part of this source code package.
r"""
Send notification messages to Slack
===================================

Use a slack webhook to send notification messages
"""
from cmk.notification_plugins.utils import (
    format_link,
    host_url_from_context,
    post_request,
    process_by_status_code,
    service_url_from_context,
)

COLORS = {
    "CRITICAL": "#EE0000",

This file has been truncated. show original

My last idea to get more information would be to increase the log Level and test it again.
Log Level for the Notification system can be increased via Global Settings.

system · June 27, 2024, 6:25pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.