Hey, I am new to CheckMK and using raw edition and have a query related to it.
Can we monitor CheckMK own services by itself and trigger alert when it itself goes down. Is it possible ?
Since, as per my understanding if the core itself goes down the agent installed on its own server will not be able to send the information to the core. Do I need to monitor by different core or can it be done by itself. Please provide the information
This can be done using a distributed setup and you have the “remote” site monitoring the CheckMK instance health on the “other” site.
Ie…
Site A - Checkmk local (master) on hosta.
Site B - Checkmk remote on hostb.
Configure hosta@SiteA to be monitored by Site B.
Configure hostb@SiteB to be monitored by Site A.
If Checkmk goes down on hosta@SiteA notifications will be triggered from SiteB.
A simple solution (which I use) is to set up an HTTP (web service) check from the checkmk host where it ‘pings’ healthchecks.io and if it stops pinging healthchecks.io notifies me that checkmk is down.
A little late to the party, but giving a local option, assuming you have an MTA or other SMTP email connection on the CheckMK server to send email itself
Cronjobs
1 * * * * /bin/bash /root/restart_partly_running_sites.sh
*/15 * * * * /root/monitor_omd_and_site-.py
monitor_omd_and_site-craig.py
#!/usr/bin/env python3
#WARNING: AI-assisted coding
import subprocess
import smtplib
import re
import argparse
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
# Function to check the overall state from omd status
def check_overall_state(verbose=False):
# Run the omd status command
result = subprocess.run(['omd', 'status', '<SITENAME>'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# Use regex to match "Overall state: running" with arbitrary whitespace after the colon
if re.search(r"Overall state:\s*running", result.stdout):
return True, result.stdout # The site is running
else:
return False, result.stdout # The site is not running
# Function to check the systemd service status for omd
def check_systemd_service(verbose=False):
# Run the systemctl status command for omd
result = subprocess.run(['systemctl', 'status', 'omd'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# Initialize the string to store the relevant lines
service_output = ""
# Process the stdout to get the second line and the last 15 lines
output_lines = result.stdout.splitlines()
# Get the second line (index 1), if it exists
if len(output_lines) > 2:
service_output += f"{output_lines[2]}\n"
# Get the last 10 lines (or fewer if there aren't 10 lines)
last_10_lines = '\n'.join(output_lines[-10:])
service_output += f"{last_10_lines}\n"
# Check if the service is "Active: active"
if "Active: active" in result.stdout:
return True, service_output # The service is active
else:
return False, service_output # The service is not active
# Function to send an email using a local unencrypted port 25 relay
def send_email(body, subject, to_email, verbose=False):
from_email = '<EMAIL_3>' # The sender's email address
msg = MIMEMultipart()
msg['From'] = from_email
msg['To'] = to_email
msg['Subject'] = subject
msg.attach(MIMEText(body, 'plain'))
# Send the email using SMTP server over port 25 (unencrypted)
try:
# Local relay address <LOCAL_MTA> on port 25
server = smtplib.SMTP('<LOCAL_MTA>', 25)
server.sendmail(from_email, to_email, msg.as_string())
server.quit()
if verbose:
print("Email sent successfully!")
except Exception as e:
if verbose:
print(f"Failed to send email: {e}")
else:
# Fail silently if email fails
pass
# Main function
def main():
# Parse command-line arguments
parser = argparse.ArgumentParser(description="Check the OMD site and service status.")
parser.add_argument('-v', '--verbose', action='store_true', help="Enable verbose output (stdout/stderr).")
args = parser.parse_args()
# Check omd status
site_running, site_output = check_overall_state(verbose=args.verbose)
# Check omd systemd service status
service_active, service_output = check_systemd_service(verbose=args.verbose)
# Only send an email if there is a failure
if not site_running and not service_active:
body = (f"Both the site and the OMD service are in a failed state.\n\n"
f"Site Status Output:\n{site_output}\n\n"
f"OMD Service Status Output:\n{service_output}")
subject = "Site and OMD Service Status: Both Failed"
to_emails = ["<EMAIL_1>", "<EMAIL_2>"]
for email in to_emails:
send_email(body, subject, email, verbose=args.verbose)
if args.verbose:
print("Site and omd service both failed")
elif not site_running:
body = (f"The site is stopped or not running.\n\n"
f"Site Status Output:\n{site_output}")
subject = "Site Status: Stopped"
to_emails = ["<EMAIL_1>", "<EMAIL_2>"]
for email in to_emails:
send_email(body, subject, email, verbose=args.verbose)
if args.verbose:
print("Site failed")
elif not service_active:
body = (f"The OMD service is inactive or failed.\n\n"
f"OMD Service Status Output:\n{service_output}")
subject = "OMD Service Status: Inactive or Failed"
to_emails = ["<EMAIL_1>", "<EMAIL_2>"]
for email in to_emails:
send_email(body, subject, email, verbose=args.verbose)
if args.verbose:
print("Systemd service failed")
elif service_active and site_running and args.verbose:
print(f"Both site and service are running.\n{site_output}\n\n{service_output}")
if __name__ == '__main__':
main()
Restart sites script:
#!/bin/bash
service_name="omd"
#Checks if the systemd service is active and restart if not
if systemctl is-active --quiet "${service_name}.service" ; then
echo "foobar" > /dev/null
else
systemctl restart "${service_name}"
fi
#Loop through all sites on this host and start any that are partly running. Any fully stopped should not be started by this script
for SITE in $(omd sites --bare); do
if [[ $(omd status $SITE|grep 'partially running' -c) -ge 1 ]]; then
(date; omd restart $SITE; echo -e "\n\n-------------------------------------------\n\n") >> /tmp/restart_cmk_sites.log
fi
done