Systemd Service Summary is reporting too long times for activating or reloading a systemd service

Troubleshooting
,
1

CMK version: Raw 2.3.0p25-1 (Server and agent)
OS version: Debian 12

EDIT: I have reworked this post to be more understandable.

I dont have a problem with phpsessionclean.service especially.
In my case, Systemd Service Summary is reporting too long times for activating or reloading a systemd service because the agent sends in the same output:

  • first the timestamp of the last state change and
  • a moment later the new state "activating.

My gues ist: the timestamp comes from the output of the second command in the agent section <<<section_systemd >>>

systemctl list-unit-files --full --no-legend --no-pager --plain --type service --type socket | tr -s ' '

and the state (change) comes from the output of the third command in the agent section <<<section_systemd >>>

systemctl --all --type service --type socket --full --no-legend --no-pager --plain | sed '/^$/q' | tr -s ' '

My question: Is the agent sporadically sending non consistent output for the systemd services which leads to this notification?

Examples:

Local time: 20:29

Active: inactive (dead) since Wed 2025-02-19 20:09:00 CET; 29min ago)

phpsessionclean.service loaded activating start start Clean php session files

leads to: phpsesssionclean activating for: 29 minutes 0 seconds

and just arrived (until now I got only the notification):

Service apache2 reloading for: 11 days 0 hours

This apache2.service isn’t reloading for 11 days :smile:

To find the cause I have a tcpdump for localhost port 6556 and agent in legacy mode.
At the time of the last notification I found this in the tcpdump capture on agent output (strong markup from me):

… phpsessionclean.service - Clean php session files
Loaded: loaded (/lib/systemd/system/phpsessionclean.service; static)
Active: inactive (dead) since Wed 2025-02-19 20:09:00 CET; 29min ago
TriggeredBy: … phpsessionclean.timer
Process: 3761458 ExecStart=/usr/lib/php/sessionclean (code=exited, status=0/SUCCESS)
Main PID: 3761458 (code=exited, status=0/SUCCESS) CPU: 201ms

and in the same agent output:

phpsessionclean.service loaded activating start start Clean php session files

I appreciate your help and say Thank you in advance.

1 Like
2

Hey @mbohal ,

welcome back to the Forums :slight_smile:

Monitoring checkmk with tcpdump, how neat is that… :star_struck:

I was writing a long text about systemctl status phpsessionclean.service --type service --type socket --no-pager --lines 0 | tr -s ' ' but then I read your text again and I’ve seen you already have that output…

TL;DR

You could/should filter phpsessionclean.service from the output, because that service is happy to be dead and get triggered by the .timer.

Insights

The agent sends the output of systemd to the server here:

section_systemd() {
    if inpath systemctl; then
        echo '<<<systemd_units>>>'
        # use plain to force ASCII output that is simpler to parse
        echo "[list-unit-files]"
        systemctl list-unit-files --full --no-legend --no-pager --plain --type service --type socket | tr -s ' '
        echo "[status]"
        systemctl status --all --type service --type socket --no-pager --lines 0 | tr -s ' '
        echo "[all]"
        systemctl --all --type service --type socket --full --no-legend --no-pager --plain | sed '/^$/q' | tr -s ' '
    fi
}

Agent and Server don’t know that a service “can” be dead thats why i said you should disable/filter the service.

Kind regards,
René

3

Hey jochum,

thank your for your reply and your explanation. But I don’t have a problem with phpsessionclean.service especially, but Systemd Service Summary is sporadically reporting too long times for activating or reloading a systemd service.

Please take a look at my reworked initial post and my received notification:

Service apache2 reloading for: 11 days 0 hours
4

This is the necrobump of the century, but strangely enough, we see the same behavior over here as well, on CheckMK’s own monitoring node.

image
image1584×235 42.1 KB

I’m pretty sure that Apache is alive considering I can access the dashboard, so this is essentially monitoring noise :frowning: