"Persistent" ACK comments cannot be removed

CMK version: CEE 2.1.0p13
OS version: Ubuntu 20.04 LTS

Error message:


Not sure how we got into this situation, but there are “persistent” comments in our site, originally entered alongside acknowledgements - but we cannot get rid of them!
Service is back to OK for months…

The “Remove comments” button does nothing, artificially setting the service to non-OK and adding a new ACK simply adds a second comment with the same “E. Type Acknowledgement” icon and once the service returns OK the second comment goes away correctly - but the original one remains.

I think you’ve activated the option “persistent comment” when you acknowledged the problem.


I believe it is correct that a “persistent” comment does not disappear once the service goes back to OK - this is the difference to a “sticky” ACK comment.

→ but according to documentation (and common sense) I should be able to remove it nevertheless?
Does this happen for me only or is there a bug that prevents persistent comments from being removed?
I can trigger it at will, fake a check result, add ACK with “persistent” set, wait for check to go back to OK on next check interval (or fake OK result), find the comment still there (expected), find out there is no way to delete it.


Oh no!

I ran into the same problem. Even if I remove the service and rescan (remove all and find new) it does not disappear.

Seems to be a bug not fixed. No workarounds or explanation provided.

Report to feedback@checkmk.com and hope for the best.

#Not able to delete some comments#

Hi all! I checked with the team – the issue is known and there is a work in progress in fixing it :ok_hand:

I’ll keep you updated – as soon as there’s an update on resolving it, I’ll let you know in this thread.


Nice! :slight_smile: Hopefully you release it soon :slight_smile:

1 Like

Great news @Sara, thanks! :slight_smile:

Hi @Sara
I have the same problem with: 2.1.0p18.cme running in Ubuntu 20.04.
Is there any news about this issue?
Thanks, Peter

No news for now, unfortunately, but I am following the progress: as soon as I get any updates, I will let you know in this thread

2.1.0p22 CEE - issue still persist.

I mean this shouldn’t be rocket science and the issue is open since August 2022 - where I sent this information to CheckMK support.

It’s a pain in the a** for our engineers with the old comments. We have comments back to June 2022 on very important services.


@Sara: Is there a way to remove the acknowledgment on the command line? Where are they stored? I assume inside redis, but I have no idea how to talk to redis and remove the old comments there.

As a workaround, I am now used to add an expiration time on new acknowledgments/comments, but the old ones are really disturbing.

@PeterE no they are not in Redis. They are stored in the state of the core, which you do want to manipulate manually. Maybe someone else can provide a workaround, but you want to be very cautious here.

What I have to say: Why would you use persistent comments in the first place? I never found a good use case for this. Normal comments maybe. But with an acknowledgement you do you want a comment that persists after the acknowledgement perished.

We are looking into the issue at hand regardless.

@robin.gierse We do not use “persistent” comments. The comments are only flagged as ‘sticky’, which is the default setting, and I never flag comments as ‘persistent’. Still, they persist in the system and cannot be deleted. It would be ok to flush all comments / acknowledgments from the system, if that is the easiest and safest solution.

Is this the file var/check_mk/core/state.pb ? It contains the messages I am trying to get rid of, but that file is a binary. Is there a dump/restore procedure which would allow me to remove the comments?

I now tried to reproduce the problem, and I cannot. Normal Ack’s can be deleted. :sweat_smile:
I am using “Checkmk Managed Services Edition 2.1.0p18”.
Perhaps the problem was solved. I will watch out for old ‘acknowledge messages’ when they pop up again and will try to delete them as well. They only pop up if something is not OK, which can take a while.
Not sure, if it solves the problem reported by @bitwiz as well, so please do not close the thread.

Just to confirm we still have this issue with Persistent comments as you noted so keep that separate in the future :slight_smile:.

Unable to remove about 3k comments.

Response to @robin.gierse

“What I have to say: Why would you use persistent comments in the first place?”

We use it in multiple scenarios both with automation and manually. We use persistent both with ACK’s to avoid important info/case number/status getting lost after state changes.

Under normal conditions (before this bug) old persistent comments were sporadically deleted manually after about 100 days trough a dashboard.

If you have any input for an improved work-flow i’m all ears.

Good news – there is a werk with a fix: Fixed removal of persistent acknowledgements :partying_face:

Not sure about when it will be released but it is definitely not far now.

1 Like

Thanks for the info, great news indeed! :slight_smile: