CMK version: CEE 2.1.0p13 OS version: Ubuntu 20.04 LTS
Error message:
Not sure how we got into this situation, but there are “persistent” comments in our site, originally entered alongside acknowledgements - but we cannot get rid of them!
Service is back to OK for months…
The “Remove comments” button does nothing, artificially setting the service to non-OK and adding a new ACK simply adds a second comment with the same “E. Type Acknowledgement” icon and once the service returns OK the second comment goes away correctly - but the original one remains.
I believe it is correct that a “persistent” comment does not disappear once the service goes back to OK - this is the difference to a “sticky” ACK comment.
→ but according to documentation (and common sense) I should be able to remove it nevertheless?
Does this happen for me only or is there a bug that prevents persistent comments from being removed?
I can trigger it at will, fake a check result, add ACK with “persistent” set, wait for check to go back to OK on next check interval (or fake OK result), find the comment still there (expected), find out there is no way to delete it.
@Sara: Is there a way to remove the acknowledgment on the command line? Where are they stored? I assume inside redis, but I have no idea how to talk to redis and remove the old comments there.
As a workaround, I am now used to add an expiration time on new acknowledgments/comments, but the old ones are really disturbing.
@PeterE no they are not in Redis. They are stored in the state of the core, which you do want to manipulate manually. Maybe someone else can provide a workaround, but you want to be very cautious here.
What I have to say: Why would you use persistent comments in the first place? I never found a good use case for this. Normal comments maybe. But with an acknowledgement you do you want a comment that persists after the acknowledgement perished.
@robin.gierse We do not use “persistent” comments. The comments are only flagged as ‘sticky’, which is the default setting, and I never flag comments as ‘persistent’. Still, they persist in the system and cannot be deleted. It would be ok to flush all comments / acknowledgments from the system, if that is the easiest and safest solution.
Is this the file var/check_mk/core/state.pb ? It contains the messages I am trying to get rid of, but that file is a binary. Is there a dump/restore procedure which would allow me to remove the comments?
I now tried to reproduce the problem, and I cannot. Normal Ack’s can be deleted.
I am using “Checkmk Managed Services Edition 2.1.0p18”.
Perhaps the problem was solved. I will watch out for old ‘acknowledge messages’ when they pop up again and will try to delete them as well. They only pop up if something is not OK, which can take a while.
Not sure, if it solves the problem reported by @bitwiz as well, so please do not close the thread.
Update: In the meantime, I found out, that the old ack messages are still there and I cannot delete them. So, probably, I have marked them as persistent. Looking forward to werk 15255!
“What I have to say: Why would you use persistent comments in the first place?”
We use it in multiple scenarios both with automation and manually. We use persistent both with ACK’s to avoid important info/case number/status getting lost after state changes.
Under normal conditions (before this bug) old persistent comments were sporadically deleted manually after about 100 days trough a dashboard.
If you have any input for an improved work-flow i’m all ears.
I’ve upgraded to 2.1.0p25 and the problem disappeared for me. I can now delete the old persistent acknowledgments. I had to delete the acknowledgments service-by-service, however. I tried to select all “OK” services and do a bulk delete of the acknowledgments, but it did not work. I had only about 20 services with this problem, so doing it by hand was ok for me.
Hopefully you release it soon 
