Local site MYSITE - Livestatus error Malformed raw response output

[mccheck]

Hello there
since today i got the error (title/screenshot). can you help me troubleshoot? is there a way to “clear” this log?

CMK version: Checkmk Raw Edition 2.1.0p9
OS version: Debian 11

Error message: Local site MYSITE - Livestatus error Malformed raw response output

OMD[monitoring]:/$ omd status
agent-receiver: running
mkeventd:       running
rrdcached:      running
npcd:           running
nagios:         running
apache:         running
redis:          running
crontab:        running
-----------------------
Overall state:  running

OMD[monitoring]:/$ cmk --paths
Files copied or created during installation
  Main components of check_mk                    : /omd/sites/monitoring/share/check_mk/modules/
  Checks                                         : /omd/sites/monitoring/share/check_mk/checks/
  Notification scripts                           : /omd/sites/monitoring/share/check_mk/notifications/
  Inventory plugins                              : /omd/sites/monitoring/share/check_mk/inventory/
  Agents for operating systems                   : /omd/sites/monitoring/share/check_mk/agents/
  Documentation files                            : /omd/sites/monitoring/share/doc/check_mk/
  Check_MK's web pages                           : /omd/sites/monitoring/share/check_mk/web/
  Check manpages (for check_mk -M)               : /omd/sites/monitoring/share/check_mk/checkman/
  Binary plugins (architecture specific)         : /omd/sites/monitoring/lib/
  Templates for PNP4Nagios                       : /omd/sites/monitoring/share/check_mk/pnp-templates/
  Startscript for Nagios daemon                  : /omd/sites/monitoring/etc/init.d/core
  Path to Nagios executable                      : /omd/sites/monitoring/bin/nagios

Configuration files edited by you
  Main configuration file of Nagios              : /omd/sites/monitoring/tmp/nagios/nagios.cfg
  Directory where Nagios reads all *.cfg files   : /omd/sites/monitoring/etc/nagios/conf.d/
  Directory that contains main.mk                : /omd/sites/monitoring/etc/check_mk/
  Directory containing further *.mk files        : /omd/sites/monitoring/etc/check_mk/conf.d/
  Directory where Apache reads all config files  : /omd/sites/monitoring/etc/apache/
  Users/Passwords for HTTP basic authentication  : /omd/sites/monitoring/etc/htpasswd

Data created by Nagios/Check_MK at runtime
  File into which Nagios configuration is written: /omd/sites/monitoring/etc/nagios/conf.d/check_mk_objects.cfg
  Path to Nagios status.dat                      : /omd/sites/monitoring/tmp/nagios/status.dat
  Base working directory for variable data       : /omd/sites/monitoring/var/check_mk/
  Checks found by inventory                      : /omd/sites/monitoring/var/check_mk/autochecks/
  Precompiled host checks                        : /omd/sites/monitoring/var/check_mk/precompiled/
  Stored snmpwalks (output of --snmpwalk)        : /omd/sites/monitoring/var/check_mk/snmpwalks/
  Current state of performance counters          : /omd/sites/monitoring/tmp/check_mk/counters/
  Cached output from agents                      : /omd/sites/monitoring/tmp/check_mk/cache/
  Unacknowledged logfiles of logwatch extension  : /omd/sites/monitoring/var/check_mk/logwatch/

Sockets and pipes
  Nagios' command pipe                           : /omd/sites/monitoring/tmp/run/nagios.cmd
  Nagios' check results directory                : /omd/sites/monitoring/tmp/nagios/checkresults
  Socket of Check_MK's livestatus module         : /omd/sites/monitoring/tmp/run/live

Locally installed addons
  Locally installed checks                       : /omd/sites/monitoring/local/share/check_mk/checks/
  Locally installed notification scripts         : /omd/sites/monitoring/local/share/check_mk/notifications/
  Locally installed inventory plugins            : /omd/sites/monitoring/local/share/check_mk/inventory/
  Locally installed check man pages              : /omd/sites/monitoring/local/share/check_mk/checkman/
  Locally installed agents and plugins           : /omd/sites/monitoring/local/share/check_mk/agents/
  Locally installed Multisite addons             : /omd/sites/monitoring/local/share/check_mk/web/
  Locally installed PNP templates                : /omd/sites/monitoring/local/share/check_mk/pnp-templates/
  Locally installed documentation                : /omd/sites/monitoring/local/share/doc/check_mk/
  Locally installed localizations                : /omd/sites/monitoring/local/share/check_mk/locale/
OMD[monitoring]:/$

thanks a lot

edit: i forgott, it’s happening, after i do a “fake check” on a host, to test notifications…
i did already a reboot, its not helping

[mccheck]

Problem found, there was a plugin from me writing garbage in the nagios.log… i fixed it.

[schlarbm]

I’m having the same problem - what kind of garbage was the plugin writing?

[mccheck]

check you nagios.log (last 4 hours) for some unexpected entries

root@checkmk:~# omd su YOURSITE
OMD[YOURSITE]:~$ cd ~/var/nagios
OMD[YOURSITE]:~/var/nagios$ cat nagios.log

[schlarbm]

@mccheck

check you nagios.log (last 4 hours) for some unexpected entries

That’s still too many lines for me to manually check…

However, I found the reason for the problem in my case:
I had some acknowledgement comments that contained fancy unicode emojis and apparently that broke in the step of decoding the livestatus response.

I fixed it by looking through grep comment_data ~/var/nagios/retention.dat and manually fixing the broken lines there.

I’ve reported that to the developers (FEED-7279).

[AppealingVocally]

I get this error when I open one of the top alerters in the past 7 days:

I see no files in ~/var/nagios.

Absolutely no clue how to debug this.

Running CFE 2.1.0p9 in a Docker container.

[finalduty (Andy)]

I was having the same issue as schlarbm and found that some of the custom checks we’d written were outputting emoji, which were causing this problem. After removing the emoji the issue went away