Version 2.0.0p3: PING Service NO Service Graphs

KNT · April 22, 2021, 9:29am

Hello!
Next strange issue regarding Service Graph of PING-Service. With version 1.6 I had nice graphs, in version 2.0.0p3 graphs disappeared with notice: “No historic metrics recorded but performance data is available. Maybe performance data processing is disabled.”

Graphs also not appearing in the notification mails.
In 1.6 it worked:

Had a similar problem with almost all graphs not showing in version p2, got fixed with p3 : Checkmk 2.0.0p2 - Graphs not loading - #5 by KNT

Any hint for me?

albzundy · July 5, 2021, 1:36pm

I had the same problem with 2.0.0p1.
I updated to 2.0.0p6 but the problem still exists:

@ KNT do you found a solution meanwhile ?

andreas-doehler · July 6, 2021, 7:46pm

If you use also the RAW edition, you can take look at the log files of the pnp4nagios / rrdcached.
I think there can be found some error messages.
→ ~/var/pnp4nagios/log/ and ~/var/log/rrdcached.log

One other thing - processing of performance data is switched on in your site?

albzundy · July 7, 2021, 7:15am

Yes performance data is enabled:
grafik
or is there any other place where I can control this?

In the /omd/sites/tphmon/var/pnp4nagios/log/perfdata.log are lots of these lines with different hostnames, in this example only 2 visible:
The oldest log entry is from 2020-09-29 and continious like this

2021-07-06 14:10:04 [3952] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__pl.rrd 1625573386:80
2021-07-06 14:10:04 [3952] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625573386.000000 when last update time is 1625573387.000000 (minimum one second step)
2021-07-06 14:10:04 [3952] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__rtmax.rrd 1625573386:0.290
2021-07-06 14:10:04 [3952] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625573386.000000 when last update time is 1625573387.000000 (minimum one second step)
2021-07-06 14:10:04 [3952] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__rtmin.rrd 1625573386:0.290
2021-07-06 14:10:04 [3952] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625573386.000000 when last update time is 1625573387.000000 (minimum one second step)
2021-07-06 14:52:07 [4824] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW23003/Check_MK_execution_time.rrd 1625575918:6.070
2021-07-06 14:52:07 [4824] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625575918.000000 when last update time is 1625575918.000000 (minimum one second step)
2021-07-06 14:52:07 [4824] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW23003/Check_MK_user_time.rrd 1625575918:0.050
2021-07-06 14:52:07 [4824] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625575918.000000 when last update time is 1625575918.000000 (minimum one second step)
2021-07-06 14:52:07 [4824] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW23003/Check_MK_system_time.rrd 1625575918:0.000
2021-07-06 14:52:07 [4824] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625575918.000000 when last update time is 1625575918.000000 (minimum one second step)
2021-07-06 14:52:07 [4824] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW23003/Check_MK_children_user_time.rrd 1625575918:0.000
2021-07-06 14:52:07 [4824] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625575918.000000 when last update time is 1625575918.000000 (minimum one second step)
2021-07-06 14:52:07 [4824] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW23003/Check_MK_children_system_time.rrd 1625575918:0.000
2021-07-06 14:52:07 [4824] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625575918.000000 when last update time is 1625575918.000000 (minimum one second step)
2021-07-06 14:52:07 [4824] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW23003/Check_MK_cmk_time_snmp.rrd 1625575918:6.020
2021-07-06 14:52:07 [4824] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625575918.000000 when last update time is 1625575918.000000 (minimum one second step)
2021-07-06 14:52:07 [4824] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW23003/Check_MK_cmk_time_agent.rrd 1625575918:0.000
2021-07-06 14:52:07 [4824] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625575918.000000 when last update time is 1625575918.000000 (minimum one second step)
2021-07-07 06:00:16 [27226] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__rta.rrd 1625630399:0.343
2021-07-07 06:00:16 [27226] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625630399.000000 when last update time is 1625630402.000000 (minimum one second step)
2021-07-07 06:00:16 [27226] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__pl.rrd 1625630399:80
2021-07-07 06:00:16 [27226] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625630399.000000 when last update time is 1625630402.000000 (minimum one second step)
2021-07-07 06:00:16 [27226] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__rtmax.rrd 1625630399:0.343
2021-07-07 06:00:16 [27226] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625630399.000000 when last update time is 1625630402.000000 (minimum one second step)
2021-07-07 06:00:16 [27226] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__rtmin.rrd 1625630399:0.343
2021-07-07 06:00:16 [27226] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625630399.000000 when last update time is 1625630402.000000 (minimum one second step)
2021-07-07 07:59:18 [25062] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__rta.rrd 1625637539:0.328
2021-07-07 07:59:18 [25062] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625637539.000000 when last update time is 1625637543.000000 (minimum one second step)
2021-07-07 07:59:18 [25062] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__pl.rrd 1625637539:80
2021-07-07 07:59:18 [25062] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625637539.000000 when last update time is 1625637543.000000 (minimum one second step)
2021-07-07 07:59:18 [25062] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__rtmax.rrd 1625637539:0.328
2021-07-07 07:59:18 [25062] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625637539.000000 when last update time is 1625637543.000000 (minimum one second step)
2021-07-07 07:59:18 [25062] [0] RRDs::update /omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPELIB02/_HOST__rtmin.rrd 1625637539:0.328
2021-07-07 07:59:18 [25062] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/MYSITE/tmp/run/rrdcached.sock: illegal attempt to update using time 1625637539.000000 when last update time is 1625637543.000000 (minimum one second step)

a LOT errors

The other log file in this directory npcd.log has not much entries:

root@tphcmk:/omd/sites/MYSITE/var/pnp4nagios/log# cat npcd.log 
[09-29-2020 19:58:58] NPCD: npcd Daemon (0.6.26) started with PID=16648
[09-29-2020 19:58:58] NPCD: Please have a look at 'npcd -V' to get license information
[09-29-2020 19:58:58] NPCD: HINT: load_threshold is disabled - ('0.000000')
[10-22-2020 10:06:32] NPCD: Caught Termination Signal - Astalavista... baby
[10-22-2020 10:06:36] NPCD: npcd Daemon (0.6.26) started with PID=5062
[10-22-2020 10:06:36] NPCD: Please have a look at 'npcd -V' to get license information
[10-22-2020 10:06:36] NPCD: HINT: load_threshold is disabled - ('0.000000')
[03-02-2021 18:38:25] NPCD: Caught Termination Signal - Astalavista... baby
[03-02-2021 18:38:33] NPCD: npcd Daemon (0.6.26) started with PID=13267
[03-02-2021 18:38:33] NPCD: Please have a look at 'npcd -V' to get license information
[03-02-2021 18:38:33] NPCD: HINT: load_threshold is disabled - ('0.000000')
[03-17-2021 16:07:17] NPCD: Caught Termination Signal - Astalavista... baby
[03-17-2021 16:14:08] NPCD: npcd Daemon (0.6.26) started with PID=21742
[03-17-2021 16:14:08] NPCD: Please have a look at 'npcd -V' to get license information
[03-17-2021 16:14:08] NPCD: HINT: load_threshold is disabled - ('0.000000')
[04-14-2021 12:08:11] NPCD: Caught SIGHUP - reloading configuration
[04-14-2021 12:08:11] NPCD: Configuration reload succesfull.
[04-14-2021 12:08:11] NPCD: Caught Termination Signal - Astalavista... baby
[04-14-2021 12:08:51] NPCD: npcd Daemon (0.6.26) started with PID=1019
[04-14-2021 12:08:51] NPCD: Please have a look at 'npcd -V' to get license information
[04-14-2021 12:08:51] NPCD: HINT: load_threshold is disabled - ('0.000000')
[07-05-2021 15:25:53] NPCD: Caught Termination Signal - Astalavista... baby
[07-05-2021 15:26:41] NPCD: npcd Daemon (0.6.26) started with PID=3319
[07-05-2021 15:26:41] NPCD: Please have a look at 'npcd -V' to get license information
[07-05-2021 15:26:41] NPCD: HINT: load_threshold is disabled - ('0.000000')

In the rrdcached path are two filles:

rrd.journal.1625628410.927215
rrd.journal.1625635610.926825

tail rrd.journal.1625628410.927215 :

update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHUSV16002/Temperature_Battery_temp.rrd 1625635589:17
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHUSV16002/Temperature_External_1_temp.rrd 1625635589:18
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHUSV16002/Uptime_uptime.rrd 1625635589:16081147
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHUSV16002/Check_MK_execution_time.rrd 1625635589:3.970
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHUSV16002/Check_MK_user_time.rrd 1625635589:0.120
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHUSV16002/Check_MK_system_time.rrd 1625635589:0.100
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHUSV16002/Check_MK_children_user_time.rrd 1625635589:0.080
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHUSV16002/Check_MK_children_system_time.rrd 1625635589:0.120
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHUSV16002/Check_MK_cmk_time_snmp.rrd 1625635589:3.550
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHUSV16002/Check_MK_cmk_time_agent.rrd 1625635589:0.000

tail rrd.journal.1625635610.926825 :

update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPEICA01/_HOST__rta.rrd 1625641544:0.000
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPEICA01/_HOST__pl.rrd 1625641544:100
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPEICA01/_HOST__rtmax.rrd 1625641544:0.000
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHIPEICA01/_HOST__rtmin.rrd 1625641544:0.000
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW16011/Check_MK_execution_time.rrd 1625641544:6.710
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW16011/Check_MK_user_time.rrd 1625641544:0.300
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW16011/Check_MK_system_time.rrd 1625641544:0.140
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW16011/Check_MK_children_user_time.rrd 1625641544:0.110
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW16011/Check_MK_children_system_time.rrd 1625641544:0.140
update /opt/omd/sites/MYSITE/var/pnp4nagios/perfdata/TPHSW16011/Check_MK_cmk_time_snmp.rrd 1625641544:6.root@tphcmk:/omd/sites/MYSITE/var/rrdcached#

jan.filo · July 12, 2021, 7:02pm

Hi,
i had similar problem that graphs wont render or would have big holes in them. The fix was increasing
Maximum concurrent Checkmk fetchers and Maximum concurrent Checkmk checkers. You can check in sidebar Core statistics if Fetcher helper or checker helper are constantly on 80% or bigger you need to change value for them

albzundy · July 13, 2021, 8:33am

Hello Jan, thanks for your answer but we are using the free RAW edition. These options are not present.

In this topic someone else was looking for these settings, too:

https://forum.checkmk.com/t/2-0-fetcher-and-checker/23785/3

andreas-doehler · July 13, 2021, 9:38am

npcd.log looks ok
rrd.journal also is ok
But the perfdata.log shows that there is a problem to write the perfdata to the files.
There could be different problems - first one is that pnp4nagios cannot write the perfdata dump files to the rrd files and tries every interval to write the dump files again. The dump files grow over the time and no new values are written as the process crashes avery interval at the same file.

You can look inside “~/var/pnp4nagios/spool/” if there are some files or nothing (nothing or only 1-2 is ok).

albzundy · July 13, 2021, 9:53am

Hello Andreas,

thank you very much for analyzing the log files
The directory ~/var/pnp4nagios/spool/ is a bit strange…

When I write ls -al there is sometimes only one file visible, sometimes no files are shown. But the files overall are a lot more.
Here you can see different outputs from ls -al:

system · July 13, 2022, 9:54am

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.