Instead of Graphs | Error messages are displayed on email notification of checkmk

PrachiKatakwar · January 6, 2021, 6:48am

Hi Team,

Appreciate the forum, its really helpful for us to investigate the issue and keeps up the motivation in checkmk.

Sorry , its a long post, I have written all my brain doubts here.

The issue is coming after 20th Oct 2020, the checkmk server was restarted and after that the email notifications are displayed like below with error message instead of graphs.
Checkmk version is Raw 1.6.Op6

To resolve this , I first looked at omd status, site name is checkmk, single site all components installed in Centos 7 VM.

The nagios is running in above , but if I do systemctl status nagios, its not running, restarted multiple times but not starting up

Now I checked cat /var/log/maillog

Now checked: cat /omd/sites/checkmk/var/log/notify.log.1
cat /omd/sites/checkmk/var/log/nagios.log

5 cat /omd/sites/checkmk/var/pnp4nagios/log/perfdata.log , could find the rrdcached error

6.cat /omd/sites/checkmk/var/pnp4nagios/log/npcd.log

cd /omd/sites/checkmk/var/log/apache/error_log
df -h, doesnt seem any memory issue

On reading various forums , could find above logs but literally scratching my hair to resolve it, Also Yes I know the checkmk version is too old, so whether we should upgrade it and so the issues will automatically be resolved or should we resolve the issues first and then upgrade.

Need your sincere advice and would be thankful for your guidance.

Best Regards,
Prachi Katakwar

PrachiKatakwar · January 7, 2021, 6:29am

Hi Team,

GodMorgon,

Request your guidance, so many things going in my mind.

Any pointers on above issue, while reading several bogs I tried to change the timeout, random_delay and flushed_timeout values in rrdcached.conf from 3600, 1800 and 7200 to 300 600 and 1200 respectively as shown in the below screenshot, but nothing fruitful came

If we see the error on email notification carefully , could guess its related to something rrdtool command, may be I need to give rrdtool flush cache or may be delete all the rrd files…but since I am new to checkmk and CentOS 7 as well , so bit confused.

/Prachi

anonymous15 · January 7, 2021, 7:58am

Looks to me like your rrd is corrupt or deleted.
Does this happen for all hosts/services ?
Did you already tried a omd restart ?

PrachiKatakwar · January 7, 2021, 8:17am

Hi ,

Thank you so much for the reply, I have been continuously checking if someone would reply:)

Yes almost for all hosts , sometimes the graph comes in the email notification but mostly I get this error

I have done omd restart multiple times but it doesnt help …

Nagios is not running , Is this the issue…and rrd cache also corrupted.

Shall I delete rrd files, or give some command with rrdtool

Please suggest .

Best Regards
/Prachi

anonymous15 · January 7, 2021, 8:26am

What does omd status shows ? Can you also show the output of the command ? omd config show

If the rrd data (means the historical performace data) is not needed , then we can also think of deleting it but lets consider this as a last option.

The error message in the screenshot (the red one) is not clearly visible. Can you Zoomify it or just paste the text of that message ?

PrachiKatakwar · January 7, 2021, 8:41am

Hi,

Yes surely.

omd status:

But if I do systemctl nagios status, its failed and I have done Nagios restart multiple times but it doesnt come up.

omd config show

My sitename is checkmk
So , If I do omd config checkmk
I get this interactive screen

The error which I am getting in email notifications is this

PrachiKatakwar · January 7, 2021, 8:57am

anonymous15 · January 7, 2021, 9:01am

This should be run as site user. LIke su - checkmk and then run this command.

Can you try cmk -R --verbose as site-user and share the output ?

Did you installed any plugin for the sensors ? It is possible that the plugin output don’t give valid data and the check is trying to process this not valid data.
On the monitoring server you can also do the following test.(Please run it as site user)

$ cmk --debug -vvII sekatenessus

If this is running without error then you can only compare the plugin output.

PrachiKatakwar · January 7, 2021, 9:40am

Hi ,

Thank you the quick turnaround, you guys are great.

This is the output for omd config show

Attached the output for cmk -R --verboseoutput cmk -R --verbose.txt (95.2 KB)

Did you installed any plugin for the sensors ? It is possible that the plugin output don’t give valid data and the check is trying to process this not valid data.

How to check that , but since last 1 year no plugin is installed.

Attached the output
cmk --debug -vvII sekatenessusOutput cmk --debug --vvII sekatenessus.txt (34.4 KB)

Please guide.

PrachiKatakwar · January 7, 2021, 10:36am

Morover If we see nagios log, its coming like this

cat /omd/sites/checkmk/var/log/nagios.log

[1610012943] SERVICE NOTIFICATION: eakshgu;seliicvh00637-idrac;Check_MK Discovery;WARNING;bulk mail;WARN - 8 unmonitored services (dell_idrac_disks:6, dell_idrac_raid:2)(!), no vanished services found, no new host labels
[1610012944] EXTERNAL COMMAND: _LOG;SERVICE NOTIFICATION RESULT: eakshgu;seliicvh00637-idrac;Check_MK Discovery;OK;bulk mail;Unable to fetch number of graphs: Unable to fetch graph infos:

XML file "/omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00637-idrac/Check_MK_Discovery.xml" not found. <a href="http://docs.pnp4nagios.org/faq/6">Read FAQ online</a>

;Spooled mail to local mail transmission agent – Unable to fetch number of graphs: Unable to fetch graph infos:

XML file "/omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00637-idrac/Check_MK_Discovery.xml" not found. <a href="http://docs.pnp4nagios.org/faq/6">Read FAQ online</a>

[1610012944] SERVICE NOTIFICATION RESULT: eakshgu;seliicvh00637-idrac;Check_MK Discovery;OK;bulk mail;Unable to fetch number of graphs: Unable to fetch graph infos:

XML file "/omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00637-idrac/Check_MK_Discovery.xml" not found. <a href="http://docs.pnp4nagios.org/faq/6">Read FAQ online</a>

;Spooled mail to local mail transmission agent – Unable to fetch number of graphs: Unable to fetch graph infos:

XML file "/omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00637-idrac/Check_MK_Discovery.xml" not found. <a href="http://docs.pnp4nagios.org/faq/6">Read FAQ online</a>

[1610012993] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013053] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013114] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013174] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013234] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013294] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013354] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013414] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013474] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013535] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013595] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013655] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013715] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013775] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013835] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013895] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610013955] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014015] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014075] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014135] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014196] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014256] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014316] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014376] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014436] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014496] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014556] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014616] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014677] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014737] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014797] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014857] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014917] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610014977] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610015037] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610015098] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610015158] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610015218] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610015271] Auto-save of retention data completed successfully.
[1610015278] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610015338] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610015398] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610015459] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)
[1610015519] livestatus: Timeperiod cache not updated, there are no timeperiods (yet)

anonymous15 · January 7, 2021, 11:09am

Were there any recent Hardware/Software changes ? On what OS you run Checkmk ?

Also, can you share rrdcached error from the perfdata.log ? From the screenshot, its not clearly visible,

PrachiKatakwar · January 7, 2021, 3:58pm

Hi,

Sorry for the delay in reply, Yes there were some changes on 20th Oct in our environment, not typically on Checkmk but that day since we didnt keep the VM on maintenance mode , so it threw lot of email notifications and after restarting the server the graphs with error notification like below started coming.

The red error I am writing:
ERROR: ’ is not a valid function name in
RRDtool 1.7.1

Usage: rrdtool options command_options
valid commands: create, update, updatev, graph, graphv,dump,restore,

For more information read RRD manpages
Ok u:0.00 s0.01 r:0.00
/omd/sites/checkmk/bin/rrdtool graph --daemon=unix:omd/sites/checkmk/tmp/rrdcached.sock - --slope-mode --color vertical-label=

We are running checkmk on CentOS 7.

rrdcatched error from perfdata.log

2020-12-29 12:41:08 [8462] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00356-idrac/System_Board_Pwr_Consumption_power.rrd 1609242051:400
2020-12-29 12:41:08 [8462] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242051.000000 when last update time is 1609242051.000000 (minimum one second step)
2020-12-29 12:41:08 [8462] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00356-idrac/Temperature_CPU1_temp.rrd 1609242051:71
2020-12-29 12:41:08 [8462] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242051.000000 when last update time is 1609242051.000000 (minimum one second step)
2020-12-29 12:41:08 [8462] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00356-idrac/Temperature_CPU2_temp.rrd 1609242051:64
2020-12-29 12:41:08 [8462] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242051.000000 when last update time is 1609242051.000000 (minimum one second step)
2020-12-29 12:41:08 [8462] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00356-idrac/Temperature_System_Board_Inlet_temp.rrd 1609242051:20
2020-12-29 12:41:08 [8462] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242051.000000 when last update time is 1609242051.000000 (minimum one second step)
2020-12-29 12:41:08 [8462] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00356-idrac/Uptime_uptime.rrd 1609242051:11932135
2020-12-29 12:41:08 [8462] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242051.000000 when last update time is 1609242051.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_in.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_inucast.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_innucast.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_indisc.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_inerr.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_out.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_outucast.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_outnucast.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_outdisc.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_outerr.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Interface_6_outqlen.rrd 1609242106:0
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/System_Board_Pwr_Consumption_power.rrd 1609242106:476
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Temperature_CPU1_temp.rrd 1609242106:62
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Temperature_CPU2_temp.rrd 1609242106:66
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Temperature_System_Board_Exhaust_temp.rrd 1609242106:36
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Temperature_System_Board_Inlet_temp.rrd 1609242106:25
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:42:09 [13574] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00657-idrac/Uptime_uptime.rrd 1609242106:11583661
2020-12-29 12:42:09 [13574] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242106.000000 when last update time is 1609242106.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_in.rrd 1609242199:268.111159
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_inucast.rrd 1609242199:4.158987
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_innucast.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_indisc.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_inerr.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_out.rrd 1609242199:197.388177
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_outucast.rrd 1609242199:1.010444
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_outnucast.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_outdisc.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_outerr.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Interface_7_outqlen.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/System_Board_Pwr_Consumption_power.rrd 1609242199:292
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Temperature_CPU1_temp.rrd 1609242199:72
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Temperature_CPU2_temp.rrd 1609242199:61
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Temperature_System_Board_Inlet_temp.rrd 1609242199:21
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00424-idrac/Uptime_uptime.rrd 1609242199:12698877
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_in.rrd 1609242199:219.857552
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_inucast.rrd 1609242199:3.327259
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_innucast.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_indisc.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_inerr.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_out.rrd 1609242199:214.959536
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_outucast.rrd 1609242199:0.965067
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_outnucast.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_outdisc.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_outerr.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Interface_7_outqlen.rrd 1609242199:0
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/System_Board_Pwr_Consumption_power.rrd 1609242199:304
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Temperature_CPU1_temp.rrd 1609242199:73
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Temperature_CPU2_temp.rrd 1609242199:62
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Temperature_System_Board_Inlet_temp.rrd 1609242199:21
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:43:40 [21141] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00505-idrac/Uptime_uptime.rrd 1609242199:11934837
2020-12-29 12:43:40 [21141] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242199.000000 when last update time is 1609242199.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_in.rrd 1609242249:24.792955
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_inucast.rrd 1609242249:0.281466
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_innucast.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_indisc.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_inerr.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_out.rrd 1609242249:74.135396
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_outucast.rrd 1609242249:0.25332
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_outnucast.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_outdisc.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_outerr.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Interface_6_outqlen.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/System_Board_Pwr_Consumption_power.rrd 1609242249:350
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Temperature_CPU1_temp.rrd 1609242249:49
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Temperature_CPU2_temp.rrd 1609242249:48
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Temperature_System_Board_Exhaust_temp.rrd 1609242249:33
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Temperature_System_Board_Inlet_temp.rrd 1609242249:23
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00646-idrac/Uptime_uptime.rrd 1609242249:11261206
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_in.rrd 1609242249:25.672636
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_inucast.rrd 1609242249:0.290813
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_innucast.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_indisc.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_inerr.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_out.rrd 1609242249:73.049988
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_outucast.rrd 1609242249:0.248869
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_outnucast.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_outdisc.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_outerr.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_outqlen.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/System_Board_Pwr_Consumption_power.rrd 1609242249:462
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Temperature_CPU1_temp.rrd 1609242249:74
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Temperature_CPU2_temp.rrd 1609242249:63
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Temperature_System_Board_Exhaust_temp.rrd 1609242249:36
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Temperature_System_Board_Inlet_temp.rrd 1609242249:25
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Uptime_uptime.rrd 1609242249:11579800
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:45:56 [29826] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00631-idrac/_HOST__rta.rrd 1609242346:0.254
2020-12-29 12:45:56 [29826] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242346.000000 when last update time is 1609242346.000000 (minimum one second step)
2020-12-29 12:45:56 [29826] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00631-idrac/_HOST__pl.rrd 1609242346:0
2020-12-29 12:45:56 [29826] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242346.000000 when last update time is 1609242346.000000 (minimum one second step)
2020-12-29 12:45:56 [29826] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00631-idrac/_HOST__rtmax.rrd 1609242346:0.441
2020-12-29 12:45:56 [29826] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242346.000000 when last update time is 1609242346.000000 (minimum one second step)
2020-12-29 12:45:56 [29826] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00631-idrac/_HOST__rtmin.rrd 1609242346:0.189
2020-12-29 12:45:56 [29826] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242346.000000 when last update time is 1609242346.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_in.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_inucast.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_innucast.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_indisc.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_inerr.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_out.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_outucast.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_outnucast.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_outdisc.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_outerr.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_outqlen.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/System_Board_Pwr_Consumption_power.rrd 1609242354:320
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Temperature_CPU1_temp.rrd 1609242354:74
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Temperature_CPU2_temp.rrd 1609242354:62
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Temperature_System_Board_Inlet_temp.rrd 1609242354:22
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Uptime_uptime.rrd 1609242354:27842036
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)

BR
/Prachi

anonymous15 · January 9, 2021, 12:24am

If I understand you correctly, this issue with the graphs is with all the services/host. How important is the historical performance data to you ? This means if you delete the rrd, historical performance data is gone.

Apart from deleting the rrd, I don;t have any other idea. As a test you can try deleting the rrd of a service check of a Host. For example Check_MK service check of a particualr host and in the next interval of the service check, it will be created automatically. Then check if the issue still persists.

andreas-doehler · January 9, 2021, 6:20am

Only one idea.

the performance data is shown correctly inside PNP4Nagios?
- If not the solution from @anonymous15 with deleting the performance data would the only one
was the notification template for the HTML mail modified?

From your posts

This is no real error message for instance as the “Check_MK Discovery” service has no graphs.

PrachiKatakwar · January 9, 2021, 10:41am

Hi @anonymous15/@andreas-doehler,

First of all, would appreciate that you guys are really amazing, as we come from all different parts of the world but connecting here and resolving the issue is very encouraging and motivating for us as well.
Big Thanks to you.

Yes , we can absolutely delete the rrd, as old data is not of much importance but upcoming data going forward is important as after resolving this error graph issue , my next step is to upgrade raw checkmk current version 1.6 Op6 to Raw 1.6 Op19.

@andreas-doehler : Output On giving the command cat /omd/sites/checkmk/var/pnp4nagios/log/perfdata.log

2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Interface_6_outqlen.rrd 1609242249:0
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/System_Board_Pwr_Consumption_power.rrd 1609242249:462
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Temperature_CPU1_temp.rrd 1609242249:74
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Temperature_CPU2_temp.rrd 1609242249:63
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Temperature_System_Board_Exhaust_temp.rrd 1609242249:36
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Temperature_System_Board_Inlet_temp.rrd 1609242249:25
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:44:25 [24833] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00659-idrac/Uptime_uptime.rrd 1609242249:11579800
2020-12-29 12:44:25 [24833] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242249.000000 when last update time is 1609242249.000000 (minimum one second step)
2020-12-29 12:45:56 [29826] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00631-idrac/_HOST__rta.rrd 1609242346:0.254
2020-12-29 12:45:56 [29826] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242346.000000 when last update time is 1609242346.000000 (minimum one second step)
2020-12-29 12:45:56 [29826] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00631-idrac/_HOST__pl.rrd 1609242346:0
2020-12-29 12:45:56 [29826] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242346.000000 when last update time is 1609242346.000000 (minimum one second step)
2020-12-29 12:45:56 [29826] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00631-idrac/_HOST__rtmax.rrd 1609242346:0.441
2020-12-29 12:45:56 [29826] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242346.000000 when last update time is 1609242346.000000 (minimum one second step)
2020-12-29 12:45:56 [29826] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00631-idrac/_HOST__rtmin.rrd 1609242346:0.189
2020-12-29 12:45:56 [29826] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242346.000000 when last update time is 1609242346.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_in.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_inucast.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_innucast.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_indisc.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_inerr.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_out.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_outucast.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_outnucast.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_outdisc.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_outerr.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Interface_7_outqlen.rrd 1609242354:0
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/System_Board_Pwr_Consumption_power.rrd 1609242354:320
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Temperature_CPU1_temp.rrd 1609242354:74
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Temperature_CPU2_temp.rrd 1609242354:62
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Temperature_System_Board_Inlet_temp.rrd 1609242354:22
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)
2020-12-29 12:45:56 [29827] [0] RRDs::update /omd/sites/checkmk/var/pnp4nagios/perfdata/seliicvh00352-idrac/Uptime_uptime.rrd 1609242354:27842036
2020-12-29 12:45:56 [29827] [0] RRDs::update ERROR rrdcached@unix:/omd/sites/checkmk/tmp/run/rrdcached.sock: illegal attempt to update using time 1609242354.000000 when last update time is 1609242354.000000 (minimum one second step)

was the notification template for the HTML mail modified?

No , but now on doing cat /omd/sites/checkmk/var/log/notify.1.log the output is coming as below:

2021-01-08 18:59:50 * notifying epkolan, eakshgu, ehampat, qsuslju, eanajkd via mail, parameters: (no parameters), bulk: yes
2021-01-08 18:59:50 --> storing for bulk notification epkolan|mail|60|1000|host|sekate-cv-ta-vpod09
2021-01-08 18:59:50 - created bulk directory /omd/sites/checkmk/var/check_mk/notify/bulk/epkolan/mail/60,1000,host,sekate-cv-ta-vpod09
2021-01-08 18:59:50 - stored in /omd/sites/checkmk/var/check_mk/notify/bulk/epkolan/mail/60,1000,host,sekate-cv-ta-vpod09/86640f8e-37a2-433e-8fa7-c0e153827eb9
2021-01-08 18:59:50 --> storing for bulk notification eakshgu|mail|60|1000|host|sekate-cv-ta-vpod09
2021-01-08 18:59:50 - created bulk directory /omd/sites/checkmk/var/check_mk/notify/bulk/eakshgu/mail/60,1000,host,sekate-cv-ta-vpod09
2021-01-08 18:59:50 - stored in /omd/sites/checkmk/var/check_mk/notify/bulk/eakshgu/mail/60,1000,host,sekate-cv-ta-vpod09/93bb6332-b5a5-4b9c-98b1-532627fe7482
2021-01-08 18:59:50 --> storing for bulk notification ehampat|mail|60|1000|host|sekate-cv-ta-vpod09
2021-01-08 18:59:50 - created bulk directory /omd/sites/checkmk/var/check_mk/notify/bulk/ehampat/mail/60,1000,host,sekate-cv-ta-vpod09
2021-01-08 18:59:50 - stored in /omd/sites/checkmk/var/check_mk/notify/bulk/ehampat/mail/60,1000,host,sekate-cv-ta-vpod09/09cc44cf-5632-418c-aaaa-e96cc35b65b4
2021-01-08 18:59:50 --> storing for bulk notification qsuslju|mail|60|1000|host|sekate-cv-ta-vpod09
2021-01-08 18:59:50 - created bulk directory /omd/sites/checkmk/var/check_mk/notify/bulk/qsuslju/mail/60,1000,host,sekate-cv-ta-vpod09
2021-01-08 18:59:50 - stored in /omd/sites/checkmk/var/check_mk/notify/bulk/qsuslju/mail/60,1000,host,sekate-cv-ta-vpod09/4e9f5be2-9628-454e-a1c3-f431445ee6d8
2021-01-08 18:59:50 --> storing for bulk notification eanajkd|mail|60|1000|host|sekate-cv-ta-vpod09
2021-01-08 18:59:50 - created bulk directory /omd/sites/checkmk/var/check_mk/notify/bulk/eanajkd/mail/60,1000,host,sekate-cv-ta-vpod09
2021-01-08 18:59:50 - stored in /omd/sites/checkmk/var/check_mk/notify/bulk/eanajkd/mail/60,1000,host,sekate-cv-ta-vpod09/42800ac7-3b3b-44ce-863b-c17a4a9cc555
2021-01-08 19:00:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/epkolan/mail/60,1000,host,sekate-cv-ta-vpod09 is not ripe yet (age: 12, count: 1)!
2021-01-08 19:00:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/ehampat/mail/60,1000,host,sekate-cv-ta-vpod09 is not ripe yet (age: 12, count: 1)!
2021-01-08 19:00:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/eanajkd/mail/60,1000,host,sekate-cv-ta-vpod09 is not ripe yet (age: 12, count: 1)!
2021-01-08 19:00:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/qsuslju/mail/60,1000,host,sekate-cv-ta-vpod09 is not ripe yet (age: 12, count: 1)!
2021-01-08 19:00:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/eakshgu/mail/60,1000,host,sekate-cv-ta-vpod09 is not ripe yet (age: 12, count: 1)!
2021-01-08 19:01:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/epkolan/mail/60,1000,host,sekate-cv-ta-vpod09 is ripe: age 72 >= 60
2021-01-08 19:01:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/ehampat/mail/60,1000,host,sekate-cv-ta-vpod09 is ripe: age 72 >= 60
2021-01-08 19:01:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/eanajkd/mail/60,1000,host,sekate-cv-ta-vpod09 is ripe: age 72 >= 60
2021-01-08 19:01:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/qsuslju/mail/60,1000,host,sekate-cv-ta-vpod09 is ripe: age 72 >= 60
2021-01-08 19:01:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/eakshgu/mail/60,1000,host,sekate-cv-ta-vpod09 is ripe: age 72 >= 60
2021-01-08 19:01:02 Sending out 5 ripe bulk notifications
2021-01-08 19:01:02 -> epkolan/mail /omd/sites/checkmk/var/check_mk/notify/bulk/epkolan/mail/60,1000,host,sekate-cv-ta-vpod09
2021-01-08 19:01:03 mail: Spooled mail to local mail transmission agent
2021-01-08 19:01:03 -> ehampat/mail /omd/sites/checkmk/var/check_mk/notify/bulk/ehampat/mail/60,1000,host,sekate-cv-ta-vpod09
2021-01-08 19:01:03 mail: Spooled mail to local mail transmission agent
2021-01-08 19:01:03 -> eanajkd/mail /omd/sites/checkmk/var/check_mk/notify/bulk/eanajkd/mail/60,1000,host,sekate-cv-ta-vpod09
2021-01-08 19:01:04 mail: Spooled mail to local mail transmission agent
2021-01-08 19:01:04 -> qsuslju/mail /omd/sites/checkmk/var/check_mk/notify/bulk/qsuslju/mail/60,1000,host,sekate-cv-ta-vpod09
2021-01-08 19:01:04 mail: Spooled mail to local mail transmission agent
2021-01-08 19:01:04 -> eakshgu/mail /omd/sites/checkmk/var/check_mk/notify/bulk/eakshgu/mail/60,1000,host,sekate-cv-ta-vpod09
2021-01-08 19:01:04 mail: Spooled mail to local mail transmission agent
2021-01-08 19:03:22 ----------------------------------------------------------------------
2021-01-08 19:03:22 Got raw notification (sekatevc01;Filesystem /storage/log) context with 52 variables
2021-01-08 19:03:22 Raw context:
CONTACTEMAIL=
CONTACTNAME=check-mk-notify
CONTACTPAGER=
DATE=2021-01-08
HOSTACKAUTHOR=
HOSTACKCOMMENT=
HOSTADDRESS=10.61.0.10
HOSTALIAS=sekatevc01
HOSTATTEMPT=1
HOSTCHECKCOMMAND=check-mk-host-ping!-w 200.00,80.00% -c 500.00,100.00%
HOSTDOWNTIME=0
HOSTGROUPNAMES=SVC,CV-TE-vCenters
HOSTNAME=sekatevc01
HOSTNOTIFICATIONNUMBER=0
HOSTOUTPUT=OK - 10.61.0.10: rta 0.130ms, lost 0%
HOSTPERFDATA=rta=0.130ms;200.000;500.000;0; pl=0%;80;100;; rtmax=0.167ms;;;; rtmin=0.105ms;;;;
HOSTPROBLEMID=0
HOSTSTATE=UP
HOSTSTATEID=0
HOSTTAGS=/wato/virtualisation/cv-te/vcenters/ auto-piggyback ip-v4 ip-v4-only lan no-agent prod site:checkmk snmp snmp-only snmp-v2
HOST_ADDRESS_4=10.61.0.10
HOST_ADDRESS_6=
HOST_ADDRESS_FAMILY=4
LASTHOSTSTATE=UP
LASTHOSTSTATECHANGE=1610082712
LASTHOSTSTATEID=0
LASTHOSTUP=1610128764
LASTSERVICEOK=1610128460
LASTSERVICESTATE=WARNING
LASTSERVICESTATECHANGE=1610128641
LASTSERVICESTATEID=1
LONGDATETIME=Fri Jan 8 19:03:22 CET 2021
LONGHOSTOUTPUT=
LONGSERVICEOUTPUT=
NOTIFICATIONAUTHOR=
NOTIFICATIONAUTHORALIAS=
NOTIFICATIONAUTHORNAME=
NOTIFICATIONCOMMENT=
NOTIFICATIONTYPE=PROBLEM
SERVICEACKAUTHOR=
SERVICEACKCOMMENT=
SERVICEATTEMPT=3
SERVICECHECKCOMMAND=check_mk-hr_fs
SERVICEDESC=Filesystem /storage/log
SERVICEGROUPNAMES=
SERVICENOTIFICATIONNUMBER=1
SERVICEOUTPUT=WARN - 70.11% used (30.96 of 44.16 GB), (warn/crit at 70.0%/85.0%), trend: +2.78 GB / 24 hours
SERVICEPERFDATA=/storage/log=31706.308594;31654.689063;38437.836719;0;45220.984375 fs_size=45220.984375;;;; growth=-12145.968472;;;; trend=2841.979194;;;0;1884.207682
SERVICEPROBLEMID=68023
SERVICESTATE=WARNING
SERVICESTATEID=1
SHORTDATETIME=2021-01-08 19:03:22
2021-01-08 19:03:22 Setting HOSTNOTIFICATIONNUMBER for notification from ‘0’ to ‘1’
2021-01-08 19:03:22 Computed variables:
CONTACTS=eanajkd,eakshgu,ehampat,epkolan,qsuslju
HOSTFORURL=sekatevc01
HOSTOUTPUT_HTML=(This macro is deprecated: see Werk#7427) OK - 10.61.0.10: rta 0.130ms, lost 0%
HOSTSHORTSTATE=UP
HOSTURL=/check_mk/index.py?start_url=view.py%3Fview_name%3Dhoststatus%26host%3Dsekatevc01%26site%3Dcheckmk
LASTHOSTSHORTSTATE=UP
LASTHOSTSTATECHANGE_REL=0d 12:51:30
LASTHOSTUP_REL=0d 00:03:58
LASTSERVICEOK_REL=0d 00:09:02
LASTSERVICESHORTSTATE=WARN
LASTSERVICESTATECHANGE_REL=0d 00:06:01
LONGSERVICEOUTPUT_HTML=(This macro is deprecated: see Werk#7427)
MICROTIME=1610129002842814
MONITORING_HOST=sekatecheckmk
OMD_ROOT=/omd/sites/checkmk
OMD_SITE=checkmk
PREVIOUSHOSTHARDSHORTSTATE=UP
PREVIOUSHOSTHARDSTATE=UP
PREVIOUSSERVICEHARDSHORTSTATE=OK
PREVIOUSSERVICEHARDSTATE=OK
SERVICEFORURL=Filesystem%20/storage/log
SERVICEOUTPUT_HTML=(This macro is deprecated: see Werk#7427) WARN - 70.11% used (30.96 of 44.16 GB), (warn/crit at 70.0%/85.0%), trend: +2.78 GB / 24 hours
SERVICESHORTSTATE=WARN
SERVICEURL=/check_mk/index.py?start_url=view.py%3Fview_name%3Dservice%26host%3Dsekatevc01%26service%3DFilesystem%20/storage/log%26site%3Dcheckmk
WHAT=SERVICE
2021-01-08 19:03:22 Preparing rule based notifications
2021-01-08 19:03:22 Found 0 user specific rules
2021-01-08 19:03:22 Global rule ‘SVC’…
2021-01-08 19:03:22 -> matches!
2021-01-08 19:03:22 - adding notification of epkolan, eakshgu, ehampat, qsuslju, eanajkd via mail
2021-01-08 19:03:22 Global rule ‘CV-PL’…
2021-01-08 19:03:22 -> does not match: The rule requires WATO folder ‘cv-pl’, but the host is in ‘virtualisation/cv-te/vcenters’
2021-01-08 19:03:22 Global rule ‘TE Infra’…
2021-01-08 19:03:22 -> does not match: The rule requires WATO folder ‘infra’, but the host is in ‘virtualisation/cv-te/vcenters’
2021-01-08 19:03:22 Global rule ‘Notify all contacts of a host/service via HTML email’…
2021-01-08 19:03:22 -> does not match: This rule is disabled
2021-01-08 19:03:22 Executing 1 notifications:
2021-01-08 19:03:22 * notifying epkolan, eakshgu, ehampat, qsuslju, eanajkd via mail, parameters: (no parameters), bulk: yes
2021-01-08 19:03:22 --> storing for bulk notification epkolan|mail|60|1000|host|sekatevc01
2021-01-08 19:03:22 - created bulk directory /omd/sites/checkmk/var/check_mk/notify/bulk/epkolan/mail/60,1000,host,sekatevc01
2021-01-08 19:03:22 - stored in /omd/sites/checkmk/var/check_mk/notify/bulk/epkolan/mail/60,1000,host,sekatevc01/199990d6-463f-47ec-a94e-42fd51c2fe96
2021-01-08 19:03:22 --> storing for bulk notification eakshgu|mail|60|1000|host|sekatevc01
2021-01-08 19:03:22 - created bulk directory /omd/sites/checkmk/var/check_mk/notify/bulk/eakshgu/mail/60,1000,host,sekatevc01
2021-01-08 19:03:22 - stored in /omd/sites/checkmk/var/check_mk/notify/bulk/eakshgu/mail/60,1000,host,sekatevc01/1505e6b4-fdc9-4610-a142-e70aab960508
2021-01-08 19:03:22 --> storing for bulk notification ehampat|mail|60|1000|host|sekatevc01
2021-01-08 19:03:22 - created bulk directory /omd/sites/checkmk/var/check_mk/notify/bulk/ehampat/mail/60,1000,host,sekatevc01
2021-01-08 19:03:22 - stored in /omd/sites/checkmk/var/check_mk/notify/bulk/ehampat/mail/60,1000,host,sekatevc01/be33bc8a-8ec1-4e40-a791-c505533a6458
2021-01-08 19:03:22 --> storing for bulk notification qsuslju|mail|60|1000|host|sekatevc01
2021-01-08 19:03:22 - created bulk directory /omd/sites/checkmk/var/check_mk/notify/bulk/qsuslju/mail/60,1000,host,sekatevc01
2021-01-08 19:03:22 - stored in /omd/sites/checkmk/var/check_mk/notify/bulk/qsuslju/mail/60,1000,host,sekatevc01/23a5497b-bef7-4682-bc88-fa6ca2834895
2021-01-08 19:03:22 --> storing for bulk notification eanajkd|mail|60|1000|host|sekatevc01
2021-01-08 19:03:22 - created bulk directory /omd/sites/checkmk/var/check_mk/notify/bulk/eanajkd/mail/60,1000,host,sekatevc01
2021-01-08 19:03:22 - stored in /omd/sites/checkmk/var/check_mk/notify/bulk/eanajkd/mail/60,1000,host,sekatevc01/37803564-426d-4ff6-b7f9-03dd86c66430
2021-01-08 19:04:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/epkolan/mail/60,1000,host,sekatevc01 is not ripe yet (age: 39, count: 1)!
2021-01-08 19:04:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/ehampat/mail/60,1000,host,sekatevc01 is not ripe yet (age: 39, count: 1)!
2021-01-08 19:04:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/eanajkd/mail/60,1000,host,sekatevc01 is not ripe yet (age: 39, count: 1)!
2021-01-08 19:04:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/qsuslju/mail/60,1000,host,sekatevc01 is not ripe yet (age: 39, count: 1)!
2021-01-08 19:04:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/eakshgu/mail/60,1000,host,sekatevc01 is not ripe yet (age: 39, count: 1)!
2021-01-08 19:05:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/epkolan/mail/60,1000,host,sekatevc01 is ripe: age 99 >= 60
2021-01-08 19:05:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/ehampat/mail/60,1000,host,sekatevc01 is ripe: age 99 >= 60
2021-01-08 19:05:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/eanajkd/mail/60,1000,host,sekatevc01 is ripe: age 99 >= 60
2021-01-08 19:05:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/qsuslju/mail/60,1000,host,sekatevc01 is ripe: age 99 >= 60
2021-01-08 19:05:02 Bulk /omd/sites/checkmk/var/check_mk/notify/bulk/eakshgu/mail/60,1000,host,sekatevc01 is ripe: age 99 >= 60
2021-01-08 19:05:02 Sending out 5 ripe bulk notifications
2021-01-08 19:05:02 -> epkolan/mail /omd/sites/checkmk/var/check_mk/notify/bulk/epkolan/mail/60,1000,host,sekatevc01
2021-01-08 19:05:03 mail: Spooled mail to local mail transmission agent
2021-01-08 19:05:03 -> ehampat/mail /omd/sites/checkmk/var/check_mk/notify/bulk/ehampat/mail/60,1000,host,sekatevc01
2021-01-08 19:05:03 mail: Spooled mail to local mail transmission agent
2021-01-08 19:05:03 -> eanajkd/mail /omd/sites/checkmk/var/check_mk/notify/bulk/eanajkd/mail/60,1000,host,sekatevc01
2021-01-08 19:05:04 mail: Spooled mail to local mail transmission agent
2021-01-08 19:05:04 -> qsuslju/mail /omd/sites/checkmk/var/check_mk/notify/bulk/qsuslju/mail/60,1000,host,sekatevc01
2021-01-08 19:05:04 mail: Spooled mail to local mail transmission agent
2021-01-08 19:05:04 -> eakshgu/mail /omd/sites/checkmk/var/check_mk/notify/bulk/eakshgu/mail/60,1000,host,sekatevc01
2021-01-08 19:05:04 mail: Spooled mail to local mail transmission agent

PrachiKatakwar · January 9, 2021, 11:03am

Hi @anonymous15 / @andreas-doehler:

For deleting rrd files, our checkmk is hosting around 600 servers.

So , rrd files in /omd/sites/checkmk/var/pnp4nagios/perfdata path for 1 server are these many:

[root@sekatecheckmk perfdata]# cd sekatenessus/
[root@sekatecheckmk sekatenessus]# ls
Check_MK_children_system_time.rrd Disk_IO_SUMMARY_disk_write_ios.rrd HOST.xml Interface_2_innucast.rrd Interface_3_outdisc.rrd PING_rtmin.rrd
Check_MK_children_user_time.rrd Disk_IO_SUMMARY_disk_write_ql.rrd Interface_1_indisc.rrd Interface_2_in.rrd Interface_3_outerr.rrd PING.xml
Check_MK_cmk_time_agent.rrd Disk_IO_SUMMARY_disk_write_throughput.rrd Interface_1_inerr.rrd Interface_2_inucast.rrd Interface_3_outnucast.rrd Processor_Queue_load15.rrd
Check_MK_execution_time.rrd Disk_IO_SUMMARY.xml Interface_1_innucast.rrd Interface_2_outdisc.rrd Interface_3_outqlen.rrd Processor_Queue_load1.rrd
Check_MK_system_time.rrd DotNet_Memory_Management__Global__percent.rrd Interface_1_in.rrd Interface_2_outerr.rrd Interface_3_out.rrd Processor_Queue_load5.rrd
Check_MK_user_time.rrd DotNet_Memory_Management__Global_.xml Interface_1_inucast.rrd Interface_2_outnucast.rrd Interface_3_outucast.rrd Processor_Queue.xml
Check_MK.xml Filesystem_C___C__.rrd Interface_1_outdisc.rrd Interface_2_outqlen.rrd Interface_3.xml System_Time_offset.rrd
CPU_utilization_user.rrd Filesystem_C___fs_size.rrd Interface_1_outerr.rrd Interface_2_out.rrd Memory_and_pagefile_memory.rrd System_Time.xml
CPU_utilization_util.rrd Filesystem_C___growth.rrd Interface_1_outnucast.rrd Interface_2_outucast.rrd Memory_and_pagefile_mem_total.rrd Uptime_uptime.rrd
CPU_utilization.xml Filesystem_C___trend.rrd Interface_1_outqlen.rrd Interface_2.xml Memory_and_pagefile_pagefile.rrd Uptime.xml
Disk_IO_SUMMARY_disk_average_read_wait.rrd Filesystem_C__.xml Interface_1_out.rrd Interface_3_indisc.rrd Memory_and_pagefile_pagefile_total.rrd Web_Service_Default_Web_Site_connections.rrd
Disk_IO_SUMMARY_disk_average_write_wait.rrd _HOST__pl.rrd Interface_1_outucast.rrd Interface_3_inerr.rrd Memory_and_pagefile.xml Web_Service_Default_Web_Site.xml
Disk_IO_SUMMARY_disk_read_ios.rrd _HOST__rta.rrd Interface_1.xml Interface_3_innucast.rrd PING_pl.rrd
Disk_IO_SUMMARY_disk_read_ql.rrd _HOST__rtmax.rrd Interface_2_indisc.rrd Interface_3_in.rrd PING_rta.rrd
Disk_IO_SUMMARY_disk_read_throughput.rrd _HOST__rtmin.rrd Interface_2_inerr.rrd Interface_3_inucast.rrd PING_rtmax.rrd

Which one to delete and what is the command ?
Is it simply
rm

Please guide me…being a beginner I am trying understand the concept:)

BR
/Prachi

andreas-doehler · January 9, 2021, 5:07pm

Please don’t post so many log files if there is nothing inside what is helpful. The notification log has no error inside - how does the generated mail for the filesystem “/storage/log” looks like?

Please answer the questions before posting some log files

The following message can come from a I/O problem on your server. That means the old performance data could not be written to disk and it is already the next data in the queue. How looks the disk I/O load on your monitoring machine?

PrachiKatakwar · January 9, 2021, 6:47pm

Hi @andreas-doehler,

I am sorry for long logs files.
how does the generated mail for the filesystem “/storage/log” looks like?

Checked the var/log/maillog — cant find any errors.

the performance data is shown correctly inside PNP4Nagios?
Yes , I cant see anything wrong except the rrdcatched error coming in perfdata.log.

How looks the disk I/O load on your monitoring machine?
My Monitoring machine is CentOS 7 VM on VMware.
Using Top command could see below
27717 checkmk 20 0 74312 28272 4952 S 10.0 0.4 0:00.30 python
27473 checkmk 20 0 74312 28276 4952 S 6.6 0.4 0:00.20 python
27478 checkmk 20 0 74336 28548 4952 S 6.6 0.4 0:00.20 python
27479 checkmk 20 0 74312 28276 4952 S 6.6 0.4 0:00.20 python
27716 checkmk 20 0 74312 28276 4952 S 6.3 0.4 0:00.19 python
27712 checkmk 20 0 74336 28544 4952 S 6.0 0.4 0:00.18 python
27482 checkmk 20 0 74336 28544 4952 S 5.6 0.4 0:00.17 python
27254 checkmk 20 0 74336 28548 4952 S 5.0 0.4 0:00.24 python
27258 checkmk 20 0 74336 28548 4952 S 5.0 0.4 0:00.24 python
27969 checkmk 20 0 72560 27580 4936 R 5.0 0.3 0:00.15 python
27968 checkmk 20 0 71372 26348 4916 R 4.3 0.3 0:00.13 python
27973 checkmk 20 0 68392 23604 4916 R 4.3 0.3 0:00.13 python
27964 checkmk 20 0 68140 23288 4912 R 4.0 0.3 0:00.12 python
27245 checkmk 20 0 74336 28544 4952 S 3.7 0.4 0:00.18 python

I don’t know what is wrong in system,
/Prachi

andreas-doehler · January 9, 2021, 7:56pm

You should look inside this mail that was generated not in the mail log.
As there is no error message inside the notify.log it means the mail was generated correctly from CMK.
Now you need to check how does the mail looks like in your mail program - is there a graph or not?

Again why do you look inside logs if I ask that you should check the graphs inside PNP4Nagios itself.
You should look at a graph that generated errors in mail and try to open this graph inside PNP to test if it is shown there.

I see nothing about the disk I/O in your output. As you use a VM i would say with 100% certainty that you have I/O problems in your system. And this I/O problems lead to the RRD::update errors.

PrachiKatakwar · January 12, 2021, 5:20am

Hi @andreas-doehler/@anonymous15 ,

GoodMorning,

Today again in the morning I got the below error for sekatenessus, now its like not coming for all the servers so from last 2 days while I was monitoring it didnt come but today suddenly in the morning this came along with the email notification