Hi Mike,
since we don't know how many hosts/services/tags are in the configuration, I can only guess at the moment.
It could be DNS related. Can you send me the profile file generated by the command
cmk --profile -vv -B
The command itself will take a bit longer, but the profile file should show the problem directly.
Regards
Andreas
···
Von: "Andreas Döhler" <andreas.doehler@gmail.com>
An: "Rochford, Mike" <MRochford@starkstate.edu>
CC: checkmk-en@lists.mathias-kettner.de
Gesendet: Dienstag, 19. November 2019 22:36:09
Betreff: Re: [Check_mk (english)] Check_MK very slow when activating changes
Hi Mike,
then this should not be a problem with the reloading really. I think my last post is obsolete with this. But you can try if you find the time.
It is possible that someone from the tribe29 team is reading this and knows exactly what the problem is.
Best regards
Andreas
Am Di., 19. Nov. 2019 um 21:48 Uhr schrieb Rochford, Mike < [ mailto:MRochford@starkstate.edu | MRochford@starkstate.edu ] >:
Andreas,
I will try the below steps when I get a chance later tonight or tomorrow.
I just ran an activation so that I could get the complete debug and it just completed. Looks like the section that takes the longest is the cmc_all_hosts at 1673.99 seconds. From this I would assume that the process is hanging here? Not sure what that means or how to go about troubleshooting it.
OMD[ssc]:~$ cmk --debug -vvO
Waiting for exclusive lock on /omd/sites/ssc/etc/check_mk/ [ http://main.mk/ | main.mk ] .
Generating configuration for core (type cmc)...
Time needed [cmc_global]: 0.00 sec
time period '24X7': 2 time points
Time needed [cmc_timeperiods]: 0.00 sec
Time needed [cmc_contacts]: 0.00 sec
Time needed [cmc_all_hosts]: 1673.99 sec
Time needed [cmc_groups]: 0.00 sec
Time needed [cmc_groups]: 0.00 sec
Time needed [cmc_groups]: 0.00 sec
Time needed [cmc_stringlist]: 0.00 sec
Time needed [cmc_contactlists]: 0.00 sec
Not importing state from Nagios, /omd/sites/ssc/var/nagios/retention.dat not found.
/omd/sites/ssc/var/check_mk/core/config written.
OK
Packing config...OK
Reloading monitoring core...OK
Thanks,
Mike Rochford
IT Manager • Information Technology
330-494-6170 ext 4244 • B224
[ mailto:mrochford@starkstate.edu | mrochford@starkstate.edu ] • [ https://www.starkstate.edu/ |
starkstate.edu ]
6200 Frank Ave NW • North Canton, OH 44720
[ https://www.starkstate.edu/ ] [ Stark State College - Startseite | Facebook ] [ https://twitter.com/TheStarkState ] [ starkstatecollege - YouTube ] [ https://www.instagram.com/starkstate/ ]
From: Andreas Döhler [mailto: [ mailto:andreas.doehler@gmail.com | andreas.doehler@gmail.com ] ]
Sent: Tuesday, November 19, 2019 3:33 PM
To: Rochford, Mike < [ mailto:MRochford@STARKSTATE.EDU | MRochford@STARKSTATE.EDU ] >
Cc: [ mailto:checkmk-en@lists.mathias-kettner.de | checkmk-en@lists.mathias-kettner.de ]
Subject: Re: [Check_mk (english)] Check_MK very slow when activating changes
Ok then it should be a check blocking the core helpers to end in time.
Or a other problem that all helper processes can be reloaded.
If you do a test sometime with "omd stop cmc" and then a "cmk --debug -vvU" and now the "omd start cmc".
Takes this also so long?
If this doesn't work I have no idea where you can look further.
Then you need to write to the guys in Munich. But the first thing what they would ask is, "is it possible to upgrade to the latest 1.5 version first" 
Best regards
Andreas
Am Di., 19. Nov. 2019 um 21:20 Uhr schrieb Rochford, Mike < [ mailto:MRochford@starkstate.edu | MRochford@starkstate.edu ] >:
Andreas,
We don’t use the Nagios core we use the CMC core. So I don’t think the below issue applies to our situation.
When I run the command to commit the changes here is the output. It hangs at this point. This displays instantly and then will sit at this point for close to 45 minutes. After that it completes the remaining commands almost instantly and completes successfully. We don’t see any errors or anything in the logs. Just that it takes an extremely long time to commit the changes.
OMD[ssc]:~$ cmk --debug -vvO
Waiting for exclusive lock on /omd/sites/ssc/etc/check_mk/ [ https://urldefense.proofpoint.com/v2/url?u=http-3A__main.mk&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=1vzWWJP41D7YfByMkCtlUdEhu_mhaY2pP6bURyi2XOA&s=npRkvoLZ6x6QuatUaVdVtyXyyupgEsHeaQnCoHYchp8&e= | main.mk ] .
Generating configuration for core (type cmc)...
Time needed [cmc_global]: 0.00 sec
time period '24X7': 2 time points
Time needed [cmc_timeperiods]: 0.00 sec
Time needed [cmc_contacts]: 0.00 sec
Thanks,
Mike Rochford
IT Manager • Information Technology
330-494-6170 ext 4244 • B224
[ mailto:mrochford@starkstate.edu | mrochford@starkstate.edu ] • [ https://www.starkstate.edu/ | starkstate.edu ]
6200 Frank Ave NW • North Canton, OH 44720
[ https://www.starkstate.edu/ ] [ https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_starkstate&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=1vzWWJP41D7YfByMkCtlUdEhu_mhaY2pP6bURyi2XOA&s=4qeCiV0K0TkaO_4n4wkv60T_O8SqgevHIziqalwn-wQ&e= ] [ https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_TheStarkState&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=1vzWWJP41D7YfByMkCtlUdEhu_mhaY2pP6bURyi2XOA&s=Az_5OT83A5SnZb6wZztPzZcBVGyssmltU3Hd6NmmG7g&e= ] [ https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_user_starkstatecollege&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=1vzWWJP41D7YfByMkCtlUdEhu_mhaY2pP6bURyi2XOA&s=IvrlraUbuCAr3ujw-cZqgCgx3VHnPHH-eDgGN1Jefd8&e= ] [ https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instagram.com_starkstate_&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=1vzWWJP41D7YfByMkCtlUdEhu_mhaY2pP6bURyi2XOA&s=JeBs4vG1OyuHiyzThKm7FAOi7YUvsK8bFSuuTDDOW0Y&e= ]
From: Andreas Döhler [mailto: [ mailto:andreas.doehler@gmail.com | andreas.doehler@gmail.com ] ]
Sent: Tuesday, November 19, 2019 3:03 PM
To: Rochford, Mike < [ mailto:MRochford@STARKSTATE.EDU | MRochford@STARKSTATE.EDU ] >
Cc: [ mailto:checkmk-en@lists.mathias-kettner.de | checkmk-en@lists.mathias-kettner.de ]
Subject: Re: [Check_mk (english)] Check_MK very slow when activating changes
Hi Mike,
if you do your activation with the "--debug -vv" switches on where does it stays so long?
I think this sounds like the activation problem with the Nagios core we discussed here for some time.
If the system hangs at "config check of core" then this thread from the mailing list can help - I hope [ https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mathias-2Dkettner.de_pipermail_checkmk-2Den_2019-2DApril_027617.html&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=tEN9Iu-2-wDhxaeZx0stG8abIfwDq1cWiI7gEpVDahg&s=kAz3O6N7HRMJYhIFqiFEjNLuAvtpNAXOdFqHF0yDx_A&e= | https://lists.mathias-kettner.de/pipermail/checkmk-en/2019-April/027617.html ]
Best regards
Andreas
Am Di., 19. Nov. 2019 um 15:59 Uhr schrieb Rochford, Mike < [ mailto:MRochford@starkstate.edu | MRochford@starkstate.edu ] >:
BQ_BEGIN
Recently we have been seeing that activating changes in Check_MK version 1.5.0p9 is taking 30-45 minutes. In the past the changes would be activated in seconds. We are not seeing any errors in the GUI or from the command line. I’ve activated changes from both the GUI and the command line with the additional debug commands and no errors are displayed. The activations will finish but it’s taking 30-45 minutes to complete.
It doesn’t seem to matter how many changes are being activated. The time it takes is the same for 1 change or 10 changes. There have not been any changes or updates to our server or the Check_MK software when these delays started occurring.
Thanks,
Mike Rochford
IT Manager • Information Technology
330-494-6170 ext 4244 • B224
[ mailto:mrochford@starkstate.edu | mrochford@starkstate.edu ] • [ https://www.starkstate.edu/ | starkstate.edu ]
6200 Frank Ave NW • North Canton, OH 44720
[ https://www.starkstate.edu/ ] [ https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_starkstate&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=tEN9Iu-2-wDhxaeZx0stG8abIfwDq1cWiI7gEpVDahg&s=w22f0R22vPIKB9IUS11yY6DfTiEN-B6EiXO1td8nydE&e= ] [ https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_TheStarkState&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=tEN9Iu-2-wDhxaeZx0stG8abIfwDq1cWiI7gEpVDahg&s=u4mTczt7q3qhKBJXu1mE2RlkesZfXYIE8tSw_Qf_NtM&e= ] [ https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_user_starkstatecollege&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=tEN9Iu-2-wDhxaeZx0stG8abIfwDq1cWiI7gEpVDahg&s=3pi12uhRNd7SWzX5CvCjOxVeDSLojcz-SwfIJriHlY&e= ] [ https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instagram.com_starkstate&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=tEN9Iu-2-wDhxaeZx0stG8abIfwDq1cWiI7gEpVDahg&s=kJFya6CYvR0_GJxWq0-OMmutKF9gjKXxoDcL8nIFqCc&e= ]
_______________________________________________
checkmk-en mailing list
[ mailto:checkmk-en@lists.mathias-kettner.de | checkmk-en@lists.mathias-kettner.de ]
Manage your subscription or unsubscribe
[ https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mathias-2Dkettner.de_cgi-2Dbin_mailman_listinfo_checkmk-2Den&d=DwMFaQ&c=BsKAWNA-yiy1Hzspd2_ParwRTlDx91DmTVgZVqjcD8I&r=OlJQIXuPxLNCyrE21zBdOEaqie0TLN0M5zO_RLiPia4&m=tEN9Iu-2-wDhxaeZx0stG8abIfwDq1cWiI7gEpVDahg&s=8LlHlvT2TFzSheWgwTgXaPkA-DpS_6HjY_geXwxDerI&e= | https://lists.mathias-kettner.de/cgi-bin/mailman/listinfo/checkmk-en ]
BQ_END
_______________________________________________
checkmk-en mailing list
checkmk-en@lists.mathias-kettner.de
Manage your subscription or unsubscribe
https://lists.mathias-kettner.de/cgi-bin/mailman/listinfo/checkmk-en