CMK version is 2.5.0p8 pro/ultimate - OS docker image
With 2.5 we found a new “nice” feature. Very long wait time at config generation if you have some/many “no-ip” hosts or hosts with “FQDN” instead of IPs.
We found the problem inside the dev systems with many “no-ip” hosts.
After issue a “cmk --debug -vvU” to generate a new core config you wait a very long time if you have bad luck.
What happens?
- In the background CMK tries to resolve every hostname also if there is “no-ip” set - why?
- Normally this would be not so bad but inside a docker dev container you have some search entry like “localdomain” or “lxd” that are not existing
- Result is a timeout from your DNS server for these queried domains. (CMK checks all hostnames with the short name and the search domain)
- If you remove the search domain from inside the container it is already 10 times faster as it only gets en “NXDomain” reply from your local DNS for the short name query
- It is only possible to eliminate the DNS lookup if you inside some IP for the “no-ip” hosts like 127.0.0.1
The problem exists on your system if a config generation waits some time at the point
“Creating cmc protobuf configuration…” or you a “cmk -D hostname” for a single host and it takes some seconds. With IP set this is nearly instant.
2.4 had not these DNS lookup problems for “no-ip” hosts.
tcpdump inside the container
tcpdump -nn udp port 53
tcpdump: verbose output suppressed, use -v\[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:11:00.689065 IP 172.17.0.2.47905 > 192.168.150.2.53: 32848+ A? OSG-1-5.localdomain. (37)
16:11:05.694373 IP 172.17.0.2.47905 > 192.168.150.2.53: 32848+ A? OSG-1-5.localdomain. (37)
16:11:10.700106 IP 172.17.0.2.35433 > 192.168.150.2.53: 63407+ A? OSG-1-5. (25)
16:11:10.724529 IP 192.168.150.2.53 > 172.17.0.2.35433: 63407 NXDomain 0/1/0 (100)
16:11:10.725124 IP 172.17.0.2.44354 > 192.168.150.2.53: 51685+ A? OSG-R.105. (27)
16:11:10.756709 IP 192.168.150.2.53 > 172.17.0.2.44354: 51685 NXDomain 0/1/0 (102)
16:11:10.756870 IP 172.17.0.2.51343 > 192.168.150.2.53: 55109+ A? OSG-R.105.localdomain. (39)
You see the localdomain queries need nearly 5 seconds timeout before CMK continues.
Next is where CMK stops at config generation without further output what it does.
Starting full compilation for all hosts
Creating global helper config...Trying to acquire lock on /omd/sites/cmk/var/check\_mk/core/helper\_config/578/rrd\_config/cmc\_log\_rrdcreation
Got lock on /omd/sites/cmk/var/check\_mk/core/helper\_config/578/rrd\_config/cmc\_log\_rrdcreation
Releasing lock on /omd/sites/cmk/var/check\_mk/core/helper\_config/578/rrd\_config/cmc\_log\_rrdcreation
Released lock on /omd/sites/cmk/var/check\_mk/core/helper\_config/578/rrd\_config/cmc\_log\_rrdcreation
Trying to acquire lock on /omd/sites/cmk/var/check\_mk/core/helper\_config/578/inventory\_plugins\_index.json
Got lock on /omd/sites/cmk/var/check\_mk/core/helper\_config/578/inventory\_plugins\_index.json
Releasing lock on /omd/sites/cmk/var/check\_mk/core/helper\_config/578/inventory\_plugins\_index.json
Released lock on /omd/sites/cmk/var/check\_mk/core/helper\_config/578/inventory\_plugins\_index.json
OK
Creating cmc protobuf configuration…
“cmk -D” example with IP and “no-ip”
OMD[cmk]:~$ time cmk -D test123
test123
Addresses: 127.0.0.1
Tags: [address_family:ip-v4-only], [agent:no-agent], [criticality:prod], [ip-v4:ip-v4], [networking:lan], [piggyback:auto-piggyback], [ping:ping], [site:cmk], [snmp_ds:no-snmp]
Labels: [cmk/site:cmk]
Host groups: check_mk
Contact groups: all
Agent mode: No agent
Type of agent:
Process piggyback data
PING only
Services:
checktype item params description groups
--------- ---- ------ ----------- ------
real 0m2.584s
user 0m2.061s
sys 0m0.495s
OMD[cmk]:~$ time cmk -D test123
test123
Addresses: No IP
Tags: [address_family:no-ip], [agent:no-agent], [criticality:prod], [networking:lan], [piggyback:auto-piggyback], [site:cmk], [snmp_ds:no-snmp]
Labels: [cmk/site:cmk]
Host groups: check_mk
Contact groups: all
Agent mode: No agent
Type of agent:
Process piggyback data
PING only
Services:
checktype item params description groups
--------- ---- ------ ----------- ------
real 0m12.715s
user 0m2.179s
sys 0m0.481s
