Rename single host stucks (over 1 hour)

Troubleshooting
, ,
1

CMK version:
2.1.0p10 CEE
OS version:
Appliance
Error message:
There is not really an error Message but the task shows this status:
Runtime 1:34:11 (estimated duration: 0:00:14)

Renaming WATO configuration...
Renaming host(s) in folders...
Renaming host(s) in cluster nodes...
Renaming host(s) in parents...
Renaming host(s) in rulesets...
Renaming host(s) in BI aggregations...
Renaming host(s) in base configuration, rrd, history files, etc.
This might take some time and involves a core restart...
2

Hi Marco,

how large is your environment and how long is your monitoring running ?
checkmk has to go through every log line of the monitoring history and change the entries of the hostname there.
If possible, i would generally not recommend to rename hosts but to delete and recreate, as renaming is a massive effort for the server and i´ve seen monitoring die because of the cpu usage during this process.

3

Hi,

the site where this issue is, was cloned from another Site without logs and rrds and is running maybe a month.
I already read that in the forum, but wanted to try it to keep the history (on the Main Site)
I don’t see an high usage of any resources on the server

image
image1764×669 224 KB

4

The rename process is the last entry in your ps list, maybe you can attach an strace to see what is doing at the moment.
CPU doesnt seem to be the problem :slight_smile:

5

I’m not strong with strace at all, but for me it looks like the Process is doing nothing.
I tried it with and waited a Minute but nothing more appeared

root@checkmk:~# strace -p 2548
strace: Process 2548 attached
flock(5, LOCK_EX

also

root@checkmk:~# strace -e trace=read,write -p 2548
strace: Process 2548 attached

shows nothing.
lsof of the pid:

root@checkmk:~# lsof -p 2548
COMMAND  PID       USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
python3 2548 serviceapi  cwd    DIR   0,21     4096          2 /
python3 2548 serviceapi  rtd    DIR   0,21     4096          2 /
python3 2548 serviceapi  txt    REG   8,18    14304    4194684 /omd/versions/2.1.0p10.cee/bin/python3.9
python3 2548 serviceapi  mem    REG   8,18    19872    4219989 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_multiprocessing.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18  2856704    4195350 /omd/versions/2.1.0p10.cee/lib/libprotobuf.so.29.0.1
python3 2548 serviceapi  mem    REG   8,18   347432    4302956 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/google/protobuf/pyext/_message.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18   104760    4197007 /omd/versions/2.1.0p10.cee/lib/libgcc_s.so.1
python3 2548 serviceapi  mem    REG   8,18  2161248    4196994 /omd/versions/2.1.0p10.cee/lib/libstdc++.so.6.0.29
python3 2548 serviceapi  mem    REG   8,18    14368    4302935 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/google/protobuf/internal/_api_implementation.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    37728    4219961 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_csv.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18   266464    4219992 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/pyexpat.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14464    4219994 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_statistics.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18   306664    4219927 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_decimal.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18   128120    4219946 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_ctypes.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18  1115552    4219950 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/unicodedata.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18   844768    4236194 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so
python3 2548 serviceapi  mem    REG   8,18    47600    4235373 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/bcrypt/_bcrypt.abi3.so
python3 2548 serviceapi  mem    REG   8,18    18336    4237320 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Cipher/_raw_aesni.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    30600    4237253 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Cipher/_raw_aes.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    18328    4237255 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Cipher/_raw_ocb.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14216    4237234 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Hash/_ghash_clmul.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14144    4237228 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Hash/_ghash_portable.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14056    4237150 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Util/_cpuid_c.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14232    4236634 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Protocol/_scrypt.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14216    4237293 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Cipher/_Salsa20.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14240    4237165 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Hash/_MD5.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    22440    4237167 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Hash/_SHA256.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    18336    4237156 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Hash/_SHA1.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14224    4237217 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Hash/_BLAKE2s.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14056    4237135 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Util/_strxor.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14248    4237304 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Cipher/_raw_ctr.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14160    4237334 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Cipher/_raw_ofb.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14240    4237259 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Cipher/_raw_cfb.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   0,21    35224       1613 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
python3 2548 serviceapi  mem    REG   8,18   206536    4221095 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/_cffi_backend.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    53160    4219963 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_blake2.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    59336    4219953 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_hashlib.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   0,21    30776        543 /usr/lib/x86_64-linux-gnu/libuuid.so.1.3.0
python3 2548 serviceapi  mem    REG   8,18    14152    4237313 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Cipher/_raw_cbc.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14056    4237322 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/Cryptodome/Cipher/_raw_ecb.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    14464    4219951 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_uuid.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18   757312    4195342 /omd/versions/2.1.0p10.cee/lib/libnetsnmp.so.40.1.0
python3 2548 serviceapi  mem    REG   8,18    48360    4261699 /omd/versions/2.1.0p10.cee/lib/python3.9/site-packages/netsnmp/client_intf.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    44328    4219971 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/binascii.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18  3049384    4197002 /omd/versions/2.1.0p10.cee/lib/libcrypto.so.1.1
python3 2548 serviceapi  mem    REG   8,18   609576    4196998 /omd/versions/2.1.0p10.cee/lib/libssl.so.1.1
python3 2548 serviceapi  mem    REG   8,18   182048    4219984 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    48920    4219932 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_json.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    23040    4219935 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_posixsubprocess.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    32384    4219944 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_sha512.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    23176    4219957 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_random.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   0,21   158400        455 /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.4
python3 2548 serviceapi  mem    REG   8,18    15008    4219982 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_bisect.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    19184    4219934 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/grp.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   0,21    74688        249 /usr/lib/x86_64-linux-gnu/libbz2.so.1.0.4
python3 2548 serviceapi  mem    REG   0,21   117184         39 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
python3 2548 serviceapi  mem    REG   8,18    45600    4219929 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_lzma.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    36624    4219952 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/zlib.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    51296    4219948 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_multibytecodec.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18   121568    4219987 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_datetime.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    19616    4219940 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_queue.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18   134816    4219942 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_pickle.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    31008    4219922 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/termios.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    58888    4219974 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_struct.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    70464    4219969 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/array.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    28288    4219968 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_bz2.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    91824    4219923 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_socket.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    31104    4219960 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_heapq.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    22912    4219962 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/fcntl.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    40728    4219967 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/select.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   8,18    70312    4219939 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/math.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   0,21   200752        434 /usr/lib/locale/C.UTF-8/LC_CTYPE
python3 2548 serviceapi  mem    REG   0,21  3373216        415 /usr/lib/locale/locale-archive
python3 2548 serviceapi  mem    REG   0,21  1820400         43 /usr/lib/x86_64-linux-gnu/libc-2.28.so
python3 2548 serviceapi  mem    REG   0,21  1579448         41 /usr/lib/x86_64-linux-gnu/libm-2.28.so
python3 2548 serviceapi  mem    REG   0,21    14720         37 /usr/lib/x86_64-linux-gnu/libutil-2.28.so
python3 2548 serviceapi  mem    REG   0,21    14592         35 /usr/lib/x86_64-linux-gnu/libdl-2.28.so
python3 2548 serviceapi  mem    REG   0,21   146968         33 /usr/lib/x86_64-linux-gnu/libpthread-2.28.so
python3 2548 serviceapi  mem    REG   0,21    43328        389 /usr/lib/x86_64-linux-gnu/libcrypt-2.28.so
python3 2548 serviceapi  mem    REG   8,18    14632    4219970 /omd/versions/2.1.0p10.cee/lib/python3.9/lib-dynload/_opcode.cpython-39-x86_64-linux-gnu.so
python3 2548 serviceapi  mem    REG   0,21    26402        422 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
python3 2548 serviceapi  mem    REG   8,18  3555528    4197001 /omd/versions/2.1.0p10.cee/lib/libpython3.9.so.1.0
python3 2548 serviceapi  mem    REG   0,21   165632         30 /usr/lib/x86_64-linux-gnu/ld-2.28.so
python3 2548 serviceapi    0r  FIFO   0,12      0t0 1188448005 pipe
python3 2548 serviceapi    1w  FIFO   0,12      0t0 1188448006 pipe
python3 2548 serviceapi    2w  FIFO   0,12      0t0 1188448007 pipe
python3 2548 serviceapi    3rW  REG   8,18      106    3416325 /omd/sites/serviceapi/etc/check_mk/main.mk
python3 2548 serviceapi    4w   CHR    1,3      0t0       1028 /dev/null
python3 2548 serviceapi    5r   REG   8,18       29    3416903 /omd/sites/serviceapi/etc/check_mk/multisite.mk
6

Same issue here when renaming a host. Any ideas how to debug this problem?

7

Maybe this helps: the rename process wants to lock the file /omd/sites/SITE/etc/check_mk/multisite.mk, but it is used by the apache service.
Use ‘fuser /omd/sites/SITE/etc/check_mk/multisite.mk’ to find the apache-process using the multisite-file and kill it. Now you can see with strace that the renaming-process ends successfully and you can start the apache server with omd start apache (as site-user)
This is only a workaround…

8

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.