High IO after updating to 2.4 and disk is filling up slowly until full

CMK version: 2.4.0p3.cee
OS version: Debian GNU/Linux 11 (bullseye)

Total DISK READ:         0.00 B/s | Total DISK WRITE:       382.03 M/s
Current DISK READ:       0.00 B/s | Current DISK WRITE:     358.67 M/s
    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 645047 be/4 root        0.00 B/s    0.00 B/s  0.00 % 85.83 % [kworker/u65:3+flush-8:0]
 651033 be/4 NOLI        0.00 B/s   21.09 K/s  0.00 %  2.20 % python3 /omd/sites/sitename/bin/fetcher
 651525 be/4 NOLI        0.00 B/s    7.03 K/s  0.00 %  1.74 % apache2 -f /omd/sites/sitename/etc/apache/apache.conf
 650848 be/4 NOLI        0.00 B/s  382.00 M/s  0.00 %  0.00 % python3 /omd/sites/sitename/bin/mkeventd --syslog --syslog-fd 3 --snmptrap --snmptrap-fd 5
      1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
      2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
      3 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_gp]
root@checkmk:~# lsof | grep '(deleted)'
cafservic   1732                       dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2045 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2061 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2062 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2063 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2064 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2065 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2066 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2067 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2068 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2069 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2070 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2071 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2072 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2073 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2074 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2075 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2076 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2077 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2079 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2080 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2081 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2082 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2083 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2084 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2085 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2086 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2087 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2088 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2089 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2090 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2091 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2092 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2093 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2094 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2095 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2096 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2097 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2098 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2099 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2100 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2101 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2552 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2553 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2554 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2555 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2556 ZMQbg/Rea      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2557 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2558 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2559 ZMQbg/IO/      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2594 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2595 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2596 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2597 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2598 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2599 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
cafservic   1732   2600 cafservic      dcscaf   29u      REG                8,2            0    7864326 /tmp/#7864326 (deleted)
apache2   218384                         root    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)
apache2   624319                     www-data    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)
apache2   633810                     www-data    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)
apache2   635174                     www-data    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)
apache2   636338                     www-data    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)
apache2   640832                     www-data    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)
apache2   641412                     www-data    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)
apache2   645600                     www-data    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)
apache2   648243                     www-data    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)
gunicorn: 650842                         sitename    8u      REG                8,2            0    7864331 /tmp/wgunicorn-gsslq6vp (deleted)
gunicorn: 650846                         sitename    8u      REG                8,2            0    7864331 /tmp/wgunicorn-gsslq6vp (deleted)
python3   650848                         sitename    7u      REG                8,2     48059832   12845075 /opt/omd/sites/sitename/var/mkeventd/history/history.sqlite-wal (deleted)
python3   650848                         sitename    8ur     REG                8,2        98304   12845077 /opt/omd/sites/sitename/var/mkeventd/history/history.sqlite-shm (deleted)
python3   650848                         sitename   12u      REG                8,2  67484127232    7340079 /var/tmp/etilqs_2c49620e4c850724 (deleted)
python3   650848 650849 StatusSer        sitename    7u      REG                8,2     48059832   12845075 /opt/omd/sites/sitename/var/mkeventd/history/history.sqlite-wal (deleted)
python3   650848 650849 StatusSer        sitename    8ur     REG                8,2        98304   12845077 /opt/omd/sites/sitename/var/mkeventd/history/history.sqlite-shm (deleted)
python3   650848 650849 StatusSer        sitename   12u      REG                8,2  67481890816    7340079 /var/tmp/etilqs_2c49620e4c850724 (deleted)
python3   650848 650850 EventServ        sitename    7u      REG                8,2     48059832   12845075 /opt/omd/sites/sitename/var/mkeventd/history/history.sqlite-wal (deleted)
python3   650848 650850 EventServ        sitename    8ur     REG                8,2        98304   12845077 /opt/omd/sites/sitename/var/mkeventd/history/history.sqlite-shm (deleted)
python3   650848 650850 EventServ        sitename   12u      REG                8,2  67482873856    7340079 /var/tmp/etilqs_2c49620e4c850724 (deleted)
gunicorn: 650899                         sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899                         sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650900 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650900 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650901 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650901 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650902 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650902 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650903 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650903 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650904 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650904 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650905 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650905 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650906 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650906 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650907 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650907 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650908 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650908 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650909 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650909 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650899 650910 cmk-autom        sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650899 650910 cmk-autom        sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
gunicorn: 650911                         sitename   24u      REG                8,2            0    7864332 /tmp/wgunicorn-auufzhcj (deleted)
gunicorn: 650912                         sitename   25u      REG                8,2            0    7864333 /tmp/wgunicorn-17rfo9la (deleted)
apache2   651640                     www-data    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)
apache2   651657                     www-data    9u      REG                8,2            0    7864330 /tmp/.ZendSem.Emk60m (deleted)

Good Morning @torsteingehc,

if you’re using CEE, could you open a support request?

Sunny Greetings
Hartmut

1 Like