Frequent coredumps when checkmk fetcher is cleaning up snmp response data

Troubleshooting
,
1

CMK version: 2.5.0p12
OS version: Oracle Linux 9

Error message:

We frequently see python coredumps in our checkmk server’s system logfile. After initial analysis we found out:

The stack trace shows the crash occurring in the Net-SNMP library while freeing a PDU after a successful SNMP GET:

malloc_printerr()
_int_free()
free()
snmp_free_varbind()
snmp_free_pdu()
netsnmp_get()

The process aborts inside libnetsnmp.so.40 (which comes shipped with the site) and appears to be triggered during cleanup rather than during communication with the SNMP device itself.

Entries in cmc.log matching the time of the core dumps show error messages like:

[fetcher pool] [service "<HOSTNAME>;Check_MK"] [helper 1044419] unexpected message on stderr: free(): invalid pointer
[fetcher pool] [service "<HOSTNAME>;Check_MK Discovery"] [helper 1325170] unexpected message on stderr: free(): invalid size

Could this result from a misconfiguration or is it some kind of bug?
Thanks a lot in advance!

Best Regards,
Sven

2

One lead worth checking before anything else: Red Hat rebased OpenSSL to 3.5 in RHEL 9.7 (Oracle Linux 9 follows RHEL), and the site ships its own libcrypto.so.3. See Werk #18935. Net-SNMP links against libcrypto, so a mismatch there can produce exactly this picture — memory allocated by one library, freed by the other, aborting in snmp_free_varbind() / snmp_free_pdu() with the device itself being fine.

Quick check as site user:

OMD[site]:~$ ldd lib/libnetsnmp.so.40 | grep -i crypto
OMD[site]:~$ echo $LD_LIBRARY_PATH
OMD[site]:~$ openssl version -a
$ rpm -q openssl openssl-libs && cat /etc/oracle-release

Also: do the crashing hosts use SNMPv3? If all crashes happen on v2c, it’s probably something else.

Just guessing: As an immediate workaround you can switch the affected hosts to the Classic SNMP backend (Setup > Agents > SNMP rules > Hosts using a specific SNMP Backend) — slower, but a crash there no longer takes the fetcher down.

3

Hello Joerg, thank you for your reply!
We checked the outputs of the commands you suggested (as site user):

$ rpm -q openssl openssl-libs && cat /etc/oracle-release
openssl-3.5.8-1.0.1.el9_8.x86_64
openssl-libs-3.5.8-1.0.1.el9_8.x86_64
Oracle Linux Server release 9.8

$ echo $LD_LIBRARY_PATH
/omd/sites/SITE/local/lib:/omd/sites/SITE/lib

$ openssl version -a
OpenSSL 3.5.7 9 Jun 2026 (Library: OpenSSL 3.5.7 9 Jun 2026)

$ ldd /usr/lib64/libnetsnmp.so.40.1.0 | grep crypto
libcrypto.so.3 => /omd/sites/SITE/lib/libcrypto.so.3 (0x00007f5089400000)

So there indeed appears to be a mismatch between the openssl via rpm installation and the site-supplied openssl (3.5.7 vs 3.5.8)

The errors seems to happen on SNMPv3 Devices but as far as I can tell is not limited to them.

Thank you!
Regards, Sven