Hi !!!
Why does the virtual appliance have no swap file defined and how can I fix this ? We are not allowed to mount extra disks and the default disk /dev/sdb is used for the cluster, any other partition is really small. As a quick fix we have used the /rw mount but this is only 4G so we used a swap of only 2G in size …
I cannot talk for the checkmk people but I would expect that a checkmk server that starts swapping is of no use because it is only busy with moving data in and out of the disk instead of doing its math. The services and hosts will go all stale if that happens.
To my experience servers that should respond quickly usually don’t have swap enabled.
if you run out off memory the OOM Killer kicks in, and kills your cmc … 
Adding 100G memory is a solution, but not very economic …
If this happens the system by itself has a real problem.
In my monitoring installations i also run very small swap space.
We run nearly all our virtual machines with just 100MB of swap space.
If a VM starts swapping it has not enough RAM configured for the application.
if you run out of memory the OOM Killer (Out Of Memory) start killing processes and usually it kills the cmk/cmc process leaving the system useless.
I agree the system gets ultra slow when it start to swap, but I prefer a slow system instead of no system
Iow I’m still clueless on how to proceed, we are not allowed to mount extra disks on the virtual appliance, so how should I proceed ?
I would first inspect why the system runs out of memory.
If this happend in the past time on one of my systems, something was wrong inside the system.
You really should find out, why the system is running out of memory. This should not happen.
As a workaround you can create a file and use it as swap space. Here is an example on how to do it: https://linuxize.com/post/create-a-linux-swap-file/
But again: Find the problem, don’t work around the error.
our master cluster had a split brain, this escalated in several discoveries that went bezerk and some were running several hours.
Once the split brain was solved everything normalized
Look at the bright side, all nodes have now free memory upgrades 