Proxmox/DRBD cluster with a third diskless node reports an error

CMK version: 2.5.0p10
OS version: Debian 13

Proxmox VE 9.3
DRBD 9.3

Error message: Connection State: Established, Roles: Primary/Secondary, Diskstates: UpToDate/Diskless (Secondary/Diskless)

Description:

We have a Proxmox/DRBD cluster with three nodes. Only the first two nodes have local storage that is synchronized with DRBD; the third node is used for the quorum but also has sufficient computing power and is therefore diskless. The DRBD volumes are therefore diskless on the third node. However, Check-MK reports all volumes as faulty because it interprets “diskless” as an error. In our case, though, being diskless with respect to the third node is not an error.
Question: Is there a solution for configuring Check-MK so that the diskless configuration for the third node is recognized as error-free?

Walter

Edit: added the screen shot:

For the shown CRIT states not the “Diskless” state is the reason. It is the change from “Secondary/Diskless” to “UpToDate/Diskless” that you get a CRIT.
At discovery time the DRBD on this node was a secondary and now it is active.

I tried a “Rescan” without success and “Remove all and find new” without success.

This is the service view for one host:

image

How to solve this problem?

That’s strange as an update of the check parameters should solve it.
Can you have a look inside the discovery at the current check parameters and also do a “cmk -D hostname” to see the discovered parameters?

The relevant “Service discovery rules” for this Host:

Process discovery........................................................................................................................................................................................................

2 Rules


Process name: Proxmox %2

Process Matching: ^([0-9a-z\/_-]*)\/?(pve-firewall|pvestatd|pve-ha-crm|pve-ha-lrm|pvescheduler|pvefw-logger|proxmox-firewall|pve-lxc-syscalld).*$

Default parameters for detected services: 

CPU rescale maximum load: 100% is all cores at full load
Levels for process count: 1 processes, 1 processes, 1 processes, 1 processes,

Process name: Proxmox %2

Process Matching: ^([0-9a-z\/_-]*)\/?(pvedaemon|pveproxy).*$
Default parameters for detected services: 


CPU rescale maximum load: 100% is all cores at full load
Levels for process count: 1 processes, 1 processes, 50 processes, 100 processes

The “cmk -d hostname” returns this for the first DRBD devices:

drbd_disk                      drbd1001                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1001 disk
drbd_net                       drbd1001                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1001 net
drbd_stats                     drbd1001                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1001 stats
drbd                           drbd1001                    {‘roles_inventory’: [‘Secondary’, ‘Secondary’], ‘diskstates_inventory’: [‘UpToDate’, ‘Diskless’]}                                                                                                                                                                                                                                                                                                        DRBD drbd1001 status
drbd_disk                      drbd1002                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1002 disk
drbd_net                       drbd1002                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1002 net
drbd_stats                     drbd1002                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1002 stats
drbd                           drbd1002                    {‘roles_inventory’: [‘Secondary’, ‘Secondary’], ‘diskstates_inventory’: [‘UpToDate’, ‘Diskless’]}                                                                                                                                                                                                                                                                                                        DRBD drbd1002 status
drbd_disk                      drbd1003                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1003 disk
drbd_net                       drbd1003                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1003 net
drbd_stats                     drbd1003                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1003 stats
drbd                           drbd1003                    {‘roles_inventory’: [‘Secondary’, ‘Secondary’], ‘diskstates_inventory’: [‘UpToDate’, ‘Diskless’]}                                                                                                                                                                                                                                                                                                        DRBD drbd1003 status
drbd_disk                      drbd1004                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1004 disk
drbd_net                       drbd1004                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1004 net
drbd_stats                     drbd1004                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1004 stats
drbd                           drbd1004                    {‘roles_inventory’: [‘Secondary’, ‘Secondary’], ‘diskstates_inventory’: [‘UpToDate’, ‘Diskless’]}                                                                                                                                                                                                                                                                                                        DRBD drbd1004 status
drbd_disk                      drbd1005                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1005 disk
drbd_net                       drbd1005                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1005 net
drbd_stats                     drbd1005                    {}                                                                                                                                                                                                                                                                                                                                                                                                       DRBD drbd1005 stats
drbd                           drbd1005                    {‘roles_inventory’: [‘Secondary’, ‘Secondary’], ‘diskstates_inventory’: [‘UpToDate’, ‘Diskless’]}


Edit to make it more redable.

They are not relevant - the important ones are the rule “DR:BD roles and diskstates”, i think there is somewhere defined that this DRBD should be “Secondary/Diskless”.

I found a “Service monitoring rule” “DR:BD roles and diskstates” and added this rule to the 3 servers. Unfortunately the rule do not change anything.

Can you show the rule?

Do you mean this:

In this rule nothing is defined.
This will not work.

I added this rule only after you asked your question. What do I need to configure in this rule, and why do I need it?

You don’t need to configure this rule - you should check if there is a rule existing that defines some parameters.
Your “cmk -d hostname” output shows that you have the discovered roles and diskstates of

The check shows that it is expecting “Secondary/Diskless” as disk state. This must be somewhere defined (normally a rule). Or where the new discovered parameters not activated?

As an temporary solution, I moved all of these serveries “Connection State: Established, Roles: Secondary/Secondary, Diskstates: UpToDate/Diskless (Secondary/Diskless)” in to the “Disabled Services” for these hosts. I will investigate later this week.