Filesystems like /hedvig/d13 are not discovered with df check but with mount check

Troubleshooting
, ,
1

I’m trying to monitor a Commvault Hyperscale-X appliance wich is based on RHEL. All is good, but I’m unable to add mounted filesystems like /hedvig/d… they are just not displayed after discovery. This is some kind of software defined storage solution, but in the end all those d* disks are mounted xfs filesystems.

The funny thing is that the df module does not show them, but the mount module does. So I’m able to check the mount options but not the size. Other fs are discovered as expected.

Example:

/dev/mapper/hedvigmetavg-hedvigd2         300G   95G  205G  32% /hedvig/d2
/dev/sdf                                   17T   16T  843G  95% /hedvig/d6
/dev/sdn                                   17T   16T  891G  95% /hedvig/d14
/dev/sds                                   17T   16T  914G  95% /hedvig/d19
/dev/sdj                                   17T   16T  838G  96% /hedvig/d10
/dev/sdi                                   17T   16T  844G  95% /hedvig/d9
/dev/sdd                                   17T   16T  885G  95% /hedvig/d4
/dev/sdh                                   17T   16T  859G  95% /hedvig/d8
/dev/sdy                                   17T   16T  946G  95% /hedvig/d25
/dev/sdt                                   17T   16T  910G  95% /hedvig/d20
/dev/sdw                                   17T   16T  914G  95% /hedvig/d23
/dev/sdz                                   17T   16T  983G  95% /hedvig/d26
/dev/sdaa                                  17T   16T  872G  95% /hedvig/d3
/dev/sdx                                   17T   16T 1017G  94% /hedvig/d24
/dev/sdu                                   17T   16T  970G  95% /hedvig/d21
/dev/sdq                                   17T   16T  994G  95% /hedvig/d17
/dev/sdm                                   17T   16T  933G  95% /hedvig/d13
/dev/sdk                                   17T   16T  1.1T  94% /hedvig/d11
/dev/sdv                                   17T   16T  948G  95% /hedvig/d22
/dev/sdp                                   17T   16T  968G  95% /hedvig/d16
/dev/sdo                                   17T   16T  987G  95% /hedvig/d15
/dev/sdl                                   17T   16T  1.1T  94% /hedvig/d12
/dev/sdr                                   17T   16T  1.1T  94% /hedvig/d18
/dev/sde                                   17T   16T  839G  96% /hedvig/d5
/dev/sdg                                   17T   16T  838G  96% /hedvig/d7

Environment:

**CMK version:**  2.1.0p1
**OS version:** Server Debian 11.4, Clients unknown RHEL version

**Error message:** none

**Output of “cmk --debug -vvn hostname”:** (If it is a problem with checks or plugins)


Checkmk version 2.1.0p1
Try license usage history update.
Trying to acquire lock on /omd/sites/backup_team/var/check_mk/license_usage/next_run
Got lock on /omd/sites/backup_team/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/backup_team/var/check_mk/license_usage/history.json
Got lock on /omd/sites/backup_team/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/backup_team/var/check_mk/license_usage/history.json
Released lock on /omd/sites/backup_team/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/backup_team/var/check_mk/license_usage/next_run
Released lock on /omd/sites/backup_team/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7f56f0489040]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(sdes1601-dp.xxx.xxx, base_path=/omd/sites/backup_team/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 46 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 10.24.56.221:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Reading data from agent
Write data to cache file /omd/sites/backup_team/tmp/check_mk/cache/sdes1601-dp.xxx.xxx
Trying to acquire lock on /omd/sites/backup_team/tmp/check_mk/cache/sdes1601-dp.xxx.xxx
Got lock on /omd/sites/backup_team/tmp/check_mk/cache/sdes1601-dp.xxx.xxx
Releasing lock on /omd/sites/backup_team/tmp/check_mk/cache/sdes1601-dp.xxx.xxx
Released lock on /omd/sites/backup_team/tmp/check_mk/cache/sdes1601-dp.xxx.xxx
Closing TCP connection to 10.24.56.221:6556
[cpu_tracking] Stop [7f56f0489040 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=2.2200000025331974))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7f56f0489430]
[PiggybackFetcher] Fetch with cache settings: NoCache(sdes1601-dp.xxx.xxx, base_path=/omd/sites/backup_team/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for 'sdes1601-dp.xxx.xxx'. Skip processing.
No piggyback files for '10.24.56.221'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7f56f0489430 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_bonding:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<ovs_bonding:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<multipath>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1668015688,30)>>> / Transition HostSectionParser -> HostSectionParser
<<<ipmi:cached(1668015697,300):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<ipmi_discrete:cached(1668015690,300):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<logwatch>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'chrony', 'cifsmounts', 'cmk_agent_ctl_status', 'cpu', 'df', 'diskstat', 'ipmi', 'ipmi_discrete', 'job', 'kernel', 'labels', 'lnx_bonding', 'lnx_if', 'local', 'logwatch', 'md', 'mem', 'mounts', 'multipath', 'nfsmounts', 'ovs_bonding', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
  -> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7f56f03cb610]
value store: synchronizing
Trying to acquire lock on /omd/sites/backup_team/tmp/check_mk/counters/sdes1601-dp.xxx.xxx
Got lock on /omd/sites/backup_team/tmp/check_mk/counters/sdes1601-dp.xxx.xxx
value store: loading from disk
Releasing lock on /omd/sites/backup_team/tmp/check_mk/counters/sdes1601-dp.xxx.xxx
Released lock on /omd/sites/backup_team/tmp/check_mk/counters/sdes1601-dp.xxx.xxx
Bonding Interface bond1 Status: up, Mode: IEEE 802.3ad Dynamic link aggregation, enpsf3/88:E9:A4:5E:C4:44 up, enpsf5/88:E9:A4:5E:E4:44 up
Bonding Interface bond2 Status: up, Mode: IEEE 802.3ad Dynamic link aggregation, enpsf4/88:E9:A4:5E:C4:45 up, enpsf6/88:E9:A4:5E:E4:45 up
CPU load             15 min load: 69.60, 15 min load per core: 1.09 (64 cores)
CPU utilization      Total CPU: 45.23%
Check_MK Agent       Version: 2.1.0p1, OS: linux, Agent plugins: 2, Local checks: 0
Disk IO SUMMARY      Read: 501 MB/s, Write: 101 MB/s, Latency: 6 milliseconds
Filesystem /         15.47% used (20.65 of 133.51 GB), trend: +478.92 kB / 24 hours
Filesystem /boot     15.78% used (149.35 of 946.73 MB), trend: 0.00 B / 24 hours
Filesystem /boot/efi 0.38% used (3.62 of 951.12 MB), trend: 0.00 B / 24 hours
Filesystem /flache/metadatadir 0.11% used (32.97 MB of 29.79 GB), trend: 0.00 B / 24 hours
Filesystem /hedvig/d2 30.2% used (90.56 of 299.85 GB), trend: +23.31 GB / 24 hours
Filesystem /hedvig/hpod/data 0.15% used (45.54 MB of 29.79 GB), trend: +1.15 MB / 24 hours
Filesystem /hedvig/hpod/log 0.28% used (85.58 MB of 29.79 GB), trend: +14.21 MB / 24 hours
Filesystem /mnt/d2   0.75% used (457.88 MB of 59.58 GB), trend: +27.68 MB / 24 hours
Filesystem /mnt/d3   14.49% used (116.53 of 804.43 GB), trend: +3.17 GB / 24 hours
Filesystem /mnt/d4   14.33% used (115.29 of 804.43 GB), trend: +2.27 GB / 24 hours
Filesystem /mnt/d5   14.14% used (113.75 of 804.43 GB), trend: +1.28 GB / 24 hours
Filesystem /mnt/f1   0.11% used (34.06 MB of 29.79 GB), trend: -12.79 kB / 24 hours
Filesystem /opt      8.12% used (14.45 of 178.01 GB), trend: +391.58 kB / 24 hours
Filesystem /opt/commvault/MediaAgent64/IndexCache 3.01% used (80.77 GB of 2.62 TB), trend: -1.19 GB / 24 hours
Filesystem /var      34.81% used (7.74 of 22.25 GB), trend: -114.17 MB / 24 hours
Filesystem /var/log  36.63% used (24.45 of 66.75 GB), trend: +5.81 GB / 24 hours
Filesystem /ws/ddb   26.46% used (788.26 GB of 2.91 TB), trend: +9.08 GB / 24 hours
IPMI Sensor Summary  141 sensors - 122 OK - 19 skipped
Interface 02         [bond1], (up), MAC: 88:E9:A4:5E:C4:44, Speed: 50 GBit/s, In: 20.9 MB/s (0.33%), Out: 46.2 MB/s (0.74%)
Interface 03         [bond2], (up), MAC: 88:E9:A4:5E:C4:45, Speed: 50 GBit/s, In: 246 MB/s (3.94%), Out: 231 MB/s (3.70%)
Interface 04         [bond1.180], (up), MAC: 88:E9:A4:5E:C4:44, Speed: 50 GBit/s, In: 20.0 MB/s (0.32%), Out: 44.7 MB/s (0.72%)
Interface 05         [bond1.400], (up), MAC: 88:E9:A4:5E:C4:44, Speed: 50 GBit/s, In: 231 B/s (<0.01%), Out: 12.1 B/s (<0.01%)
Interface 08         [enpsf3], (up), MAC: 88:E9:A4:5E:C4:44, Speed: 25 GBit/s, In: 12.9 MB/s (0.41%), Out: 33.4 MB/s (1.07%)
Interface 09         [enpsf4], (up), MAC: 88:E9:A4:5E:C4:45, Speed: 25 GBit/s, In: 88.9 MB/s (2.85%), Out: 5.06 B/s (<0.01%)
Interface 10         [enpsf5], (up), MAC: 88:E9:A4:5E:E4:44, Speed: 25 GBit/s, In: 8.01 MB/s (0.26%), Out: 12.7 MB/s (0.41%)
Interface 11         [enpsf6], (up), MAC: 88:E9:A4:5E:E4:45, Speed: 25 GBit/s, In: 157 MB/s (5.04%), Out: 231 MB/s (7.40%)
Kernel Performance   Process Creations: 12.18/s, Context Switches: 527929.33/s, Major Page Faults: 18.32/s, Page Swap in: 18.75/s, Page Swap Out: 19.40/s
Log /var/log/messages No error messages
Memory               Total virtual memory: 68.98% - 377.91 GB of 547.85 GB
NTP Time             Offset: 0.0357 ms, Stratum: 2, Time since last sync: 3 minutes 33 seconds
Systemd Service Summary Total: 181, Disabled: 22, Failed: 2
TCP Connections      Established: 3237
Uptime               Up since May 23 2022 13:31:55, Uptime: 170 days 6 hours
No piggyback files for 'sdes1601-dp.xxx.xxx'. Skip processing.
No piggyback files for '10.24.56.221'. Skip processing.
[cpu_tracking] Stop [7f56f03cb610 - Snapshot(process=posix.times_result(user=0.16000000000000014, system=0.009999999999999981, children_user=0.0, children_system=0.0, elapsed=0.16999999806284904))]
[agent] Success, execution time 2.4 sec | execution_time=2.390 user_time=0.160 system_time=0.010 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=2.220
2

Hi,

Are you running the CheckMK agent with root privileges? If not it can happen that df does not have sufficient rights to show all filesystems.

Regards,
Louis

3

It’s run from xinetd and has “user = root” in config. Even as non root user df shows the mounted filesystems on that servers. And mount command/module is also showing them.

4

Anything else I can do to debug?

5

What happens if you run (as user root) the CheckMK agent directly on the client itself? Do you see the output of the df command then?

6

Looks ok to me.

<<<check_mk>>>
Version: 2.1.0p1
AgentOS: linux
..
<<<df>>>
tmpfs                                   tmpfs   263887420           0  263887420       0% /dev/shm
tmpfs                                   tmpfs   263887420      190796  263696624       1% /run
tmpfs                                   tmpfs   263887420           0  263887420       0% /sys/fs/cgroup
/dev/mapper/raidvg-root                 xfs     139990236    18639860  121350376      14% /
/dev/mapper/hedvigmetavg-mntd2          xfs      62478536      420084   62058452       1% /mnt/d2
/dev/mapper/metadatavg-ddb              xfs    3124082272   849672324 2274409948      28% /ws/ddb
/dev/mapper/hedvigmetavg-mntd4          xfs     843503196   170286484  673216712      21% /mnt/d4
/dev/mapper/hedvigmetavg-hedvigd2       xfs     314419200    39659548  274759652      13% /hedvig/d2
/dev/mapper/raidvg-var                  xfs      23331708     8932448   14399260      39% /var
/dev/mapper/hedvigmetavg-mntd3          xfs     843503196   170290836  673212360      21% /mnt/d3
/dev/mapper/hedvigmetavg-hedvighpodlog  xfs      31237220       60132   31177088       1% /hedvig/hpod/log
/dev/mapper/hedvigmetavg-flachemetadata xfs      31237220       33760   31203460       1% /flache/metadatadir
/dev/mapper/hedvigmetavg-mntd5          xfs     843503196   170443700  673059496      21% /mnt/d5
/dev/mapper/hedvigmetavg-mntf1          xfs      31237220       35464   31201756       1% /mnt/f1
/dev/mapper/raidvg-var_log              xfs      69995120    14579828   55415292      21% /var/log
/dev/mapper/hedvigmetavg-hedvighpoddata xfs      31237220       43124   31194096       1% /hedvig/hpod/data
/dev/mapper/raidvg-opt                  xfs     186653648    17623380  169030268      10% /opt
/dev/sdc2                               xfs        969452      152952     816500      16% /boot
/dev/sdc1                               vfat       973952        3712     970240       1% /boot/efi
/dev/mapper/metadatavg-index            xfs    2811673228   132659500 2679013728       5% /opt/commvault/MediaAgent64/IndexCache
/dev/sdaa                               xfs   17576241152 13433205880 4143035272      77% /hedvig/d3
/dev/sdo                                xfs   17576241152 13388719092 4187522060      77% /hedvig/d15
/dev/sdx                                xfs   17576241152 13338973784 4237267368      76% /hedvig/d24
/dev/sdr                                xfs   17576241152 13533033976 4043207176      77% /hedvig/d18
/dev/sdl                                xfs   17576241152 13405761704 4170479448      77% /hedvig/d12
/dev/sdk                                xfs   17576241152 13257611020 4318630132      76% /hedvig/d11
/dev/sdh                                xfs   17576241152 13375371516 4200869636      77% /hedvig/d8
/dev/sdf                                xfs   17576241152 13449866084 4126375068      77% /hedvig/d6
/dev/sdi                                xfs   17576241152 13364733552 4211507600      77% /hedvig/d9
/dev/sdj                                xfs   17576241152 13351422920 4224818232      76% /hedvig/d10
/dev/sde                                xfs   17576241152 13405034676 4171206476      77% /hedvig/d5
/dev/sdd                                xfs   17576241152 13487436400 4088804752      77% /hedvig/d4
/dev/sdn                                xfs   17576241152 13367518624 4208722528      77% /hedvig/d14
/dev/sdg                                xfs   17576241152 13406959288 4169281864      77% /hedvig/d7
/dev/sdz                                xfs   17576241152 13309181280 4267059872      76% /hedvig/d26
/dev/sdp                                xfs   17576241152 13333215564 4243025588      76% /hedvig/d16
/dev/sdt                                xfs   17576241152 13324565028 4251676124      76% /hedvig/d20
/dev/sdw                                xfs   17576241152 13349159016 4227082136      76% /hedvig/d23
/dev/sds                                xfs   17576241152 13391461296 4184779856      77% /hedvig/d19
/dev/sdv                                xfs   17576241152 13356981764 4219259388      76% /hedvig/d22
/dev/sdm                                xfs   17576241152 13341724684 4234516468      76% /hedvig/d13
/dev/sdy                                xfs   17576241152 13333793508 4242447644      76% /hedvig/d25
/dev/sdq                                xfs   17576241152 13496156476 4080084676      77% /hedvig/d17
/dev/sdu                                xfs   17576241152 13378713100 4197528052      77% /hedvig/d21
tmpfs                                   tmpfs    52777488           0   52777488       0% /run/user/0
7

Yes, it does. So, for now we’ve confirmed that the agent at least does what it’s supposed to do. Next thing is to see what arrives at your CheckMK server. So from the CheckMK server try something like this:
nc -v <client IP> 6556

Then check the output. If the df data is in there, then try something like this as the site user:
cmk -d <clientname>

Where <clientname> is the name of your client as it’s know in CheckMK. Again see if the df output is shown.

8

Seems not everything is returned from agent? Here’s the nc output.

<<<df>>>
tmpfs                                   tmpfs   263887420         0  263887420       0% /dev/shm
tmpfs                                   tmpfs   263887420    207040  263680380       1% /run
tmpfs                                   tmpfs   263887420         0  263887420       0% /sys/fs/cgroup
/dev/mapper/raidvg-root                 xfs     139990236  12546332  127443904       9% /
/dev/mapper/hedvigmetavg-mntd3          xfs     843503196 170395928  673107268      21% /mnt/d3
/dev/mapper/metadatavg-ddb              xfs    3124082272 838544340 2285537932      27% /ws/ddb
/dev/mapper/hedvigmetavg-flachemetadata xfs      31237220     33760   31203460       1% /flache/metadatadir
/dev/mapper/raidvg-var                  xfs      23331708   8582524   14749184      37% /var
/dev/mapper/hedvigmetavg-hedvigd2       xfs     314419200  32735192  281684008      11% /hedvig/d2
/dev/mapper/hedvigmetavg-mntd2          xfs      62478536    418420   62060116       1% /mnt/d2
/dev/mapper/hedvigmetavg-mntf1          xfs      31237220     34876   31202344       1% /mnt/f1
/dev/mapper/hedvigmetavg-hedvighpodlog  xfs      31237220     58252   31178968       1% /hedvig/hpod/log
/dev/mapper/hedvigmetavg-mntd5          xfs     843503196 170391700  673111496      21% /mnt/d5
/dev/mapper/hedvigmetavg-mntd4          xfs     843503196 170402008  673101188      21% /mnt/d4
/dev/mapper/hedvigmetavg-hedvighpoddata xfs      31237220     42540   31194680       1% /hedvig/hpod/data
/dev/mapper/raidvg-var_log              xfs      69995120  15067732   54927388      22% /var/log
/dev/sdc2                               xfs        969452    152928     816524      16% /boot
/dev/sdc1                               vfat       973952      3712     970240       1% /boot/efi
/dev/mapper/raidvg-opt                  xfs     186653648  14371492  172282156       8% /opt
/dev/mapper/metadatavg-index            xfs    2811673228  36598332 2775074896       2% /opt/commvault/MediaAgent64/IndexCache
<<<df>>>

As well as the agent run as site user


<<<df>>>
tmpfs                                   tmpfs   263887420         0  263887420       0% /dev/shm
tmpfs                                   tmpfs   263887420   4229416  259658004       2% /run
tmpfs                                   tmpfs   263887420         0  263887420       0% /sys/fs/cgroup
/dev/mapper/raidvg-root                 xfs     139990236  21657500  118332736      16% /
/dev/mapper/hedvigmetavg-hedvigd2       xfs     314419200  46909748  267509452      15% /hedvig/d2
/dev/mapper/hedvigmetavg-mntd4          xfs     843503196 125003144  718500052      15% /mnt/d4
/dev/mapper/hedvigmetavg-mntd3          xfs     843503196 125020000  718483196      15% /mnt/d3
/dev/mapper/hedvigmetavg-mntf1          xfs      31237220     34876   31202344       1% /mnt/f1
/dev/mapper/hedvigmetavg-hedvighpodlog  xfs      31237220     75916   31161304       1% /hedvig/hpod/log
/dev/mapper/hedvigmetavg-hedvighpoddata xfs      31237220     56220   31181000       1% /hedvig/hpod/data
/dev/mapper/hedvigmetavg-flachemetadata xfs      31237220     33760   31203460       1% /flache/metadatadir
/dev/mapper/metadatavg-ddb              xfs    3124082272 813859236 2310223036      27% /ws/ddb
/dev/mapper/hedvigmetavg-mntd5          xfs     843503196 124631976  718871220      15% /mnt/d5
/dev/mapper/hedvigmetavg-mntd2          xfs      62478536    423624   62054912       1% /mnt/d2
/dev/mapper/raidvg-var                  xfs      23331708   8150504   15181204      35% /var
/dev/mapper/raidvg-opt                  xfs     186653648  15152476  171501172       9% /opt
/dev/sdc2                               xfs        969452    152932     816520      16% /boot
/dev/mapper/metadatavg-index            xfs    2811673228  83064064 2728609164       3% /opt/commvault/MediaAgent64/IndexCache
/dev/sdc1                               vfat       973952      3712     970240       1% /boot/efi
/dev/mapper/raidvg-var_log              xfs      69995120  12353632   57641488      18% /var/log
tmpfs                                   tmpfs    52777484         0   52777484       0% /run/user/0
<<<df>>>
[df_inodes_start]
tmpfs                                   tmpfs  65971855      1  65971854    1% /dev/shm
tmpfs                                   tmpfs  65971855   1794  65970061    1% /run
tmpfs                                   tmpfs  65971855     16  65971839    1% /sys/fs/cgroup
/dev/mapper/raidvg-root                 xfs    70029312 285507  69743805    1% /
/dev/mapper/hedvigmetavg-hedvigd2       xfs   157286400     81 157286319    1% /hedvig/d2
/dev/mapper/hedvigmetavg-mntd4          xfs   421957632   5201 421952431    1% /mnt/d4
/dev/mapper/hedvigmetavg-mntd3          xfs   421957632   3751 421953881    1% /mnt/d3
/dev/mapper/hedvigmetavg-mntf1          xfs    15626240     11  15626229    1% /mnt/f1
/dev/mapper/hedvigmetavg-hedvighpodlog  xfs    15626240     12  15626228    1% /hedvig/hpod/log
/dev/mapper/hedvigmetavg-hedvighpoddata xfs    15626240     16  15626224    1% /hedvig/hpod/data
/dev/mapper/hedvigmetavg-flachemetadata xfs    15626240      3  15626237    1% /flache/metadatadir
/dev/mapper/metadatavg-ddb              xfs   312560832 212744 312348088    1% /ws/ddb
/dev/mapper/hedvigmetavg-mntd5          xfs   421957632   5719 421951913    1% /mnt/d5
/dev/mapper/hedvigmetavg-mntd2          xfs    31254528     32  31254496    1% /mnt/d2
/dev/mapper/raidvg-var                  xfs    11671552  14602  11656950    1% /var
/dev/mapper/raidvg-opt                  xfs    93372416  94978  93277438    1% /opt
/dev/sdc2                               xfs      487936    375    487561    1% /boot
/dev/mapper/metadatavg-index            xfs   281304640  43863 281260777    1% /opt/commvault/MediaAgent64/IndexCache
/dev/sdc1                               vfat          0      0         0     - /boot/efi
/dev/mapper/raidvg-var_log              xfs    35014656   2769  35011887    1% /var/log
tmpfs                                   tmpfs  65971855      1  65971854    1% /run/user/0
[df_inodes_end]
[df_lsblk_start]
[df_lsblk_end]
9

Hmmm, running out of ideas here. Have you tried compeltely rescanning the host by doing “Remove all and find new”? From what I can see your output is correct so yes, the filesystem data should be in your services.
image

10

Just did this, but no change. But why is the agent locally returning the fs but not over network? Anything regarding rules to check?

11

Ah, sorry, misread the previous post. So over the network there’s an issue. Which explains why you’re not seeing your results in CheckMK. Ok, so let’s check if there may be a permission issue after all. First, let’s see if xinetd is running with the proper permissions:
ps axuwww | grep xinetd

Should return something like this:
root 1488 0.0 0.2 49972 4560 ? Ss 09:34 0:00 /usr/sbin/xinetd -stayalive -dontfork

Then the contents of /etc/xinetd.d/check_mk, should read something like this:

service check_mk
{
        type           = UNLISTED
        port           = 6556
        socket_type    = stream
        protocol       = tcp
        wait           = no
        user           = root
        server         = /usr/bin/check_mk_agent

        disable        = no
}

I’ve stripped out comments and options that are not active.

Also, I noticed that you’re still using an old version of the agent: 2.1.0p1, while 2.1.0p16 is available. It’s a long shot but you could try upgrading it.

12

First thanks for your step by step support! I’ve updated today to latest version 2.1.0p16. Is it possible that selinux is interfering? But then why only for some fs information…

root      3723  0.0  0.0  27168   964 ?        Ss   Nov10   0:20 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid

service check-mk-agent
{
    type           = UNLISTED
    port           = 6556
    socket_type    = stream
    protocol       = tcp
    wait           = no
    user           = root

    server         = /usr/bin/check_mk_agent

    per_source     = 3

    log_on_success =

    disable        = no
}
13

Looks good, so that’s not the issue. You may indeed want to try to set selinux to permissive mode for testing. I noticed that all the devices that are missing are “whole” disks, so /dev/sdf, /dev/sdg, etc. The ones showing are either partitions like /dev/sdc1 or LVM volumes.

Maybe selinux is blocking something there? I guess it’s worth a shot.

1 Like
14

And indeed its selinux… I’ll have to talk to vendor how to proceed, thanks for all the help.

<<df>>>
..
/dev/sdp xfs 17576241152 13870449836 3705791316 79% /hedvig/d16
/dev/sdv xfs 17576241152 34820 17576206332 1% /hedvig/d22
/dev/sdj xfs 17576241152 13862328112 3713913040 79% /hedvig/d10
/dev/sdt xfs 17576241152 13158073536 4418167616 75% /hedvig/d20
/dev/sdm xfs 17576241152 13227296828 4348944324 76% /hedvig/d13
...
15

Ah glad to hear that we’ve at least found the cause of the issue. I don’t know too much about selinux, but I’m afraid you may end up have to write some security context thingies to get this working.

Then again, since it’s an appliance, I hope the vendor’s willing to help you out.

Good luck,
Louis

16

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.