Mk_ceph - I just do not understand how

XeataOne · May 29, 2023, 11:27pm

Following the problem here: Proxmox mit Ceph I am in the same position as original poster.
The plugin is in the directory but I see no output in Check_MK.
I run it as sh ./mk_ceph.txt and there is no output.

I suspect that I have not followed this instruction as I really cannot figure out what it is asking me to do:

# Check Ceph storage
# Config file must contain:
# USER=client.admin
# KEYRING=/etc/ceph/ceph.client.admin.keyring

Can someone re-word this comment in another way please?

chauhan_sudhir · May 30, 2023, 10:26am

As a first step, you can try running the commands manually as being called by the agent plugin:

github.com

Checkmk/checkmk/blob/master/agents/plugins/mk_ceph#L24


      
          # Check Ceph storage
          # Config file must contain:
          # USER=client.admin
          # KEYRING=/etc/ceph/ceph.client.admin.keyring
          
          
# shellcheck disable=SC1091 # TODO: add a configuration example and remove this suppression
          # shellcheck source=agents/cfg_examples/ceph.cfg
          . "$MK_CONFDIR/ceph.cfg" 2>/dev/null
          
          
if [ -n "$USER" ] && [ -n "$KEYRING" ]; then
              CEPH_CMD="ceph -n $USER --keyring=$KEYRING"
              echo "<<<ceph_status>>>"
              $CEPH_CMD -s -f json-pretty
              if OUT="$($CEPH_CMD df detail --format json)"; then
                  echo "<<<ceph_df_json:sep(0)>>>"
                  $CEPH_CMD version --format json
                  echo "$OUT"
              else
                  # fallback for old versions if json output is not available
                  echo "<<<ceph_df>>>"
                  $CEPH_CMD df detail

XeataOne · May 30, 2023, 12:27pm

Hi, thanks.

Background: Ceph is running great on a 3-node Proxmox Cluster and the Proxmox OMD checks are running great (all green).

I’m trying to add the CEPH plugin but am just missing some fundamentals about HOW to do so. I have copied it to the plugins directory as outlined in the README but no output happens when I refresh services for this server:

These plugins can be installed in the plugins directory of the Linux agent
in /usr/lib/check_mk_agent/plugins/. Please only install the plugins that
you really need.

The VARS are clearly incorrect:

$ echo $USER
*logged_in_username*
$ echo $KEYRING
*nothing returned*

Running the ceph command with bogus data (because I don’t know what the real data should be) returns:

ceph -n 'logged_in_username' --keyring='made_up_key"
Error initializing cluster client: Error('rados_initialize failed with error code: -22')

My question is more about the config file which:

# Check Ceph storage
# Config file must contain:
# USER=client.admin
# KEYRING=/etc/ceph/ceph.client.admin.keyring

I have two ceph.conf files:

user@server:$ find / -name ceph.conf 2>/dev/null
/usr/lib/tmpfiles.d/ceph.conf
/etc/ceph/ceph.conf
/etc/pve/ceph.conf
user@server:$

Both of which contain the same data and do not have a USER entry. Both of which do contain a KEYRING entry:

[client]
	 keyring = /etc/pve/priv/$cluster.$name.keyring

My GUESS is that I should add a “USER = client.admin” into one - or both - of these ceph.conf files. However I am unclear if client.admin is meant to be entered exactly like that, or if it refers to some other username that I ought to know about - but do not.

chauhan_sudhir · May 30, 2023, 2:49pm

What do you have under /etc/pve/priv ?
Our mk_ceph agent plugin won’t understand these variables.

I am not an expert on this topic. Maybe this User Management — Ceph Documentation helps ?

herzkerl · June 14, 2023, 10:27am

I’ve been using mk_ceph with Proxmox without any issues prior to upgrading to 2.2—it hadn’t been working since the upgrade.

Running the plugin script directly didn’t give any output. When running the command manually, I’m getting an output.

ceph -n client.admin --keyring=/etc/pve/priv/ceph.client.admin.keyring -s -f json-pretty

Just found the solution and wanted to share:
There seems to be a problem with the variable $MK_CONFDIR, see

github.com

Checkmk/checkmk/blob/master/agents/plugins/mk_ceph#L21


      
          # shellcheck disable=SC2034
          CMK_VERSION="2.3.0b1"
          
          
# Check Ceph storage
          # Config file must contain:
          # USER=client.admin
          # KEYRING=/etc/ceph/ceph.client.admin.keyring
          
          
# shellcheck disable=SC1091 # TODO: add a configuration example and remove this suppression
          # shellcheck source=agents/cfg_examples/ceph.cfg
          . "$MK_CONFDIR/ceph.cfg" 2>/dev/null
          
          
if [ -n "$USER" ] && [ -n "$KEYRING" ]; then
              CEPH_CMD="ceph -n $USER --keyring=$KEYRING"
              echo "<<<ceph_status>>>"
              $CEPH_CMD -s -f json-pretty
              if OUT="$($CEPH_CMD df detail --format json)"; then
                  echo "<<<ceph_df_json:sep(0)>>>"
                  $CEPH_CMD version --format json
                  echo "$OUT"
              else

I’ve changed that line to . "/etc/check_mk/ceph.cfg" 2>/dev/null and it works again. Maybe $MK_CONFDIR has been changed in v2.2?

chauhan_sudhir · June 14, 2023, 9:09pm

So, the Checkmk agent cannot call the mk_ceph plugin at all ?
How are you calling the agent plugin ?

robin.gierse · June 15, 2023, 6:24am

@herzkerl the environment variable $MK_CONFDIR is set by the agent on initialization, so all plugins run by the agent can use it. If you run the script manually, you need to set the variable manually first. The default value is /etc/check_mk/.

Your change just works around the missing environment variable, which is not a fix or solution.

Sara · June 15, 2023, 11:39am

There is a package from the exchange that I found.
Maybe it could be useful for you?

herzkerl · June 16, 2023, 5:07pm

I only upgraded both our server and the agents to 2.2.x—I didn’t change any other settings. Why would the $MK_CONFDIR variable work prior to the upgrade, but stop working when having upgraded?

herzkerl · June 16, 2023, 5:30pm

I installed that package a while ago, but I don’t know what to do next, to be honest. I just read the documentation again and couldn’t find an answer to it. Could you please point me in the right direction?

Sara · June 21, 2023, 1:01pm

I think we can ask @r.sander for some advice on this

r.sander · June 22, 2023, 8:53am

The extension comes with an agent plugin. You either deploy that via agent bakery or manually by copying it from $OMD_ROOT/local/share/agents/plugins to /usr/lib/check_mk_agent/plugins/60 on your Ceph nodes.

The agent plugin currently has no configuration and requires python3-rados to be installed. It also needs /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring. This will be configurable in the future.

herzkerl · July 24, 2023, 10:58pm

Good news: I got it working Thank you very much for explaining, @r.sander!

One small issue remains: When the plugin is active Checkmk warns that there are duplicate checks now.

But once I deactivate the plugin, the following Checks are stale:
• Ceph OSDs
• Ceph PGs
• Ceph Pool SUMMARY

r.sander · July 25, 2023, 6:45am

You should only have one agent plugin active, either mk_ceph or ours.

herzkerl · July 25, 2023, 7:37am

That’s what I thought, but yours lacks the three Checks I mentioned above. While I don’t think ‘Ceph Pool SUMMARY’ has any useful information, the other two do:

Ceph OSDs: Epoch rate (15 minutes 0 seconds average): 0.00, OSDs: XX, Remapped PGs: 0, OSDs out: 0, 0%, OSDs down: 0, 0%
Ceph PGs: PGs: XXX, Status 'active+clean': XXX, Status 'active+clean+scrubbing+deep': X

Is there any way to keep only those two checks? Or maybe you want to put them in your plugin with the next release…

r.sander · July 25, 2023, 7:59am

The PGs are in the “Ceph Status” check. I have to look into the code to see what the other check does.

manolo1988 · September 13, 2023, 10:30am

hi,

im not able to get it working on all 3 Nodes of my Cluster
On Node1 it is running and i get
CEPH-OSDs / PGs / Pool.mgr / Status

on the other Nodes, i get following error:

“WARNING: Parsing of section ceph_status failed - please submit a crash report! (Crash-ID: a3ae09a2-5224-11ee-9de7-92250e8ac011)”

can you guys maybee help me out?

Thanks,
BR

d0zz1 · September 28, 2023, 5:15pm

Hi,

i have the same problem. Did you find a solution?

manolo1988 · September 29, 2023, 4:58am

yes, i had to put in the keyrings to the other nodes and then it worked

d0zz1 · September 29, 2023, 8:44am

Thanks! I’m using /etc/pve/priv/ceph.client.admin.keyring now instead of /etc/ceph/ceph.client.admin.keyringand now it works for all Hosts.