Percona mysql monitoring problems

Hi,

I have a Percona mysql installation on RedHat Linux that I monitor with checkmk mysql plugin.
I had checkmk 1.6.0p19 and it worked.
But I upgraded to 2.0.0p5 and also the agent with it’s mysql plugin and then suddenly checkmk won’t show any findings related to mysql.

When I run the command “check_mk_agent” on the db server, I get some output showing info for
<<<mysql_ping>>>
<<>>
<<<mysql_capacity>>>
<<<mysql_slave>>>

But that info doesn’t show on the host on the checkmk server.
The mysql.cfg contains the follwing info
[client]
user=root
password=the-password
socket=/opt/mysql/admin/pid/mysql.sock

I had to put socket, since the percona installation has the config file in a non standard location.

So I don’t know what to do or where to look for errors.
I would be very grateful for any help in pointing me to logs or even “solving” the issue?

Thanks

ricardo

Hi,
I tried to revert back to agent 1.6.0p19, but now it doesn’t work either.
The output local from the agent gives some line with mysql sections. But on the checkmk server it doesn’t show upp anything related to the mysql plugin.

I’m totally out of ideas, so I would appreciate any help or tips to point me in the right direction. Either to look into any logs or any config tips.

Thanks in advance,

ricardo

Hi,

Any tips, anyone?
I would really like some help to fix this. Right now, we are not able to monitor mysql related metrics on this host.

Thanks
ricardo

Hi @ricardof

A successful agent output on one of my hosts (skipped multiple lines, intentionally)
looks something like this:
<<<mysql_ping>>>
mysqld is alive
<<<mysql>>>
Aborted_clients 0
Aborted_connects 0
Acl_cache_items_count 0
...

This is from a 1.5.x host/agent.

I may be misunderstanding your first post, but: If you don’t get any output, and only the lines, I’d assume
that the DB connection is not established. BTW, I don’t have to add a “socket” line
in /etc/check_mk/mysql.cfg, although my socket is here: /opt/mysql/data/db/mysql.sock

The plugin works with username and password, only. Other plugins (percona mysql nagios plugins) do need this information, I believe the mk_mysql plugin, does not. Can you try to remove
that line with “socket” from the config and retry?

HTH,
Thomas

Hi Thomas,

I get multiple lines in the agent output locally on the server. And at the end several <<>> related sections appear.
But nothing show up in the checkmk server.
So obviously the connection to the db works.

I had version 1.6.0p19 and it was working. I upgraded the server to 2.0.xxxx and also this agent and then it didn’t work.
I reverted back the agent to the working one, but no mysql plugin info in checkmk server GUI. So there I’m stuck.
Regarding the connection to the db and the mysql.cfg, I have tried the suggestions you propose but got no luck. Only by adding the socket line, I was able to have a connection to the db.
So this is not the problem. I know that this percona install was complied from source and not installed from a package so the default locations are not the expected ones. Therefore I needed the socket info in config file.

The only problem is that agent output has the info but it never reaches the checkmk server.
And I’ve ran out of ideas and opitons.

Thanks again,

ricardo

Hi @ricardof

Hmm… OK, and if you “telnet” or “ncat” from your CMK host towards your target host on port 6556?
Do you see the correct agent output?

If you see the full output you’re expecting on the server, what happens if you create a new host
in CMK? Does it “get” all services?

Thomas

Hi Thomas

Yes, I’ve tried telnet to 6556 but no mysql info. Only the basic agent output.
When I recreate the host, it shows only the services I previously monitored. And the disabled ones are still disabled but no mysql services

ricardo

Hi Ricardo,

That’s… strange. So you say you do have a full and correct mysql section when running
check_mk_agent on the host you want to monitor, but when you telnet to it, you
don’t get the mysql section? Did I understand that correctly?

Theoretically the telnet output should be the same (i.e. include the
mysql section) as the locally executed agent, unless I’m currently missing something
vital… You could also - just to exclude any possible misconfiguration of the mk_mysql plugin -
verify that it works as expected by troubleshooting it according to those instructions: Monitoring MySQL - 5.1. Diagnostic options under Linux.

Aside from the fact that - in my opinion - this shouldn’t be happening, let’s try something else, too: Can you attempt to clear the cache of this host, if it exists in OMD[<SITENAME>]:~/tmp/check_mk/cache?

Thomas

Hey Ricardo,

Here’s another thought: Are you sure, that the agent you’re contacting remotely over port tcp/6556, is the same one you execute locally? What I mean is: Can it be that you have somehow, more than one
agent installed, and you accidentally execute the “wrong” one when you telnet to your target host
on port tcp/6556?

If you try contacting the agent via ssh, do you have the same behaviour, or do you “get” the mysql section?

Thomas

Hi,
I see.
But how can I check on the server if 2 agents are installed. I have only used the RPM package to install the agent. And when I check which rpm packages that are installed, I get only 1. Then I uninstall that and all the checkmk folders are deleted.

Regarding the ssh option from checkmk server, I cannot do that since the agent server is a the customer’s place and we have a vpn tunnel betwween the checkmk server and the customers server with only traffic allowed on port 6556. So doing a check via ssh will fail.

Thanks for the input. Really good ideas!

ricardo

Hi Ricardo,

Well since one can install the agent by simply copying the shell script somewhere, you could
end up running multiple agents, this way.

What I don’t understand is, that you say you don’t have ssh, but at the same time you can connect to the server in question to run the agent locally and get a mysql output. You also apparently can check/remove an RPM, so you must have some kind of access to this machine, right?
How do you access this host to administer it?

  • Can you send the output of the agent being executed locally, and the output of the agent
    being contacted remotely?
  • Can you check, whether you run the agent remotely via xinetd or a systemd socket file?
  • Depending on the method you’re “exposing” the agent, can you check on which port its supposed to be reachable?

Thomas

Ok,

First let me explain the ssh and not ssh access.

The monitoring server has only access to the agent via 6556.
But for me to access the server itself via ssh, I need to have an vpn client connected to the customer’s network, and then I can connect through ssh from my computer.
That’s why I cannot run the checkmk ssh connection.

I haven’t copied any shell script of the agent. I’ve only installed the rpm packages every time.
It all started when I upgraded to v 2.0 of the agent and then couldn’t go back and make it work.

Let me get back regarding the outputs of your suggestions

ricardo