Jolokia implementation

Hi

I am trying to monitor my applications that are running on a tomcat 8.0 & 8.5 (different servers)

Steps taken:

  • deployed the jolokia.war file via the tomcat manager
  • Set the user to have access to the Jolokia suburi
  • Requesting the version via the URL works

{"request":{"type":"version"},"value":{"agent":"1.7.1","protocol":"7.2","config":{"listenForHttpService":"true","maxCollectionSize":"0","authIgnoreCerts":"false","agentId":"10.0.0.157-1560-2fd540d9-servlet","agentType":"servlet","policyLocation":"classpath:\/jolokia-access.xml","agentContext":"\/jolokia","mimeType":"text\/plain","discoveryEnabled":"false","streaming":"true","historyMaxEntries":"10","allowDnsReverseLookup":"true","maxObjects":"0","debug":"false","serializeException":"false","detectorOptions":"{}","dispatcherClasses":"org.jolokia.http.Jsr160ProxyNotEnabledByDefaultAnymoreDispatcher","multicastGroup":"239.192.48.84","maxDepth":"15","authMode":"basic","authMatch":"any","canonicalNaming":"true","allowErrorDetails":"true","realm":"jolokia","includeStackTrace":"true","multicastPort":"24884","useRestrictorService":"false","debugMaxEntries":"100"},"info":{"product":"tomcat","vendor":"Apache","version":"8.0.32"}},"timestamp":1637233131,"status":200}

  • Deployed the python files to /usr/lib/check_mk_agent/plugins (mk_jolokia.py & mk_jolokia_2.py)
  • Added the jolokia.cfg file to /etc/check_mk
# Settings for authentication
 protocol = https
 server   = "10.127.0.157:8080"
 user     = "my_user"
 password = my_password
 mode     = "digest"
# use "basic_preemptive" if server responds with 403 forbidden even
# though user/password are correct
 mode     = "basic_preemptive"
suburi   = "jolokia"
instance = None
  • I select “JVM tomcat threadpool levels” and add that as a rule for a specific server.
  • I need to fill in the name of the JVM / Threadpool, butr where can I find that?
    Do I need to adjust anything else?
    Have I forgotten to do something ?

Hi,

You should use either mk_jolokia.py or mk_jolokia_2.py on one system, not both. It’s depending on you installed version of python which you should use.

As for access, you’ve created the user “my_user” in your tomcat-users.xml? On my tomcat servers I have something like this for the monitoring user:

<role rolename="jolokia"/>
<user username="monitoring" password="secret_password" roles="jolokia"/>

And in /etc/check_mk/jolokia.cfg I have:

server   = "localhost"
port     = 8080
user     = "monitoring"
password = "secret_password"
mode     = "basic_preemptive"
vars     = []
suburi   = "jolokia"
instance = None

Note that the port has it’s own line and is not specified as part of the URL. Don’t know if that’s supposed to work.

Louis

1 Like

Hi Louis

I’ve deleted the mk_jolokia.py file and adjusted the port to a seperate line.

When I re-discover the VM, I get this:

Which is fine for the first step, thanks!

But I need to monitor the apps that a running on that server and not Jolokia itself


How do I set it to monitor one specific application running on Tomcat or all applications running on that Tomcat server?

(this is obviously the DEV server, I will start monitoring the prod server when everything is stable on the DEV server)

Hi Steven,

Yes, monitoring just jolokia itself wouldn’t be very useful. It should actually discover the other applications as well. Just to make sure, these applications, they’re running on port 8080 as well, right?

Next is making sure jolokia can actually see them. In the output of:
http://<your host>:8080/jolokia/list

you should be able to find the names of your applications. Does that work?

Hi Louis


Yes, that list command does seem to work, it outputs a lot of info

Yes, all my applications are running on that port 8080 and I am able to find applications that are running on my tomcat server, in that output.

Hi Steven,

Ok, so that’s good. And also not, since CheckMK should pick it up then. Just wondering if it can be a rights issue. If, on the tomcat host, you run the check_mk agent directly, in the output there should be something like this:

<<<jolokia_metrics:sep(0)>>>
8080,/jolokia^@activeSessions^@0
8080,/jolokia^@maxActiveSessions^@-1
8080,/bavik^@activeSessions^@0
8080,/bavik^@maxActiveSessions^@-1
...

If you get that, what’s the output if you contact it from the check_mk server? Should be the same, but if not, that explains it.

Louis

Also, I just noticed, you deleted mk_jolokia.py and kept mk_jolokia_2.py. If you have python3 (a bit recent machine should have this), you should have kept mk_jolokia.py, which is for python 3.

Hi Louis

There is no output like the one you posted when I run “check_mk_agent”. Only the default which has worked since it was set up.

(We are running phyton version 2.7.12)

Ah, ok, no worries. What’s the output if you run /usr/lib/check_mk_agent/plugins/mk_jolokia_2.py directly from the command line? That should give you the right output. If not, hopefully it produces an errormessage that’ll help fix this issue.

username@server:/usr/lib/check_mk_agent/plugins$ python mk_jolokia_2.py
<<<jolokia_info>>>
Error: mk_jolokia requires the requests library. Please install it on the monitored system.

Ok, so I understand it needs the requests library.
Phython’s library? Jolokia’s Library?

Far as I can see on my OS (SuSE) there’s a package python2-requests. My best guess would be for you to install the equivalent for your distribution.

1 Like

Hi Louis, thanks for the continued help, I appreciate it, I installed the requests module but then I run into an error message when I try to execute the phyton script again

user@server:/usr/lib/check_mk_agent/plugins$ pip install requests
Collecting requests
Downloading https://files.pythonhosted.org/packages/92/96/144f70b972a9c0eabbd4391ef93ccd49d0f2747f4f6a2a2738e99e5adc65/requests-2.26.0-py2.py3-none-any.whl (62kB)
100% |████████████████████████████████| 71kB 6.1MB/s
Collecting certifi>=2017.4.17 (from requests)
Downloading https://files.pythonhosted.org/packages/37/45/946c02767aabb873146011e665728b680884cd8fe70dde973c640e45b775/certifi-2021.10.8-py2.py3-none-any.whl (149kB)
100% |████████████████████████████████| 153kB 4.0MB/s
Collecting urllib3<1.27,>=1.21.1 (from requests)
Downloading https://files.pythonhosted.org/packages/af/f4/524415c0744552cce7d8bf3669af78e8a069514405ea4fcbd0cc44733744/urllib3-1.26.7-py2.py3-none-any.whl (138kB)
100% |████████████████████████████████| 143kB 7.4MB/s
Collecting idna<3,>=2.5; python_version < “3” (from requests)
Downloading https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl (58kB)
100% |████████████████████████████████| 61kB 8.3MB/s
Collecting chardet<5,>=3.0.2; python_version < “3” (from requests)
Downloading https://files.pythonhosted.org/packages/19/c7/fa589626997dd07bd87d9269342ccb74b1720384a4d739a1872bd84fbe68/chardet-4.0.0-py2.py3-none-any.whl (178kB)
100% |████████████████████████████████| 184kB 5.7MB/s
Installing collected packages: certifi, urllib3, idna, chardet, requests
Successfully installed certifi chardet idna requests urllib3
You are using pip version 8.1.1, however version 21.3.1 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
user@server:/usr/lib/check_mk_agent/plugins$ python mk_jolokia_2.py
Traceback (most recent call last):
File “mk_jolokia_2.py”, line 623, in
main()
File “mk_jolokia_2.py”, line 614, in main
for config in configs_iterable:
File “mk_jolokia_2.py”, line 587, in yield_configured_instances
custom_config = load_config(custom_config)
File “mk_jolokia_2.py”, line 606, in load_config
exec(open(conffile).read(), {}, custom_config)
File “”, line 9
protocol = http
^
IndentationError: unexpected indent
user@server:/usr/lib/check_mk_agent/plugins$

Looks like an error in the plugin code. I’m not sure. It’s getting a bit out of my league now. But what I do know about python is that it’s very picky about indentation.

Mabe if you paste the content of mk_jolokia_2.py here it may show some hints on how to correct it:

Hey Louis

Can you show me your jolokia.cfg file, it seems to have an issue in that file.
Please remove all indentifying lines and passwords.

Thank you.

Hey Louis

I’ve got some good news, it seemed to be the bad formating of the jolokia.cfg file that was hindering the execution of the file. (there where TABS that shouldn’t have been there [not added by me, but where in the file already])
Now it executes without issue.

Now I get that output that you referenced earlier:

<<<jolokia_info:sep(0)>>>
8080tomcat8.0.321.7.1
<<<jolokia_metrics:sep(0)>>>
8080,/xconnect-gnvactiveSessions0
8080,/xconnect-gnvmaxActiveSessions-1
8080,/devactiveSessions0
8080,/devmaxActiveSessions-1

Now, I did a re-discover of the VM but it doesn’t seem to have picked anything up yet.
How do I get this output, into CheckMK ?

Hi Steven,

Hmmm, glad you got that working at least. It should pick up the services automatically then. So, is this the output from running mk_jolokia_2.py directly or from calling the agent? If from the first, do you see the jolokia info if you call the agent from the CheckMK server? Something like:

nc -v <tomcat server> 6556

That should have the jolokia section. If not, it’s probably a rights issue.

Louis

Hey Louis

I set the rights to the python file to 777 and locally I can execute it now.
When I do a netcat as you suggested, I still get this output:

<<local:sep(0)>>
<<<jolokia_info>>>
Error: mk_jolokia requires the requests library. Please install it on the monitored system.

Oh, that’s why you don’t see it on the CheckMK server. It’s not getting the data. I’ve been in similar situations before and typcially it means that the environment settings in your interactive shell when you log in on the tomcat server are not the same. Most of the time that means some PATH or Library environment is not set when calling the agent over the network

Now, I’m no python expert (far from) but I guess you’ll need to find out what environmet settings you have and find a way to have those available when calling the agent via the network. Maybe adding something to /etc/bash.bashrc.local can do the trick.

Good luck,
Louis

Hi Louis

Seemed I need to execute ‘sudo -H pip install requests’ and now it works from the check_MK server as well.

Woow, and now it auto populates in the checkMK web frontend!
It detected 137 extra services!

Finally, we got there, thanks again for all the help!

Hi Steven,

Glad to hear you got it working in the end.

Happy monitoring
Louis