Prometheus with custom URL

[openmindz (Thomas Karatzoglou)]

Hello everyone,

I’m trying to set up Prometheus integration on a CRE in version 2.0.0p17 as per the “Prometheus integrieren” section in the manual, but it looks like the check doesn’t care about my custom URL, and always tries to poll

<BASE_URL> + /api/v1/targets

The URL I need to poll, is formed like this though:

<BASE_URL> + /prometheus/

Because the “targets” are here:

<BASE_URL> + /prometheus/api/v1/targets

I have found similar reports from users wanting to use a custom URL, in those three threads:

• Problems with prometheus special - #5 by marass
• Follow up for prometheus special agent
• Howto spesify cutom url for prometheus - #2 by marco

Please note, that the user in the first thread, had the issue that there
was no possibility to specify a custom URL to begin with. The second thread
was created by the same user, to inform that this works now, since he can
specify a custom URL. A follow-up posting regarding syntax was created, but
never answered.

How exactly does the custom URL need to be specified, in order to get
this to work? Does anyone have experience with this?

Thanks in advance,
Thomas

EDIT: Added Checkmk version and edition

[openmindz (Thomas Karatzoglou)]

Hello everyone,

Here is a screenshot of the rule I created. As you can see, I have
a (blurred out) “base URL” + /prometheus/:

Here is a screenshot of my dummy host, configured with “no IP”
and “Configured API integrations, no Checkmk agent”:

And here is the result:

The “/prometheus/” part is simply omitted, thus resulting in a “404”…

Am I doing something wrong?

Thanks in advance,
Thomas

[robin.gierse]

Ping @martin.hirschvogel: Can you provide some help or insight here?

[openmindz (Thomas Karatzoglou)]

Hello everyone,

I just wanted to add, that I just tried this on a CEE in version 2.0.0p17 as well,
and I have the same behaviour: The “/prometheus/” part of my URL is being
omitted.

Thomas

[openmindz (Thomas Karatzoglou)]

Hello everyone,

When I run the prometheus special agent from command line, nothing happens: the agent just seems to be hanging. I have executed this command:

OMD[test]:~/share/check_mk/agents/special$ ./agent_prometheus --debug -v --timeout 10

An output is never returned.

• Can the prometheus special agent be tested like this, or is there another way?
• Does anyone have a working prometheus rule configuration with a custom URL he/she can share?

Thanks in advance,
Thomas

[martin.hirschvogel]

If you run cmk -D prometheus you get the total command of the Prometheus special agent. Then you can adjust the parameters for debugging purposes.

[openmindz (Thomas Karatzoglou)]

Thank you @martin.hirschvogel

With the “prometheus” rule configured as above, running cmk -D prometheus actually shows my custom URL correctly:

Applied changes, but the check does not seem to poll the URL I specified. The “/prometheus/”
part is omitted, resulting in this:

Am I doing something wrong, or is this perhaps a known issue?

Regards,
Thomas

[openmindz (Thomas Karatzoglou)]

Hello everyone,

I am starting to believe that this is a bug: I cannot use a custom URL with this check.

• Am I doing something wrong?
• Does anyone have similar experience?

I have applied the following workaround: I locally installed a prometheus instance
and configured it to federate my other instance.

My rule is now configured against my local instance, where I don’t need a custom URL, since
it’s already in the prometheus configuration as target in /etc/prometheus/prometheus.yml. I get e.g. cadvisor metrics, and have configured a pod and a service.

BTW, upon discovery on my prometheus host, a check was added for “Prometheus Build”, which
fails/crashes with a KeyError exception regarding the version:

Exception KeyError (version)

Traceback

File “/omd/sites/l10/lib/python3/cmk/base/checking.py”, line 581, in get_aggregated_result
result = _aggregate_results(check_function(**kwargs))
File “/omd/sites/l10/lib/python3/cmk/base/checking.py”, line 812, in _aggregate_results
perfdata, results = _consume_and_dispatch_result_types(subresults)
File “/omd/sites/l10/lib/python3/cmk/base/checking.py”, line 856, in _consume_and_dispatch_result_types
for subr in subresults:
File “/omd/sites/l10/lib/python3/cmk/base/api/agent_based/register/check_plugins.py”, line 89, in filtered_generator
for element in generator(*args, **kwargs):
File “/omd/sites/l10/lib/python3/cmk/base/plugins/agent_based/prometheus_build.py”, line 43, in check_prometheus_build
summary=f"Version: {section[‘version’]}",

I have submitted a “Crash report”.

Thomas

[openmindz (Thomas Karatzoglou)]

Hello everyone,

Can anybody confirm whether:

• I’m doing something wrong or:
• There is a known issue, preventing this from working, i.e. it’s a bug.

I have two further questions:

• Does this still exist for Checkmk 1.6.x? I cannot find the mkp’s anymore that were delivered with feature pack 2 in 2020, I believe.
• How is the “metrics assignment” supposed to work in a “PromQL” query? This isn’t documented as far as I can see, and I cannot choose the same metric twice, ending up using totally randomly any metric that “might” fit, which btw isn’t labeled with my metric name, but with the one of the metric one selects… and it says something about the “element” not existing anymore. Here is an excerpt:

This produces graphs, but as I said, labelled with the “metric” names, and
not with mine:

How would I deduce by this graph which is which?

Thanks in advance,
Thomas

[Doc (Christian Wirtz)]

Hi @openmindz ,

I experienced the same problems. I will try to confirm this in the next days.

[bingyan]

hi
i meet too the keyError(version)

any new discoveries about this keyError(version)

[simonm (Simon)]

I think I have found the error. The API prefix was put in the wrong place (tested on v2.1.0p11):

https://customprefix192.33.22.11:443/api/v1
# should be
https://192.33.22.11:443/customprefix/api/v1 ) 

I opened a PR (agent_prometheus: fixed promql mode (init, query quoting), API prefix… by simonmeggle · Pull Request #513 · tribe29/checkmk · GitHub) for that and two other problems:

• I was not able to define a rule with solely PromQL queries. The agent got stuck / ended in a timeout.
• AFAIS, PromQL queries should not be URL encoded. The API of Prometheus works fine if the query is transferred unquoted.

If anybody wants to test this, I have changed these two files.

• place agent_prometheus.py in ~/local/lib/python3/cmk/special_agents/
• prometheus.py (utils) obviously cannot be loaded from local. I had to edit lib/python3/cmk/special_agents/utils/prometheus.py directly.

Best regards,
Simon