CheckMK 2.0 - Exception in cmk-update-agent plugin while retrieving new agent

[Notausstieg0309 (Markus Bloch)]

Hi together,

I have performed several bakery changes in my environment, So I’m adding/modifiying agent rules, etc.

One of my monitored hosts now shows since 1 day this check result:

So I have checked the log file /var/lib/check_mk_agent/cmk-update-agent.log on the monitored host. It shows the following:

Any ideas how to debug this?

I have already tried:

• manually removing / reinstalling the baked agent via *.deb package
• enforce a new backed agent by changing an affecting rule for the host and bake+sign
• reregister the host via ./cmk-update-agent register
• checked ~/var/log/web.log as well as ~/var/log/apache/error_log but no entries
• checked ~/var/log/apache/access_log:
  

Anybody an idea?

Thank you

Best Regards
Markus

[athomaidis (anastasios)]

Hi Markus,

did you try to remove the state file from /etc?
It’s named cmk-update-agent.state.

After that please run cmk-update-agent register.

Br
Anastasios

[Notausstieg0309 (Markus Bloch)]

@athomaidis

did you try to remove the state file from /etc?
It’s named cmk-update-agent.state.

Yes, indeed. I have removed the state file before re-registering the host:

But the problem remains.

[athomaidis (anastasios)]

Did you check the syslog in /var/log/?

[Notausstieg0309 (Markus Bloch)]

Yes, but no helpful informations as well:

I think the most interesting information is in the cmk-update-agent.log in my first post, but the cmk-update-agent plugin is a 64bit binary with an integrated python environment. I have no idea how i can debug this.

Thanks for your help

Best Regards
Markus

[Notausstieg0309 (Markus Bloch)]

I have performed a tcpdump on the site hosting machine while performing a manual run of cmk-update-agent on the affected host.

I see, that the request is delivered by the systemwide apache2 service via “Transfer-Encoding: chunked”, which means, there is no Content-Length header set:

But when I go further and examine the HTTP request from the system-wide apache to the OMD site instance apache, the request is been delivered with a content length correctly:

When Apache decides to switch to gzip encoding, it also uses Transfer-Encoding: chunked to deliver the response as it compresses by chunks and directly sends it to the client (in this case, the cmk-update-agent script).

As a workaround I have disabled compression for the vhost-definition that serves all OMD sites:

    # CheckMK
    SetEnv no-gzip dont-vary
    Include /omd/apache/*.conf

With this, the error does not popup anymore.

But in general, cmk-update-agent should not rely on the Content-Length header, as it can be missing in case of chunked encoding.

[miclubcomau (MiClub)]

Hi,
I’ve just upgraded our CheckMK to 2.0.0p3, and having exactly the same issue with one host.

I’ve tried the SetEnv in the apache conf, but still not able to have the host connect.

Also been through the steps of removing the .state file and reregister the host, but no go.

The host is external to the LAN, but worked fine on v1.6.

Does anyone have any other suggestions on this please?

TIA.

[miclubcomau (MiClub)]

Hi,

FYI, I found the issue, as we have a NGINX Rev Proxy in between the host and CMK, it was doing the gziping of head info.

So this said, I guess it is confirmed that the gzip of chunked encoded data causes the agent to mark the python-request as bad content, ie the error “Content Length”.

Thank you.

[sebkir]

We are planning to release a fix for this in Checkmk 2.0.0p6. I will keep you posted.

[Notausstieg0309 (Markus Bloch)]

Great news! Thank you very much.