Automatic Agent Update Registration Failing with Internal error: 'list' object has no attribute 'update'

scotsie · July 13, 2022, 1:12am

CMK version: 2.0.0p18 (CME)
OS version: CheckMK Server CentOS 7, Agent Server CentOS 7

I have a single host out of a group of hosts where the agent was deployed via automation (salt master server) that reports the agent is not registered. When I attempt to register manually it fails reporting the following at the command line.

Going to register agent at deployment server
 Unexpected answer from Checkmk server: Missing json data. Maybe we are talking to an agent bakery from before Checkmk 2.0 ?
See syslog or Logfile at /var/lib/check_mk_agent/cmk-update-agent.log for details.

To make sure the wrong agent wasn’t installed or an install error did not occur, I removed the checkmk agent using yum remove and deleted the .state and log files.
I checked the bakery on the checkmk server side and confirmed the host name in question was listed next to the baked agent version. I pulled a fresh RPM file and installed using yum install.

Afterwards, I ran registration again using the following syntax with the same result.

# /usr/bin/cmk-update-agent register -v -s corpnms01.corporate.local -i corpnms -H web03-2.prod.corporate.local -U automation -S [redacted]
Updated the certificate store "/var/lib/check_mk_agent/cas/all_certs.pem" with 1 certificate(s)
Going to register agent at deployment server
Unexpected answer from Checkmk server: Missing json data. Maybe we are talking to an agent bakery from before Checkmk 2.0 ?
See syslog or Logfile at /var/lib/check_mk_agent/cmk-update-agent.log for details.

/var/log/messages

Jul 12 20:46:44 web03-2 journal: [cmk-update-agent] ERROR: Unexpected answer from Checkmk server: Missing json data. Maybe we are talking to an agent bakery from before Checkmk 2.0 ?

/var/lib/check_mk_agent/cmk-update-agent.log

2022-07-12 20:31:22,221 INFO: Updated the certificate store "/var/lib/check_mk_agent/cas/all_certs.pem" with 1 certificate(s)
2022-07-12 20:31:22,221 INFO: Starting Update mode as plugin.
2022-07-12 20:31:22,221 DEBUG: No state file found yet. New state data will be saved to /etc/cmk-update-agent.state
2022-07-12 20:31:22,223 DEBUG: Saved deployment status to /etc/cmk-update-agent.state.
2022-07-12 20:31:22,223 DEBUG: Caught Exception:
Traceback (most recent call last):
  File "cmk_update_agent.py", line 1208, in _do_update_as_plugin
  File "cmk_update_agent.py", line 1242, in _do_update_agent
Exception: The agent updater is not registered at the deployment server
2022-07-12 20:31:22,223 DEBUG: Writing agent section to stdout:
<<<check_mk>>>
AgentUpdate: last_check None last_update None aghash None pending_hash None update_url https://corpnms01.prod.corporate.local/corpnms/check_mk error The agent updater is not registered at the deployment server

2022-07-12 20:31:22,223 DEBUG: Sending new state data to agent bakery
2022-07-12 20:31:22,223 DEBUG: Fetching content (using requests): https://corpnms01.prod.corporate.local/corpnms/check_mk/deploy_agent.py
2022-07-12 20:31:22,374 DEBUG: Response from agent bakery:
{'result_code': 1, 'result': 'Missing host name', 'severity': 'error'}
2022-07-12 20:31:22,374 DEBUG: Caught error from Agent Bakery. Details:
Missing host name
2022-07-12 20:31:22,374 DEBUG: Caught Exception:
Traceback (most recent call last):
  File "cmk_update_agent.py", line 1236, in _push_status
  File "cmk_update_agent.py", line 1353, in _fetch_agent_info
  File "cmk_update_agent.py", line 316, in fetch_data_from_server
  File "cmk_update_agent.py", line 349, in _do_request
  File "cmk_update_agent.py", line 395, in _check_for_response_error
Exception: Agent Bakery: Missing host name
2022-07-12 20:31:22,375 DEBUG: Done.
2022-07-12 20:31:41,451 DEBUG: Starting Checkmk Agent Updater v2.0.0p18
2022-07-12 20:31:41,451 DEBUG: Successfully read /etc/cmk-update-agent.state.
2022-07-12 20:31:41,452 DEBUG: Successfully read /etc/check_mk/cmk-update-agent.cfg.
2022-07-12 20:31:41,452 DEBUG: Updating the certificate store "/var/lib/check_mk_agent/cas/all_certs.pem"...
2022-07-12 20:31:41,455 INFO: Updated the certificate store "/var/lib/check_mk_agent/cas/all_certs.pem" with 1 certificate(s)
2022-07-12 20:31:41,455 DEBUG: Starting Registration Mode.
2022-07-12 20:31:41,455 INFO: Going to register agent at deployment server
2022-07-12 20:31:41,455 DEBUG: Fetching content (using requests): https://corpnms01.prod.corporate.local/corpnms/check_mk/register_agent.py
2022-07-12 20:31:42,510 DEBUG: Response from agent bakery:
Internal error: 'list' object has no attribute 'update'

An internal error occured while processing your request. You can report this issue to the Checkmk team to help fixing this issue. Please open the crash report page and use the form for reporting the problem.
{"result_code": 1, "result": "'list' object has no attribute 'update'", "severity": "error"}
2022-07-12 20:31:42,510 DEBUG: Caught Exception:
Traceback (most recent call last):
  File "cmk_update_agent.py", line 1889, in main
  File "cmk_update_agent.py", line 878, in run
  File "cmk_update_agent.py", line 1001, in _run_mode
  File "cmk_update_agent.py", line 1108, in _register_agent
  File "cmk_update_agent.py", line 316, in fetch_data_from_server
  File "cmk_update_agent.py", line 349, in _do_request
  File "cmk_update_agent.py", line 398, in _check_for_response_error
Exception: Unexpected answer from Checkmk server: Missing json data. Maybe we are talking to an agent bakery from before Checkmk 2.0 ?
2022-07-12 20:31:42,511 ERROR: Unexpected answer from Checkmk server: Missing json data. Maybe we are talking to an agent bakery from before Checkmk 2.0 ?

Given the error ‘missing host name’ I also tried registering using --hostname as well as without any options to have it prompt.

Thanks in advance for any guidance.

Sincerely,

Scotsie

robin.gierse · July 13, 2022, 6:32am

Are you actually talking to a Checkmk server?
Is the Checkmk server running the same version as the agent?
Is there any proxy or web application firewall between the agent and server?

scotsie · July 13, 2022, 9:06am

Yes, the fqdn specified in the register command is the checkmk server.
Yes, the agent was signed, baked and pulled from the same checkmk server as item #1.
No proxy but these are in different datacenters so firewalls are a factor however, the peer servers that registered fine are in the same subnet.

robin.gierse · July 13, 2022, 3:47pm

Might still be worthwhile to ask the firewall admins, if they do web application firewalling. Newer firewalls can do that, and we have seen the weirdest effects there.

scotsie · July 14, 2022, 6:05am

Fair point, I have a love/hate with the network sometimes. No changes reported but something suddenly works (or not). It’s mostly frustrating because I deployed 6 servers in the same subnet and 5 of 6 registered back to the checkmk server fine.
I’ll do some port and trace testing and report back.

Thanks for your time.

scotsie · August 3, 2022, 12:37pm

Took me a little bit to get back to this.
Performed some testing from the CLI on the cmk server and host.
DNS is resolving properly between the two hosts.

A telnet from the cmk server to the host port 6556 successfully retrieves the agent info’

OMD[corpnms]:~$ cmk -d web03-2.prod.corporate.local
<<<check_mk>>>
Version: 2.0.0p18
AgentOS: linux
Hostname: web03-2
<snip>

A telnet from the host to the cmk server to port 80/443 is also successful.

[root@web03-2 ~]# telnet corpnms01.corporate.local 80
Trying 192.168.44.93...
Connected to corpnms01.corporate.local.
Escape character is '^]'.
^]

[root@web03-2 ~]# telnet corpnms01.corporate.local 443
Trying 192.168.44.93...
Connected to corpnms01.corporate.local.
Escape character is '^]'.
^]

Manual registration from the host showed the same results as the earlier log entries.

Going to register agent at deployment server
Unexpected answer from Checkmk server: Missing json data. Maybe we are talking to an agent bakery from before Checkmk 2.0 ?
See syslog or Logfile at /var/lib/check_mk_agent/cmk-update-agent.log for details.

I performed an update to the CMK Server to release 2.0.0p26.cme, baked and signed the agent and retried registration.
No difference.

With the missing attribute and reference to earlier releases, I deleted the host from check_mk completely, committed the changes and then added it back.
Registration still failed.

Finally, I removed the agent again, deleted the /etc/cmk-update-agent.state file and related /var/lib/check_mk folder.
Downloaded the updated agent 2.0.0p26 version and installed. Manual registration was finally successful.

Going to register agent at deployment server
Successfully registered agent of host "web03-2.prod.corporate.local" for deployment.
You can now update your agent by running 'cmk-update-agent -v'
Saved your registration settings to /etc/cmk-update-agent.state.

So honestly unable to isolate what was causing it to fail but the:

Update of the checkmk server release
2nd round of agent removal/install
allowed it to register.

Hope this helps someone else.

gilbertcane · October 18, 2022, 12:22pm

Python lists cannot be divided into separate lists based on characters that appear in the values of a list. This is unlike strings which values can be separated into a list. The AttributeError is an exception thrown when an object does not have the attribute you tried to access. The ‘list’ object has no attribute ‘split’ and you’re trying to call python split() function on the whole list of lines, and you can’t split a list of strings, only a string. So, you need to split each line, not the whole thing.

To solve the above problem, you need to iterate over the strings in the list to get individual strings; then, you can call the split() function.

system · October 18, 2023, 12:22pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.