Unable to use ansible collection against a checkmk cluster

Troubleshooting
, , , ,
1

Hi All,
I recently wanted to use ansible collection in order to automate my configurations over a checkmk cluster. My current version is Enterprise Edition 2.1.0p30.
I simply tried with a folder creation but got the error msg:

fatal: [checkmk_cluster]: FAILED! => {“ansible_facts”: {“discovered_interpreter_python”: “/usr/bin/python3”}, “changed”: false, “msg”: “Error calling API. HTTP code -1. Details: N/A.”}

The issue was, that it succeeded when I tried against my backup standalone checkmk with the same version. I used the curl command to confirm, that the automation secret was correct for the checkmk cluster.
Could someone please advise if ansible collection is not working for checkmk cluster?
Thank you.

2

Hi @vu.nguyen! What exactly does “Checkmk Cluster” mean to you? Do you have two Checkmk Appliances clustered, or is it something home-grown?

Generally, I see no reason, why it should not work.

3

Hi @robin.gierse ,
Yes, I have two physical Checkmk Appliances clustered. I tried executing ansible playbook over either cluster IP or active IP with no help. Honestly the http code -1 is strange and I can’t find out any glue about it.
FYI, my ansible running from a WSL Ubuntu remotely. In fact I do not see any issue for that, it can be executed successfully over another standalone checkmk.
And this is the curl command to display folder. It worked therefore failure API calling should not be a credential issue.

curl -G
–request GET
–header “Accept: application/json”
–header “Authorization: Bearer $USERNAME $PASSWORD”
–data-urlencode ‘parent=~’
–data-urlencode ‘recursive=true’
–data-urlencode ‘show_hosts=false’
“$API_URL/domain-types/folder_config/collections/all”

4

The error code -1 indicates a timeout or a generic connection error, I am not certain.
Does running ansible with -vvv provide additional insights?
Can you see the requests in the site’s web.log?

5

Hi @robin.gierse ,
Thanks a lot for your support.
In the site’s web.log, I did not see any relevant requests for both Checkmk systems regarding ansible automate.
I tried running hostgroup creation ansible with -vvv and saw a difference when Checkmk clustered reported Failed to connect to the host via ssh: Shared connection to IP closed. ESTABLISH SSH CONNECTION FOR USER: root while standalone one only reported Shared connection to IP closed. ESTABLISH SSH CONNECTION FOR USER: root
I spent much time on this and should no ssh issues. Both Checkmk systems are now on the same network. All ssh config/version are same, and logs are also same.
Here is the output for Checkmk clustered:

ansible-playbook [core 2.14.6]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/cmk/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/cmk/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /mnt/c/cmk/inventory/kemp-cmk as it did not pass its verify_file() method
auto declined parsing /mnt/c/cmk/inventory/kemp-cmk as it did not pass its verify_file() method
Parsed /mnt/c/cmk/inventory/kemp-cmk inventory source with ini plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: hostgroup.yml ******************************************************************************************************************************************************************************************
1 plays in hostgroup.yml

PLAY [Create Host Groups] *************************************************************************************************************************************************************************************

TASK [Create a single host group] *****************************************************************************************************************************************************************************
task path: /mnt/c/cmk/hostgroup.yml:7
<10.165.31.147> ESTABLISH SSH CONNECTION FOR USER: root
<10.165.31.147> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/cmk/.ansible/cp/18abc3596d"' 10.165.31.147 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<10.165.31.147> (0, b'/root\n', b'')
<10.165.31.147> ESTABLISH SSH CONNECTION FOR USER: root
<10.165.31.147> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/cmk/.ansible/cp/18abc3596d"' 10.165.31.147 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1689852493.270862-67-217590670387403 `" && echo ansible-tmp-1689852493.270862-67-217590670387403="` echo /root/.ansible/tmp/ansible-tmp-1689852493.270862-67-217590670387403 `" ) && sleep 0'"'"''
<10.165.31.147> (0, b'ansible-tmp-1689852493.270862-67-217590670387403=/root/.ansible/tmp/ansible-tmp-1689852493.270862-67-217590670387403\n', b'')
<kempCMK> Attempting python interpreter discovery
<10.165.31.147> ESTABLISH SSH CONNECTION FOR USER: root
<10.165.31.147> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/cmk/.ansible/cp/18abc3596d"' 10.165.31.147 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'python3.11'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.10'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<10.165.31.147> (0, b'PLATFORM\nLinux\nFOUND\n/usr/bin/python3.7\n/usr/bin/python3\n/usr/bin/python2.7\n/usr/bin/python\n/usr/bin/python\nENDFOUND\n', b'')
<10.165.31.147> ESTABLISH SSH CONNECTION FOR USER: root
<10.165.31.147> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/cmk/.ansible/cp/18abc3596d"' 10.165.31.147 '/bin/sh -c '"'"'/usr/bin/python3.7 && sleep 0'"'"''
<10.165.31.147> (0, b'{"platform_dist_result": ["debian", "10.13", ""], "osrelease_content": "PRETTY_NAME=\\"Debian GNU/Linux 10 (buster)\\"\\nNAME=\\"Debian GNU/Linux\\"\\nVERSION_ID=\\"10\\"\\nVERSION=\\"10 (buster)\\"\\nVERSION_CODENAME=buster\\nID=debian\\nHOME_URL=\\"https://www.debian.org/\\"\\nSUPPORT_URL=\\"https://www.debian.org/support\\"\\nBUG_REPORT_URL=\\"https://bugs.debian.org/\\"\\n"}\n', b'<stdin>:29: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5\n')
Using module file /home/cmk/.ansible/collections/ansible_collections/checkmk/general/plugins/modules/host_group.py
<10.165.31.147> PUT /home/cmk/.ansible/tmp/ansible-local-62qrz4xr66/tmpswq9ztzp TO /root/.ansible/tmp/ansible-tmp-1689852493.270862-67-217590670387403/AnsiballZ_host_group.py
<10.165.31.147> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/cmk/.ansible/cp/18abc3596d"' '[10.165.31.147]'
<10.165.31.147> (0, b'sftp> put /home/cmk/.ansible/tmp/ansible-local-62qrz4xr66/tmpswq9ztzp /root/.ansible/tmp/ansible-tmp-1689852493.270862-67-217590670387403/AnsiballZ_host_group.py\n', b'')
<10.165.31.147> ESTABLISH SSH CONNECTION FOR USER: root
<10.165.31.147> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/cmk/.ansible/cp/18abc3596d"' 10.165.31.147 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1689852493.270862-67-217590670387403/ /root/.ansible/tmp/ansible-tmp-1689852493.270862-67-217590670387403/AnsiballZ_host_group.py && sleep 0'"'"''
<10.165.31.147> (0, b'', b'')
<10.165.31.147> ESTABLISH SSH CONNECTION FOR USER: root
<10.165.31.147> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/cmk/.ansible/cp/18abc3596d"' -tt 10.165.31.147 '/bin/sh -c '"'"'/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1689852493.270862-67-217590670387403/AnsiballZ_host_group.py && sleep 0'"'"''
<10.165.31.147> (1, b'\r\n{"changed": false, "failed": true, "msg": "Error calling API. HTTP code -1. Details: N/A", "invocation": {"module_args": {"server_url": "http://10.165.31.147/", "site": "kemp", "automation_user": "automation", "automation_secret": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "validate_certs": false, "title": "My Host Group", "name": "my_host_group", "state": "present", "groups": []}}}\r\n', b'Shared connection to 10.165.31.147 closed.\r\n')
<10.165.31.147> Failed to connect to the host via ssh: Shared connection to 10.165.31.147 closed.
<10.165.31.147> ESTABLISH SSH CONNECTION FOR USER: root
<10.165.31.147> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/home/cmk/.ansible/cp/18abc3596d"' 10.165.31.147 '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1689852493.270862-67-217590670387403/ > /dev/null 2>&1 && sleep 0'"'"''
<10.165.31.147> (0, b'', b'')
fatal: [kempCMK]: FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "invocation": {
        "module_args": {
            "automation_secret": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "automation_user": "automation",
            "groups": [],
            "name": "my_host_group",
            "server_url": "http://10.165.31.147/",
            "site": "kemp",
            "state": "present",
            "title": "My Host Group",
            "validate_certs": false
        }
    },
    "msg": "Error calling API. HTTP code -1. Details: N/A"
}

PLAY RECAP ****************************************************************************************************************************************************************************************************
kempCMK                : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
6

Now I see the problem!
You need to run the tasks, that talk to the REST API (all modules essentially) as delegate_to: localhost, probably. I assume you are running Ansible on your local machine and that machine can also reach the Checkmk webinterface, right? Then adding the above directive to all Ansible tasks that use a checkmk.general module (not a role!) should fix your issue.

7

Hi @robin.gierse ,
Thanks a lot. You saved my life :+1:. It worked fine now.

1 Like