REST API 'Show the monitored services of a host' does not show the right status of a service

CMK version: 2.2.0p3
OS version:Debian 11.7

My goal is to make a python script that monitors only green service checks if a host is new to checkmk.

I have made a “Execute a service discovery” API call with “fix_all” mode waited for the service discovery to complete and then gave the list of services which I get as a response basically to the “Show the monitored services of a host” API call. Between these two api calls I have activated all pending changes too.

The Problem is if I look into the states (is the service OK, WARNING, CRIT) I get either everything is OK which is not true in the web interface or it works half the time as expected. My guess is that my server is too slow to process all api calls but I am not sure.

I looked into: Hosts show Check_MK service for SNMP monitoring as "CRIT" "Null"

But it’s not really a solution beside maybe rolling back to a later version.

Can you provide a little bit of output what you receive after your API call and how your output is compared to the web interface?

I am getting a json object back from the “Show the monitored services of a host” call which looks like this but for every service in a host:

{"links": [{"domainType": "link", "rel": "self", "href": "<url>/check_mk/api/1.0/objects/service/checkmk-dev-OMD%2520bvd_lan%2520status", "method": "GET", "type": "application/json"}], "domainType": "service", "id": "checkmk-dev-OMD bvd_lan status", "title": "Service OMD bvd_lan status", "members": {}, "extensions": {"description": "OMD bvd_lan status", "host_name": "checkmk-dev", "state_type": 1, "state": 0, "last_check": 1691735710}}

I get the “state” value which you can see here is 0. And most of the time it is 0 for all services.

Here is maybe a bit of a cleaner look:

Here is also some relevant skript parts:

def execute_a_service_discovery(url: str, hostname: str, mode: str):
    option = "/domain-types/service_discovery_run/actions/start/invoke"

    url += option

    payload = {
            'host_name': hostname,
            'mode': mode
            }

    r = requests.post(url, headers=headers, json=payload)
    return r

def show_the_monitored_services(url: str, hostname: str, service_description: str):
    option = "/objects/host/"+hostname+"/actions/show_service/invoke?service_description="+service_description

    url += option
    
    r = requests.get(url, headers=headers)
    return r

def update_the_phase_service(url: str, hostname: str, check_type: str, service_item: str, target_phase: str):
    option = "/objects/host/"+hostname+"/actions/update_discovery_phase/invoke"

    url += option

    payload = {
            'check_type': check_type,
            'service_item': service_item,
            'target_phase': target_phase
            }

    print("UPDATE: ", json.dumps(payload), "FOR HOST: ", hostname)

    r = requests.put(url, headers=headers, json=(payload))
    return r

def service_monitor(hostname: str):
    service_to_body = {}
    #execute_a_service_discovery(url, hostname, "refresh")
    tmp = execute_a_service_discovery(url, hostname, "fix_all").text

    activate_pending_changes(url)

    json_response_extensions = find_json_values_by_key('extensions',tmp)

    for index,x in enumerate(json_response_extensions):
       
        if index == len(json_response_extensions)-1:
            continue

        # TODO: Expectation that all hosts have an agent installed on them for the following line to work
        service_to_body[x['service_name']] = x


    #print("RESPOND ACTIVATE CHANGES 1: ", activate_pending_changes(url))
    return service_to_body

def get_all_hosts(url: str, effective_attributes='false'):
    option = "/domain-types/host_config/collections/all?effective_attributes="+effective_attributes

    url += option

    r = requests.get(url, headers=headers)
    return r
    

def monitor_initial_cleanup():
    hosts = find_json_values_by_key('id', get_all_hosts(url).text) 
    
    print("HOSTS: ", hosts)
    
    for hostname in hosts:
        services = service_monitor(hostname)
        
        for service_name, body in services.items():
            monitored_response = show_the_monitored_services(url, hostname, service_name).text
            
            #print("MONITORED RESPONSE: ", monitored_response)

            #activate_pending_changes(url)
            #sleep(2)

            state = find_json_values_by_key('state', monitored_response)
          
            if len(state) != 0:
                print("HOST: ", hostname, " SERVICE: ", service_name, " STATUS: ", state[0])

            if len(state) != 0 and state[0] != 0:
                print("RESPONSE: ", update_the_phase_service(url, hostname, body['check_plugin_name'], body['service_item'], 'undecided'))


if __name__ == '__main__':
     monitor_initial_cleanup()

You are sure that your function find_json_values_by_key get the correct value?
In my test i set a service to warning and get the correct result.

{
  "links": [
    {... }
  ],
  "domainType": "service",
  "id": "HOSTNAME-Check_MK",
  "title": "Service Check_MK",
  "members": {},
  "extensions": {
    "description": "Check_MK",
    "host_name": "HOSTNAME",
    "state_type": 0,
    "state": 1,
    "last_check": 1691739464
  }
}

State type is soft as i set this service to WARN manually.

{
  "links": [
    {...}
  ],
  "domainType": "service",
  "id": "HOSTNAME-Check_MK",
  "title": "Service Check_MK",
  "members": {},
  "extensions": {
    "description": "Check_MK",
    "host_name": "HOSTNAME",
    "state_type": 1,
    "state": 0,
    "last_check": 1691739694
  }
}

One check later it is again green as also shown inside GUI.

URL used was

/api/1.0/objects/host/HOSTNAME/actions/show_service/invoke?service_description=Check_MK

Hmm… I checked the find_json_values_by_key function multiple times and it gives me correct values. It basically goes through the entire json object and appends a list of all the values of that key I search. If you wanna see it here it is:

def find_json_values_by_key(key, json_repr):
    results = []

    def _decode_dict(a_dict):
        try:
            results.append(a_dict[key])
        except KeyError:
            pass
        return a_dict

    json.loads(json_repr, object_hook=_decode_dict) # Return value ignored.
    return results

I just build a little sleep of 2 sec into my code in the monitor_initial_cleanup function (if you look closely in the script i send I commented the line out with the sleep(2)) and it works better and gives me more often correct values than before maybe it’s related to the server or something with the network/requests?