Using REST API to get counts of hosts - filtering or reducing fields

General
1

Hi. I am working to try and get some reporting from various different systems in our company and one of them is CheckMK. I have REST API access and can see that I can get a list of all hosts using the

/domain-types/host_config/collection/all

endpoint. I can then use jq to extract the folder and host name from the JSON fields to combine into a path that I can then grep against and check the folder (we have folders per team, but each folder then contains a rigid structure of three folders into which hosts are organised and I need to count the total hosts in each lowest level folder:

Main
    -> Folder 1
               -> Folder x
               -> Folder y
               -> Folder z
    -> Folder 2
               -> Folder x
               -> Folder y
               -> Folder z

I need to count “total hosts in folder x from all branches”, “total hosts in folder x from all branches” etc.

I don’t think this "export all hosts with data is being particularly efficient though. I am also trying to avoid iterating through the structure and making multiple API calls too.

Is there any way to provide a “folder contains” type request to check MK or even better, any way to reduce the number of fields returned in the endpoint i am currently using? At the moment the host list is 1300+ hosts and takes about 6-7 seconds to transfer the data, then another 4 to process with jq, then only fractions of seconds to process through grep once the list is just 1300 lines of text.

I have several large instances to run through like this and this is before I have even approached the services lists…

2

You could use the API endpoint /domain-types/host/collections/all and add a query parameter that searches on filename. This should contain the folder name.

3

Hi, just trying this out at the moment. when I add ?columns=filename to that api query i get:

400 Bad Request: these fields have problems - columns

without it I get a list of all hosts on my test system (only 12 hosts) but on the live system I just get an “empty” response with the json framework but no results (both running same version.

Looking in the API docs it says this is a custom variable.

4

turns out I had missed the -G from my curl command which meant the encoded data wasn’t being sent. I ended up with this in bash to test:

DATA=$(curl \
  --request GET \
  -H "Authorization: Bearer <username> <secret>" \
  -G \
  -k \
  -s \
  -H "Content-Type: application/json" \
  --data-urlencode 'columns=filename'
  --data-urlencode 'columns=name')
echo $DATA | jq .

Which works on the smaller test site but doesn’t give any results on the live one. Is there some difference between the permissions required to make a call to

domain-types/hosts-config

and

domain-types/host

or am i missing something? same depth of folder structure in each site. I was expecting to get a list of 1300 hosts. I’m confused!

5

I’d use a view in checkmk instead, so you could collect all your data in one API call.

I have a view in checkmk that I use regularly, that contains the hostname, the WATO folder path and the host labels. We use that for automation.

Any view in Checkmk can be called using output_format=[json/python/csv]

6

Is that not using the old web-api though or have i misunderstood?

7

That is the Multisite-API. The deprecated Web-API was only for configuration tasks.

I had success with the following code from the Python example:

resp = session.get(
    f"{API_URL}/domain-types/host/collections/all",
    params={
        "query": '{"op": "~", "left": "filename", "right": "/wato/folder"}',
        "columns": ['name', 'filename'],
    },
)
8

Thanks All. I went with a REST API call in the end, to keep the code as generic as possible and allow me to audit the innaccurate and inconsistent views and hosts/folders in our existing multi-server and multi-site deployment. (I can’t guarantee that specific views or folders exist, the reason I am doing the audit in the first place. Eventually I will be leaving the script running periodically to report non-compliant folder structures etc reporting to a central DB or repository of some sort.

Its not the prettiest of code as I am certainly no programmer, but it works :slight_smile: Now to convert it from monolithic (checking multiple times for different folders existing and the number of hosts in each) to something more elegant.

RANDOM FACT: In my actual script I went from fully monolithic wrong API call per count (see initial post where i was using host_config instead of “host” method), to an extension of the below, which got it down from 27secs for one site to run one stat, down to 2.8sec to get all 6 stats from 3 sites on a server by only getting the data once per site and converting that from JSON to strings and just operating on those instead.

Andy

CHECK_MK_SERVER="https://servername/sitename"

BEARER_TOKEN="username apikey"

API_ENDPOINT="$CHECK_MK_SERVER/check_mk/api/1.0/domain-types/host/collections/all"

# Grab the data from the API in JSON format - ignoring HTTPS errors
# and selecting the columns "name", "filename" (folder path) and "num_services"
LIVE_DATA=$(curl -H "Authorization: Bearer $BEARER_TOKEN" "$API_ENDPOINT" \
 --request GET \
 -G \
 -k \
 -s \
 -H "Content-Type: application/json" \
 --data-urlencode 'columns=name' \
 --data-urlencode 'columns=filename' \
 --data-urlencode 'columns=num_services' \
)

#Convert the JSON to a string list so can grep and do things with it easily in bash
STRING_LIST=$(echo $LIVE_DATA | jq -r '.value[] | "\(.extensions.filename)/\(.extensions.name)"')

# Grab just the values of the number of services per host
STRING_LIST_SERVICES=$(echo $LIVE_DATA | jq -r '.value[] | "\(.extensions.num_services)"')

# Add up the number of services into TOTAL_SERVICES
SERVICE_ARRAY=($STRING_LIST_SERVICES)
TOTAL_SERVICES=0
for SERVICE_COUNT in ${SERVICE_ARRAY[@]}; do
 TOTAL_SERVICES=$(( $TOTAL_SERVICES + $SERVICE_COUNT ))

# Count the total hosts in to ALL_HOSTS
ALL_HOSTS=$(echo "$STRING_LIST" | wc -l)

# Count all hosts with xxxxx in the folder/file path into XXXXX_HOSTS
XXXXX_HOSTS=$(echo "$STRING_LIST" | grep "xxxxx/" | wc -l)

# Count all hosts with yyyyy in the folder/file path into YYYYY_HOSTS
YYYYY_HOSTS=$(echo "$STRING_LIST" | grep "yyyyy/" | wc -l)