Upgrading CheckMK 2.1 => 2.3

General
, , , ,
1

Hello all,
it is upgrade time !

I am currently running a distributed monitoring architecture with roughly 50 remote site.
The central site has 2.1.0.16 version and the remote site have 2.1.0.15 (some has 2.1.0.16 too).

In the next few weeks, we plan to upgrade the central and the remote site to 2.3.0.10 (while I am writing this the 2.3.0.11 is being released by 2 h, so before choosing this as the target version i’d like to wait a couple of weeks, just to be sure no bug are being reported).

Test executed
During this week I have created a clone of the central site, and connected a remote that I have created just for this test. Both the central and the remote site has 2.1.0.16 version, I have performed firstly and update 2.1.0.16 → 2.1.0.46 and then an upgrade 2.1.0.46 → 2.2.0.31 and 2.2.0.31 → 2.3.0.10. I have tried, in a couple of run, to stop every site, perform the update/upgrade and then starting every site and to update/upgrade firstly the remote site and then the central site, both ways are described here.
Both run went successful.

I am not planning to update the agent, so the monitored VM will still run the 2.1.0.15 (some 2.1.0.16) agent version.
During the test I have experienced some error/warning caused by incompatible MKPs, these errors can be ignored since we are not using them, so I am planning to disable and remove these MKPs before the productive run.

I feel worried about how the old plugin will interact with the upgraded site.
Under this aspect I have tested the mk_oracle, mk_sap_hana and a couple of custom check and i see no problem, the output is correctly interpretated and shown on the dashboard.

Question
So, there are some other test i can perform before the productive run ? Or there is some precaution I have to take ? (backup + snapshot are already counted in :smiley: )

Updating every site could take a while, so it is mandatory start every site after upgrade, or can i simply install every package, perform an upgrade from 2.1 to 2.3 and then start the site ?

In the documentation it is mentioned that the 2.3 version could lead to an +20% CPU usage, how could this be forecasted ? I suppose it is depend on the number of host/checks, is there a way how could this be calculated ?

And finally, the big question, Is there any downside in upgrading from 2.1 to 2.3 ?

If something is unclear, please do not to esitate to ask.

Kr,
Davide

2

Hi Davide,

In general as already discussed here: Different version for Master & Slave - #3 by erik you can only upgrade from from one minor version to the next minor version and you have to upgrade all remote sites first before you can upgrade the master.

We run a three tier environment with test, preprod and prod which is advisable in bigger environments.

On test I run two sites running real checks and one site in simulation mode which is a 1:1 copy of a real site connected to master. Unfortunately simualtion mode is not as good as it was until 1.6 but it helps to do some basic checks.

I always start testing upgrades with the simulated site. Make a screenshot of the dashbard to document current number of alerts. We need that later for comparision. Then do a manual backup and run the upgrade. In case of issues during upgrade possibly I have to restore the site from backup and fix it on master. If there are issues with the configuration you can always run cmk-update-config with -v or even --debug. It also helps to troubleshoot.

If simulation site upgrade runs without issues I check for any UNKNOW, STALE alerts or even crashes. Also compare the figures of the dashboard so that you have the same number of services, hosts or alerts. With the last upgrade we had the issue that SNMP based checks had been gone. This way you can detect it. Also test things like activate changes, LDAP auth etc. whatever is configured.
If simulation site is OK we proceed with the other remote sites connected and do the same tests.
In the next step we upgrade PrepProd remote site and then the first sites in the prod env. If all runs well we continue upgrading all remote sites.
While that runs we do a backup of master from test envirnment and restore it in another vm to test upgrade. You may can skip that but we have a couple of people working even in test env and I dont want to disturb them too much. Atfter upgrade of master in Test environment we check if there are any compatibility issues with the old agent. Mainly such problems should already visible while upgrading the simulation site because it uses cached agent output from old agent.

If upgrade of master in Test env is finally done we also upgrade PreProd and then Prod env.
Before we bake new agents we disable automatic agent updates. After, we enable automatic agent updates region by region to see of we have any issues.

!!! Always do a backup on CLI before you do any upgrade. !!!

To automate the remote sites we run the following script:

#!/usr/bin/env bash
HOSTNAME=$(hostname)
SITECODE=${HOSTNAME:2:3}
SITECODE=${SITECODE^^}
VERSION=2.1.0p42
OVERSION=$( /usr/bin/omd version -b ${SITECODE})

if [ ${OVERSION} = ${VERSION}.cee ]
then
    echo -e "!!!!!! Site ${SITECODE} is already on version ${VERSION} !!!!!"
    exit 1
fi
echo "Updating site ${SITECODE} from version ${OVERSION} to checkmk version:"
if /usr/bin/omd versions -b | grep $VERSION.cee
then
    echo -e "################# Backup site ${SITECODE} to tmp ################"
    if /usr/bin/omd backup ${SITECODE} /tmp/Backup_${SITECODE}.tgz
    then
        echo -e "+ Backup successful"
        backup=1
    else
        echo -e "! Could not make a backup of site ${SITECODE}"
        echo -e "! Start site ${SITECODE} with version ${OVERSION}"
        /usr/bin/omd start ${SITECODE}
        exit 1
    fi

    echo -e "################# Stopping site ${SITECODE} in OMD ################"
     # Sometimes site could not be stopped with only one attempt. We try three times
     stopcounter=0
     until /usr/bin/omd stop ${SITECODE}
     do
         echo -e "! Failed stopping ${SITECODE} in OMD"
         ((stopcounter++))
         if (( stopcounter >= 3 ))
         then
            echo -e "! Failed stopping ${SITECODE} in OMD after ${stopcounter} attempts"
            /usr/bin/omd start ${SITECODE}
            exit 1
         fi
      done
      echo -e "+ Successful stopped ${SITECODE} in OMD"

    echo -e "\n################# Updating site ${SITECODE} to ${VERSION} ################\n"
    if /usr/bin/omd -f -V $VERSION.cee update --conflict=install ${SITECODE}
    then
        echo -e "+ Successful update site ${SITECODE} to Version ${VERSION}"
        update=1
    else
        echo -e "! Failed update site ${SITECODE} to Version ${VERSION}"
        update=0
    fi

    echo -e "\n################# Rebuilding checkmk config ################\n"
    if /usr/bin/su - ${SITECODE} -c 'cmk -U'
    then
        echo -e "+ Successful ran cmk -U"
        cmk=1
    else
        echo -e "! Failed running cmk -U"
        cmk=0
    fi
    success=$((${backup} * ${update} * ${cmk}))
    if [ ${success} -eq 0 ]
    then
        echo -e "############ restore site ${SITECODE} after failed update ################"
        omd restore --reuse --kill /tmp/Backup_${SITECODE}.tgz
    fi
 echo -e "################# Starting site after update ################"
    if /usr/bin/omd start ${SITECODE}
    then
        echo -e "+ Successful started site ${SITECODE^^} with Version $(omd version -b ${SITECODE})\n"
        sitestart=1
    else
        echo -e "! Failed to start site ${SITECODE^^} with Version $(omd version -b ${SITECODE})\n"
        sitestart=0
    fi
    success=$((${backup} * ${update} * ${cmk} * ${sitestart}))
    summary="Backup Site:\t${backup}\nUpdate Site:\t${update}\nRebuild config:\t${cmk}\nStart site:\t${sitestart}\n"
    if [ ${success} -gt 0 ]
    then
        echo -e "################# Update completed without errors ################"
        #rm -f /tmp/Backup_${SITECODE}.tgz
        echo -e ${summary}
        exit 0
    else
        echo -e "################# Update completed with errors ###############"
        echo -e ${summary}
        exit 2
    fi
fi
echo -e "############ Version ${VERSION} not available in omd #############"
exit 1
4

Hi Mike,
thanks for the exhaustive response.

Unfortunately I do not have a copy 1:1 of every remote site, i greatly appreciate the advice that you gave to me, so soon or later this architecture model will be implemented.

But this leads me to other question: so you got 3 sites (for every customer, i suppose) that points to the same group of hosts, this does not require more CPU from the monitored host ? And, the sites are hosted on the same VM ? If so, what is the size of the VM ? Currently our remote site has 2vCPU and 4 GB RAM.

For the compatibility of the old agent I have tested the relevant plugin and no errors are being reported.

Thanks again,
Davide

6

We too. We just selected on site which we believe running almost all technology we monitor in the company.

7

We run CEE and my ‘customers’ are purely internal. As said, one site is in simulation mode, so no agent contacted. The other two´s are one for local IT and one for engineering IT. We mainly have test systems configured. Some systems are also just simulated. Its a bit a mix of all.
The master of test system runs on a separate VM in our datacenter and the remote sites are running as VM on a test ESX in my site datacenter.

But, we have the situation that we monitor hosts from two monitoring systems and until now I got no complains that it has an impact on the monitored systems.

regards

Michael