Understanding agent timeouts and caching for mk-sql and other plug-ins

General
1

Can someone help me understand how Checkmk handles timeouts, asynchronous execution, and caching for agent plug-ins and local checks?

My current issue concerns mk-sql.exe for Microsoft SQL Server, but I have similar questions about mk_oracle on Linux.

I manage the agents exclusively through the Agent Bakery. There is no manual configuration on the monitored hosts.

For MSSQL, I configured the individual sections as follows:

  • General instance status: synchronous, as this is the only available option
  • Backups: asynchronous and cached
  • Availability groups: asynchronous and cached
  • Clusters: asynchronous and cached
  • Cache age: 600 seconds

I also have the following generic agent rule:

Set timeouts for plug-ins and local checks

  • Type: Plug-in
  • Script pattern: mk-sql*
  • Timeout: 360 seconds

The following rules are currently not configured:

  • Set execution mode for plug-ins and local checks
  • Service check timeout (Micro Core)

As soon as I enable the Backups section, the Checkmk service reports:

[agent] Fetcher for host "serverXXX" timed out after 30 seconds CRIT,
[piggyback] Fetcher for host "serverXXX" timed out after 30 seconds CRIT,
Missing monitoring data for all plugins WARN,
execution time 0.0 sec

What I do not understand is where these 30 seconds come from. I have not found any rule in my configuration that uses this value.

The SQL cluster is relatively large and not particularly well optimised. When I run the plug-in manually:

C:\ProgramData\checkmk\agent\plugins\mk-sql.exe -l -v -c C:\ProgramData\checkmk\agent\config\mk-sql.yml

one query belonging to the Backups section takes around two to three minutes for one specific SQL instance. This is why I increased the plug-in timeout to 360 seconds. However, the fetcher still fails after 30 seconds.

I am trying to understand how the following settings interact:

  1. Where does the 30-second fetcher timeout come from?
  2. Does the mk-sql* timeout rule apply to the complete plug-in process or separately to its individual sections?
  3. If one section must run synchronously while other sections are configured as asynchronous, how does Checkmk execute mk-sql.exe overall?
  4. Does the cache age define how often the query is executed, how long existing output remains valid, or both?
  5. Is the generic “Set execution mode for plug-ins and local checks” rule required when the MSSQL rule already defines synchronous or asynchronous execution per section?
  6. Is the Micro Core service check timeout relevant to the agent fetcher timeout, or does it affect a different stage?
  7. What would be the recommended configuration for a slow Backups query so that it can complete and be cached without causing the entire agent fetch to fail?

I have read the documentation and inline help, but I am still missing the overall relationship between these settings.