Service discovery fails for datadog host

[3c2c2ff5]

CMK version: 2.3.0p3
OS version: Debian Bookworm

Error message: [special_datadog] Agent exited with code 1: Agent failed - please submit a crash report!

Output of “cmk --debug -vvn hostname”: (If it is a problem with checks or plugins)

[special_datadog] Agent exited with code 1: Agent failed - please submit a crash report!
[piggyback] Success (but no data found for this host), execution time 1.4 sec | execution_time=1.450 user_time=0.010 system_time=0.000 children_user_time=0.540 children_system_time=0.060 cmk_time_ds=0.840 cmk_time_agent=0.000

I get the following error:

  File "/omd/sites/prod/lib/python3/cmk/special_agents/v0_unstable/agent_common.py", line 148, in _special_agent_main_core
    return main_fn(args)
  File "/omd/sites/prod/lib/python3/cmk/special_agents/agent_datadog.py", line 740, in agent_datadog_main
    {"monitors": _monitors_section, "events": _events_section, "logs": _logs_section}[section](
  File "/omd/sites/prod/lib/python3/cmk/special_agents/agent_datadog.py", line 713, in _logs_section
    logs = list(
  File "/omd/sites/prod/lib/python3/cmk/special_agents/agent_datadog.py", line 533, in query_logs
    queried_logs = list(self._execute_query())
  File "/omd/sites/prod/lib/python3/cmk/special_agents/agent_datadog.py", line 555, in _execute_query
    yield from (Log.model_validate(raw_log) for raw_log in response["data"])
  File "/omd/sites/prod/lib/python3/cmk/special_agents/agent_datadog.py", line 555, in <genexpr>
    yield from (Log.model_validate(raw_log) for raw_log in response["data"])
  File "/omd/sites/prod/lib/python3.12/site-packages/pydantic/main.py", line 503, in model_validate
    return cls.__pydantic_validator__.validate_python(

This query doesn’t work from cmk --debug -vvn datadog-instance:

/omd/sites/prod/share/check_mk/agents/special/agent_datadog datadog-instance api_key app_key https://api.datadoghq.eu --monitor_tags source:checkmk --monitor_monitor_tags --event_max_age 600 --event_tags source:checkmk --event_tags_show --event_syslog_facility 1 --event_syslog_priority 1 --event_service_level 0 --log_max_age 600 --log_query '' --log_indexes '*' --log_text message:message --log_syslog_facility 1 --log_service_level 0 --sections monitors events logs

the option –monitor_monitor_tags seems to break the check, when modifying it to:
–monitor_tags it works but the <<<datadog_events:sep(0)>>> section fails.

I configured the datadog integration according to the tow sources:

Integrating Datadog

Open observability: How to integrate Datadog in Checkmk | Checkmk

Working with the Datadog integration in Checkmk

One can see that metriks are coming in:

But service discovery fails:

There are events coming in but host inventory doesn’t work. I am not sure what I am missing here?

Thanks

[3c2c2ff5]

I fixed it by removing all the monitor_tags and event_tags and unchecked the Fetch Logs option, which seems to break the check by itself, furthermore I am a bit confused in regard of the tags, I tried service:mytag, source:mytag and mytag to restrict monitors to fetch, but no luck. For now either all monitors or none. The same for the Events.

Now I need to wrap my head around the logs and the search query.