Since upgrade to 2.3.0 the BI isn't working anymore

Troubleshooting
1

Hey Guys

Since I’m upgraded from 2.2 to 2.3 I have an isse with my BI rules

CMK version: 2.3.0p20 CEE
OS version: Debian 12

Error message:

Internal error: host_name

An internal error occured while processing your request. You can report this issue to the Checkmk team to help fixing this issue.

Exception:

KeyError (host_name)

Traceback

  File "/omd/sites/site/lib/python3/cmk/gui/wsgi/applications/checkmk.py", line 188, in _process_request
    resp = page_handler()
  File "/omd/sites/site/lib/python3/cmk/gui/wsgi/applications/utils.py", line 113, in _call_auth
    func()
  File "/omd/sites/site/lib/python3/cmk/gui/pages.py", line 187, in wrapper
    return hc().handle_page()
  File "/omd/sites/site/lib/python3/cmk/gui/pages.py", line 51, in handle_page
    self.page()
  File "/omd/sites/site/lib/python3/cmk/gui/pages.py", line 146, in <lambda>
    "page": lambda self: self._wrapped_callable[0](),
  File "/omd/sites/site/lib/python3/cmk/gui/views/page_show_view.py", line 100, in page_show_view
    process_view(GUIViewRenderer(view, show_buttons=True))
  File "/omd/sites/site/lib/python3/cmk/gui/views/page_show_view.py", line 174, in process_view
    _process_regular_view(view_renderer)
  File "/omd/sites/site/lib/python3/cmk/gui/views/page_show_view.py", line 192, in _process_regular_view
    _show_view(view_renderer, unfiltered_amount_of_rows, rows)
  File "/omd/sites/site/lib/python3/cmk/gui/views/page_show_view.py", line 412, in _show_view
    view_renderer.render(
  File "/omd/sites/site/lib/python3/cmk/gui/view_renderer.py", line 193, in render
    has_done_actions = do_actions(
  File "/omd/sites/site/lib/python3/cmk/gui/views/command/form.py", line 156, in do_actions
    _commands, confirm_options, confirm_dialog_options, executor = core_command(
  File "/omd/sites/site/lib/python3/cmk/gui/views/command/form.py", line 66, in core_command
    result = cmd.action(cmdtag, spec, row, row_nr, action_rows)
  File "/omd/sites/site/lib/python3/cmk/gui/views/command/base.py", line 166, in action
    result = self._action(cmdtag, spec, row, row_index, action_rows)
  File "/omd/sites/site/lib/python3/cmk/gui/views/command/commands.py", line 1845, in _action
    cmdtag, specs, len_action_rows = self._downtime_specs(cmdtag, row, action_rows, spec)
  File "/omd/sites/site/lib/python3/cmk/gui/views/command/commands.py", line 2053, in _downtime_specs
    len_action_rows = len({row["host_name"] for row in action_rows})

Local Variables

'aggr_hosts': [('Max recursion depth reached',
                                  'Max recursion depth reached'),
                                 ('Max recursion depth reached',
                                  'Max recursion depth reached'),
                                 ('Max recursion depth reached',
                                  'Max recursion depth reached'),
                                 ('Max recursion depth reached',
                                  'Max recursion depth reached'),
                                 ('Max recursion depth reached',
                                  'Max recursion depth reached'),
                                 ('Max recursion depth reached',
                                  'Max recursion depth reached'),
                                 ('Max recursion depth reached',
                                  'Max recursion depth reached'),
                                 ('Max recursion depth reached',
                                  'Max recursion depth reached'),
                                 ('Max recursion depth reached',
                                  'Max recursion depth reached'),
                                 ('Max recursion depth reached',
                                  'Max recursion depth reached')],

The BI rules are not compley - all rules are structured like this:

-Hostnamexy
–Hostname_xy_Network
–Hostname_xy_Systems

Is there anything I can do to fix this or is this a CheckMK issue?

Greetz
Ovrld

2

Hi Ovrld,

when does this error appear,

  • when viewing the resulting BI aggregation in general,
  • when viewing the resulting BI aggregation in a certain way, or
  • when doing something else?

Can you attach the (anonymized) URL of the site, please?

Best regards
Lars

3

Hey Lars

Here are the answers:

  • when viewing the resulting BI aggregation in general
    No - I can see the aggregrations. It’s only when I try to set an aggregation in a downtime. Even when there’s only one host in the aggregation

Here you can see an example of the aggregation:

image
image487×785 66.4 KB

Url:
https://monitoring.domain.de/sitename/check_mk/view.py?view_name=aggr_all_maint&_show_filter_form=0

I have the same when I make a really simple BI with only two hosts in one rule

4

Hi!

Yes, I can reproduce that error in my 2.3.0p21 environment.
I will create an internal ticket for it. Thanks for reporting!

Best regards
Lars

3 Likes
5

Hi Lars,

do you have any news regarding the internal ticket you opened?

Many thanks

1 Like
6

Welcome in the forum, Alessandro,

There’s no update on the ticket, yet. Let’s see, when it will be picked for a sprint.

Best regards
Lars

2 Likes
7

We got two updates but the issue is’nt fixed yet :frowning:

8

Today I upgraded to 2.3.0p25 and the issue still isn’t fixed yet

@lars.getwan
Can you try setting the priority higher? The problem persists since p20 or earlier - and we have a lot of problems in our production environment without BI.

9

Hi!

I increased the priority, but our developers are currently very busy preparing the 2.4.0 release.

Best regards
Lars

10

Hi Lars - is there a werk that is open for this? I submitted a bug report today experiencing the same issue and would like to track it if possible.

Thanks.

11

Hi Brian,

there’s an internal ticket that I created. The ticket will later result into a werk, as soon as it is fixed by the development team.

Best regards
Lars

1 Like
12

Checkmk Enterprise Edition 2.3.0p28 and still no fix here :frowning:

I mentioned this issue with 2.3.0p20 116 days ago (almost 4 months ago) and I mentioned, that this in an important feature :confused:

@lars.getwan - Can you please get any pressure on the ticket to get it solved as soon as possible?

13

Dear @Overlord ,

Please note that ticket prioritisation follows specific processes. A ticket’s priority may increase, for example, if additional relevant information is provided.

If you have a support contract, you may also find it helpful to report the issue directly through the support portal to communicate about it directly.

1 Like
14

Push - I’m on Checkmk Enterprise Edition 2.3.0p34 and it’s still not working (Since 11/24) :frowning:

15 
Internal error: host_name
An internal error occured while processing your request. You can report this issue to the Checkmk team to help fixing this issue.

Since November 11 2024 I’m on it. Still waiting :frowning:

p35… no fix :confused: Enterprise subscription with 12000 Services…

16

p36… still no fix.

I mention this only for people like me who hope that it will be fixed someday.

17

p37… I had hoped because I saw this in the changelog:

BI
* 18362 FIX: Prevent Race-Condition where aggregations could be compiled twice...

But no, issue still persist

18

I put the problem to ChatGPT, which provided a workaround. I implemented and tested the solution within five minutes.

  1. omd su sitename
  2. nano ~/local/lib/python3/cmk/gui/plugins/views/bi_downtime_fix.py
# ~/local/share/check_mk/web/plugins/views/bi_downtime_fix.py

import logging
from cmk.gui.views.command import commands as _cmd

log = logging.getLogger(__name__)

# Idempotenz: nur patchen, wenn noch nicht gepatcht
if not getattr(_cmd.CommandScheduleDowntimes, "_bi_fix_applied", False):
    _orig = _cmd.CommandScheduleDowntimes._downtime_specs

    def _patched(self, cmdtag, row, action_rows, spec):
        try:
            return _orig(self, cmdtag, row, action_rows, spec)
        except KeyError as e:
            # Nur unseren bekannten Fall anfassen
            if str(e) != "'host_name'":
                raise

            # Nur, wenn es BI-Rows sind (aggr_hosts vorhanden)
            has_bi = any("aggr_hosts" in (r or {}) for r in (action_rows or []))
            if not has_bi:
                # Kein BI-Context -> originaler Fehler
                raise

            # Aus BI-Zeilen aggr_hosts -> (site, host) ableiten
            hosts = set()
            for r in action_rows or []:
                for site, host in r.get("aggr_hosts", []):
                    if host:
                        hosts.add((site, host))

            if not hosts:
                # Nichts ableitbar -> originaler Fehler
                raise

            patched_rows = [{"site": s, "host_name": h} for (s, h) in hosts]
            return _orig(self, cmdtag, row, patched_rows, spec)

    _cmd.CommandScheduleDowntimes._downtime_specs = _patched
    _cmd.CommandScheduleDowntimes._bi_fix_applied = True
    log.warning("bi_downtime_fix loaded: BI downtime host expansion active")
  1. omd restart apache

After this the BI is working again.

I do not guarantee the functionality of this patch or take responsibility for any issues it may cause. I am not a developer, and certainly not a Python developer. I simply provided the errors from the crash report to ChatGPT, and ChatGPT immediately generated the code as a solution.

The fix should be verified after every upgrade and removed once the CheckMK team has resolved the issue.

Finally fixed… I thought I was going to hit a full year of waiting x)

19

Same Problem after updating 2.4.0p10 to 2.4.0p14 today. Our BI is broken now. So this bug is still not completely fixed :confused:

20

I can confirm that: The BI is now completely broken. That’s quite an achievement — managing to go an entire year without releasing a fix for a problem, only to end up making it stop working altogether.

I tested it on a CheckMK instance without my BI fix just to be sure — turns out it wasn’t me who broke it, it was the CheckMK team.

I rolled back to p38 and will wait until it’s fixed — or I’ll fix it myself.

Exception: KeyError (site)

Traceback:

 File "/omd/sites/sitename/lib/python3/cmk/gui/wsgi/applications/checkmk.py", line 188, in _process_request
    resp = page_handler()
  File "/omd/sites/sitename/lib/python3/cmk/gui/wsgi/applications/utils.py", line 113, in _call_auth
    func()
  File "/omd/sites/sitename/lib/python3/cmk/gui/pages.py", line 187, in wrapper
    return hc().handle_page()
  File "/omd/sites/sitename/lib/python3/cmk/gui/pages.py", line 51, in handle_page
    self.page()
  File "/omd/sites/sitename/lib/python3/cmk/gui/pages.py", line 146, in <lambda>
    "page": lambda self: self._wrapped_callable[0](),
  File "/omd/sites/sitename/lib/python3/cmk/gui/views/page_show_view.py", line 100, in page_show_view
    process_view(GUIViewRenderer(view, show_buttons=True))
  File "/omd/sites/sitename/lib/python3/cmk/gui/views/page_show_view.py", line 174, in process_view
    _process_regular_view(view_renderer)
  File "/omd/sites/sitename/lib/python3/cmk/gui/views/page_show_view.py", line 192, in _process_regular_view
    _show_view(view_renderer, unfiltered_amount_of_rows, rows)
  File "/omd/sites/sitename/lib/python3/cmk/gui/views/page_show_view.py", line 412, in _show_view
    view_renderer.render(
  File "/omd/sites/sitename/lib/python3/cmk/gui/view_renderer.py", line 271, in render
    layout.render(
  File "/omd/sites/sitename/lib/python3/cmk/gui/views/layout/layouts.py", line 773, in render
    cell.paint(row, render_link_to_view)
  File "/omd/sites/sitename/lib/python3/cmk/gui/painter/v0/base.py", line 699, in paint
    tdclass, content = self.render(row, link_renderer)
  File "/omd/sites/sitename/lib/python3/cmk/gui/painter/v0/base.py", line 537, in render
    tdclass, content = self.render_content(row)
  File "/omd/sites/sitename/lib/python3/cmk/gui/painter/v0/base.py", line 691, in render_content
    return painter.render(row, self)
  File "/omd/sites/sitename/lib/python3/cmk/gui/bi/view.py", line 852, in render
    return paint_aggregated_tree_state(row)
  File "/omd/sites/sitename/lib/python3/cmk/gui/bi/view.py", line 829, in paint_aggregated_tree_state
    return renderer.css_class(), renderer.render()
  File "/omd/sites/sitename/lib/python3/cmk/gui/bi/foldable_tree_renderer.py", line 67, in render
    self._show_tree()
  File "/omd/sites/sitename/lib/python3/cmk/gui/bi/foldable_tree_renderer.py", line 96, in _show_tree
    self._show_subtree(
  File "/omd/sites/sitename/lib/python3/cmk/gui/bi/foldable_tree_renderer.py", line 248, in _show_subtree
    with self._show_node(tree, show_host, mousecode=mc, img_class=css_class):
  File "/omd/sites/sitename/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/omd/sites/sitename/lib/python3/cmk/gui/bi/foldable_tree_renderer.py", line 387, in _show_node
    must_escape=not active_config.sites[self._row["site"]].get("is_trusted", False),

Local Variables:

{'addclass': [],
 'class_': ['content', 'state', 'state0'],
 'effective_state': {'acknowledged': False,
                     'in_downtime': False,
                     'in_service_period': True,
                     'output': '',
                     'state': 0},
 'icon_name': None,
 'icon_title': None,
 'img_class': 'closed',
 'mousecode': 'cmk.bi.toggle_subtree(this, 1);',
 'self': <cmk.gui.bi.foldable_tree_renderer.FoldableTreeRendererTree object at 0x7f9aa2ca3a70>,
 'show_host': False,
 'tree': ({'acknowledged': False,
           'in_downtime': False,
           'in_service_period': True,
           'output': '',
           'state': 0},
          None,
          {'aggr_group_tree': ['Hosts'],
           'aggr_type': 'multi',
           'aggregation_id': 'default_aggregation',
           'docu_url': '',
           'downtime_aggr_warn': False,
           'frozen_marker': None,
           'node_visualization': {'ignore_rule_styles': False,
                                  'layout_id': 'builtin_default',
                                  'line_style': 'round'},
           'nodes': [{'docu_url': 'Max recursion depth reached',
                      'frozen_marker': 'Max recursion depth reached',
                      'nodes': 'Max recursion depth reached',
                      'reqhosts': 'Max recursion depth reached',
                      'rule_id': 'Max recursion depth reached',
                      'rule_layout_style': 'Max recursion depth reached',
                      'title': 'Max recursion depth reached',
                      'type': 'Max recursion depth reached'},
                     {'docu_url': 'Max recursion depth reached',
                      'frozen_marker': 'Max recursion depth reached',
                      'nodes': 'Max recursion depth reached',
                      'reqhosts': 'Max recursion depth reached',
                      'rule_id': 'Max recursion depth reached',
                      'rule_layout_style': 'Max recursion depth reached',
                      'title': 'Max recursion depth reached',
                      'type': 'Max recursion depth reached'},
                     {'docu_url': 'Max recursion depth reached',
                      'frozen_marker': 'Max recursion depth reached',
                      'nodes': 'Max recursion depth reached',
                      'reqhosts': 'Max recursion depth reached',
                      'rule_id': 'Max recursion depth reached',
                      'rule_layout_style': 'Max recursion depth reached',
                      'title': 'Max recursion depth reached',
                      'type': 'Max recursion depth reached'},
                     {'docu_url': 'Max recursion depth reached',
                      'frozen_marker': 'Max recursion depth reached',
                      'nodes': 'Max recursion depth reached',
                      'reqhosts': 'Max recursion depth reached',
                      'rule_id': 'Max recursion depth reached',
                      'rule_layout_style': 'Max recursion depth reached',
                      'title': 'Max recursion depth reached',
                      'type': 'Max recursion depth reached'},
                     {'docu_url': 'Max recursion depth reached',
                      'frozen_marker': 'Max recursion depth reached',
                      'nodes': 'Max recursion depth reached',
                      'reqhosts': 'Max recursion depth reached',
                      'rule_id': 'Max recursion depth reached',
                      'rule_layout_style': 'Max recursion depth reached',
                      'title': 'Max recursion depth reached',
                      'type': 'Max recursion depth reached'}],
           'reqhosts': [('Max recursion depth reached',
                         'Max recursion depth reached')],
           'rule_id': 'host',
           'rule_layout_style': {'style_config': {}, 'type': 'none'},
           'title': 'Host 2k19nfs',
           'type': 2,
           'use_hard_states': False},
          [({'acknowledged': 'Max recursion depth reached',
             'in_downtime': 'Max recursion depth reached',
             'in_service_period': 'Max recursion depth reached',
             'output': 'Max recursion depth reached',
             'state': 'Max recursion depth reached'},
            None,
            {'docu_url': 'Max recursion depth reached',
             'frozen_marker': 'Max recursion depth reached',
             'nodes': 'Max recursion depth reached',
             'reqhosts': 'Max recursion depth reached',
             'rule_id': 'Max recursion depth reached',
             'rule_layout_style': 'Max recursion depth reached',
             'title': 'Max recursion depth reached',
             'type': 'Max recursion depth reached'},
            ['Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached']),
           ({'acknowledged': 'Max recursion depth reached',
             'in_downtime': 'Max recursion depth reached',
             'in_service_period': 'Max recursion depth reached',
             'output': 'Max recursion depth reached',
             'state': 'Max recursion depth reached'},
            None,
            {'docu_url': 'Max recursion depth reached',
             'frozen_marker': 'Max recursion depth reached',
             'nodes': 'Max recursion depth reached',
             'reqhosts': 'Max recursion depth reached',
             'rule_id': 'Max recursion depth reached',
             'rule_layout_style': 'Max recursion depth reached',
             'title': 'Max recursion depth reached',
             'type': 'Max recursion depth reached'},
            ['Max recursion depth reached', 'Max recursion depth reached']),
           ({'acknowledged': 'Max recursion depth reached',
             'in_downtime': 'Max recursion depth reached',
             'in_service_period': 'Max recursion depth reached',
             'output': 'Max recursion depth reached',
             'state': 'Max recursion depth reached'},
            None,
            {'docu_url': 'Max recursion depth reached',
             'frozen_marker': 'Max recursion depth reached',
             'nodes': 'Max recursion depth reached',
             'reqhosts': 'Max recursion depth reached',
             'rule_id': 'Max recursion depth reached',
             'rule_layout_style': 'Max recursion depth reached',
             'title': 'Max recursion depth reached',
             'type': 'Max recursion depth reached'},
            ['Max recursion depth reached']),
           ({'acknowledged': 'Max recursion depth reached',
             'in_downtime': 'Max recursion depth reached',
             'in_service_period': 'Max recursion depth reached',
             'output': 'Max recursion depth reached',
             'state': 'Max recursion depth reached'},
            None,
            {'docu_url': 'Max recursion depth reached',
             'frozen_marker': 'Max recursion depth reached',
             'nodes': 'Max recursion depth reached',
             'reqhosts': 'Max recursion depth reached',
             'rule_id': 'Max recursion depth reached',
             'rule_layout_style': 'Max recursion depth reached',
             'title': 'Max recursion depth reached',
             'type': 'Max recursion depth reached'},
            ['Max recursion depth reached', 'Max recursion depth reached']),
           ({'acknowledged': 'Max recursion depth reached',
             'in_downtime': 'Max recursion depth reached',
             'in_service_period': 'Max recursion depth reached',
             'output': 'Max recursion depth reached',
             'state': 'Max recursion depth reached'},
            None,
            {'docu_url': 'Max recursion depth reached',
             'frozen_marker': 'Max recursion depth reached',
             'nodes': 'Max recursion depth reached',
             'reqhosts': 'Max recursion depth reached',
             'rule_id': 'Max recursion depth reached',
             'rule_layout_style': 'Max recursion depth reached',
             'title': 'Max recursion depth reached',
             'type': 'Max recursion depth reached'},
            ['Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached',
             'Max recursion depth reached'])])}