Other Integrations(RabbitMQ) not working

mommalongnips · April 25, 2023, 10:18pm

We are running CMK Enterprise 2.1.0p25 on RHEL 8. We are having issues getting the RabbitMQ plugin to work at all.

The rule is configured under Setup → Agent → Other Integrations → RabbitMQ as follows:

A TCPDump from the host shows that it isn’t even trying to connect to the port of the rabbit server

The host is configured to ‘Configured API Integrations and CheckMK Agent’
Here is the output of the cmk --debug -vnn hostname with some information redacted:

Checkmk version 2.1.0p25
Try license usage history update.
Trying to acquire lock on /omd/sites/aces/var/check_mk/license_usage/next_run
Got lock on /omd/sites/aces/var/check_mk/license_usage/next_run
Trying to acquire lock on /omd/sites/aces/var/check_mk/license_usage/history.json
Got lock on /omd/sites/aces/var/check_mk/license_usage/history.json
Next run time has not been reached yet. Abort.
Releasing lock on /omd/sites/aces/var/check_mk/license_usage/history.json
Released lock on /omd/sites/aces/var/check_mk/license_usage/history.json
Releasing lock on /omd/sites/aces/var/check_mk/license_usage/next_run
Released lock on /omd/sites/aces/var/check_mk/license_usage/next_run
+ FETCHING DATA
  Source: SourceType.HOST/FetcherType.TCP
[cpu_tracking] Start [7fdc3eecef70]
[TCPFetcher] Fetch with cache settings: DefaultAgentFileCache(**REDACTED**, base_path=/omd/sites/aces/tmp/check_mk/cache, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=False, use_outdated=False, simulation=False)
Not using cache (Too old. Age is 22 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to **REDACTED**:6556 (5.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b'16')
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Write data to cache file /omd/sites/aces/tmp/check_mk/cache/**REDACTED**
Trying to acquire lock on /omd/sites/aces/tmp/check_mk/cache/**REDACTED**
Got lock on /omd/sites/aces/tmp/check_mk/cache/**REDACTED**
Releasing lock on /omd/sites/aces/tmp/check_mk/cache/**REDACTED**
Released lock on /omd/sites/aces/tmp/check_mk/cache/**REDACTED**
Closing TCP connection to **REDACTED**:6556
[cpu_tracking] Stop [7fdc3eecef70 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=1.4500000001862645))]
  Source: SourceType.HOST/FetcherType.PIGGYBACK
[cpu_tracking] Start [7fdc3eece580]
[PiggybackFetcher] Fetch with cache settings: NoCache(**REDACTED**, base_path=/omd/sites/aces/tmp/check_mk/data_source_cache/piggyback, max_age=MaxAge(checking=0, discovery=120, inventory=120), disabled=True, use_outdated=False, simulation=False)
Not using cache (Cache usage disabled)
[PiggybackFetcher] Execute data source
No piggyback files for '**REDACTED**'. Skip processing.
No piggyback files for '**REDACTED**'. Skip processing.
Not using cache (Cache usage disabled)
[cpu_tracking] Stop [7fdc3eece580 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
  Source: SourceType.HOST/FetcherType.TCP
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<df>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq>>> / Transition HostSectionParser -> HostSectionParser
<<<postfix_mailq_status:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1682459575,120)>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
No persisted sections
  -> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'chrony', 'cifsmounts', 'cmk_agent_ctl_status', 'cpu', 'df', 'diskstat', 'job', 'kernel', 'labels', 'lnx_if', 'local', 'md', 'mem', 'mounts', 'nfsmounts', 'postfix_mailq', 'postfix_mailq_status', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  Source: SourceType.HOST/FetcherType.PIGGYBACK
No persisted sections
  -> Add sections: []
Received no piggyback data
Received no piggyback data
[cpu_tracking] Start [7fdc3eeb6100]
value store: synchronizing
Trying to acquire lock on /omd/sites/aces/tmp/check_mk/counters/**REDACTED**
Got lock on /omd/sites/aces/tmp/check_mk/counters/**REDACTED**
value store: loading from disk
Releasing lock on /omd/sites/aces/tmp/check_mk/counters/**REDACTED**
Released lock on /omd/sites/aces/tmp/check_mk/counters/**REDACTED**
CPU load             15 min load: 3.04, 15 min load per core: 0.25 (12 cores)
CPU utilization      Total CPU: 24.88%
Check_MK Agent       Version: 2.1.0p20, OS: linux, Agent plugins: 1, Local checks: 0
Disk IO SUMMARY      Read: 0.00 B/s, Write: 3.08 MB/s, Latency: 464 microseconds
Filesystem /         27.86% used (10.31 of 37.02 GB), trend: +25.06 MB / 24 hours
Filesystem /boot     24.99% used (253.39 of 1014.00 MB), trend: -8.51 MB / 24 hours
Filesystem /home     0.18% used (32.55 MB of 18.07 GB), trend: 0.00 B / 24 hours
Interface ens192     [2], (up), MAC: 00:50:56:B2:4A:80, Speed: 10 GBit/s, In: 4.26 MB/s (0.34%), Out: 4.19 MB/s (0.34%)
Interface lo         [1], (up), MAC: 00:00:00:00:00:00, Speed: unknown, In: 0.00 B/s, Out: 0.00 B/s
Kernel Performance   Process Creations: 6.25/s, Context Switches: 45467.83/s, Major Page Faults: 0.00/s, Page Swap in: 0.00/s, Page Swap Out: 0.00/s
Memory               Total virtual memory: 24.49% - 3.76 GB of 15.37 GB, 9 additional details available
NTP Time             Offset: 0.0457 ms, Stratum: 3, Time since last sync: 1 minute 39 seconds
Number of threads    697, Usage: 0.74%
Postfix Queue        Deferred queue length: 0, Active queue length: 0
Systemd Service Summary Total: 119, Disabled: 6, Failed: 0
TCP Connections      Established: 348
Uptime               Up since Mar 28 2023 22:50:37, Uptime: 27 days 23 hours
No piggyback files for '**REDACTED**'. Skip processing.
No piggyback files for '**REDACTED**'. Skip processing.
[cpu_tracking] Stop [7fdc3eeb6100 - Snapshot(process=posix.times_result(user=0.05999999999999983, system=0.010000000000000009, children_user=0.0, children_system=0.0, elapsed=0.07000000029802322))]
[agent] Success, execution time 1.5 sec | execution_time=1.520 user_time=0.070 system_time=0.010 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=1.440

Any thoughts on what might be happening? Thanks in advanced!

gstolz · April 26, 2023, 6:30am

Can you check what “cmk -D ” says? It looks like the rabbitMQ rule is not being picked up, but from your screenshot, I’m not sure how that could happen.

Maybe the rabbitMQ Instance (first checkbox) is actually mandatory?

mommalongnips · April 26, 2023, 3:34pm

@gstolz Thanks for the reply!

When running a cmk -D it spits out tons of information but nothing about rabbitmq. When filtering the results for rabbitmq or rabbit I don’t see any relevant information. I can post the results if you’d like!

I did try putting something in the first box with no luck

Thanks!

gstolz · April 27, 2023, 6:22am

Thought about this for a while, because your rule looked good + you said that the host is configured to use ‘Configured API Integrations and CheckMK Agent’ so it made no sense, that the rabbitmq integration is not used.

Then I saw it

your rule is deactivated^^

mommalongnips · April 27, 2023, 3:47pm

You’re my hero!

I feel like such an idiot. I’ve been using CheckMK for 7+ years and I have no idea how I overlooked that or how I accidentally checked it in the first place…

Works like a charm! I spent days digging through logs and nothing made any sense.

I need to buy you a beer!

system · April 26, 2024, 3:47pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.