Mk_oracle plugin timing out for one database but not the other

ianmurphy · November 29, 2023, 11:42am

Hi there community,

Have an mk_oracle plugin running on one of our database servers. The server has two databases on it and one reports fine in the dashboard, the other times out. I can’t see any obvious reasons why (database and server resources are good and on another DB server with two databases they both return data with no issues).

Any troubleshooting advice appreciated!

–

CMK version: 2.2.0p8 (raw edition)
OS version: Red Hat 9

Error message: Timed out plugin(s): (!)

Output of “cmk --debug -vvn hostname”:

OMD[stlnagios]:~$ cmk --debug -vvn <database server>
Checkmk version 2.2.0p8
+ FETCHING DATA
  Source: SourceInfo(hostname='<database server>', ipaddress='192.168.11.20', ident='agent', fetcher_type=<FetcherType.TCP: 8>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f06655ea790]
Read from cache: AgentFileCache(<database server>, path_template=/omd/sites/stlnagios/tmp/check_mk/cache/{hostname}, max_age=MaxAge(checking=0, discovery=120, inventory=120), simulation=False, use_only_cache=False, file_cache_mode=6)
Not using cache (Too old. Age is 400 sec, allowed is 0 sec)
[TCPFetcher] Execute data source
Connecting via TCP to 192.168.11.20:6556 (15.0s timeout)
Detected transport protocol: TransportProtocol.TLS (b'16')
Reading data from agent via TLS socket
Reading data from agent
Detected transport protocol: TransportProtocol.PLAIN (b'<<')
Closing TCP connection to 192.168.11.20:6556
Write data to cache file /omd/sites/stlnagios/tmp/check_mk/cache/<database server>
Trying to acquire lock on /omd/sites/stlnagios/tmp/check_mk/cache/<database server>
Got lock on /omd/sites/stlnagios/tmp/check_mk/cache/<database server>
Releasing lock on /omd/sites/stlnagios/tmp/check_mk/cache/<database server>
Released lock on /omd/sites/stlnagios/tmp/check_mk/cache/<database server>
[cpu_tracking] Stop [7f06655ea790 - Snapshot(process=posix.times_result(user=0.010000000000000009, system=0.0, children_user=0.0, children_system=0.0, elapsed=7.25))]
  Source: SourceInfo(hostname='<database server>', ipaddress='192.168.11.20', ident='piggyback', fetcher_type=<FetcherType.PIGGYBACK: 4>, source_type=<SourceType.HOST: 1>)
[cpu_tracking] Start [7f06655f8590]
Read from cache: NoCache(<database server>, path_template=/dev/null, max_age=MaxAge(checking=0.0, discovery=0.0, inventory=0.0), simulation=False, use_only_cache=False, file_cache_mode=1)
[PiggybackFetcher] Execute data source
No piggyback files for '<database server>'. Skip processing.
No piggyback files for '192.168.11.20'. Skip processing.
[cpu_tracking] Stop [7f06655f8590 - Snapshot(process=posix.times_result(user=0.0, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.0))]
+ PARSE FETCHER RESULTS
<<<check_mk>>> / Transition NOOPParser -> HostSectionParser
<<<cmk_agent_ctl_status:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_agent_plugins_lnx:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<labels:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<df_v2>>> / Transition HostSectionParser -> HostSectionParser
<<<systemd_units>>> / Transition HostSectionParser -> HostSectionParser
<<<nfsmounts_v2:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<cifsmounts>>> / Transition HostSectionParser -> HostSectionParser
<<<mounts>>> / Transition HostSectionParser -> HostSectionParser
<<<ps_lnx>>> / Transition HostSectionParser -> HostSectionParser
<<<mem>>> / Transition HostSectionParser -> HostSectionParser
<<<cpu>>> / Transition HostSectionParser -> HostSectionParser
<<<uptime>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if>>> / Transition HostSectionParser -> HostSectionParser
<<<lnx_if:sep(58)>>> / Transition HostSectionParser -> HostSectionParser
<<<tcp_conn_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<multipath>>> / Transition HostSectionParser -> HostSectionParser
<<<diskstat>>> / Transition HostSectionParser -> HostSectionParser
<<<kernel>>> / Transition HostSectionParser -> HostSectionParser
<<<md>>> / Transition HostSectionParser -> HostSectionParser
<<<vbox_guest>>> / Transition HostSectionParser -> HostSectionParser
<<<job>>> / Transition HostSectionParser -> HostSectionParser
<<<chrony:cached(1701257744,120)>>> / Transition HostSectionParser -> HostSectionParser
<<<mrpe>>> / Transition HostSectionParser -> HostSectionParser
<<<mrpe>>> / Transition HostSectionParser -> HostSectionParser
<<<local:sep(0)>>> / Transition HostSectionParser -> HostSectionParser
<<<logwatch>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_instance>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_sessions>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_logswitches>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_undostat>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_recovery_area>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_processes>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_recovery_status>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_longactivesessions>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_dataguard_stats>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_performance>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_locks>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_systemparameter>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_tablespaces>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_rman>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_jobs>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_resumable>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_iostats>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_instance>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_processes>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_asm_diskgroup>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_instance:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_sessions:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_logswitches:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_undostat:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_recovery_area:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_processes:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_recovery_status:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_longactivesessions:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_dataguard_stats:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_performance:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_locks:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_systemparameter:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_tablespaces:cached(1701257343,0):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_rman:cached(1701257343,0):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_jobs:cached(1701257343,0):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_resumable:cached(1701257343,0):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_performance:cached(1701257343,0):sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_instance:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_sessions:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_logswitches:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_undostat:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_recovery_area:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_processes:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_recovery_status:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_longactivesessions:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_dataguard_stats:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_performance:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_locks:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<oracle_systemparameter:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
<<<checkmk_cached_plugins:sep(124)>>> / Transition HostSectionParser -> HostSectionParser
  HostKey(hostname='<database server>', source_type=<SourceType.HOST: 1>)  -> Add sections: ['check_mk', 'checkmk_agent_plugins_lnx', 'checkmk_cached_plugins', 'chrony', 'cifsmounts', 'cmk_agent_ctl_status', 'cpu', 'df_v2', 'diskstat', 'job', 'kernel', 'labels', 'lnx_if', 'local', 'logwatch', 'md', 'mem', 'mounts', 'mrpe', 'multipath', 'nfsmounts_v2', 'oracle_asm_diskgroup', 'oracle_dataguard_stats', 'oracle_instance', 'oracle_iostats', 'oracle_jobs', 'oracle_locks', 'oracle_logswitches', 'oracle_longactivesessions', 'oracle_performance', 'oracle_processes', 'oracle_recovery_area', 'oracle_recovery_status', 'oracle_resumable', 'oracle_rman', 'oracle_sessions', 'oracle_systemparameter', 'oracle_tablespaces', 'oracle_undostat', 'ps_lnx', 'systemd_units', 'tcp_conn_stats', 'uptime', 'vbox_guest']
  HostKey(hostname='<database server>', source_type=<SourceType.HOST: 1>)  -> Add sections: []
Received no piggyback data
[cpu_tracking] Start [7f0664994390]
value store: synchronizing
Trying to acquire lock on /omd/sites/stlnagios/tmp/check_mk/counters/<database server>
Got lock on /omd/sites/stlnagios/tmp/check_mk/counters/<database server>
value store: loading from disk
Releasing lock on /omd/sites/stlnagios/tmp/check_mk/counters/<database server>
Released lock on /omd/sites/stlnagios/tmp/check_mk/counters/<database server>
CPU load             15 min load: 0.25, 15 min load per core: 0.12 (2 cores)
CPU utilization      Total CPU: 4.96%
Check_MK Agent       Version: 2.2.0p8, OS: linux, Agent plugins: 2, Local checks: 0, Timed out plugin(s): <failing database>(!)
Disk IO SUMMARY      Read: 318 kB/s, Write: 131 kB/s, Latency: 515 microseconds
Filesystem /         Used: 15.20% - 4.09 GiB of 26.9 GiB, trend per 1 day 0 hours: -4.12 MiB, trend per 1 day 0 hours: -0.01%
Filesystem /boot     Used: 26.26% - 266 MiB of 1014 MiB, trend per 1 day 0 hours: +0 B, trend per 1 day 0 hours: +0%
Filesystem /ora01    Used: 27.64% - 8.13 GiB of 29.4 GiB, trend per 1 day 0 hours: +78.9 MiB, trend per 1 day 0 hours: +0.26%, Time left until disk full: 276 days 2 hours
Filesystem /ora01/arch Used: 19.89% - 5.85 GiB of 29.4 GiB, trend per 1 day 0 hours: +2.25 GiB, trend per 1 day 0 hours: +7.65%, Time left until disk full: 10 days 11 hours
Filesystem /ora01/oradata Used: 58.07% - 99.4 GiB of 171 GiB, trend per 1 day 0 hours: +685 MiB, trend per 1 day 0 hours: +0.39%, Time left until disk full: 107 days 8 hours
Interface ens5       [2], (up), MAC: 12:5C:06:15:B7:7B, Speed: unknown, In: 38.0 kB/s (0.03%/125 MB/s), Out: 41.8 kB/s (0.03%/125 MB/s)
Kernel Performance   Process Creations: 4.73/s, Context Switches: 5748.85/s, Major Page Faults: 24.05/s, Page Swap in: 41.01/s, Page Swap Out: 0.00/s
Log /ora01/diag/rdbms/<working database>/<working database>/trace/alert_<working database>.log No error messages
Log /ora01/diag/rdbms/<failing database>/<failing database>/trace/alert_<failing database>.log No error messages
Log /var/log/messages 1 CRIT messages (Last worst: "Nov 29 03:32:09 <database server> amazon-ssm-agent[1283]: 2023-11-29 03:32:09 INFO [EC2Identity] Checking disk for registration info")
Log /var/log/secure  No error messages
Log /var/log/sync/s3_sync_<working database>_arl_backups.log No error messages
Log /var/log/sync/s3_sync_<working database>_db_backups.log No error messages
Log /var/log/sync/s3_sync_<failing database>_arl_backups.log No error messages
Log /var/log/sync/s3_sync_<failing database>_db_backups.log No error messages
Memory               Total virtual memory: 4.87% - 1.23 GiB of 25.4 GiB, 9 additional details available
Mount options of /   Mount options exactly as expected
Mount options of /boot Mount options exactly as expected
Mount options of /ora01 Mount options exactly as expected
Mount options of /ora01/arch Mount options exactly as expected
Mount options of /ora01/oradata Mount options exactly as expected
NFS mount /mnt/oracle/backup Source: 127.0.0.1:/, Used: <0.01% - 812 GiB of 8.00 EiB, trend per 1 day 0 hours: +124 GiB, trend per 1 day 0 hours: +<0.01%, Time left until disk full: 189719 years 310 days
NTP Time             Offset: 0.0214 ms, Stratum: 5, Time since last sync: 5 minutes 36 seconds
Number of threads    312, Usage: 0.25%
ORA <working database> Instance Database Name <working database>, Status OPEN, Role PRIMARY, Version 19.3.0.0.0, Logins allowed, Log Mode archivelog, Force Logging no
ORA <working database> Locks   No locks existing
ORA <working database> Logswitches 3 log switches in the last 60 minutes (warn/crit below -1/-1) (warn/crit at 50/100)
ORA <working database> Long Active Sessions 0
ORA <working database> Performance DB Time: 0.02/s, DB CPU: 0.01/s, DB Non-Idle Wait: 0.01/s, Maximum SGA Size: 2.00 GiB, Buffer Cache Size: 1.50 GiB, Shared Pool Size: 480 MiB, Redo Buffers: 7.51 MiB, Java Pool Size: 0 B, Large Pool Size: 16.0 MiB, Streams Pool Size: 0 B, Shared IO Pool Size: 112 MiB, Buffer hit ratio: 99.1%, Library cache hit ratio: 94.0%
ORA <working database> Processes 67 of 1500 processes are used: 4.47%
ORA <working database> Recovery Area 0 B out of 0 B used (0.0%, warn/crit at 70.0%/90.0%), 0 B reclaimable
ORA <working database> Recovery Status primary database, oldest Checkpoint 39 minutes 35 seconds ago
ORA <working database> Sessions 2.82%, 64 of 2272 sessions
ORA <working database> Undo Retention Undo retention: 44 minutes 5 seconds, Active undo blocks: 1152, Max concurrent transactions: 2, Max querylen: 31 minutes 4 seconds, Space errors: 0
ORA <working database> Uptime  Up since 2023-11-29 09:40:50, uptime: 1:55:02
ORA <working database>.AGBIP_BIPLATFORM Tablespace ONLINE (PERMANENT), Size: 128 MiB, 0.14% used (44.9 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.AGBIP_IAS_OPSS Tablespace ONLINE (PERMANENT), Size: 60.0 MiB, 0.17% used (56.1 MiB of max. 32.0 GiB), Free: 31.9 GiB, Space left: 31.9 GiB
ORA <working database>.AGBIP_IAS_TEMP Tablespace ONLINE (TEMPORARY), Size: 100 MiB, 0% used (0 B of max. 32.0 GiB), Free: 32.0 GiB
ORA <working database>.AGBIP_IAU Tablespace ONLINE (PERMANENT), Size: 60.0 MiB, 0.15% used (48.4 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.AGBIP_MDS Tablespace ONLINE (PERMANENT), Size: 100 MiB, 0.04% used (12.3 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.AGBIP_STB Tablespace ONLINE (PERMANENT), Size: 10.0 MiB, <0.01% used (1.88 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.AGBIP_WLS Tablespace ONLINE (PERMANENT), Size: 60.0 MiB, 0.03% used (9.75 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.ARCHIVELOG RMAN Backup Time since last backup: 35 days 9 hours
ORA <working database>.ARGUS_AEXP_DATA_01 Tablespace ONLINE (PERMANENT), Size: 640 MiB, 1.57% used (514 MiB of max. 32.0 GiB), Free: 31.5 GiB, Space left: 31.5 GiB
ORA <working database>.ARGUS_AEXP_INDEX_01 Tablespace ONLINE (PERMANENT), Size: 128 MiB, 0.01% used (4.00 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.ARGUS_AL_DATA_01 Tablespace ONLINE (PERMANENT), Size: 1.00 GiB, 2.63% used (861 MiB of max. 32.0 GiB), Free: 31.2 GiB, Space left: 31.2 GiB
ORA <working database>.ARGUS_AL_INDEX_01 Tablespace ONLINE (PERMANENT), Size: 128 MiB, 0.08% used (26.0 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.ARGUS_APP.DBMS_JOB_1 Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 2 seconds, Next Run: 30-NOV-23 09.40.58.000000 AM +00:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.ARGUS_APP.DBMS_JOB_2 Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 11.30.58.000000 AM +00:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.ARGUS_APP.DBMS_JOB_47 Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 09.40.59.000000 PM +00:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.ARGUS_APP.DBMS_JOB_67 Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 6 seconds, Next Run: 30-NOV-23 09.40.59.000000 AM +00:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.ARGUS_DATA_01 Tablespace ONLINE (PERMANENT), Size: 18.2 GiB, 53.91% used (17.3 GiB of max. 32.0 GiB), Free: 14.7 GiB, Space left: 14.7 GiB
ORA <working database>.ARGUS_DATA_02 Tablespace ONLINE (PERMANENT), Size: 512 MiB, 1.23% used (404 MiB of max. 32.0 GiB), Free: 31.6 GiB, Space left: 31.6 GiB
ORA <working database>.ARGUS_DATA_03 Tablespace ONLINE (PERMANENT), Size: 256 MiB, 0.70% used (228 MiB of max. 32.0 GiB), Free: 31.8 GiB, Space left: 31.8 GiB
ORA <working database>.ARGUS_DATA_04 Tablespace ONLINE (PERMANENT), Size: 640 MiB, 1.56% used (510 MiB of max. 32.0 GiB), Free: 31.5 GiB, Space left: 31.5 GiB
ORA <working database>.ARGUS_DATA_05 Tablespace ONLINE (PERMANENT), Size: 384 MiB, 0.65% used (212 MiB of max. 32.0 GiB), Free: 31.8 GiB, Space left: 31.8 GiB
ORA <working database>.ARGUS_INDEX_01 Tablespace ONLINE (PERMANENT), Size: 512 MiB, 1.36% used (446 MiB of max. 32.0 GiB), Free: 31.6 GiB, Space left: 31.6 GiB
ORA <working database>.ARGUS_INDEX_02 Tablespace ONLINE (PERMANENT), Size: 256 MiB, 0.25% used (81.3 MiB of max. 32.0 GiB), Free: 31.9 GiB, Space left: 31.9 GiB
ORA <working database>.ARGUS_INDEX_03 Tablespace ONLINE (PERMANENT), Size: 256 MiB, 0.38% used (125 MiB of max. 32.0 GiB), Free: 31.9 GiB, Space left: 31.9 GiB
ORA <working database>.ARGUS_INDEX_04 Tablespace ONLINE (PERMANENT), Size: 128 MiB, 0.32% used (105 MiB of max. 32.0 GiB), Free: 31.9 GiB, Space left: 31.9 GiB
ORA <working database>.ARGUS_INDEX_05 Tablespace ONLINE (PERMANENT), Size: 256 MiB, 0.42% used (139 MiB of max. 32.0 GiB), Free: 31.9 GiB, Space left: 31.9 GiB
ORA <working database>.ARGUS_INDEX_06 Tablespace ONLINE (PERMANENT), Size: 128 MiB, 0.32% used (104 MiB of max. 32.0 GiB), Free: 31.9 GiB, Space left: 31.9 GiB
ORA <working database>.BIP_OWNER.DBMS_JOB_65 Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 11.31.00.000000 AM +00:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.BIP_OWNER.DBMS_JOB_66 Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 11.31.00.000000 AM +00:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.BIP_OWNER.DBMS_JOB_68 Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 11.31.00.000000 AM +00:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.BIP_OWNER.DBMS_JOB_69 Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 11.31.00.000000 AM +00:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.CONTROLFILE RMAN Backup Time since last backup: 35 days 9 hours
ORA <working database>.DB_INCR_0 RMAN Backup Time since last backup: 38 days 9 hours
ORA <working database>.DB_INCR_1 RMAN Backup Time since last backup: 35 days 9 hours, Incremental SCN 1454837387
ORA <working database>.DLP_DATA_01 Tablespace ONLINE (PERMANENT), Size: 40.0 MiB, 0.06% used (19.3 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.DLP_DATA_02 Tablespace ONLINE (PERMANENT), Size: 140 MiB, 0.37% used (121 MiB of max. 32.0 GiB), Free: 31.9 GiB, Space left: 31.9 GiB
ORA <working database>.DLP_DATA_03 Tablespace ONLINE (PERMANENT), Size: 14.7 GiB, 43.78% used (14.0 GiB of max. 32.0 GiB), Free: 18.0 GiB, Space left: 18.0 GiB
ORA <working database>.DLP_DATA_04 Tablespace ONLINE (PERMANENT), Size: 10.0 MiB, <0.01% used (1.12 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.DLP_DATA_05 Tablespace ONLINE (PERMANENT), Size: 10.0 MiB, 0.01% used (3.38 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.DLP_INDEX_01 Tablespace ONLINE (PERMANENT), Size: 150 MiB, 0.43% used (142 MiB of max. 32.0 GiB), Free: 31.9 GiB, Space left: 31.9 GiB
ORA <working database>.DLP_INDEX_02 Tablespace ONLINE (PERMANENT), Size: 80.0 MiB, 0.21% used (67.9 MiB of max. 32.0 GiB), Free: 31.9 GiB, Space left: 31.9 GiB
ORA <working database>.DLP_INDEX_03 Tablespace ONLINE (PERMANENT), Size: 30.0 MiB, 0.08% used (27.8 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.DLP_INDEX_04 Tablespace ONLINE (PERMANENT), Size: 40.0 MiB, 0.10% used (32.1 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.DLP_INDEX_05 Tablespace ONLINE (PERMANENT), Size: 60.0 MiB, 0.16% used (51.5 MiB of max. 32.0 GiB), Free: 31.9 GiB, Space left: 31.9 GiB
ORA <working database>.DLP_INDEX_06 Tablespace ONLINE (PERMANENT), Size: 30.0 MiB, 0.08% used (27.8 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.DLP_LOB_01 Tablespace ONLINE (PERMANENT), Size: 10.0 MiB, <0.01% used (1.00 MiB of max. 32.0 GiB), Free: 32.0 GiB, Space left: 32.0 GiB
ORA <working database>.DLP_OWNER.DBMS_JOB_46 Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 11.30.58.000000 AM +00:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.ESM_DATA_01 Tablespace ONLINE (PERMANENT), Size: 896 MiB, 1.56% used (512 MiB of max. 32.0 GiB), Free: 31.5 GiB, Space left: 31.5 GiB
ORA <working database>.ESM_INDEX_01 Tablespace ONLINE (PERMANENT), Size: 512 MiB, 1.00% used (326 MiB of max. 32.0 GiB), Free: 31.7 GiB, Space left: 31.7 GiB
ORA <working database>.ORACLE_OCM.MGMT_CONFIG_JOB Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 1 second, Next Run: 30-NOV-23 01.01.01.602732 AM -04:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.ORACLE_OCM.MGMT_STATS_CONFIG_JOB Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 1 second, Next Run: 01-DEC-23 01.01.01.567001 AM -04:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.SIGMATAU_EXPORT Tablespace ONLINE (PERMANENT), Size: 5.00 GiB, 14.64% used (4.68 GiB of max. 32.0 GiB), Free: 27.3 GiB, Space left: 27.3 GiB
ORA <working database>.SYS.BSLN_MAINTAIN_STATS_JOB Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 03-DEC-23 12.00.00.484042 AM -07:00, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.SYS.CLEANUP_NON_EXIST_OBJ Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 09.40.59.352003 PM +00:00,  no log information found (ignored)
ORA <working database>.SYS.CLEANUP_ONLINE_IND_BUILD Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 04.37.54.056300 AM -07:00,  no log information found (ignored)
ORA <working database>.SYS.CLEANUP_ONLINE_PMO Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 07.40.31.109928 AM -04:00,  no log information found (ignored)
ORA <working database>.SYS.CLEANUP_TAB_IOT_PMO Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 04.38.04.399558 AM -07:00,  no log information found (ignored)
ORA <working database>.SYS.CLEANUP_TRANSIENT_PKG Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 07.40.21.000000 AM -04:00,  no log information found (ignored)
ORA <working database>.SYS.CLEANUP_TRANSIENT_TYPE Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 29-NOV-23 09.40.59.148765 PM +00:00,  no log information found (ignored)
ORA <working database>.SYS.DRA_REEVALUATE_OPEN_FAILURES Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 01-JAN-70 12.00.00.000000 AM +00:00,  no log information found (ignored)
ORA <working database>.SYS.FGRAUTOPURGE_JOB Job Job-State: DISABLED, Enabled: No (ignored), Next Run: 01-JAN-70 12.00.00.000000 AM +00:00,  no log information found (ignored)
ORA <working database>.SYS.FILE_SIZE_UPD Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 15-OCT-24 04.40.44.480993 AM -04:00,  no log information found (ignored)
ORA <working database>.SYS.HM_CREATE_OFFLINE_DICTIONARY Job Job-State: DISABLED, Enabled: No (ignored), Next Run: 01-JAN-70 12.00.00.000000 AM +00:00,  no log information found (ignored)
ORA <working database>.SYS.LOAD_OPATCH_INVENTORY Job Job-State: DISABLED, Enabled: No (ignored), Next Run: 01-JAN-70 12.00.00.000000 AM +00:00,  no log information found (ignored)
ORA <working database>.SYS.ORAAUTOTASK_CLEAN Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 30-NOV-23 03.00.00.899862 AM PST8PDT, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.SYS.PMO_DEFERRED_GIDX_MAINT_JOB Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 30-NOV-23 02.00.00.398035 AM US/PACIFIC,  no log information found (ignored)
ORA <working database>.SYS.PURGE_LOG Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 30-NOV-23 03.00.00.669286 AM PST8PDT, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.SYS.RSECLEAN_RECOVERABLE_SCRIPT Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 30-NOV-23 12.00.00.252546 AM PST8PDT, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.SYS.SMCLEAN_AUTO_SPLIT_MERGE Job Job-State: SCHEDULED, Enabled: Yes, Last Duration: 0 seconds, Next Run: 30-NOV-23 12.00.00.750894 AM PST8PDT, Last Run Status: SUCCEEDED (ignored disabled Job)
ORA <working database>.SYS.XMLDB_NFS_CLEANUP_JOB Job Job-State: DISABLED, Enabled: No (ignored), Next Run: 01-JAN-70 12.00.00.000000 AM +00:00,  no log information found (ignored)
ORA <working database>.SYSAUX Tablespace ONLINE (PERMANENT), Size: 36.9 GiB, 54.69% used (35.0 GiB of max. 64.0 GiB), Free: 29.0 GiB, Space left: 29.0 GiB, 2 data files (2 avail, 2 autoext)
ORA <working database>.SYSTEM Tablespace ONLINE (PERMANENT), Size: 3.74 GiB, 11.52% used (3.69 GiB of max. 32.0 GiB), Free: 28.3 GiB, Space left: 28.3 GiB
ORA <working database>.TEMP Tablespace ONLINE (TEMPORARY), Size: 1.00 GiB, 0% used (0 B of max. 16.0 GiB), Free: 16.0 GiB
ORA <working database>.UNDOTBS1 Tablespace ONLINE (UNDO), Size: 2.16 GiB, 1.74% used (569 MiB of max. 32.0 GiB), Free: 31.4 GiB
ORA <working database>.USERS Tablespace ONLINE (PERMANENT), Size: 5.52 GiB, 15.85% used (5.07 GiB of max. 32.0 GiB), Free: 26.9 GiB, Space left: 26.9 GiB
ORA <failing database> Instance Database Name <failing database>, Status OPEN, Role PRIMARY, Version 19.3.0.0.0, Logins allowed, Log Mode archivelog, Force Logging no
ORA <failing database> Locks   No locks existing
ORA <failing database> Logswitches 3 log switches in the last 60 minutes (warn/crit below -1/-1) (warn/crit at 50/100)
ORA <failing database> Long Active Sessions 0
ORA <failing database> Performance DB Time: -7.34/s, DB CPU: -2.72/s, DB Non-Idle Wait: -4.62/s, Maximum SGA Size: 2.00 GiB, Buffer Cache Size: 848 MiB, Shared Pool Size: 320 MiB, Redo Buffers: 7.51 MiB, Java Pool Size: 16.0 MiB, Large Pool Size: 16.0 MiB, Streams Pool Size: 0 B, Shared IO Pool Size: 64.0 MiB, Buffer hit ratio: 96.7%, Library cache hit ratio: 82.1%
ORA <failing database> Processes 55 of 1000 processes are used: 5.50%
ORA <failing database> Recovery Area 0 B out of 0 B used (0.0%, warn/crit at 70.0%/90.0%), 0 B reclaimable
ORA <failing database> Recovery Status primary database, oldest Checkpoint 29 seconds ago
ORA <failing database> Sessions 3.55%, 54 of 1522 sessions
ORA <failing database> Undo Retention Undo retention: 15 minutes 0 seconds, Active undo blocks: 608, Max concurrent transactions: 3, Max querylen: 0 seconds, Space errors: 0
ORA <failing database> Uptime  Up since 2023-11-29 11:35:18, uptime: 0:00:34
Process /usr/sbin/sshd -D -oCiphersaes256-gcm@openssh.comchacha20-poly1305@openssh.comaes256-ctraes256-cbcaes128-gcm@openssh.comaes128-ctraes128-cbc -oMACshmac-sha2-256-etm@openssh.comhmac-sha1-etm@openssh.comumac-128-etm@openssh.comhmac-sha2-512-etm@openssh.comhmac-sha2-256hmac-sha1umac-128@openssh.comhmac-sha2-512 -oGSSAPIKexAlgorithmsgss-curve25519-sha256-gss-nistp256-sha256-gss-group14-sha256-gss-group16-sha512-gss-gex-sha1-gss-group14-sha1- -oKexAlgorithmscurve25519-sha256curve25519-sha256@libssh.orgecdh-sha2-nistp256ecdh-sha2-nistp384ecdh-sha2-nistp521diffie-hellman-group-exchange-sha256diffie-hellman-group14-sha256diffie-hellman-group16-sha512diffie-hellman-group18-sha512diffie-hellman-group-exchange-sha1diffie-hellman-group14-sha1 -oHostKeyAlgorithmsecdsa-sha2-nistp256ecdsa-sha2-nistp256-cert-v01@openssh.comecdsa-sha2-nistp384ecdsa-sha2-nistp384-cert-v01@openssh.comecdsa-sha2-nistp521ecdsa-sha2-nistp521-cert-v01@openssh.comssh-ed25519ssh-ed25519-cert-v01@openssh.comrsa-sha2-256rsa-sha2-256-cert-v01@openssh.comrsa-sha2-512rsa-sha2-512-cert-v01@openssh.comssh-rsassh-rsa-cert-v01@openssh.com -oPubkeyAcceptedKeyTypesecdsa-sha2-nistp256ecdsa-sha2-nistp256-cert-v01@openssh.comecdsa-sha2-nistp384ecdsa-sha2-nistp384-cert-v01@openssh.comecdsa-sha2-nistp521ecdsa-sha2-nistp521-cert-v01@openssh.comssh-ed25519ssh-ed25519-cert-v01@openssh.comrsa-sha2-256rsa-sha2-256-cert-v01@openssh.comrsa-sha2-512rsa-sha2-512-cert-v01@openssh.comssh-rsassh-rsa-cert-v01@openssh.com -oCASignatureAlgorithmsecdsa-sha2-nistp256ecdsa-sha2-nistp384ecdsa-sha2-nistp521ssh-ed25519rsa-sha2-256rsa-sha2-512ssh-rsa Processes: 1, Virtual memory: 85.7 MiB, Resident memory: 5.72 MiB, CPU: 0%, Running for: 47 days 1 hour
Process sssd         Processes: 1, Virtual memory: 191 MiB, Resident memory: 7.12 MiB, CPU: 0%, Running for: 41 days 2 hours
Systemd Service Summary Total: 164, Disabled: 19, Failed: 2, 1 static service failed (systemd-modules-load)(!!)
Systemd Socket Summary Total: 24, Disabled: 3, Failed: 0
TCP Connections      Established: 16
Uptime               Up since Oct 13 2023 10:33:27, Uptime: 47 days 1 hour
login_<working database>       OK - dummy login connected
login_<failing database>       OK - dummy login connected
No piggyback files for '<database server>'. Skip processing.
No piggyback files for '192.168.11.20'. Skip processing.
[cpu_tracking] Stop [7f0664994390 - Snapshot(process=posix.times_result(user=0.030000000000000027, system=0.0, children_user=0.0, children_system=0.0, elapsed=0.029999999329447746))]
[agent] Success, [piggyback] Success (but no data found for this host), execution time 7.3 sec | execution_time=7.280 user_time=0.040 system_time=0.000 children_user_time=0.000 children_system_time=0.000 cmk_time_agent=7.240

ianmurphy · November 29, 2023, 2:04pm

Hi again. Have resolved this by setting the following the /usr/lib/check_mk_agent/plugins/mk_oracle file

MK_ORA_DEBUG=true
MK_ORA_LOGGING=true

Rescanned the host in the console after this and the other database is no longer showing up, and all it’s checks are available.

Weird, and I can’t explain it, but I’ll take it!

burgeau · December 24, 2023, 3:31am

In most cases when I have had this problem it is due to query duration for the Oracle checks. Typically the tablespace or RMAN check depending on your database config.
‘
Do you have any details on database version, size etc?

The mk_oracle log file would be a good place to start to see where the time is spent.

ianmurphy · January 2, 2024, 3:28pm

Hi Mark. Yes it did seem like a timeout issue as have seen those manifest in other types of checks. As I mentioned, I seem to have “fixed” it the MK_ORA_DEBUG and MK_ORA_LOGGING settings set to “true”. Not sure why but it’s working so I’m happy!

Cheers,

IanM

system · January 1, 2025, 3:28pm

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.