Monitoring SAP R/3 + SAP HANA

General
1

Hi everyone,

A bit of introduction first: In the last couple months, I was given the task to find a monitoring tool that could monitor SAP R/3 and SAP HANA environments. I then researched and found about Check_MK. From then on, I have been reading documentation and experimenting different ways to configure things, etc.
Finally, a couple weeks ago, I started to properly try to configure and implement the plugins to monitor the previously mentioned environments. Unfortunately, it did not work as intended.

For the HANA plugin, when I create a rule on WATO for it and install the baked agent, all I get are the following services, plus, the Check_MK Discovery service becomes unavailable, as seen next::
https://i.imgur.com/V8raXK9.png
I have also tried to manually edit the file, placing it on the plugins folder and running it, but that wouldn’t even give me any services at all; I assume I just didn’t configure it right, which is ok.
Something I recently realised is I may have to run the script on terminal with some specific arguments, and it will then provide me with the needed info, although I don’t really think this is how it’s supposed to work, it’s the only thing that comes to mind. Please clarify me on this matter as well.

Now for the R/3 part: I have followed the article on https://checkmk.com/cms_legacy_sap_r3.html; I successfully installed all prerequesites, and configured/installed the plugin with all the credentials, on the Check_MK’s server machine, and all I get is this service:
image
It was supposed to monitor another machine, which is on the same network. I made sure to properly configure the IP address of the aforementioned server. On a small note, I have also tried the rule-based approach, as well as the manual file editing (I also tried to configure it both with, and without the sap.cfg file), which is what the article suggests, but none of them gave me the expected results: they both just generate one service.
Something I didn’t try was to install the prerequesites and the plugin on the to-be-monitored-machine, because it is a very critical server, and I would rather not do it, unless it is the only option. Apart from that, I have no clue on possible alternatives.

Lastly, here are some technical details if needed:
Check_MK version: 1.6.0p11 CEE
Check_MK server OS: Ubuntu 18.04.4 LTS
SAP server OS: SLES 12 SP3

Feel free to ask me any further details you might need to know.
Thanks in advance! :slight_smile:

2

Hi @agc and welcome to the checkmk community.

oh well 
 SAP and monitoring, this is kinda difficult topic.

First to your monitoring of HANA systems. Do you have a scale-up or scale-out system, which version are you running and have you a multi-tenant environment?
In general you need some preparations to monitor an HANA database, like creating a user with proper roles inside the database and optional store the credentials to hdbuserstore.

For monitoring the ABAP side of an SAP system, the legacy sap R3 check exists but I personally not recommend to use this, because this plugin is using the CCMS and this isn’t supported anymore by SAP since, I guess, 10 years. You also have to configure a lot of stuff to get your metrics inside SAP to be collected and then collected by checkmk.
I would rather recommend to use your SAP Solution Manager and the Monitoring and Alerting Infrastructure with the 3rd party alerting connector to write a log file which is monitored by checkmk via event console.

3

Hello, and thanks :slight_smile:

It is a scale-up, multi-tenant system running version 2.00.042.00.1564994110.
I have talked to my superiors about the new user matter, let’s see how that goes. But I do have a question about this: why doesn’t the admin user work, since it supposedly has all permissions/roles?

As for the ABAP part, thanks in advance for you suggestions. I have already mentioned it and it should be discussed soon. Someone suggested we could use Solution Manager to monitor everything, including HANA, but is that recommended? Does it provide alerts and reports aswell as Check_MK does? Does it allow the monitoring of the same services? How much more, or less complete is it?

Pardon me for the newbie questions, I’m just a trainee, and although my chiefs are very well into this matter (SAP/HANA, but not that much into monitoring), I thought I should get feedback from other sources too.

4

The admin user works but is quite risky to have all permissions on a database for just monitoring purposes. You also won’t giva any user SAP_ALL to your systems :stuck_out_tongue_winking_eye:

With the parameters you wrote it’s possible to monitor a lot of functions of the HANA database, all you need is a user with the MONITORING role. You can provide the credentials at the rule directly or let your DBA add it to the hdbuserstore so you never have the password clear text anywhere.

The following services are monitored:

SAP HANA Backup <SID> <INSTANCENR> - complete data backup
SAP HANA Backup <SID> <INSTANCENR> - log backup
SAP HANA Disk <SID> <INSTANCENR> - Data
SAP HANA Disk <SID> <INSTANCENR> - Log
SAP HANA Disk <SID> <INSTANCENR> - Trace
SAP HANA Events <SID> <INSTANCENR>
SAP HANA License <SID> <INSTANCENR>
SAP HANA Replication Status
SAP HANA Status <SID> <INSTANCENR>
SAP HANA Version <SID> <INSTANCENR>

The HANA monitoring from SAP is okay, but doesn’t provide the best informations yet. Additional you will get some trouble to well configure the HANA system inside LMDB, because SLD doesn’t provide all informations needed at the moment. The checkmk plugin also provide this information state based and the Solution Manager only works event based. And you are also depending on the administrators of solution manager to patch all the modules to have the latest monitoring available from SAP.

Also the initial configuration to run the monitoring via Solution Manager is huge, you have to have diagnostic agents on each host, have configured the database monitoring connection inside the SAP system, have to run the Managed System Configuration inside Solution Manager for this system, have to enable the MAI, configure the MAI for your thresholds and i am sure, I missed some steps so far. Therefor it’s much easier for the beginning to just install a checkmk agent on the host and create a database user with the right role.

And all in all you will also get some free help around for the check plugin and don’t have to pay a SAP consultant. :slight_smile:

5

Thank you for your input. We’ll probably stick to Check_MK then, for the HANA part at least. Will discuss it for ABAP yet.

The admin user works but is quite risky to have all permissions on a database for just monitoring purposes.

Yeah, I get it, and agree. But if “the admin user works” then why did I get the result mentioned in the first post? (https://i.imgur.com/V8raXK9.png) Shouldn’t it create the services you just typed in? Or does it just create them with the monitoring role? I got confused on that part :thinking:

6

I can’t check your image from imgur, because it’s blocked from my company :slight_smile:

Can you please check to call the plugin directly from command line with MK_CONFDIR=/etc/check_mk /usr/lib/check_mk_agent/plugins/sap_hana. It should tell you why it cant gather more information.

7

Oh, I didn’t know. What kind of company blocks imgur anyway lol

Well, because of being a new member, I can only upload one image per post, but basically I got some services created within Check_MK. The services’ names correspond to the repeated lines on the following image. So for example, I got a service named “File Welcome to the SAP HANA Database interactive terminal.” Another one called “File Type :\h for help with commands” And the last one called “File \q to quit”. They all have a status of “WARN - File stat time failed”. I also got a log service, called “Log SAP HANA Alerts for HDB 02”. The logs it gives me are the names of the other services, all 3 of them, once a minute. I hope you can understand.

Edit:
Additionally, the Check_MK Discovery service, goes into an unknown state, and if I try to do a full services scan, I get the following error message:

Starting job

+ FETCHING DATA
[agent] Execute data source
[piggyback] Execute data source
No piggyback files for ‘ip_adress’. Skip processing.
No piggyback files for ‘ip_address’. Skip processing.
WARNING: Exception while parsing agent section ‘sap_hana_status’: IndexError(‘list index out of range’,)
File “/omd/sites/site_name/lib/python/cmk_base/data_sources/host_sections.py”, line 299, in _update_with_parse_function
return parse_function(section_content)
File “/omd/sites/site_name/share/check_mk/checks/sap_hana_status”, line 43, in parse_sap_hana_status
‘version’: line[2],
systemd_units does not support discovery. Skipping it.
ps_lnx does not support discovery. Skipping it.
ps.perf does not support discovery. Skipping it.
Completed.

Output from running the plugin from terminal

image
image649×1726 34.2 KB

8

Oh, seems like there is a problem with the login and execution of the SQLs. Let me check this, maybe I find the problem.

The information of the databses and instances are gathered by the command /usr/sap/hostctrl/exe/lssap.

9

Can you check, if your user and password to login against database are provided inside /etc/check_mk/sap_hana.cfg?

With this credentials you can check if the connection works via
su - <sidadm> -c "/usr/sap/<SID>/HDB<INSTANCE>/exe/hdbsql -C -F\; -x -a -n localhost -i <INSTANCE> -u <USER> -p <PASSWORD> \"select version from sys.m_database\""
or with hdbuserstore via
su - <sidadm> -c "/usr/sap/<SID>/HDB<INSTANCE>/exe/hdbsql -C -F\; -x -a -n localhost -i <INSTANCE> -U <STOREKEY> \"select version from sys.m_database\""

Actually i don’t can figure out, why you are getting an interactice terminal instead of just executing the SQL and exit with output.

10

Yes, the credentials are on sap_hana.cfg file.
I ran the command and it does work, it returns HANA version as expected, but the services remain the same. :frowning:

11

Have you set the Run as to Agent at the rule SAP HANA?

grafik
grafik711×232 3.51 KB

12

It was set to run as instance, I just changed to run as agent, reinstalled the new baked agent, and now the services I previously told you about are gone, but no new ones were discovered.

Edit: I did got these errors on the service discovery page:

systemd_units does not support discovery. Skipping it.
ps_lnx does not support discovery. Skipping it.
ps.perf does not support discovery. Skipping it.
13

No I mean, just uncheck the Run as section. Running at against the instance is right.

14

Oh, ok. So I did that and the services are back again :frowning:

15

Can you please run the plugin in debug mode from command line and send me the output as PM? The output is really long and I guess the forum cant handle such long posts. Don’t forget to censor senstitive data like passwords at the output.

MK_CONFDIR=/etc/check_mk/ /usr/lib/check_mk_agent/pluginsmk_sap_hana --debug

16

I ran the plugin in debug mode and got the output, however, both posts and messages are limited to 32000 characters, and the output is longer than that :sweat_smile: I can upload it to pastebin or some other site you can access; what site should that be?

PS: For some reason I couldn’t PM you.

17

Oh, okay, thats bad. I guess we have no sharing platforms available allowed by my company.
Can you just send the first ~100 lines? I guess important informations are there to analyse the problem further.

18

Here it is:

+ shift
+ test -n ''
+ '[' -e /etc/check_mk//sap_hana.cfg ']'
+ . /etc/check_mk//sap_hana.cfg
++ USER=<username>
++ PASSWORD=<password>
+ '[' -z '' ']'
+ '[' -z <username> ']'
+ '[' -z <password> ']'
+ ps -efw
+ grep '[H]DB.*sapstartsrv'
+ read -r line
++ echo '<sidadm>     2511      1  0 Apr17 ?        00:00:59 /usr/sap/<SID>/HDB<INSTANCE>/exe/sapstartsrv pf=/usr/sap/HDB/SYS/profile/HDB_HDB<INSTANCE>_vhcalhdbdb -D -u <sidadm>'
++ sed 's/.*\/\(.*\)\/.*\/exe\/sapstartsrv.*/\1/g'
+ SID=<SID>
++ echo '<sidadm>     2511      1  0 Apr17 ?        00:00:59 /usr/sap/<SID>/HDB<INSTANCE>/exe/sapstartsrv pf=/usr/sap/HDB/SYS/profile/HDB_HDB<INSTANCE>_vhcalhdbdb -D -u <sidadm>'
++ sed -e 's/.*\/\(.*\)\/exe\/sapstartsrv.*/\1/g' -e 's/[^0-9]//g'
+ INSTANCE=<INSTANCE>
++ echo '<sidadm>     2511      1  0 Apr17 ?        00:00:59 /usr/sap/<SID>/HDB<INSTANCE>/exe/sapstartsrv pf=/usr/sap/HDB/SYS/profile/HDB_HDB<INSTANCE>_vhcalhdbdb -D -u <sidadm>'
++ grep -Po '^\s*(\w+)'
+ INSTANCE_USER=<sidadm>
+ '[' -z <INSTANCE> ']'
+ '[' -z <SID> ']'
+ do_query <SID> <INSTANCE> <sidadm>
+ local SID=<SID>
+ local INSTANCE=<INSTANCE>
+ local INSTANCE_USER=<sidadm>
+ local 'INSTANCE_NAME=[[HDB <INSTANCE>]]'
+ '[' -z <hostname> ']'
++ hostname
+ HOSTNAME=<hostname>
+ echo '<<<sap_hana_status:sep(59)>>>'
<<<sap_hana_status:sep(59)>>>
+++ query_sap_hana_status
+++ cat
++ mk_hdbsql HDB <instance> <sidadm> 'SELECT Name, Status, Value FROM M_SYSTEM_OVERVIEW Where NAME='\''Version'\'' or
NAME='\''All Started'\'''
++ local sid=HDB
++ local instance=<instance>
++ local instance_user=<sidadm>
++ local query
+++ echo 'SELECT Name, Status, Value FROM M_SYSTEM_OVERVIEW Where NAME='\''Version'\'' or
NAME='\''All Started'\'''
+++ tr '\n' ' '
++ query='SELECT Name, Status, Value FROM M_SYSTEM_OVERVIEW Where NAME='\''Version'\'' or NAME='\''All Started'\'' '
++ local credentials
++ local hdbsql_path=/usr/sap/<SID>/HDB<INSTANCE>/exe/hdbsql
++ '[' -n '' ']'
++ credentials='-u <username> -p <password>'
++ '[' '' = agent ']'
+++ su - <sidadm> -c '/usr/sap/<SID>/HDB<INSTANCE>/exe/hdbsql -C -F\; -x -a -n localhost -i <INSTANCE> -u <username> -p <password> "SELECT Name, Status, Value FROM M_SYSTEM_OVERVIEW Where NAME='\''Version'\'' or NAME='\''All Started'\'' "'
++ result='
Welcome to the SAP HANA Database interactive terminal.
                                           
Type:  \h for help with commands          
       \q to quit                         '
++ rc=0
++ echo '
Welcome to the SAP HANA Database interactive terminal.
                                           
Type:  \h for help with commands          
       \q to quit                         '
++ sed -e 's/^;//' -e 's/;$//'
++ return 0
+ status_output='
Welcome to the SAP HANA Database interactive terminal.
                                           
Type:  \h for help with commands          
       \q to quit                         '
+ echo '[[HDB <instance>]]'
[[HDB <instance>]]
+ echo '
Welcome to the SAP HANA Database interactive terminal.
                                           
Type:  \h for help with commands          
       \q to quit                         '

Welcome to the SAP HANA Database interactive terminal.
                                           
Type:  \h for help with commands          
       \q to quit                         
+ echo '<<<sap_hana_backup:sep(59)>>>'
<<<sap_hana_backup:sep(59)>>>
+ echo '[[HDB <instance>]]'
[[HDB <instance>]]
++ query_sap_hana_backup_snapshots
++ cat
+ mk_hdbsql HDB <instance> <sidadm> 'Select TOP 1 entry_type_name, sys_end_time, state_name, comment, message from
M_BACKUP_CATALOG where entry_type_name = '\''data snapshot'\'' AND state_name <>
'\''running'\'' order by sys_start_time desc'
+ local sid=HDB
+ local instance=<instance>
+ local instance_user=<sidadm>
+ local query
++ echo 'Select TOP 1 entry_type_name, sys_end_time, state_name, comment, message from
M_BACKUP_CATALOG where entry_type_name = '\''data snapshot'\'' AND state_name <>
'\''running'\'' order by sys_start_time desc'
++ tr '\n' ' '
+ query='Select TOP 1 entry_type_name, sys_end_time, state_name, comment, message from M_BACKUP_CATALOG where entry_type_name = '\''data snapshot'\'' AND state_name <> '\''running'\'' order by sys_start_time desc '
+ local credentials
+ local hdbsql_path=/usr/sap/<SID>/HDB<INSTANCE>/exe/hdbsql
+ '[' -n '' ']'
+ credentials='-u <username> -p <password>'
+ '[' '' = agent ']'
++ su - <sidadm> -c '/usr/sap/<SID>/HDB<INSTANCE>/exe/hdbsql -C -F\; -x -a -n localhost -i <INSTANCE> -u <username> -p <password> "Select TOP 1 entry_type_name, sys_end_time, state_name, comment, message from M_BACKUP_CATALOG where entry_type_name = '\''data snapshot'\'' AND state_name <> '\''running'\'' order by sys_start_time desc "'
+ result='
Welcome to the SAP HANA Database interactive terminal.
                                           
Type:  \h for help with commands          
       \q to quit                         '
+ rc=0
+ echo '
Welcome to the SAP HANA Database interactive terminal.
                                           
Type:  \h for help with commands          
       \q to quit                         '
+ sed -e 's/^;//' -e 's/;$//'

Welcome to the SAP HANA Database interactive terminal.
                                           
Type:  \h for help with commands          
       \q to quit                         
+ return 0
19

Maybe this will help. I had some trouble getting the HANA plugin to work also. I’m running as agent and in the plugin I had to change the line:

result=“$(${hdbsql_path} -C -F; -x -a -n localhost -i “${instance}” “${credentials}” “$query” 2>/dev/null)”

to:

result=“$(${hdbsql_path} -C -F; -x -a -n localhost -i “${instance}” ${credentials} “$query” 2>/dev/null)”

(I removed the quotes around ${credentials}). I I didn’t do that it would pass -u -p as a single string to hdbsql and it didn’t like this.

Hopefully this will help you with your issue as well.

1 Like
20

I’ve set it to run as agent and did your suggested change to the plugin and yes, it did work! Thank you both for your help, I very much appreciate it :slight_smile:

1 Like