Upgrade problems with rrd graph/file permissions

Troubleshooting
#1

Hello,

I just copied my instance via omd cp oldsite newsite and upgraded the new copy from 1.6.0p29 to 2.0.0p27. I got a few errors regarding autochecks and permission errors regarding rrd graphs.
The permission error is at step 8/17 Migrating fs_used name…:

PermissionError: [Errno 13] Permission denied: ‘/opt/omd/sites/OLDSITENAME/var/pnp4nagios/perfdata/HOST/Filesystem____.rrd’ → ‘/opt/omd/sites/OLDSITENAME/var/pnp4nagios/perfdata/HOST/Filesystem___fs_used.rrd’

What can I do to fix this? Other graphs like CPU is working…

Regards
CheckMan

#2

Looking for something similar on the forum, I found this:

#4

After digging deeper into the permission problem I found that some of the hostfolders in /opt/omd/sites/OLDSITENAME/var/pnp4nagios/perfdata have different permissions accross the folders. The newer host folders have 750 the older ones 775. But I never changed them by hand. How can this happen? And what is the right permission for the host folders?

Another question is, why is the upgrade of the new site trying to convert rrd graphs of the old sites path?
Shouldn’t the new site use their own rrds which should be located in /opt/omd/sites/NEWSITENAME/var/pnp4nagios/perfdata? There would be at least the owner the right one…

#5

There were bugs in older versions of the omd cp command.
You might want to try and upgrade the site first and then do a copy, or use the backup mechanism to “copy” the site.

#6

Tried with the backup & restore method but got the same problem again:

-| 8/17 Migrating fs_used name…
-| + “Migrating fs_used name” failed
-| Traceback (most recent call last):
-| File “/omd/sites/MYNEWSITE/lib/python3/cmk/update_config.py”, line 158, in run
-| step_func()
-| File “/omd/sites/MYNEWSITE/lib/python3/cmk/update_config.py”, line 199, in update_fs_used_name
-| cmk.update_rrd_fs_names.update()
-| File “/omd/sites/MYNEWSITE/lib/python3/cmk/update_rrd_fs_names.py”, line 186, in update
-| update_service_info(config_cache, get_hostnames(config_cache))
-| File “/omd/sites/MYNEWSITE/lib/python3/cmk/update_rrd_fs_names.py”, line 176, in update_service_info
-| update_files(hostname, service.description, service.item, ‘pnp4nagios’))
-| File “/omd/sites/MYNEWSITE/lib/python3/cmk/update_rrd_fs_names.py”, line 95, in update_files
-| entry_to_rename = update_pnp_info_files(perfvar, ‘fs_used’, filepath)
-| File “/omd/sites/MYNEWSITE/lib/python3/cmk/update_rrd_fs_names.py”, line 156, in update_pnp_info_files
-| os.rename(rrdfile, rrdfilenew)
-| PermissionError: [Errno 13] Permission denied: '/opt/omd/sites/MYOLDSITE/var/pnp4nagios/perfdata/MYHOST/Filesystem_C___C_.rrd’ → ‘/opt/omd/sites/MYOLDSITE/var/pnp4nagios/perfdata/MYHOST/Filesystem_C___fs_used.rrd’

I don’t want to update our main site. This is not just a 5 minute job. We need to migrate a few checks and also want to get a migration way which works for us in a reasonable time. Without days or weeks of a not working monitoring.

It could be so easy if this copy/backup mechanisms would work. I could let my old monitoring running and test the new/migrated one… :frowning:

#7

Hi,
if i remember right I had same problem with permisions when updating my site to 2.0. The problem was that some files had owner not as omd user but as root user dunno why. I switched to omd user and migrating worked.

Best regards,
JF

1 Like
#8

My problem currently is that the old filepath of the original site is used. This should not happen after copying to a new site or restoring a backup to a new site.

#9

I would go with the suggestion by @jan.filo, probably a permission error, that prevents proper copy and/or backup/restore.

#10

But this error happens at the update process of the copied site not at the copying process of it.
So it’s a update problem for me… the new site (copying process completed) tries to access rrds graphs of its origin at the update. Doesn’t make sense for me.

#11

Well, how do you know, the problematic paths are not a result of an improper copy job?

#12

I don’t know this. But does an update change the paths to the right one for the future?

What happens if I delete the old site (for whatever reason)? The rrds get also removed and the new site has no graphs anymore?

How can I verify that the paths are correct? Is this in any configfile?

#13

Ok looks like the path to the rrd files is written into the xml files. And maybe this is the problem here because of a wrong path in this files.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> check_mk-winperf_if /omd/sites/OLDSITENAME/var/pnp4nagios/perfdata/HOSTNAME/Interface_3_in.rrd

As you can see the old site name is written into the xml file in the new sites xml files.
Is this as intended? Or is it a bug?

#14

Okay, I feel like you have several issues here.

First, in the old site, double-check all permissions, as @jan.filo suggested.
Next, do a backup and restore to a new site.
Then upgrade the restored site.
If there are issues with the RRDs afterwards, take a look here.

Another approach would be to back up the old site and then upgrade the old site instead of the backup.
That way you have a backup just in case, but you might avoid the path issue.

2 Likes
#15

Big thanks for your help!
I will take the second approach.

But someone from CheckMK should take a look into this. The copy process looks simply broken in that case.

#16

As said, prior to Checkmk 2.1. there were bugs in the copy command, which we fixed.

1 Like