Backups don't work with 2.x

Troubleshooting
1

Here is the error:

Failed to start the job: Traceback (most recent call last): File "/omd/sites/home/bin/mkbackup", line 2024, in <module> main() File "/omd/sites/home/bin/mkbackup", line 2019, in main mode.runner(mode_args, opt_dict) File "/omd/sites/home/bin/mkbackup", line 553, in <lambda> runner=lambda args, opts: mode_backup(args[0], opts=opts), File "/omd/sites/home/bin/mkbackup", line 617, in mode_backup acquire_backup_lock() File "/omd/sites/home/bin/mkbackup", line 202, in acquire_backup_lock makedirs(str(lock_file_path.parent), group="omd", mode=0o770) File "/omd/sites/home/bin/mkbackup", line 284, in makedirs makedir(path, user, group, mode) File "/omd/sites/home/bin/mkbackup", line 294, in makedir os.mkdir(path) PermissionError: [Errno 13] Permission denied: '/var/lock/mkbackup'

Here are the (default) permissions on CentOS 7:

[root@vm-checkmk ~]# ls -la /var/lock/
total 0
drwxr-xr-x.  5 root root  100 Apr 19 12:40 .
drwxr-xr-x. 29 root root 1000 Apr 19 12:40 ..
drwxrwxr-x.  2 root lock   40 Apr 19 12:40 lockdev
drwx------.  2 root root   40 Apr 19 12:42 lvm
drwxr-xr-x.  2 root root   80 Apr 19 12:40 subsys

If I create the subfolder mkbackup with omd as the group owner, the backup runs successfully but that folder is eventually flushed and the problem comes back. I could obviously chance /var/lock’s permissions but I don’t think that’s a suitable solution. What’s wrong here?

Same error on 2.0.0, 2.0.0p1, p2 and p3.

2

This can be replicated on a fresh CentOS 7 system with a fresh Checkmk installation.

3

I honestly don’t know the directory /var/lock/ but I think backups should not go there. Maybe use /opt/ or mount something dedicated under /mnt/foo. /var/ in Linux is for variable or volatile data, according to the HFS. That is not entirely true to be fair, but you get the idea.

What you definitely need to to is ensure that the parent folder in which you are backing up (/var/lock/mkbackup in your example) is owned by the site user. That folder obviously needs to stay and then you are good to go.

4

I didn’t chose the directory… Checkmk does. They changed it in 1.6.0p21: mkbackup: Fix permission issues when backing up different sites

Not sure how I’m supposed to change this and whether or not I change it, it’s still a bug shipped with Checkmk.

5

Dear Thorian93,

the folder /var/lock/ is used to “lock” files or backup jobs while they are in use.
in this case:
checkmk creates a “lockfile” /var/lock/mkbackup/mkbackup.lock that the backup job is only executed once.
after the job is completed, the lockfile will be deleted.

i solved my problem with the Link of dnLL: mkbackup: Fix permission issues when backing up different sites in creating the subfolder.

cd /var/lock
mkdir mkbackup
chown yourcheckmkuser.omd mkbackup

after that my backup worked.

btw.: if your system restarts or fails during backup it might happen that the lockfile wasn’t deleted automatically. In this case you have to delete the lockfile mkbackup.lock manually.

6

I guess I sat on my brains while thinking about this. :wink:

Of course I know the /var/lock, I just didn’t connect the dots.

I think I got confused where @dnLL was putting his backup. I thought his whole backup was written to /var/lock. But now it seems to actually be an issue inside Checkmk’s design there.

7

Thank you for your reply. Yes, creating the directory allows checkmk to execute the backup but I noticed the directory gets deleted eventually (unsure if that’s checkmk or CentOS). So I would have to add something to the crontab to recreate that directory which isn’t ideal.

8

You could use tmpfilesd for a clean approach to your dirty solution. :wink:

But in the end I agree that you at least need feedback from the tribe whether this is an CentOS or Checkmk issue.

2 Likes
9

From my understanding the directory is just wiped after every reboot. Not just on CentOS, see https://askubuntu.com/q/243606/104514

10

All right so here is my kind of permanent fix:

[root@vm-checkmk ~]# cat /usr/lib/tmpfiles.d/checkmk.conf
#Type Path            Mode UID      GID    Age Argument
d /var/lock/mkbackup  0775 root     omd    -   -

It’s elegant, the only thing I dislike about this is having to monitor all the werks in case the tribe comes up with another solution…

1 Like
11

I think is a rather good fix and I do not really see a problem if the tribe fixes the problem. Either they ensure the same state as you do, or they change something altogether with won’t be affected by your workaround. The only thing you might want to do is remove the workaround once it is not necessary anymore.

Anyway, I agree it is not the golden solution but I think we can call it silver or at least bronze. :wink:

If you liked my suggestion I appreciate a like. :slight_smile:

1 Like
12

I just deleted the file following this: mkbackup: Fix locking problems
Will see if it’s fixed.

13

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed. Contact an admin if you think this should be re-opened.