How to backup Checkmk running in Docker

General
,
1

Hi

I run Checkmk Raw in a Docker container. What is the best way to backup Checkmk in this scenario?

Currently I am mounting the /omd/sites directory into the container using the -v /data/cmk/monitoring:/omd/sites parameter as documented here: Installation als Docker-Container
I use Duplicati to backup the data at /data/cmk/monitoring on a daily basis. I assume this is basically enough to backup Checkmk running in Docker since I can stop/remove the container and start it again and everything is still in place. However sometimes, I get such kind of errors from Duplicati when it tries to backup the files (while Checkmk is also running of course):

2023-12-12 00:00:08 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for "/host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593", storing empty metadata
UnixSupport.File+FileAccesException: Unable to access the file "/host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593" with method llistxattr, error: ENOENT (2)
  at UnixSupport.File.GetExtendedAttributes (System.String path, System.Boolean isSymlink, System.Boolean followSymlink) [0x00087] in <3a2c307381104a73ba105ad3712d666f>:0
  at Duplicati.Library.Common.IO.SystemIOLinux.GetMetadata (System.String file, System.Boolean isSymlink, System.Boolean followSymlink) [0x0000d] in <6e4f34e913fd4d89b628a5a76045f19c>:0
  at Duplicati.Library.Snapshots.NoSnapshotLinux.GetMetadata (System.String localPath, System.Boolean isSymlink, System.Boolean followSymlink) [0x00000] in <acb4fdca179441ff87fffb9a1b9a62c8>:0
  at Duplicati.Library.Main.Operation.Backup.MetadataGenerator.GenerateMetadata (System.String path, System.IO.FileAttributes attributes, Duplicati.Library.Main.Options options, Duplicati.Library.Snapshots.ISnapshotService snapshot) [0x0001b] in <e60bc008dd1b454d861cfacbdd3760b9>:0
2023-12-12 00:00:08 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593
System.IO.FileNotFoundException: Could not find file "/host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593"
File name: '/host/mnt/volume1/data/docker/volumes/check-mk-raw/omd/sites/cmk/var/pnp4nagios/spool/perfdata.1702335593'
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <d13c8b563008422a8c5aaec0a74089cc>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at System.IO.File.Open (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) [0x00000] in <d13c8b563008422a8c5aaec0a74089cc>:0
  at Duplicati.Library.Snapshots.SnapshotBase.OpenRead (System.String localPath) [0x00000] in <acb4fdca179441ff87fffb9a1b9a62c8>:0
  at Duplicati.Library.Main.Operation.Backup.FileBlockProcessor+<>c__DisplayClass1_0.<Run>b__0 (<>f__AnonymousType8`2[<Input>j__TPar,<StreamBlockChannel>j__TPar] self) [0x00217] in <e60bc008dd1b454d861cfacbdd3760b9>:0

Seems like some files are in use or not available anymore when Duplicati tries to backup them.

I am thinking of the following options:

Any advice is appreciated.

2

When a Checkmk site is running, it is quite busy working in the site’s file system. It is quite normal that files come and go, or get heavily modified, especially in the var/ and tmp/ directories. I would not backup the site this way, as the files might be corrupted after a restore.

Checkmk has the backup/restore function and the omd backup command which deal with that situation properly. The easiest way to backup a site is to use either of them and then copy the backup archive to another media. You can choose to skip the history (log files and/or performance data) to make the backup smaller.

If you still want to backup the file system, you can stop the site while the backup is running, but this gives holes in the monitoring history. To minimize those holes, you can stop the site, create a snapshot (e.g. with LVM2) of the file system, restart the site, and then backup and finally delete the snapshot.

As an absolute minimum, you should backup the local/, etc/ and var/check_mk/web/ directories of a Checkmk site.

2 Likes
3

OK, I see. Will propably check the built-in backup/restore function then. Thank you, @Heavy, for the detailed answer!