This repository helps setting up and restoring backups using borgmatic.
Clone this repository on the server that needs to be backed up:
git clone https://github.com/redpencilio/app-borgmatic.gitEnsure mu-cli is available on the machine.
Navigate to the cloned repository:
cd app-borgmaticGenerate an SSH key pair using mu-cli. The key will be authorized on the remote storage box. Therefore you will need to enter the user's password of the storage box interactively.
mu script project-scripts ssh-key add <user>@<host>:<port>Next, move the generated SSH key pair to the server's .ssh/ directory:
rm -r ~/.ssh/id_borgmatic/ ; mv id_borgmatic{,.pub} ~/.ssh/Borgmatic is now authorized to execute borg commands on the remote storage box using the generated SSH key pair ~/.ssh/id_borgmatic.
This step needs to be executed only once during the initial setup of the backups. It doesn't need to be repeated in the future when new applications are added to the backup config.
When using Hetzner storage boxes for backup, ensure to enable SSH access (which includes Borg support) on the storage box.
Generate a Borgmatic configuration for the application that needs to be backed up using mu-cli.
mu script project-scripts generate-backup-config app <user>@<host>:<port> <hostname> <app-name>E.g.
mu script project-scripts generate-backup-config app u1234-sub1@u1234.your-storagebox.de:23 abb-croco app-mandatendatabankThe script will generate a new config in ./config/borgmatic.d/<app-name>.yml and update docker-compose.override.yml accordingly.
Open the config file and make sure source_directories contain the folders that need to be backed up.
Next, (re)up the Borgmatic stack:
docker compose up -dInitialize the backup repository for the application:
docker compose exec borgmatic borgmatic init --repository <app-name> --encryption repokey --append-onlyFinally, export the repository key and store it somewhere save together with the passphrase and repository path generated by the generate-backup-config script. You will need those to be able to restore backups.
docker compose exec borgmatic borgmatic key export --repository <app-name>This step needs to be repeated for each application that requires a backup. For backups of app-http-logger, see How to backup app-http-logger.
If not done yet, setup SSH access from the machine to the backup server following the guide 'Setup SSH key pair to connect to the remote storage box (only once)' in How to setup backups for a semantic.works application.
Next, generate a config to backup HTTP logs generated by app-http-logger using mu-cli:
mu script project-scripts generate-backup-config http-log <user>@<host>:<port> <hostname> <app-name>E.g.
mu script project-scripts generate-backup-config http-log u1234-sub1@u1234.your-storagebox.de:23 abb-croco app-http-loggerThe script will generate a new config in ./config/borgmatic.d/<app-name>.yml and update docker-compose.override.yml accordingly.
Next, (re)up the Borgmatic stack:
docker compose up -dInitialize the backup repository for the application:
docker compose exec borgmatic borgmatic init --repository app-http-logger --encryption repokey --append-onlyFinally, export the repository key and store it somewhere save together with the passphrase and repository path generated by the generate-backup-config script. You will need those to be able to restore backups.
docker compose exec borgmatic borgmatic key export --repository app-http-loggerAssuming you followed the guides to setup backups for your application, you can manually trigger the creation of a backup via:
docker compose exec borgmatic borgmatic create --stats --repository <app-name>You can optionally provide --progress and -v 1 to have more extensive log output.
If the backup succeeds, the backup archive will be listed when executing:
docker compose exec borgmatic borgmatic list --repository <app-name>Change the backup frequency by updating the BACKUP_CRON environment variable on the borgmatic service in docker-compose.yml.
This pattern applies on all repositories configured in ./config/borgmatic.d. It's currently not possible to configure a different pattern per backup repository.
Next, up the stack again:
docker compose up -dThis how-to guide assumes docker-compose.dev.yml is automatically taken into account on your local machine.
First, we need to make sure we can access the remote backup server from our local machine. If you already have an SSH key with access to the storage box, put the key pair in ./ssh-keys/id_borgmatic{,.pub}. Otherwise, we will generate an SSH key and grant ourselves (temporary) access. Therefore, you will need to enter the password of the storage box interactively.
mu script project-scripts ssh-key add <user>@<host>:<port>When using Hetzner storage boxes, make sure to enable 'External reachability' on the (sub)account such that we can reach the storage box from a machine that is not in the Hetzner network.
Next, we will generate a minimalistic Borgmatic configuration to access the remote backup repository.
mu script project-scripts generate-restore-config <repository_path> <passphrase>E.g.
mu script project-scripts generate-restore-config ssh://u1234-sub1@u1234.your-storagebox.de:23/./abb-croco-app-mandatendatabank.borg my-secret-passphraseNext, start the application stack.
docker compose up -dYou should now be able to access the backup repository on the remote server via the borgmatic-restore service.
docker compose exec borgmatic-restore borgmatic listIn order to restore files, exec in the borgmatic-restore service:
docker compose exec borgmatic-restore bashInside the container, mount a repository archive (e.g. latest):
borgmatic mount --archive latest --mount-point /mntYou can now inspect the files in /mnt/. Use the bind-mounted /restore folder in the container to copy files to ./data/restore on the host machine.
E.g.
cp /mnt/data/app-mandatendatabank/docker-compose.yml /restoreWhen done, unmount the archive and exit the container.
umount /mnt && exitDon't forget, when you're finished, to remove the SSH key access from the backup server again:
mu script project-scripts ssh-key rm <user>@<host>:<port>This how-to-guide assumes a metrics stack including a node-exporter service is running on your server (e.g. in /data/metrics). How to setup such a stack is explained in app-server-monitor.
The app-borgmatic stack includes a borgmatic-exporter container to export metrics for Prometheus. The metrics are written to a text file in ./data/borgmatic-exporter/metrics/, ready to be fed to a node-exporter textfile collector.
Open the docker-compose.yml of the metrics stack and update the node-exporter service with a new volume mount and matching command argument:
services:
exporter:
image: quay.io/prometheus/node-exporter
...
volumes:
- ...
- /data/app-borgmatic/data/borgmatic-exporter/metrics:/data/borgmatic-metrics:ro
command:
- "--collector.textfile.directory=/data/borgmatic-metrics"Next, restart the exporter service:
docker compose up -d exporterThe Borgmatic metrics will now be collected my Prometheus and will be available for visualization in Grafana.
Any borgmatic command (see Borgmatic command-line reference) can be executed by using exec in the borgmatic-restore container.
docker compose exec borgmatic-restore borgmatic <any-borgmatic-command>By default, the command will be executed on all repositories configured in ./config/borgmatic.d. To specify a specific repository, use the --repository option.
E.g.
docker compose exec borgmatic-restore borgmatic list --repository app-mandatendatabankAny borg command (see borg usage guide) can be executed through borgmatic by executing:
docker compose exec borgmatic-restore borgmatic borg <any-borg-command>This has the advantage of using the borgmatic configuration, simplifying the underlying borg command.