Schedule Unattended Remote Backups Using a Hardened Restic Container
About • Built With • Prerequisites • Testing • Deployment • Usage • Contributing • Credits • Donate • License
Restic is a fast and secure backup program. It supports many backends for storing backups natively, including AWS S3, Openstack Swift, Backblaze B2, Microsoft Azure Blob Storage, and Google Cloud Storage. Restic-unattended is a helper utility written in Go to run automated restic backups using a built-in scheduler. Running as an unprivileged and hardened Docker container, restic-unattended simplifies the management of credentials and other sensitive data by using Docker secrets.
Looking for testers. Restic-unattended has been integration tested with Backblaze B2. Your feedback on the integration with any other supported backend is much appreciated.
The project uses the following core software components:
- Cobra - Go library to generate CLI applications (including Viper and pflag)
- Cron - Go library to schedule jobs using cron notation
- Dbm - Helper utility to build, harden, and deploy Docker images
- Docker - Open-source container platform
- Restic - Secure backup program
Restic-unattended can run on any Docker-capable host. The setup has been tested locally on macOS Big Sur and in production on a server running Ubuntu 20.04 LTS. Cloud storage has been tested with Backblaze B2, although other storage providers are supported too.
-
Docker Engine and Docker Compose are required - restic-unattended is intended to be deployed as a Docker container using Docker Compose for convenience. Docker Swarm is a prerequisite to enable Docker secrets, however, the use of Docker secrets itself is optional. This reference guide explains how to initialize Docker Swarm on your host.
-
A storage provider is required - Restic supports several storage providers out of the box. Cloud providers include Amazon S3, Minio Server, Wasabi, OpenStack Swift, Backblaze B2, Microsoft Azure Blob Storage, and Google Cloud Storage. Next to that, local backups are supported too, as well as storage via SFTP, a REST server, or rclone. See the restic documentation for more details.
It is recommended to test the services locally before deploying them to a production environment. Below four steps enable you to run the services on your local machine and validate they are working correctly. The configuration examples use Backblaze B2 as a storage provider. Check the restic documentation for the configuration of other storage providers.
The first step is to clone the repository to a local folder. Assuming you are in the working folder of your choice, clone the repository files with git clone. Git automatically creates a new folder restic-unattended and copies the files to this directory. Change your working folder to be prepared for the next steps.
local:~$ git clone --recurse-submodules https://github.com/markdumay/restic-unattended.git
local:~$ cd restic-unattendedThe repository uses dbm to simplify the build and deployment process. Set up an alias to simplify the execution of dbm.
local:~/restic-unattended$ alias dbm="dbm/dbm.sh" Add the same line to your shell settings (e.g. ~/.zshrc on macOS or ~/.bashrc on Ubuntu with bash login) to make the alias persistent.
The docker-compose.yml file uses environment variables to simplify the configuration. You can use the sample file in the repository as a starting point. The provided sample configuration applies to the production environment.
local:~/restic-unattended$ mv sample.env .envPending on your selected storage provider, you will need to specify the tokens and/or account credentials for restic to be able to connect with the provider. You can either specify the credentials as environment variables or as Docker secrets. In this example, we will use Docker secrets. See the project Wiki for a detailed overview of both configuration options.
As regular Docker containers do not support external Swarm secrets, we will create local secret files for testing purposes. The credentials are stored in plain text, so this is not recommended for production. Add the secrets to docker/docker-compose.yml and docker/docker-compose.dev.yml to provide the credentials for the restic-unattended container. The docker-compose.dev.yml extends the base file docker-compose.yml to simplify the debugging, building, and running of Docker images.
Mounting file-based secrets from your working directory does not work with user namespaces enabled. See the documentation for more details.
Ensure the following configuration settings are defined in docker-compose.yml. Please note that the sensitive environment variables now have a _FILE suffix and point to the location /run/secrets/. Docker mounts secrets to this location within the container by default.
version: "3.7"
secrets:
RESTIC_REPOSITORY:
external: true
RESTIC_PASSWORD:
external: true
B2_ACCOUNT_ID:
external: true
B2_ACCOUNT_KEY:
external: true
[...]
services:
restic:
[...]
environment:
- RESTIC_REPOSITORY_FILE=/run/secrets/RESTIC_REPOSITORY
- RESTIC_PASSWORD_FILE=/run/secrets/RESTIC_PASSWORD
- B2_ACCOUNT_ID_FILE=/run/secrets/B2_ACCOUNT_ID
- B2_ACCOUNT_KEY_FILE=/run/secrets/B2_ACCOUNT_KEY
secrets:
- RESTIC_REPOSITORY
- RESTIC_PASSWORD
- B2_ACCOUNT_ID
- B2_ACCOUNT_KEYThe development configuration overrides the base configuration defined in the previous section. Point to the file-based secrets with the below configuration in docker-compose.dev.yml.
secrets:
RESTIC_REPOSITORY:
file: secrets/RESTIC_REPOSITORY
external: false
RESTIC_PASSWORD:
file: secrets/RESTIC_PASSWORD
external: false
B2_ACCOUNT_ID:
file: secrets/B2_ACCOUNT_ID
external: false
B2_ACCOUNT_KEY:
file: secrets/B2_ACCOUNT_KEY
external: falseThe final step is to create the file-based secrets themselves. Replace the XXX values with your credentials.
local:~/restic-unattended$ mkdir secrets
local:~/restic-unattended$ printf XXXXX > secrets/RESTIC_REPOSITORY
local:~/restic-unattended$ printf XXXXX > secrets/RESTIC_PASSWORD
local:~/restic-unattended$ printf XXXXXXXXXXXXXXXXXXXXXXXXX > secrets/B2_ACCOUNT_ID
local:~/restic-unattended$ printf XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > secrets/B2_ACCOUNT_KEYThe repository contains a helper script to run the Docker container. Use the below commands to build and run an image with built-in shell support for debugging.
local:~/restic-unattended$ dbm dev build
local:~/restic-unattended$ dbm dev up -tThe script then validates the host machine, identifies the targeted image, and brings up the container and network. You should now be logged in to the container's shell. Try a few commands, such as restic-unattended list to verify everything is working as expected. See the Wiki for guidance on how to conduct a more elaborate integration test from within the container.
The steps for deploying in production are slightly different than for local testing. The next four steps highlight the changes.
Unchanged
The sample configuration writes log messages and errors with a timestamp to the standard output. It also defines several constraints following recommendations from the Docker Bench for Security. Finally, the production image defines a RESTIC_CMD to enable unattended backups. Ensure the docker-compose.prod.yml file captures the following settings.
services:
restic:
[...]
environment:
- RESTIC_LOGLEVEL=${RESTIC_LOGLEVEL}
- RESTIC_LOGFORMAT=${RESTIC_LOGFORMAT}
- RESTIC_TIMESTAMP=${RESTIC_TIMESTAMP}
command: "${RESTIC_CMD}"
deploy:
[...]
resources:
limits:
cpus: "${RESTIC_LIMIT_CPU}"
memory: "${RESTIC_LIMIT_MEM}"
reservations:
cpus: "${RESTIC_RESERVATION_CPU}"
memory: "${RESTIC_RESERVATION_MEM}"Adjust the logging settings and the deployment configuration for the CPU and allocated memory in the .env file as needed. The defined command runs a scheduled backup every 15 minutes and removes obsolete snapshots following a policy every day at 01:00 am.
RESTIC_LOGLEVEL=info
RESTIC_LOGFORMAT=pretty
RESTIC_LIMIT_CPU='0.25'
RESTIC_LIMIT_MEM='100M'
RESTIC_RESERVATION_CPU='0.05'
RESTIC_RESERVATION_MEM='6M'
RESTIC_CMD=restic-unattended schedule '0/15 * * * *' -p=/data/backup --forget='0 1 * * *' --keep-last=5 --keep-daily=7 --keep-weekly=13 --sustainedInstead of file-based secrets, you will now create more secure secrets. Docker secrets can be easily created using pipes. Do not forget to include the final -, as this instructs Docker to use piped input. Update the values as needed.
local:~/restic-unattended$ printf XXXXX | docker secret create RESTIC_REPOSITORY -
local:~/restic-unattended$ printf XXXXX | docker secret create RESTIC_PASSWORD -
local:~/restic-unattended$ printf XXXXXXXXXXXXXXXXXXXXXXXXX | docker secret create B2_ACCOUNT_ID -
local:~/restic-unattended$ printf XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX | docker secret create B2_ACCOUNT_KEY -If you do not feel comfortable copying secrets from your command line, you can use the wrapper create_secret.sh. This script prompts for a secret and ensures sensitive data is not displayed on your console. The script is available in the folder ./docker-secret of your repository.
local:~/restic-unattended$ ./docker-secret/create_secret.sh RESTIC_REPOSITORY
local:~/restic-unattended$ ./docker-secret/create_secret.sh RESTIC_PASSWORD
local:~/restic-unattended$ ./docker-secret/create_secret.sh B2_ACCOUNT_ID
local:~/restic-unattended$ ./docker-secret/create_secret.sh B2_ACCOUNT_KEYDocker Swarm is needed to support external Docker secrets. As such, the services will be deployed as a Docker stack in production. The helper script dbm generates the configuration using the applicable .yml files and deploys the services to the restic stack.
local:~/restic-unattended$ dbm prod build
local:~/restic-unattended$ dbm prod deployCheck the status of the Docker service with the following command. The built-in health check kicks in after 5 minutes by default, by which the status should have changed from starting to healthy.
local:~/restic-unattended$ docker psRun the following command to inspect the status of the Docker stack itself.
local:~/restic-unattended$ docker stack services resticYou should see the value 1/1 for REPLICAS for the restic service if the stack was initialized correctly. You can view the service logs with docker service logs restic_restic once the service is up and running. Debugging Swarm services can be quite tedious. If for some reason your service does not initiate properly, you can get its task ID with docker service ps restic_restic. Running docker inspect <task-id> might give you clues to what is happening. Use docker stack rm restic to remove the Docker stack entirely.
Restic-unattended is intended to run from within a Docker container as an unattended service. As such, the most common use case is to define a schedule command in the docker/docker-compose.yml file. In production, it is recommended to add the --sustained flag to ensure restic-unattended keeps running despite errors. Be sure to use the [""] notation for the cmd specification, as the production-ready image has no built-in shell.
Restic-unattended can also run from the command line (either from the host or from within a development container). The below list describes several scenarios.
- From source (command line) - the host machine requires Go 1.16 or later to be installed. From the
srcdirectory, rungo run main.gofollowed by a specific command. - From source (Visual Studio Code) - an
example-launch.jsonfile is included in the repository. Copy the configuration to.vscode/launch.jsonand setenvandargsas needed. The Go language tools need to be installed too. - From within a container - spin up a development container with
dbm dev up -t. Runrestic-unattendedfrom within the container.
Several commands and flags are supported, which are described in the project Wiki. They can also be inspected by using either restic-unattended -h or restic-unattended <command> -h.
- Clone the repository and create a new branch
local:~$ git checkout https://github.com/markdumay/restic-unattended.git -b name_for_new_branch
- Make and test the changes
- Submit a Pull Request with a comprehensive description of the changes
The scheduler routine of restic-unattended is inspired by the following blog article:
- OpsDash by RapidLoop - Job Queues in Go
The restic-unattended codebase is released under the MIT license. The README.md file, and files in the "wiki" repository are licensed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.