Linux backup using rclone and restic running in Docker containers4 January 2022
I want to back up certain files on my home server (an Intel NUC running Ubuntu) with the following criteria:
- Use Microsoft OneDrive as the storage medium
- All necessary apps run in Docker
- Support for backup features such as deduplication, retention, etc.
After some weeks’ research I decided upon restic + rclone.
The big picture
Restic provides the backup feature set.
Rclone provides the ability to use OneDrive as storage.
Rclone will run as a background service in a docker container, providing a REST interface for restic. I can’t put it any better than this direct quote from the rclone documentation:
rclone serve restic implements restic’s REST backend API over HTTP. This allows restic to use rclone as a data storage mechanism for cloud providers that restic does not support directly.https://rclone.org/commands/rclone_serve_restic/
Restic will be configured to use the rclone as a repository.
Authorising OneDrive for rclone was pretty tricky, in part due to my Docker setup.
A good starting point was the rclone documentation for the authorize command.
And once I realised that I can just use docker run and perform the configuration and authorisation in a ‘throwaway’ container, I started making some progress.
docker run -i -v /user/username/docker/rclone/config:/config rclone/rclone:latest config
Importantly, I’m mounting the host folder where I want to store the configuration for rclone into this temporary container (using the -v flag).
Now just follow rclone’s configuration wizard, obviously choosing onedrive and mostly relying on the default values for:
- OAuth Client Id (blank)
- Option client_secret (blank)
- region (global)
- Edit advanced config (no)
With Use auto config (no) being the exception.
Now we need the config token created by the authorize process. As described in the rclone documentation above, the OAuth needs to be performed in a browser. To simplify things, I’m going to run this docker command in a VNC session directly in the console on the server (mostly so that I can click on the generated link to open in the browser).
docker run --rm -i --network host rclone/rclone:latest authorize "onedrive"
What’s going on here?
- set network mode to host
- no mount necessary, as the output of this container is the access token that will be printed to the terminal
- Note, provide the backend name (“onedrive”) as the argument, not the name of the particular configuration
The authorize command supplies a clickable link which should be opened in a browser. After logging into Microsoft using the desired account, the rclone application can be authorised to access OneDrive. Once complete, the access token is displayed in the terminal.
Copy and paste the token into the (still running!) initial docker run session and continue the configuration:
- config_type: personal
- Drive OK? (yes)
- Summary (yes)
That’s the configuration of the remote repository in rclone complete!
Rclone docker setup
Just the important bits from the docker-compose file:
image: rclone/rclone container_name: rclone command: serve restic -v onedrive:backup --addr rclone:9123 volumes: - $DOCKERDIR/rclone/config:/config
Note the hostname – restic will be pointed towards http://rclone, which will work as long as both containers are running in the same docker network.
Just one simple thing: Create a folder in OneDrive called backup
The first step is to initialise a restic repository in our rclone remote target. Again, I’m going to use a throwaway container to perform the initialisation and a temporary .env file (restic.env) to provide:
- the repository location (this will be the restic backend API provided by the rclone container)
- the repository password
docker run --rm -i --network docker_default --env-file restic.env restic/restic:latest init
Docker networking is always a challenge. Note:
- connect the container to the desired docker network (where rclone is running)
- The address of the rclone API uses the hostname of the rclone container (to be found in the appropriate network)
Restic docker configuration
I decided to go for a third-party docker image which extends restic with some additional functionality, such as allowing the configuration of a crontab to automate the backup via a docker-compose environment variable.
As with rclone, here are the important bits of the docker-compose:
rbackup: image: mazzolino/restic container_name: rbackup hostname: my_hostname environment: RESTIC_BACKUP_SOURCES: "/data/..." RESTIC_REPOSITORY: "rest:http://rclone:9123/" RESTIC_PASSWORD_FILE: /run/secrets/restic_password volumes: - /host/path/to/files:/data
View rclone log
docker logs -f rclone
View rclone configuration
docker exec -it rclone rclone config show
View restic log
docker logs -f rbackup
docker exec -it rbackup restic snapshots
List contents of a snapshot
docker exec -it rbackup restic ls latest|<snapshot_id>
Find a file
docker exec -it rbackup restic find <filename>
Compare two snapshots
docker exec -it rbackup restic diff