Linux backup using rclone and restic running in Docker containers

4 January 2022 Off By nonsavant

Intro

I want to back up certain files on my home server (an Intel NUC running Ubuntu) with the following criteria:

Use Microsoft OneDrive as the storage medium
All necessary apps run in Docker
Support for backup features such as deduplication, retention, etc.

After some weeks’ research I decided upon restic + rclone.

The big picture

Restic provides the backup feature set.

Rclone provides the ability to use OneDrive as storage.

Rclone will run as a background service in a docker container, providing a REST interface for restic. I can’t put it any better than this direct quote from the rclone documentation:

rclone serve restic implements restic’s REST backend API over HTTP. This allows restic to use rclone as a data storage mechanism for cloud providers that restic does not support directly.
https://rclone.org/commands/rclone_serve_restic/

Restic will be configured to use the rclone as a repository.

Rclone configuration

Authorising OneDrive for rclone was pretty tricky, in part due to my Docker setup.

A good starting point was the rclone documentation for the authorize command.

And once I realised that I can just use docker run and perform the configuration and authorisation in a ‘throwaway’ container, I started making some progress.

docker run -i -v /user/username/docker/rclone/config:/config rclone/rclone:latest config

Importantly, I’m mounting the host folder where I want to store the configuration for rclone into this temporary container (using the -v flag).

Now just follow rclone’s configuration wizard, obviously choosing onedrive and mostly relying on the default values for:

OAuth Client Id (blank)
Option client_secret (blank)
region (global)
Edit advanced config (no)

With Use auto config (no) being the exception.

Now we need the config token created by the authorize process. As described in the rclone documentation above, the OAuth needs to be performed in a browser. To simplify things, I’m going to run this docker command in a VNC session directly in the console on the server (mostly so that I can click on the generated link to open in the browser).

docker run --rm -i --network host rclone/rclone:latest authorize "onedrive"

What’s going on here?

set network mode to host
no mount necessary, as the output of this container is the access token that will be printed to the terminal
Note, provide the backend name (“onedrive”) as the argument, not the name of the particular configuration

The authorize command supplies a clickable link which should be opened in a browser. After logging into Microsoft using the desired account, the rclone application can be authorised to access OneDrive. Once complete, the access token is displayed in the terminal.

Copy and paste the token into the (still running!) initial docker run session and continue the configuration:

config_type: personal
Drive OK? (yes)
Summary (yes)

That’s the configuration of the remote repository in rclone complete!

Rclone docker setup

Just the important bits from the docker-compose file:

image: rclone/rclone  
container_name: rclone
command: serve restic -v onedrive:backup --addr rclone:9123
volumes:
  - $DOCKERDIR/rclone/config:/config

Note the hostname – restic will be pointed towards http://rclone, which will work as long as both containers are running in the same docker network.

OneDrive setup

Just one simple thing: Create a folder in OneDrive called backup

Restic configuration

The first step is to initialise a restic repository in our rclone remote target. Again, I’m going to use a throwaway container to perform the initialisation and a temporary .env file (restic.env) to provide:

the repository location (this will be the restic backend API provided by the rclone container)
the repository password

RESTIC_REPOSITORY=rest:http://rclone:9123/
RESTIC_PASSWORD=<password>

docker run --rm -i --network docker_default --env-file restic.env restic/restic:latest init

Docker networking is always a challenge. Note:

connect the container to the desired docker network (where rclone is running)
The address of the rclone API uses the hostname of the rclone container (to be found in the appropriate network)

Restic docker configuration

I decided to go for a third-party docker image which extends restic with some additional functionality, such as allowing the configuration of a crontab to automate the backup via a docker-compose environment variable.

As with rclone, here are the important bits of the docker-compose:

  rbackup:
    image: mazzolino/restic
    container_name: rbackup
    hostname: my_hostname
    environment:
      RESTIC_BACKUP_SOURCES: "/data/..."
      RESTIC_REPOSITORY: "rest:http://rclone:9123/"
      RESTIC_PASSWORD_FILE: /run/secrets/restic_password
    volumes:
       - /host/path/to/files:/data