Blocks · Utilities

util/restic-backup

Scheduled restic backups. Backs up block volumes on a schedule from crash-consistent snapshots, following each volume's primary.

Overview

Runs restic on every node. Each replica backs up only the listed volumes whose primary is on its own node, so exactly one node backs up each volume, and after a failover the survivor's replica takes over without any action. A backup takes an LVM thin snapshot of the volume, streams the volume's bytes from it into the restic repository and drops the snapshot.

The snapshot is crash-consistent: restoring it is like restarting the block after a power cut, which every shipped block with a volume survives. restic deduplicates and encrypts, so each backup uploads only the chunks that changed. When the next backup is due is read from the repository itself, so a replica that restarts or moves keeps the schedule.

Shape
Daemonset
Reached at
Status endpoint on each node
Node package
pkgs.restic

Fetched on demand: the first time a block of this type is placed on a node, its agent fetches pkgs.restic, built from the node's own nixpkgs so the NixOS binary cache serves it. That first placement needs network access (or a binary cache on the LAN) and waits for the download; later placements on that node start at once.

Example

restic-backup.yaml
apiVersion: expanse.io/v1
kind: Block
metadata:
  name: backup
  namespace: default
spec:
  type: util/restic-backup
  strategy:
    kind: DAEMONSET
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
  config:
    repository: s3:https://s3.example.com/expanse-backups
    password: a-long-repository-password
    env:
      AWS_ACCESS_KEY_ID: GK...
      AWS_SECRET_ACCESS_KEY: ...
    volumes:
      - appdb/appdb-data
      - forge/forgejo-data
    interval: 24h
    keep:
      daily: 7
      weekly: 4
      monthly: 6
  network:
    health_check:
      readiness:
        type: PROBE_HTTP
        path: /healthz
        port: 18900
        period_seconds: 30
$ expanse ctl block apply -f restic-backup.yaml --dry-run   # validate only
$ expanse ctl block apply -f restic-backup.yaml

Configuration

Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.

KeyDefaultMeaning
repositoryrequiredAny restic repository, such as s3:https://host/bucket or sftp:user@host:/path. It is created if it does not exist.
passwordrequiredEncrypts the repository. Keep a copy outside the cluster: without it the backups cannot be read. Sealed: kept out of the stored block (block secrets).
envnoneExtra environment for restic, such as the S3 credentials. RESTIC_REPOSITORY and RESTIC_PASSWORD* are set by the block. Not sealed yet: credentials here stay in the block's config.
volumesrequired<block>/<storage> (a storage entry of a block in this namespace) or a volume name from expanse ctl volume list.
interval24hTime between backups of each volume, at least 1m.
keepdaily 7, weekly 4, monthly 6Backups kept per volume, as restic forget --keep-* counts: last, hourly, daily, weekly, monthly, yearly.
port18900Each replica's status endpoint.

Using it

The first backup of each volume runs as soon as the block starts. A storage/s3 block in the same cluster works as the repository, but it shares the cluster's fate; keep backups you need after losing the cluster outside it.

Each replica serves its view on port: GET / lists, per volume, whether its primary is on this node, the last backup and the last error. /healthz answers 503, naming the error, while a volume whose primary is here cannot be backed up.

$ curl -s http://<node>:18900/
{"blk-default-appdb-appdb-data":{"primary":true,"lastBackup":"2026-10-08T03:00:12Z","lastAttempt":"2026-10-08T03:00:09Z"}}
$ restic snapshots --host blk-default-appdb-appdb-data

Restoring a volume

A backup is the volume's whole device as one file, <volume>.img. Stop the block that uses the volume, write the image onto the volume's DRBD device on its primary, and start the block again. Deleting a block keeps its volumes:

$ expanse ctl block delete appdb
$ expanse ctl volume inspect blk-default-appdb-appdb-data   # "primary:" names the node to use
# on that node, wait until the volume is unmounted (this prints nothing):
$ findmnt /var/lib/expanse/volumes/<volume id>/mnt
# then, with the repository's RESTIC_* and credential variables set:
$ restic dump --host blk-default-appdb-appdb-data latest /blk-default-appdb-appdb-data.img | dd of=$(drbdadm sh-dev <volume id>) bs=1M iflag=fullblock oflag=direct
$ expanse ctl block apply -f appdb.yaml

Never write while the volume is mounted. The write goes through DRBD, so every replica receives it. Replace latest with a snapshot ID to restore an older backup, and pause the backup block first if a backup could start while you restore.

When a node fails

Backups follow each volume's primary. When the node holding a primary fails, the volume's new primary is on another node, and that node's replica makes the next backup. In Expanse's VM test the survivor's replica backed up again within 85 s of the crash, re-convergence included.

Limits

  • Thin volumes only: nodes without a thin pool cannot snapshot.
  • Device images, not files: to pick out files, restore into a scratch volume and mount it.
  • Crash-consistent, not application-consistent: a database restores the way it recovers from a crash.
  • Restore is manual.
  • Cluster state (/persist/expanse) is backed up separately; see Backup and disaster recovery.

Source

All block types · Fields every block shares