Overview
Runs restic on every node. Each replica backs up only the listed volumes whose primary is on its own node, so exactly one node backs up each volume, and after a failover the survivor's replica takes over without any action. A backup takes an LVM thin snapshot of the volume, streams the volume's bytes from it into the restic repository and drops the snapshot.
The snapshot is crash-consistent: restoring it is like restarting the block after a power cut, which every shipped block with a volume survives. restic deduplicates and encrypts, so each backup uploads only the chunks that changed. When the next backup is due is read from the repository itself, so a replica that restarts or moves keeps the schedule.
- Shape
- Daemonset
- Reached at
- Status endpoint on each node
- Node package
pkgs.restic
Fetched on demand: the first time a block of this type is placed on a node, its agent fetches pkgs.restic, built from the node's own nixpkgs so the NixOS binary cache serves it. That first placement needs network access (or a binary cache on the LAN) and waits for the download; later placements on that node start at once.
Example
apiVersion: expanse.io/v1 kind: Block metadata: name: backup namespace: default spec: type: util/restic-backup strategy: kind: DAEMONSET resources: requests: cpu: 100m memory: 256Mi config: repository: s3:https://s3.example.com/expanse-backups password: a-long-repository-password env: AWS_ACCESS_KEY_ID: GK... AWS_SECRET_ACCESS_KEY: ... volumes: - appdb/appdb-data - forge/forgejo-data interval: 24h keep: daily: 7 weekly: 4 monthly: 6 network: health_check: readiness: type: PROBE_HTTP path: /healthz port: 18900 period_seconds: 30
$ expanse ctl block apply -f restic-backup.yaml --dry-run # validate only $ expanse ctl block apply -f restic-backup.yaml
Configuration
Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.
| Key | Default | Meaning |
|---|---|---|
repository | required | Any restic repository, such as s3:https://host/bucket or sftp:user@host:/path. It is created if it does not exist. |
password | required | Encrypts the repository. Keep a copy outside the cluster: without it the backups cannot be read. Sealed: kept out of the stored block (block secrets). |
env | none | Extra environment for restic, such as the S3 credentials. RESTIC_REPOSITORY and RESTIC_PASSWORD* are set by the block. Not sealed yet: credentials here stay in the block's config. |
volumes | required | <block>/<storage> (a storage entry of a block in this namespace) or a volume name from expanse ctl volume list. |
interval | 24h | Time between backups of each volume, at least 1m. |
keep | daily 7, weekly 4, monthly 6 | Backups kept per volume, as restic forget --keep-* counts: last, hourly, daily, weekly, monthly, yearly. |
port | 18900 | Each replica's status endpoint. |
Using it
The first backup of each volume runs as soon as the block starts. A storage/s3 block in the same cluster works as the repository, but it shares the cluster's fate; keep backups you need after losing the cluster outside it.
Each replica serves its view on port: GET / lists, per volume, whether its primary is on this node, the last backup and the last error. /healthz answers 503, naming the error, while a volume whose primary is here cannot be backed up.
$ curl -s http://<node>:18900/ {"blk-default-appdb-appdb-data":{"primary":true,"lastBackup":"2026-10-08T03:00:12Z","lastAttempt":"2026-10-08T03:00:09Z"}} $ restic snapshots --host blk-default-appdb-appdb-data
Restoring a volume
A backup is the volume's whole device as one file, <volume>.img. Stop the block that uses the volume, write the image onto the volume's DRBD device on its primary, and start the block again. Deleting a block keeps its volumes:
$ expanse ctl block delete appdb $ expanse ctl volume inspect blk-default-appdb-appdb-data # "primary:" names the node to use # on that node, wait until the volume is unmounted (this prints nothing): $ findmnt /var/lib/expanse/volumes/<volume id>/mnt # then, with the repository's RESTIC_* and credential variables set: $ restic dump --host blk-default-appdb-appdb-data latest /blk-default-appdb-appdb-data.img | dd of=$(drbdadm sh-dev <volume id>) bs=1M iflag=fullblock oflag=direct $ expanse ctl block apply -f appdb.yaml
Never write while the volume is mounted. The write goes through DRBD, so every replica receives it. Replace latest with a snapshot ID to restore an older backup, and pause the backup block first if a backup could start while you restore.
When a node fails
Backups follow each volume's primary. When the node holding a primary fails, the volume's new primary is on another node, and that node's replica makes the next backup. In Expanse's VM test the survivor's replica backed up again within 85 s of the crash, re-convergence included.
Limits
- Thin volumes only: nodes without a thin pool cannot snapshot.
- Device images, not files: to pick out files, restore into a scratch volume and mount it.
- Crash-consistent, not application-consistent: a database restores the way it recovers from a crash.
- Restore is manual.
- Cluster state (
/persist/expanse) is backed up separately; see Backup and disaster recovery.
Source
- docs/RESTIC-BACKUP.md: the Expanse guide this page follows.
- nix/blocks/util/restic-backup: the type's identity and JSON Schema.