Every capability below is in the release, documented, and covered by NixOS VM tests that exercise real node kills, partitions and restarts. The last section lists what is deliberately not built yet, so you are never guessing.
The x86_64 ISO is a hybrid image that boots under UEFI and legacy BIOS. A full-screen installer starts automatically; headless machines can be installed over SSH with the same TUI or with a one-file unattended config.
gh:<username>), Review with a red list of what will be destroyed, Progress with a stage, bar and log tail.CONTAINS DATA; unattended installs will not touch them without --force, and --dry-run prints every command without running one.@root @nix @persist @log; the root is wiped on every reboot and only /persist survives. The rest of every disk becomes the LVM volume group expanse./persist and survives.version: 1 disks: layout: auto # auto | single | mirror devices: [/dev/sda] # empty = use every disk network: mode: dhcp ssh: authorized_keys: - ssh-ed25519 AAAA... timezone: UTC
$ expanse install --config expanse-install.yaml --dry-run $ expanse install --config expanse-install.yaml --force
Cluster state, leases and generations live in one Raft-replicated store over mTLS. cluster init bootstraps a single voter; every further node joins with a single-use token and grows the same cluster.
expanse cluster discover lists clusters and unjoined nodes on the L2 network; cluster join --discover finds the join endpoint itself.# on the leader, with its agent stopped: $ expanse cluster token create --ttl 15m --uses 1 expanse-join-… # on the new node, before its agent starts: $ expanse cluster join --address 10.0.0.11:7446 \ --token expanse-join-… joined cluster 5c0b3e2a-… as n2 (10.0.0.12:7444) $ systemctl start expansed $ expanse cluster status quorum: 2/2 nodes: 2
Volumes are DRBD 9 resources (protocol C) on LVM thin volumes, placed by the leader. Replication is a target, not a precondition: a volume lands on one node today and gains replicas as nodes join, one fully synced replica at a time.
volume diverged --choose.UnderReplicated means every replica is healthy but there are fewer than the target — the CLI and UI show "1 of 3 (no redundancy)" rather than a green tick.expanse doctor storage checks the DRBD module, VG headroom, thin-pool data and metadata, the system mirror and every DRBD resource, with a remediation hint on anything short of PASS.GPT ─┬─ ESP (FAT32) → /boot
├─ btrfs partition → @root @nix @persist @log
└─ LVM PV → VG "expanse" → thin pool → thin LVs
└─ DRBD 9 (protocol C, quorum at ≥3)
└─ filesystem or raw (block replica)$ expanse ctl volume create data --size 20Gi $ expanse ctl volume list ID NAME SIZE STATE REPLICAS NODES (PRIMARY) vol-… data 20Gi underreplicated 1/3 n1 (n1) no redundancy # … two nodes join … vol-… data 20Gi healthy 3/3 n1,n2,n3 (n1) $ expanse ctl volume resize data --size 40Gi $ expanse ctl volume snapshot data --name before-migration
A block declares a catalog type, replicas, resources, placement and networking. The leader's scheduler filters and scores nodes; each node's agent renders the unit and supervises it. Health flows back through the store and the block is promoted to RUNNING when every replica is.
maxUnavailable, maxSurge and minReadySeconds are honoured; the rolling-update VM test asserts zero failed health probes during a roll. A mid-roll crash resumes deterministically.kvm), daemonsets, and singletons fenced by a Raft lease so exactly one instance ever runs.DynamicUser, ProtectSystem=strict, PrivateTmp and friends; config is validated against the type's JSON Schema before anything is written.util/echo, web/nginx, web/static-site, web/whoami, db/redis, db/postgres, monitor/node-exporter, ai/ollama, share/smb, iscsi/target, vm/instance. A new type is four files plus one flake entry.apiVersion: expanse.io/v1 kind: Block metadata: { name: web, namespace: default } spec: type: web/nginx version: "1.27" replicas: 3 strategy: kind: active-active update: mode: rolling maxUnavailable: 1 maxSurge: 0 minReadySeconds: 10 resources: requests: { cpu: "500m", memory: "256Mi" } placement: antiAffinity: node # never two replicas on one node network: ports: - name: http port: 80 targetPort: 8080 expose: vip healthCheck: readiness: { type: http, path: /, port: 8080 }
$ expanse ctl block apply -f web.yaml $ expanse ctl block scale web --replicas 5 $ expanse ctl block logs default/web --replica 1 $ expanse ctl block explain web
Every node joins a WireGuard overlay reconciled from the store — a key rotation converges in one pass. Services are reached through VIPs held under a lease, balanced at L4 or L7, and named by cluster DNS.
<block>.<ns>.expanse.local and path prefix with idempotent retries.inet expanse table per node with dynamic sets for peers, VIPs and block ports. After bootstrap it is only ever updated element by element, so conntrack is never dropped.expanse doctor network. Twelve live checks — interfaces, overlay peers and path MTU, the port matrix, exactly-one VIP holder, ARP consistency, DNS, firewall, conntrack, time sync. External clients
│
▼ (VIP 192.168.1.100 — ARP announced by the lease holder)
┌───────────────────────────────────────────────┐
│ Physical LAN 192.168.1.0/24 │
│ n1 .11 n2 .12 n3 .13 │
└───────────────────────────────────────────────┘
│ WireGuard overlay 10.42.0.0/16 (exp0, MTU 1420)
▼ Service VIPs 10.43.0.0/16 (+ a LAN VIP pool you declare)Fleet-wide desired state is versioned inside the Raft store: hash-addressed, append-only, kept for the last 50 generations or 30 days. Rollback restores an earlier generation as a new one, so history never loses a step.
generation diff 11 12 in the CLI, or the Generations page in the web UI: added, removed and changed keys, with rollback behind a confirm dialog.nixos-rebuild switch uses, restarting only what changed and installing a boot entry.UpToDate, drain any volume primary, switch, wait for the agent to rejoin, then the next. A surviving node's reads and writes keep working throughout — proven with a background write loop, not a spot check.# first: quorum reads 3/2 and every volume is UpToDate $ expanse cluster status $ expanse ctl resource apply - \ --socket /run/expanse/agent.sock <<'EOF' nix-config:node: type: nix-config flake: path:/etc/nixos attr: nixosConfigurations.expanse-node.config.system.build.toplevel switch_mode: switch EOF # wait for expansed to return and quorum to read 3/2, # then the next node $ expanse ctl generation list $ expanse ctl generation diff 11 12 $ expanse ctl generation rollback 11 # becomes generation 13
Every node serves the web UI on :8443 as soon as its agent starts, and shows a read-only host console on tty1 in place of a login prompt — the same idea as a hypervisor's local console, for a cluster node.
expanse ctl admin reset-password.expanse console --once prints the same screen over SSH.$ expanse console --once EXPANSE 1.1.9 2026-09-28 17:42:10 UTC ┌─ n1 ─────────────────────────────────────────────────────┐ │ Hostname n1 Uptime 3d 4h 12m │ │ Health HEALTHY │ ├─ Management ─────────────────────────────────────────────┤ │ eno1 10.0.0.11 │ │ Web UI https://10.0.0.11:8443 │ ├─ Cluster ────────────────────────────────────────────────┤ │ Cluster expanse Role leader │ │ Quorum 3/2 (3 nodes) │ └──────────────────────────────────────────────────────────┘ # lost the admin password? reset it locally, shown once: $ expanse ctl admin reset-password \ --socket /run/expanse/agent.sock # trust the UI's own CA in your browser: $ scp root@n1:/persist/expanse/ca/ui-ca.pem .
Expanse does not ship its own metrics database. Every node exports Prometheus metrics for node, resource, volume and quorum health; the web UI's Health page derives the same critical alerts in-process, so the two can never disagree.
/metrics. A dedicated listener on :7447 behind the cluster CA and a bearer token you mint with expanse ctl metrics set-token.scrape_configs: - job_name: expanse scheme: https scrape_interval: 5s tls_config: ca_file: /etc/prometheus/expanse-ca.pem server_name: n1 bearer_token_file: /etc/prometheus/expanse-metrics-token static_configs: - targets: ['10.0.0.11:7447'] rule_files: - deploy/prometheus/expanse-alerts.rules.yml
Each cluster has its own Ed25519 CA; every node holds a certificate for internal mTLS that renews itself every six hours. Rotation is just "renew onto a new CA": both roots are trusted during the transition and nothing restarts.
ca rotate makes the new root primary, ca status lists nodes still pending, ca complete retires the old root — and refuses until every node has renewed, so it is safe to run early.unknown ca; zero cleartext on the join path, verified in the VM tests.$ systemctl stop expansed $ expanse cluster ca rotate && systemctl start expansed CA rotation started: the new CA is now primary; the old CA stays trusted. $ expanse cluster ca status rotating: primary CA fingerprint sha256:… pending nodes (not yet renewed): [n3] # … after every node's renewal loop has ticked … every node has renewed onto the new CA; safe to run `cluster ca complete` $ systemctl stop expansed $ expanse cluster ca complete && systemctl start expansed CA rotation complete: outgoing CA retired.
Expanse does not write a backup engine. Each node's cluster state — identity, CA, node TLS, the Raft log and the generation history — sits under /persist/expanse, which restic backs up to any S3-compatible destination. Volume bytes are captured through thin snapshots.
expanse cluster restore pulls exactly that node's state back and the daemon rejoins under its original identity, with no re-init or re-join step.expanse ctl volume snapshot takes an LVM thin snapshot; restic and dd move the bytes out and back onto the live DRBD device, checksum-equal.$ export RESTIC_REPOSITORY=s3:https://s3.example.net/backups/n1 $ export RESTIC_PASSWORD=… $ restic init # once, per node $ restic backup /persist/expanse # on whatever cadence fits # the node is rebuilt from the ISO; bring its state back: $ expanse cluster restore restored /persist/expanse from the latest backup $ systemctl start expansed # rejoins as the same node
Three shipped block types put real state on the cluster. Each one reuses the same primitives — replicated volumes, leases, VIPs and singleton scheduling — rather than a bespoke mechanism.
db/postgresEach replica runs on its own node with its own volume; PostgreSQL streaming replication is the redundancy. A lease-gated controller picks the primary, standbys clone from it with pg_basebackup, and on primary loss a survivor is promoted live with pg_promote() — no restart. Clients use one VIP routed to the primary only.
iscsi/targetOne raw DRBD-backed volume exported as a single LUN through LIO. The block is a singleton colocated with the volume's primary and exposed at a stable VIP; when its node fails the VIP moves with the new primary and the initiator's own session recovery reconnects to the same portal address.
vm/instanceA QEMU/KVM guest on a raw replicated disk, scheduled only to nodes with hardware virtualization, with a MAC pinned from the block's name so it comes back at the same address. Node loss means a cold boot elsewhere — the guest's filesystem was verified intact under sustained fsync load through the kill. There is no live migration.
Each item below is recorded in the project's own documentation as paused, deferred or a known issue. None of them is quietly missing.
PATH, which an installed node does not yet ship; util/echo runs.The quickstart goes from a USB stick to a one-node cluster with a web console, then adds nodes.