Overview
Serves an S3-compatible API with Garage, run as a single-node Garage on a replicated volume. The block is a singleton that runs where its volume's primary is, behind a stable VIP. Everything Garage keeps lives on the volume under .garage/: its metadata database, the object blocks, its node key, and the RPC secret and admin token the block generates on first start. The node that takes over after a failover is therefore the same Garage node, with the same keys and buckets.
Replication is DRBD's job, so Garage runs with replication_factor = 1, keeps its metadata in LMDB, and syncs metadata and data before acknowledging a write.
- Shape
- Singleton on a replicated volume
- Reached at
- VIP
- Node package
pkgs.garage
Fetched on demand: the first time a block of this type is placed on a node, its agent fetches pkgs.garage, built from the node's own nixpkgs so the NixOS binary cache serves it. That first placement needs network access (or a binary cache on the LAN) and waits for the download; later placements on that node start at once.
Example
apiVersion: expanse.io/v1 kind: Block metadata: name: objects namespace: default spec: type: storage/s3 replicas: 1 strategy: kind: SINGLETON resources: requests: cpu: 100m memory: 256Mi storage: - name: objects-data size: 100Gi replication: 3 mountPath: /var/lib/s3 config: port: 13900 # never 3900: the VIP holder binds that accessKeyId: GK0123456789abcdef01234567 secretAccessKey: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef buckets: [media, backups] network: ports: - name: s3 port: 3900 target_port: 13900 protocol: tcp expose: EXPOSE_VIP health_check: readiness: type: PROBE_TCP port: 13900 period_seconds: 2
Generate the key pair once and keep it: GK$(openssl rand -hex 12) and $(openssl rand -hex 32).
$ expanse ctl block apply -f s3.yaml --dry-run # validate only $ expanse ctl block apply -f s3.yaml
Configuration
Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.
| Key | Default | Meaning |
|---|---|---|
accessKeyId | required | GK followed by 24 lowercase hex digits. |
secretAccessKey | required | 64 lowercase hex digits. Fixed once the key exists on the volume: a changed secret makes the block fail to start, so deploy with a new accessKeyId instead. Sealed: kept out of the stored block (block secrets). |
buckets | [] | Buckets created on start and granted to the key (read, write, owner). Removing a name does not delete the bucket. |
region | garage | Region clients sign requests for. |
port | 13900 | Garage's S3 port, the target_port of 3900. |
rpcPort | 13901 | Garage's internal RPC port, on localhost only. |
adminPort | 13903 | Garage's admin API port, on localhost only. |
Using it
Use path-style addressing and the configured region:
$ export AWS_ACCESS_KEY_ID=GK0123456789abcdef01234567 $ export AWS_SECRET_ACCESS_KEY=<secret> $ export AWS_DEFAULT_REGION=garage $ aws --endpoint-url http://<VIP>:3900 s3 cp ./file s3://media/file
restic takes s3:http://<VIP>:3900/backups as its repository. The key may also create its own buckets. Two instances on one node need distinct port, rpcPort and adminPort.
The API is plain HTTP: requests are signed, but payloads are not encrypted. Keep the VIP on a network you trust, or put a TLS proxy such as web/caddy in front.
When a node fails
When the serving node dies, the block, its volume's primary and its VIP move together to a survivor. Requests sent meanwhile fail or time out; clients retry against the same endpoint and key. Object data is synced before Garage acknowledges a write. Garage opens LMDB with MDB_NOMETASYNC, so a crash can undo the very last metadata commit; a client that must not lose a write should read it back after a failover.
In Expanse's VM test the VIP was on a survivor 58 to 62 s after the serving node was killed, and a client retrying failed PUTs had them acknowledged again under a second later. Every acknowledged object read back intact.
Limits
- No TLS, virtual-host-style bucket addressing or Garage's website hosting.
- One access key, with read, write and owner permission on its buckets.
- No Garage multi-node replication: one block is one Garage node, replicated by DRBD.
- No quotas: size the volume instead.
Source
- docs/S3.md: the Expanse guide this page follows.
- nix/blocks/storage/s3: the type's identity and JSON Schema.