Blocks · Storage and file sharing

storage/s3

S3 object storage. An S3-compatible object store on Garage, with metadata, objects and keys on a replicated volume.

Overview

Serves an S3-compatible API with Garage, run as a single-node Garage on a replicated volume. The block is a singleton that runs where its volume's primary is, behind a stable VIP. Everything Garage keeps lives on the volume under .garage/: its metadata database, the object blocks, its node key, and the RPC secret and admin token the block generates on first start. The node that takes over after a failover is therefore the same Garage node, with the same keys and buckets.

Replication is DRBD's job, so Garage runs with replication_factor = 1, keeps its metadata in LMDB, and syncs metadata and data before acknowledging a write.

Shape
Singleton on a replicated volume
Reached at
VIP
Node package
pkgs.garage

Fetched on demand: the first time a block of this type is placed on a node, its agent fetches pkgs.garage, built from the node's own nixpkgs so the NixOS binary cache serves it. That first placement needs network access (or a binary cache on the LAN) and waits for the download; later placements on that node start at once.

Example

s3.yaml
apiVersion: expanse.io/v1
kind: Block
metadata:
  name: objects
  namespace: default
spec:
  type: storage/s3
  replicas: 1
  strategy:
    kind: SINGLETON
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
  storage:
    - name: objects-data
      size: 100Gi
      replication: 3
      mountPath: /var/lib/s3
  config:
    port: 13900          # never 3900: the VIP holder binds that
    accessKeyId: GK0123456789abcdef01234567
    secretAccessKey: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
    buckets: [media, backups]
  network:
    ports:
      - name: s3
        port: 3900
        target_port: 13900
        protocol: tcp
        expose: EXPOSE_VIP
    health_check:
      readiness:
        type: PROBE_TCP
        port: 13900
        period_seconds: 2

Generate the key pair once and keep it: GK$(openssl rand -hex 12) and $(openssl rand -hex 32).

$ expanse ctl block apply -f s3.yaml --dry-run   # validate only
$ expanse ctl block apply -f s3.yaml

Configuration

Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.

KeyDefaultMeaning
accessKeyIdrequiredGK followed by 24 lowercase hex digits.
secretAccessKeyrequired64 lowercase hex digits. Fixed once the key exists on the volume: a changed secret makes the block fail to start, so deploy with a new accessKeyId instead. Sealed: kept out of the stored block (block secrets).
buckets[]Buckets created on start and granted to the key (read, write, owner). Removing a name does not delete the bucket.
regiongarageRegion clients sign requests for.
port13900Garage's S3 port, the target_port of 3900.
rpcPort13901Garage's internal RPC port, on localhost only.
adminPort13903Garage's admin API port, on localhost only.

Using it

Use path-style addressing and the configured region:

$ export AWS_ACCESS_KEY_ID=GK0123456789abcdef01234567
$ export AWS_SECRET_ACCESS_KEY=<secret>
$ export AWS_DEFAULT_REGION=garage
$ aws --endpoint-url http://<VIP>:3900 s3 cp ./file s3://media/file

restic takes s3:http://<VIP>:3900/backups as its repository. The key may also create its own buckets. Two instances on one node need distinct port, rpcPort and adminPort.

The API is plain HTTP: requests are signed, but payloads are not encrypted. Keep the VIP on a network you trust, or put a TLS proxy such as web/caddy in front.

When a node fails

When the serving node dies, the block, its volume's primary and its VIP move together to a survivor. Requests sent meanwhile fail or time out; clients retry against the same endpoint and key. Object data is synced before Garage acknowledges a write. Garage opens LMDB with MDB_NOMETASYNC, so a crash can undo the very last metadata commit; a client that must not lose a write should read it back after a failover.

In Expanse's VM test the VIP was on a survivor 58 to 62 s after the serving node was killed, and a client retrying failed PUTs had them acknowledged again under a second later. Every acknowledged object read back intact.

Limits

  • No TLS, virtual-host-style bucket addressing or Garage's website hosting.
  • One access key, with read, write and owner permission on its buckets.
  • No Garage multi-node replication: one block is one Garage node, replicated by DRBD.
  • No quotas: size the volume instead.

Source

All block types · Fields every block shares