Blocks · Storage and file sharing

share/nfs

NFS export. One NFSv4.1/4.2 export served by NFS-Ganesha; clients reclaim their opens and locks after a failover.

Overview

Serves one NFSv4.1/4.2 export of a replicated volume with NFS-Ganesha, a userspace NFS server. The block is a singleton that runs where its volume's primary is, behind a stable VIP. Ganesha's client-recovery records live on the volume, in .nfs-state, so when the serving node fails the export moves with the data and clients reclaim their opens and locks during the grace period.

Ganesha is used instead of the kernel's nfsd because the kernel server has one export table, one grace period and one recovery directory per host.

Shape
Singleton on a replicated volume
Reached at
VIP
Node package
pkgs.nfs-ganesha

Fetched on demand: the first time a block of this type is placed on a node, its agent fetches pkgs.nfs-ganesha, built from the node's own nixpkgs so the NixOS binary cache serves it. That first placement needs network access (or a binary cache on the LAN) and waits for the download; later placements on that node start at once.

Example

nfs.yaml
apiVersion: expanse.io/v1
kind: Block
metadata:
  name: files
  namespace: default
spec:
  type: share/nfs
  replicas: 1
  strategy:
    kind: SINGLETON
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
  storage:
    - name: files-data
      size: 100Gi
      replication: 3
      mountPath: /mnt/files
  config:
    port: 12049        # never 2049: the VIP holder binds that
    pseudo: /share     # clients mount <VIP>:/share
    path: share        # exported directory inside the volume
    squash: root       # root | none | all
  network:
    ports:
      - name: nfs
        port: 2049
        target_port: 12049
        protocol: tcp
        expose: EXPOSE_VIP
    health_check:
      readiness:
        type: PROBE_TCP
        port: 12049
        period_seconds: 2
$ expanse ctl block apply -f nfs.yaml --dry-run   # validate only
$ expanse ctl block apply -f nfs.yaml

Configuration

Every key is optional. One export per block; deploy another block for another export.

Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.

KeyDefaultMeaning
port12049Ganesha's TCP listen port, the target_port of 2049.
pseudo/shareNFSv4 path clients mount.
pathshareExported directory, relative to the volume; created mode 1777 on first start. Never the volume root, which also holds .nfs-state.
readOnlyfalseExport read-only.
squashrootMap client root (root), nobody (none) or every user (all) to nobody.
gracePeriod90Seconds after a start or failover during which clients reclaim their state.

Using it

$ mount -t nfs4 -o vers=4.1,hard <VIP>:/share /mnt/files

Use hard (the default): during a failover, I/O blocks and then resumes instead of failing. Only NFSv4 over TCP is served: no NFSv3, rpcbind, NLM or UDP.

Exports use sec=sys, so the server trusts the uid a client sends, and every client that can reach the VIP may mount the export. Ganesha sees every client as the VIP address, so per-client address rules cannot work. Keep the VIP on a network you trust.

When a node fails

When the serving node dies, the block, its volume's primary and its VIP move together to a survivor, and I/O on a hard mount blocks meanwhile. The new Ganesha reads the client records from the volume and starts a grace period. Clients that were connected reclaim their opens and locks; grace ends once every recorded client has reclaimed, or when the period runs out. Files kept open across the failure stay valid, so a process writing through one descriptor carries on without remounting.

In Expanse's VM test (gracePeriod: 30) the VIP was on a survivor about 55 s after the serving node was killed, and a writer holding a file open resumed about 76 s after the kill with every acknowledged line on the volume.

Limits

  • No Kerberos (sec=krb5*), NFSv3 or per-client export rules.
  • No quotas: size the volume instead.
  • Only the Linux kernel client is exercised by the VM test.

Source

All block types · Fields every block shares