Overview
Serves one NFSv4.1/4.2 export of a replicated volume with NFS-Ganesha, a userspace NFS server. The block is a singleton that runs where its volume's primary is, behind a stable VIP. Ganesha's client-recovery records live on the volume, in .nfs-state, so when the serving node fails the export moves with the data and clients reclaim their opens and locks during the grace period.
Ganesha is used instead of the kernel's nfsd because the kernel server has one export table, one grace period and one recovery directory per host.
- Shape
- Singleton on a replicated volume
- Reached at
- VIP
- Node package
pkgs.nfs-ganesha
Fetched on demand: the first time a block of this type is placed on a node, its agent fetches pkgs.nfs-ganesha, built from the node's own nixpkgs so the NixOS binary cache serves it. That first placement needs network access (or a binary cache on the LAN) and waits for the download; later placements on that node start at once.
Example
apiVersion: expanse.io/v1 kind: Block metadata: name: files namespace: default spec: type: share/nfs replicas: 1 strategy: kind: SINGLETON resources: requests: cpu: 100m memory: 256Mi storage: - name: files-data size: 100Gi replication: 3 mountPath: /mnt/files config: port: 12049 # never 2049: the VIP holder binds that pseudo: /share # clients mount <VIP>:/share path: share # exported directory inside the volume squash: root # root | none | all network: ports: - name: nfs port: 2049 target_port: 12049 protocol: tcp expose: EXPOSE_VIP health_check: readiness: type: PROBE_TCP port: 12049 period_seconds: 2
$ expanse ctl block apply -f nfs.yaml --dry-run # validate only $ expanse ctl block apply -f nfs.yaml
Configuration
Every key is optional. One export per block; deploy another block for another export.
Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.
| Key | Default | Meaning |
|---|---|---|
port | 12049 | Ganesha's TCP listen port, the target_port of 2049. |
pseudo | /share | NFSv4 path clients mount. |
path | share | Exported directory, relative to the volume; created mode 1777 on first start. Never the volume root, which also holds .nfs-state. |
readOnly | false | Export read-only. |
squash | root | Map client root (root), nobody (none) or every user (all) to nobody. |
gracePeriod | 90 | Seconds after a start or failover during which clients reclaim their state. |
Using it
$ mount -t nfs4 -o vers=4.1,hard <VIP>:/share /mnt/files
Use hard (the default): during a failover, I/O blocks and then resumes instead of failing. Only NFSv4 over TCP is served: no NFSv3, rpcbind, NLM or UDP.
Exports use sec=sys, so the server trusts the uid a client sends, and every client that can reach the VIP may mount the export. Ganesha sees every client as the VIP address, so per-client address rules cannot work. Keep the VIP on a network you trust.
When a node fails
When the serving node dies, the block, its volume's primary and its VIP move together to a survivor, and I/O on a hard mount blocks meanwhile. The new Ganesha reads the client records from the volume and starts a grace period. Clients that were connected reclaim their opens and locks; grace ends once every recorded client has reclaimed, or when the period runs out. Files kept open across the failure stay valid, so a process writing through one descriptor carries on without remounting.
In Expanse's VM test (gracePeriod: 30) the VIP was on a survivor about 55 s after the serving node was killed, and a writer holding a file open resumed about 76 s after the kill with every acknowledged line on the volume.
Limits
- No Kerberos (
sec=krb5*), NFSv3 or per-client export rules. - No quotas: size the volume instead.
- Only the Linux kernel client is exercised by the VM test.
Source
- docs/NFS.md: the Expanse guide this page follows.
- nix/blocks/share/nfs: the type's identity and JSON Schema.