Blocks · Web and proxies

web/caddy

Caddy web server and reverse proxy. Caddy with automatic HTTPS; its certificates, ACME account and CA fail over on a replicated volume.

Overview

Runs Caddy as a web server and reverse proxy with automatic HTTPS. The block is a singleton that runs where its volume's primary is, behind one VIP that serves both 80 and 443. Caddy's data directory is on the volume: certificates and their keys, the ACME account and Caddy's internal CA. After a failover the new node serves the same certificates instead of requesting new ones, which matters under Let's Encrypt's rate limits.

The block writes the Caddyfile from sites on every start. To write your own, give a whole caddyfile instead.

Shape
Singleton on a replicated volume
Reached at
VIP on 80 and 443
Node package
pkgs.caddy

Fetched on demand: the first time a block of this type is placed on a node, its agent fetches pkgs.caddy, built from the node's own nixpkgs so the NixOS binary cache serves it. That first placement needs network access (or a binary cache on the LAN) and waits for the download; later placements on that node start at once.

Example

caddy.yaml
apiVersion: expanse.io/v1
kind: Block
metadata:
  name: edge
  namespace: default
spec:
  type: web/caddy
  replicas: 1
  strategy:
    kind: SINGLETON
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  storage:
    - name: caddy-data
      size: 1Gi
      replication: 3
      mountPath: /var/lib/caddy
  config:
    email: ops@example.com
    sites:
      - host: app.example.com
        reverseProxy: ["10.0.0.21:8080", "10.0.0.22:8080"]
      - host: status.example.com
        tls: "off"
        respond: ok
  network:
    ports:
      - name: http
        port: 80
        target_port: 18080
        protocol: tcp
        expose: EXPOSE_VIP
      - name: https
        port: 443
        target_port: 18443
        protocol: tcp
        expose: EXPOSE_VIP
    health_check:
      readiness:
        type: PROBE_TCP
        port: 18080
        period_seconds: 2
$ expanse ctl block apply -f caddy.yaml --dry-run   # validate only
$ expanse ctl block apply -f caddy.yaml

Configuration

Give either sites or caddyfile.

Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.

KeyDefaultMeaning
sitesrequired, or caddyfileSites served, one host name each. The deploy form lists existing blocks' addresses for the reverse proxy targets.
sites[].hostrequiredHost name the site answers for. A wildcard such as *.example.com needs a DNS challenge, so use caddyfile for it.
sites[].tlsacmeacme: a public certificate; internal: Caddy's own CA; off: plain HTTP only.
sites[].reverseProxyone of these twoUpstreams as host:port, round robin. A failing upstream is skipped for 30 s.
sites[].respondone of these twoA fixed one-line body, for a placeholder or a health endpoint.
emailnoneACME account email.
acmeCALet's Encrypt, then ZeroSSLACME directory URL, such as Let's Encrypt's staging server.
port18080Caddy's HTTP port, the target_port of 80.
httpsPort18443Caddy's HTTPS port, the target_port of 443.
caddyfilenoneA whole Caddyfile, used instead of everything above. Set the ports with the http_port and https_port global options. The data directory is still the volume.

Using it

Point each site's DNS name at the block's VIP (expanse ctl block get edge). port and httpsPort must differ from the exposed 80 and 443, because the VIP holder binds those on the node Caddy runs on.

A site with tls: acme or tls: internal answers HTTPS, and its plain-HTTP requests get a 308 redirect to https://<host>. ACME's HTTP challenge is answered on port 80, so for acme sites the VIP must be reachable from the internet on 80 or 443. On a private network use tls: internal; clients then need Caddy's root certificate, which is at caddy/pki/authorities/local/root.crt on the volume.

When a node fails

When the serving node dies, the block, its volume's primary and its VIP move together to a survivor, and Caddy starts there with the same data directory. Requests in flight fail and clients reconnect to the same address, which serves the same certificate. A renewal that was under way is done again on the survivor.

Caddy does not sync the files it writes, so the block flushes the volume every 2 s. A certificate issued in the last 2 s before a crash can be lost; the survivor then issues it again. In Expanse's 3-node VM test the VIP was on a survivor 55 s after the serving node was killed, and HTTPS answered again a fraction of a second later with the same certificate.

Limits

  • One Caddy serves the sites; there is no shared certificate storage for several instances.
  • Caddy's admin API is off, so a configuration change is a redeploy, which restarts Caddy.
  • HTTP/3 over UDP: VIPs forward TCP only.
  • To serve files, use web/nginx, or a caddyfile with file_server.

Source

All block types · Fields every block shares