Overview
Runs a QEMU/KVM guest, exec'd directly with no libvirt, on a raw replicated volume that the guest formats and owns. The block is a singleton colocated with its volume's primary and scheduled only to nodes with hardware virtualization. The guest's network identity is a macvtap device on the node's uplink with a MAC derived from the block's name, so it comes back at the same address after a restart on another node. There is no VIP.
The guest gets as many vCPUs as resources.requests.cpu, rounded up to whole cores, and resources.requests.memory as RAM (512 MiB if unset).
- Shape
- Singleton on a raw replicated volume
- Reached at
- The guest's own address
- Node package
- Built in (headless qemu)
Built into every node: Expanse ships a headless qemu build, so a node needs nothing extra to run a VM.
Example
apiVersion: expanse.io/v1 kind: Block metadata: name: vm1 namespace: default spec: type: vm/instance replicas: 1 strategy: kind: SINGLETON resources: requests: cpu: 1000m # 1 vCPU memory: 2Gi # guest RAM storage: - name: disk size: 10Gi replication: 3 mountPath: /mnt/vm1-disk filesystem: none # the guest owns the disk placement: requiredCapabilities: [kvm] config: image: url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2 checksumUrl: https://cloud.debian.org/images/cloud/trixie/latest/SHA512SUMS cloudInit: sshKeys: ["ssh-ed25519 AAAA... me@laptop"] user: ops # optional: a sudo user with the keys hostname: vm1 # optional: defaults to the block's name
The web UI builds the same block without YAML: Workloads → Blocks → Deploy block → vm/instance starts from the Debian 13 image and offers Ubuntu 24.04 LTS or a custom URL. In its SSH key list, gh:<user> imports a GitHub user's keys. Its First-boot setup field offers ready-made setups, starting with Pando under Docker on a Debian 13 or Ubuntu 24.04 image, so the Pando app platform deploys from the UI with no image to build; give that VM 2 CPUs, 4 GiB and a 40 GiB disk or more.
$ expanse ctl block apply -f instance.yaml --dry-run # validate only $ expanse ctl block apply -f instance.yaml
Configuration
Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.
| Key | Default | Meaning |
|---|---|---|
image | none | A cloud image (qcow2 or raw) written onto the disk on the VM's first start: url, plus checksum (SHA-256 or SHA-512 hex) or checksumUrl (a SHA256SUMS or SHA512SUMS file that lists it). Later changes do not touch the disk. Unset, the disk is used as it is. |
cloudInit | none | Handed to the guest's cloud-init on its first boot: sshKeys, user (a sudo user with the keys; unset, the keys go to the image's own user) and hostname (unset, the block's name), guestAgent (install, the default, or skip: whether the seed installs qemu-guest-agent, which tells the node the guest's addresses and runs vm exec; installing it runs the package manager's update on the first boot) and setup (packages to install, then runcmd shell commands run as root, on the first boot only). |
mac | derived from the block's name | The guest NIC's MAC address. |
bootKernel | none | A kernel for direct kernel boot. Unset, the guest boots from its disk through the BIOS. |
bootInitrd | none | The initramfs for bootKernel. |
bootCmdline | none | Kernel command line, used with bootKernel. |
guestReady | systemd | When the VM counts as RUNNING. systemd: once the guest's systemd (254 or newer) reports over vsock that it reached multi-user.target (systemd 254 and 255, as in Ubuntu 24.04, report no targets, so READY=1 alone counts). none: as soon as QEMU starts, for guests without systemd. |
cpuModel | qemu's default | The guest's CPU model. host passes the node's CPU through, including vmx/svm, so the guest can run KVM itself (nested virtualization). host is the only value accepted. |
machine | pc-i440fx-11.1 | The virtual hardware the guest sees, as a versioned QEMU machine type: pc-i440fx- or pc-q35- and a version. It stays fixed when Expanse updates QEMU, so the guest keeps the same hardware and a suspended VM can still resume. |
shutdownTimeout | 60 | Seconds a stop waits for the guest to power off after asking it to, before the VM is turned off hard. 0 turns it off at once; at most 80. |
Using it
With image set, the VM's node downloads the image into /var/lib/expanse/images on the first start, checks it, and writes it onto the disk; the block's status shows the download's progress. The disk must be at least the image's size, and the guest grows its root filesystem into the rest. The image is written only while the disk's first MiB is still zero, and that MiB is written last, so a later start or a failover boots the disk as it is and an interrupted write is redone. A qcow2 that names a backing or external data file is refused. Downloaded images stay cached for other VMs; delete them there by hand to free the space.
Each start attaches a NoCloud seed with the user, keys and hostname. cloud-init applies it on the first boot only, so change keys inside the guest after that. The guest gets its address from your DHCP server. Every VM has a QEMU guest agent port, and once the agent answers, the block's page in the web UI shows the guest's address with a ready ssh line; the node asks the agent every 15 seconds. With guestAgent: skip, the page shows the MAC address and hostname, and your router's lease list gives the address.
cloudInit.setup installs its packages, then runs its commands as one shell script in order, on the first boot only; a failover boots the same disk and runs nothing again. A failing command does not stop the rest, so end a line with || exit 1 to stop there. The VM counts as RUNNING before they finish, and their output is in the console log and in /var/log/cloud-init-output.log in the guest. The setup sits in the seed ISO and is readable in the guest, so keep passwords out of it.
cloudInit: sshKeys: ["ssh-ed25519 AAAA... me@laptop"] setup: packages: [nginx] runcmd: - systemctl enable --now nginx
The guest agent also runs commands in the guest as root, with no network or SSH key. Run vm exec as root on the node that runs the VM (its block page names the node). The output prints once the command exits, and its exit code becomes vm exec's; Ctrl-C stops waiting but leaves the command running.
$ expanse ctl vm exec vm1 -- systemctl status docker $ echo hello | expanse ctl vm exec -i vm1 -- tee /root/note
A VM started by an earlier Expanse, and still running through an upgrade, gets the agent port, its address on the page and vm exec after its next restart. cloud-init runs once, so a VM deployed before the guest agent was added needs qemu-guest-agent installed in the guest by hand.
Without image, getting a first operating system onto the empty disk is up to you: write a prebuilt image onto the volume's DRBD device on its primary before applying the block, or boot a kernel directly with bootKernel. Expanse's pando-guest-image is a worked example, for a Pando guest that boots without a registry: a prebuilt NixOS image with the Pando app platform, written onto a pre-created volume and booted with no bootKernel.
The block page's Console tab is the guest's serial console (ttyS0), live in the browser from whichever node serves the page. expanse ctl vm console attaches a terminal to it from any node, as root; Ctrl-] leaves and the guest keeps running. It opens with the console's recent output, several people can attach at once, and logging in takes an account the guest already has: cloud images have no passwords, so set one first with vm exec. A VM started by an earlier Expanse gets its console after its next restart.
$ echo debian:secret | expanse ctl vm exec -i vm1 -- chpasswd $ expanse ctl vm console vm1
The console's output also goes to the block unit's journal on the VM's node. Its readiness is the unit's status text:
$ journalctl -u expanse-block-root@default-vm1-0.service -f $ systemctl show -p StatusText --value expanse-block-root@default-vm1-0.service ready: multi-user.target reached
A macvtap guest cannot be reached from the node it runs on; use any other machine.
When a node fails
When the VM's node is lost, the disk's primary moves and a fresh QEMU cold-boots the guest on the new node. It is not a live migration: the guest's memory is lost, as after a power cut. The guest keeps its MAC and so its address, and open connections must be re-established. Under a sustained fsync write load through the kill, the guest's filesystem came back with no uncorrectable errors and every synced write present.
Limits
- No live migration.
- The console is the serial console only; there is no graphical (VNC or SPICE) console.
- Liveness probes are not run for VMs.
- A
cpuModel: hostguest is tied to its node's CPU features; keep its candidate nodes on the same CPU generation. - Tested guests: the Debian 13 and Ubuntu 24.04 cloud images, and NixOS. Other operating systems, such as Windows and the BSDs, are unverified.
- Use guests with systemd 254 or newer (Debian 12 has 252), or set
guestReady: none. - Installing from an ISO with a web console is not built yet.
Source
- docs/VMS.md: the Expanse guide this page follows.
- nix/blocks/vm/instance: the type's identity and JSON Schema.