Blocks · Virtual machines

vm/instance

Virtual machine. A QEMU/KVM guest on a raw replicated disk; node loss is a cold boot elsewhere at the same address.

Overview

Runs a QEMU/KVM guest, exec'd directly with no libvirt, on a raw replicated volume that the guest formats and owns. The block is a singleton colocated with its volume's primary and scheduled only to nodes with hardware virtualization. The guest's network identity is a macvtap device on the node's uplink with a MAC derived from the block's name, so it comes back at the same address after a restart on another node. There is no VIP.

The guest gets as many vCPUs as resources.requests.cpu, rounded up to whole cores, and resources.requests.memory as RAM (512 MiB if unset).

Shape
Singleton on a raw replicated volume
Reached at
The guest's own address
Node package
Built in (headless qemu)

Built into every node: Expanse ships a headless qemu build, so a node needs nothing extra to run a VM.

Example

instance.yaml
apiVersion: expanse.io/v1
kind: Block
metadata:
  name: vm1
  namespace: default
spec:
  type: vm/instance
  replicas: 1
  strategy:
    kind: SINGLETON
  resources:
    requests:
      cpu: 1000m       # 1 vCPU
      memory: 2Gi      # guest RAM
  storage:
    - name: disk
      size: 10Gi
      replication: 3
      mountPath: /mnt/vm1-disk
      filesystem: none   # the guest owns the disk
  placement:
    requiredCapabilities: [kvm]
  config:
    image:
      url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2
      checksumUrl: https://cloud.debian.org/images/cloud/trixie/latest/SHA512SUMS
    cloudInit:
      sshKeys: ["ssh-ed25519 AAAA... me@laptop"]
      user: ops        # optional: a sudo user with the keys
      hostname: vm1   # optional: defaults to the block's name

The web UI builds the same block without YAML: Workloads → Blocks → Deploy block → vm/instance starts from the Debian 13 image and offers Ubuntu 24.04 LTS or a custom URL. In its SSH key list, gh:<user> imports a GitHub user's keys. Its First-boot setup field offers ready-made setups, starting with Pando under Docker on a Debian 13 or Ubuntu 24.04 image, so the Pando app platform deploys from the UI with no image to build; give that VM 2 CPUs, 4 GiB and a 40 GiB disk or more.

$ expanse ctl block apply -f instance.yaml --dry-run   # validate only
$ expanse ctl block apply -f instance.yaml

Configuration

Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.

KeyDefaultMeaning
imagenoneA cloud image (qcow2 or raw) written onto the disk on the VM's first start: url, plus checksum (SHA-256 or SHA-512 hex) or checksumUrl (a SHA256SUMS or SHA512SUMS file that lists it). Later changes do not touch the disk. Unset, the disk is used as it is.
cloudInitnoneHanded to the guest's cloud-init on its first boot: sshKeys, user (a sudo user with the keys; unset, the keys go to the image's own user) and hostname (unset, the block's name), guestAgent (install, the default, or skip: whether the seed installs qemu-guest-agent, which tells the node the guest's addresses and runs vm exec; installing it runs the package manager's update on the first boot) and setup (packages to install, then runcmd shell commands run as root, on the first boot only).
macderived from the block's nameThe guest NIC's MAC address.
bootKernelnoneA kernel for direct kernel boot. Unset, the guest boots from its disk through the BIOS.
bootInitrdnoneThe initramfs for bootKernel.
bootCmdlinenoneKernel command line, used with bootKernel.
guestReadysystemdWhen the VM counts as RUNNING. systemd: once the guest's systemd (254 or newer) reports over vsock that it reached multi-user.target (systemd 254 and 255, as in Ubuntu 24.04, report no targets, so READY=1 alone counts). none: as soon as QEMU starts, for guests without systemd.
cpuModelqemu's defaultThe guest's CPU model. host passes the node's CPU through, including vmx/svm, so the guest can run KVM itself (nested virtualization). host is the only value accepted.
machinepc-i440fx-11.1The virtual hardware the guest sees, as a versioned QEMU machine type: pc-i440fx- or pc-q35- and a version. It stays fixed when Expanse updates QEMU, so the guest keeps the same hardware and a suspended VM can still resume.
shutdownTimeout60Seconds a stop waits for the guest to power off after asking it to, before the VM is turned off hard. 0 turns it off at once; at most 80.

Using it

With image set, the VM's node downloads the image into /var/lib/expanse/images on the first start, checks it, and writes it onto the disk; the block's status shows the download's progress. The disk must be at least the image's size, and the guest grows its root filesystem into the rest. The image is written only while the disk's first MiB is still zero, and that MiB is written last, so a later start or a failover boots the disk as it is and an interrupted write is redone. A qcow2 that names a backing or external data file is refused. Downloaded images stay cached for other VMs; delete them there by hand to free the space.

Each start attaches a NoCloud seed with the user, keys and hostname. cloud-init applies it on the first boot only, so change keys inside the guest after that. The guest gets its address from your DHCP server. Every VM has a QEMU guest agent port, and once the agent answers, the block's page in the web UI shows the guest's address with a ready ssh line; the node asks the agent every 15 seconds. With guestAgent: skip, the page shows the MAC address and hostname, and your router's lease list gives the address.

cloudInit.setup installs its packages, then runs its commands as one shell script in order, on the first boot only; a failover boots the same disk and runs nothing again. A failing command does not stop the rest, so end a line with || exit 1 to stop there. The VM counts as RUNNING before they finish, and their output is in the console log and in /var/log/cloud-init-output.log in the guest. The setup sits in the seed ISO and is readable in the guest, so keep passwords out of it.

    cloudInit:
      sshKeys: ["ssh-ed25519 AAAA... me@laptop"]
      setup:
        packages: [nginx]
        runcmd:
          - systemctl enable --now nginx

The guest agent also runs commands in the guest as root, with no network or SSH key. Run vm exec as root on the node that runs the VM (its block page names the node). The output prints once the command exits, and its exit code becomes vm exec's; Ctrl-C stops waiting but leaves the command running.

$ expanse ctl vm exec vm1 -- systemctl status docker
$ echo hello | expanse ctl vm exec -i vm1 -- tee /root/note

A VM started by an earlier Expanse, and still running through an upgrade, gets the agent port, its address on the page and vm exec after its next restart. cloud-init runs once, so a VM deployed before the guest agent was added needs qemu-guest-agent installed in the guest by hand.

Without image, getting a first operating system onto the empty disk is up to you: write a prebuilt image onto the volume's DRBD device on its primary before applying the block, or boot a kernel directly with bootKernel. Expanse's pando-guest-image is a worked example, for a Pando guest that boots without a registry: a prebuilt NixOS image with the Pando app platform, written onto a pre-created volume and booted with no bootKernel.

The block page's Console tab is the guest's serial console (ttyS0), live in the browser from whichever node serves the page. expanse ctl vm console attaches a terminal to it from any node, as root; Ctrl-] leaves and the guest keeps running. It opens with the console's recent output, several people can attach at once, and logging in takes an account the guest already has: cloud images have no passwords, so set one first with vm exec. A VM started by an earlier Expanse gets its console after its next restart.

$ echo debian:secret | expanse ctl vm exec -i vm1 -- chpasswd
$ expanse ctl vm console vm1

The console's output also goes to the block unit's journal on the VM's node. Its readiness is the unit's status text:

$ journalctl -u expanse-block-root@default-vm1-0.service -f
$ systemctl show -p StatusText --value expanse-block-root@default-vm1-0.service
ready: multi-user.target reached

A macvtap guest cannot be reached from the node it runs on; use any other machine.

When a node fails

When the VM's node is lost, the disk's primary moves and a fresh QEMU cold-boots the guest on the new node. It is not a live migration: the guest's memory is lost, as after a power cut. The guest keeps its MAC and so its address, and open connections must be re-established. Under a sustained fsync write load through the kill, the guest's filesystem came back with no uncorrectable errors and every synced write present.

Limits

  • No live migration.
  • The console is the serial console only; there is no graphical (VNC or SPICE) console.
  • Liveness probes are not run for VMs.
  • A cpuModel: host guest is tied to its node's CPU features; keep its candidate nodes on the same CPU generation.
  • Tested guests: the Debian 13 and Ubuntu 24.04 cloud images, and NixOS. Other operating systems, such as Windows and the BSDs, are unverified.
  • Use guests with systemd 254 or newer (Debian 12 has 252), or set guestReady: none.
  • Installing from an ISO with a web console is not built yet.

Source

All block types · Fields every block shares