Overview
Runs ollama serve on port. The on-prem LLM subsystem this type belongs to is paused, because no accelerator hardware has been available to validate it, so the type does less than its schema suggests.
Paused. Models are not pulled and are not kept: the runtime stores them in a directory private to the unit, which is discarded when it stops.
- Shape
- Replicas
- Reached at
- Node port
- Node package
pkgs.ollama
Fetched on demand: the first time a block of this type is placed on a node, its agent fetches pkgs.ollama, built from the node's own nixpkgs so the NixOS binary cache serves it. That first placement needs network access (or a binary cache on the LAN) and waits for the download; later placements on that node start at once.
Example
apiVersion: expanse.io/v1 kind: Block metadata: name: llm namespace: default spec: type: ai/ollama replicas: 1 resources: requests: cpu: 2000m memory: 8Gi config: port: 11434
$ expanse ctl block apply -f ollama.yaml --dry-run # validate only $ expanse ctl block apply -f ollama.yaml
Configuration
Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.
| Key | Default | Meaning |
|---|---|---|
port | none: always set it | Port of the Ollama API. Unset, Ollama takes a random free port. The deploy form sets 21434 and puts 11434 on the VIP. |
models | none | Model names to pull on start, per the schema; not applied. Pull models through the API instead. |
keepAlive | none | Model keep-alive; not applied. |
Using it
$ curl http://<node address>:11434/api/tags $ curl http://<node address>:11434/api/pull -d '{"model": "<model>"}'
When a node fails
A failed node's replica is placed again elsewhere and starts with no models.
Limits
- Paused: no GPU validation.
- Models are lost when the replica stops.
Source
- nix/blocks/ai/ollama: the type's identity and JSON Schema.