Overview
Runs Prometheus inside the cluster. It scrapes every voter's agent /metrics over the cluster's mutual TLS (one job per node, with the cluster CA bundle kept current through a CA rotation), scrapes node-exporter on every node, and evaluates Expanse's shipped alert rules unmodified. With a storage entry, its time-series database lives on the volume and fails over with it.
Expanse does not write its own metrics database; this is the same Prometheus you could run outside the cluster, set up for you.
- Shape
- Singleton on a replicated volume
- Reached at
- Node port, or a VIP
- Node package
pkgs.prometheus
Fetched on demand: the first time a block of this type is placed on a node, its agent fetches pkgs.prometheus, built from the node's own nixpkgs so the NixOS binary cache serves it. That first placement needs network access (or a binary cache on the LAN) and waits for the download; later placements on that node start at once.
Example
apiVersion: expanse.io/v1 kind: Block metadata: name: prometheus namespace: default spec: type: monitor/prometheus replicas: 1 strategy: kind: SINGLETON storage: - name: tsdb size: 10Gi replication: 3 mountPath: /var/lib/prometheus config: metricsToken: <the token from expanse ctl metrics set-token> scrapeInterval: 5s retention: 15d nodeExporterPort: 9100
$ expanse ctl block apply -f prometheus.yaml --dry-run # validate only $ expanse ctl block apply -f prometheus.yaml
Configuration
Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.
| Key | Default | Meaning |
|---|---|---|
metricsToken | none | The cluster metrics bearer token (expanse ctl metrics set-token). Without it only node-exporter and Prometheus itself are scraped, because the cluster stores the token only as a hash. Sealed: kept out of the stored block (block secrets). |
scrapeInterval | 5s | Scrape and rule-evaluation interval. Keep it at 5 s or below so alerts fire within their 30 s budget. |
retention | 15d | How long samples are kept. |
port | 9090 | Port of the web UI and API. |
nodeExporterPort | 9100 | Port the monitor/node-exporter daemonset listens on. |
scrapeNodeExporter | true | Scrape node-exporter on every node. |
Using it
Give the block a VIP so that monitor/grafana's prometheusUrl survives a failover. As for every VIP-exposed block, port must then differ from the port exposed on the VIP. The deploy form exposes port 80, forwarded to 9090, so the URL is http://<address>. Alertmanager is not configured; route alerts with your own.
When a node fails
With a replicated volume, the singleton restarts on the node that takes over the volume's primary, with the same time-series database. Samples during the gap are missing.
Limits
- Alertmanager routing is left to the operator.
- The scrape targets and rules are generated; there is no way to add your own scrape jobs.
Source
- docs/OBSERVABILITY.md: the Expanse guide this page follows.
- nix/blocks/monitor/prometheus: the type's identity and JSON Schema.