Blocks · Monitoring

monitor/prometheus

Prometheus. Prometheus scraping every node and evaluating Expanse's shipped alert rules, its TSDB on a replicated volume.

Overview

Runs Prometheus inside the cluster. It scrapes every voter's agent /metrics over the cluster's mutual TLS (one job per node, with the cluster CA bundle kept current through a CA rotation), scrapes node-exporter on every node, and evaluates Expanse's shipped alert rules unmodified. With a storage entry, its time-series database lives on the volume and fails over with it.

Expanse does not write its own metrics database; this is the same Prometheus you could run outside the cluster, set up for you.

Shape
Singleton on a replicated volume
Reached at
Node port, or a VIP
Node package
pkgs.prometheus

Fetched on demand: the first time a block of this type is placed on a node, its agent fetches pkgs.prometheus, built from the node's own nixpkgs so the NixOS binary cache serves it. That first placement needs network access (or a binary cache on the LAN) and waits for the download; later placements on that node start at once.

Example

prometheus.yaml
apiVersion: expanse.io/v1
kind: Block
metadata:
  name: prometheus
  namespace: default
spec:
  type: monitor/prometheus
  replicas: 1
  strategy:
    kind: SINGLETON
  storage:
    - name: tsdb
      size: 10Gi
      replication: 3
      mountPath: /var/lib/prometheus
  config:
    metricsToken: <the token from expanse ctl metrics set-token>
    scrapeInterval: 5s
    retention: 15d
    nodeExporterPort: 9100
$ expanse ctl block apply -f prometheus.yaml --dry-run   # validate only
$ expanse ctl block apply -f prometheus.yaml

Configuration

Keys go under spec.config and are checked against the type's JSON Schema when the block is applied. A default is what the block uses when the key is unset.

KeyDefaultMeaning
metricsTokennoneThe cluster metrics bearer token (expanse ctl metrics set-token). Without it only node-exporter and Prometheus itself are scraped, because the cluster stores the token only as a hash. Sealed: kept out of the stored block (block secrets).
scrapeInterval5sScrape and rule-evaluation interval. Keep it at 5 s or below so alerts fire within their 30 s budget.
retention15dHow long samples are kept.
port9090Port of the web UI and API.
nodeExporterPort9100Port the monitor/node-exporter daemonset listens on.
scrapeNodeExportertrueScrape node-exporter on every node.

Using it

Give the block a VIP so that monitor/grafana's prometheusUrl survives a failover. As for every VIP-exposed block, port must then differ from the port exposed on the VIP. The deploy form exposes port 80, forwarded to 9090, so the URL is http://<address>. Alertmanager is not configured; route alerts with your own.

When a node fails

With a replicated volume, the singleton restarts on the node that takes over the volume's primary, with the same time-series database. Samples during the gap are missing.

Limits

  • Alertmanager routing is left to the operator.
  • The scrape targets and rules are generated; there is no way to add your own scrape jobs.

Source

All block types · Fields every block shares