# 0040. The UI backend gets one metric, the controller gets zero, and monitoring stays optional in the chart

Status: Accepted · Date: 2026-10-26 · Design section: Monitoring and registry mirrors

Decided: added exactly one custom metric (`ichor_uiapi_rpc_duration_seconds`, via a single `connectrpc.com/connect` interceptor) because the UI backend exports nothing on its own; added zero custom metrics to the controller because `controller-runtime` already exports reconcile/workqueue health and `kube-state-metrics` already covers cluster-level state (pending approvals, the cluster lock, `TalosOperation` progress) from fields already in CR spec/status; shipped a `ServiceMonitor` for ichor-ui, one `PrometheusRule` with two alerts, and one Grafana dashboard, all as chart options defaulting off, the same opt-in shape `metricsCert.enabled` already established (ADR 0031).

## Context

architecture.md's "Monitoring and registry mirrors" section requires: "The controller
and UI backend export Prometheus metrics, and ServiceMonitors, alert rules
and Grafana dashboards ship with the chart as an option." It also requires
Ichor to "expose the control-plane metrics that Talos binds locally by
default (controller-manager, scheduler, etcd) and restrict scraping to a
CIDR with Talos's ingress firewall" -- that second sentence is Talos machine
config, owned by a different unit working in `pkg/render`'s system layer,
and is out of scope here; see "Left to another unit" below for what it
still needs.

ADR 0031 already settled the controller's metrics *endpoint* and its
optional certificate (`metricsCert.enabled`). What was still open: which
metrics actually need writing, whether the UI backend needs an endpoint at
all, and how many dashboards/alerts are enough.

## Decided: zero new controller metrics

`controller-runtime` v0.24.1 registers eight metrics unconditionally, with
no code required on Ichor's part (`sigs.k8s.io/controller-runtime/pkg/
internal/controller/metrics`, confirmed by reading the vendored source, not
assumed): `controller_runtime_reconcile_total` (by `controller`, `result`),
`controller_runtime_reconcile_errors_total`, `controller_runtime_terminal_
reconcile_errors_total`, `controller_runtime_reconcile_panics_total`,
`controller_runtime_reconcile_time_seconds`, `controller_runtime_max_
concurrent_reconciles`, `controller_runtime_active_workers`, and
`controller_runtime_reconcile_timeouts_total`. "The controller exports
Prometheus metrics" is true before this unit writes a single line.

The real question was which *domain-specific* facts those don't cover.
Four candidates were weighed, and all four turned out to already be
answerable without adding anything to the controller:

- **Reconcile count/latency/error rate per controller.** Exactly
  `controller_runtime_reconcile_total`, `_time_seconds`, `_errors_total`.
  Adding an Ichor-side counter here would just be a second copy of a metric
  that already exists.
- **Workqueue depth / stuck reconciles.** `controller_runtime_active_
  workers` and `_max_concurrent_reconciles` already cover "is this
  controller keeping up." Nothing to add.
- **How many machines are pending approval, right now (a gauge, not a
  counter).** `TalosMachine.spec.approvalState` (`api/v1alpha1/
  talosmachine_types.go`) is a plain enum field already on the object.
  `kube-state-metrics`'s Custom Resource State (CRS) feature reads
  arbitrary JSONPaths -- spec or status -- from any CRD and turns them into
  gauges with no code, only a YAML config the cluster operator supplies to
  their own `kube-state-metrics` deployment. Counting `approvalState:
  pending` is exactly that kind of mapping. Ichor adding its own gauge here
  would duplicate a fact already sitting in the CR, read by a tool this
  project doesn't install (architecture.md: Ichor installs no monitoring
  stack) but that most clusters already run for HPA and other autoscaling.
- **Whether the cluster lock is held, and by whom.** `pkg/clusterlock`
  (`pkg/clusterlock/clusterlock.go`) is, by its own package doc, "a single
  `coordination.k8s.io/v1` Lease per cluster." `kube-state-metrics` ships
  `kube_lease_owner` and `kube_lease_renew_time` for every `Lease` object in
  the cluster *by default* -- no CRS config, no code, nothing Ichor-side at
  all. "Is the lock stale" is `time() - kube_lease_renew_time >
  leaseDurationSeconds`; "who holds it" is `kube_lease_owner`. Already
  fully answered.
- **`TalosOperation` phase and per-machine progress.** `status.phase` and
  `status.machines[].phase` (`api/v1alpha1/talosoperation_types.go`) are
  exactly the shape CRS's `stateSet` and list-valued metric types are built
  for. Same reasoning as the approval-state case: a documented
  `kube-state-metrics` CRS config, not an Ichor-side gauge.

None of this ships from Ichor: architecture.md is explicit that the
monitoring stack itself -- including whatever reads these CR fields -- is
"a documented set of Helm values, or nothing," never something Ichor
installs. A `kube-state-metrics` CRS snippet covering `approvalState` and
`TalosOperation.status` is worth documenting for an operator who already
runs `kube-state-metrics`, but writing and shipping that config is exactly
the kind of optional-component choice architecture.md reserves for the
user, not this unit. It is not written here, to avoid the same silent-
failure shape ADR 0031 rejected for its bootstrap certificate: a config
file committed into this repo, decoupled from whatever CRS version and
schema the user's actual `kube-state-metrics` deployment runs, would look
authoritative while quietly drifting out of sync with nothing to catch it.

This also settles why no "build info" gauge (version/commit labels, a
common pattern elsewhere) was added: nothing in this unit's scope
(`cmd/`, `dist/chart/`, `config/`, `internal/uiapi/`) currently threads a
version string through `ldflags`, and wiring that up is a build-tooling
change with its own blast radius, not a one-line consequence of a
monitoring-options unit.

## Decided: one metric for the UI backend, via one interceptor

Unlike the controller, ichor-ui (`cmd/ichor-ui/main.go`) had *no* metrics
endpoint before this unit -- it is a plain `net/http` server with no
`controller-runtime` manager underneath it, so there was nothing free here.
architecture.md requires it to export something, and the one domain fact
worth having is Connect RPC latency and outcome, per procedure: is
`ApproveMachine` slow, is `StreamNodeLogs` erroring.

Checked before writing any per-handler instrumentation: `connectrpc.com/
connect` v1.21.0 ships `connect.Interceptor` (`WrapUnary`,
`WrapStreamingClient`, `WrapStreamingHandler`), read directly from the
vendored source (`connectrpc.com/connect@v1.21.0/interceptor.go`) rather
than from memory. That is the mechanism to use instead of timing code
scattered across `ClusterService`/`MachineService`/`NodeService`/
`PatchService`'s handlers. `internal/uiapi/metrics.go` adds one
`connect.Interceptor` (`MetricsInterceptor`) observing one histogram,
`ichor_uiapi_rpc_duration_seconds{procedure, code}`, registered once per
`*ServiceHandler` constructor via `connect.WithInterceptors` in `cmd/
ichor-ui/main.go`. No separate error-rate counter: the histogram's `code`
label already carries that signal (`code != "ok"`), and a second counter
duplicating part of what `_count` already gives would be exactly the kind
of metric rejected above for the controller.

`/metrics` is served from the same `http.ServeMux`, the same port, and
plain HTTP -- no second listener, no TLS, no `metricsCert`-style opt-in
certificate. ichor-ui is reached only in-cluster today (architecture.md
"Trust": a port-forward, no exposed Service), the same trust boundary the
Connect API itself already sits behind, so a second port with its own
TLS/auth story would protect nothing a scrape of the first port doesn't
already cross. `github.com/prometheus/client_golang` was already an
indirect dependency (pulled in by `controller-runtime`); using it directly
for `promhttp.Handler()` and `promauto` added zero new dependencies
(`go.mod`'s diff is a one-line promotion from indirect to direct).

## Decided: one Grafana dashboard, two alerts

**Dashboard: one, not several.** "Ichor overview" (`dist/chart/templates/
grafana/dashboard-configmap.yaml`) carries five panels: reconcile rate and
error rate by controller, reconcile duration p99, and ichor-ui's RPC
latency p99 and error rate by procedure -- everything this unit's own
metrics cover, on one screen. A dashboard is a JSON blob that rots the
moment a panel's metric name or label changes and nothing re-checks it;
splitting one screen's worth of panels into multiple dashboards would only
multiply that liability for no reader benefit. Shipped as a ConfigMap
labeled `grafana_dashboard: "1"`, the sidecar-discovery convention both
`kube-prometheus-stack`'s and Bitnami's Grafana charts support -- inert
unless the cluster's own Grafana runs that sidecar, consistent with Ichor
installing no Grafana itself.

**Alerts: two.** Both pass the "would this wake someone up" bar the task
set, and nothing else proposed did:

- `IchorControllerDown` (`up{service="ichor-controller-manager-metrics-
  service"} == 0` for 5m, `severity: critical`): the controller is the only
  writer of Talos nodes and the only enforcer of deletion protection
  (architecture.md "Trust"); down means neither is happening.
- `IchorReconcileErrorsPersisting` (`sum by (controller)
  (rate(controller_runtime_reconcile_errors_total[15m])) > 0` for 15m,
  `severity: warning`): a single failed reconcile is retried and is noise;
  a *sustained* nonzero rate over 15 minutes means something the
  controller depends on is persistently broken, not transiently slow.

Both rules are deliberately built only on metrics Ichor itself guarantees
are present (`up` from the controller's own `ServiceMonitor`,
`controller_runtime_reconcile_errors_total` from `controller-runtime`
itself) -- not on `kube_lease_*` or any CRS-derived series from the
previous section's reasoning, since those depend on a `kube-state-metrics`
deployment and config Ichor does not install or guarantee exists. An alert
rule that silently never fires because its metric was never scraped is the
same failure shape ADR 0031 already rejected for a different reason
(the bootstrap-certificate amendment): a mechanism that looks like safety
and is actually silent absence.

## Chart options, all off by default

- `prometheus.enabled` (pre-existing, ADR 0031) now also renders ichor-ui's
  own `ServiceMonitor` (`templates/prometheus/ichor-ui-metrics-monitor.
  yaml`, fixed name `ichor-ui-metrics-monitor`, matching `ichor-ui-service`'s
  own fixed-name reasoning) alongside the controller's.
- `prometheus.rules.enabled` (new): the `PrometheusRule`
  (`templates/prometheus/alert-rules.yaml`), independent of
  `prometheus.enabled` -- a user might want the `ServiceMonitor` without
  opting into alerting, or vice versa once `prometheus-operator`'s
  `PrometheusRule` CRD exists.
- `grafana.dashboards.enabled` (new): the dashboard ConfigMap
  (`templates/grafana/dashboard-configmap.yaml`).

All three default `false`, the same shape `metricsCert.enabled` already
uses (ADR 0031): rendering nothing is the default, each flag renders
exactly its own resource, verified by `helm template` both ways
(`dist/chart/chart_test.go`'s `checkMonitoringOptionsOffByDefault`, run as
part of `TestChartInvariants`'s existing default-values render, and the
new `TestMonitoringOptionsEnabled`, which turns all three on and checks the
UI `ServiceMonitor` exists, both named alerts are present in the rendered
`PrometheusRule`, and the dashboard ConfigMap's `ichor-overview.json` value
is valid JSON with at least one panel -- not just that a `ConfigMap`
exists, the same "actually render it, don't just read the template"
standard ADR 0031's own `TestMetricsCertEnabled` sets).

`config/prometheus/` (the kustomize mirror, CONTRIBUTING.md's "mirroring
the matching `config/` base when one exists") gained `ui-monitor.yaml` and
`alert-rules.yaml`, bundled behind the existing single `[PROMETHEUS]`
toggle in `config/default/kustomization.yaml` -- kustomize's coarser
one-switch convention doesn't need the chart's finer
`prometheus.enabled`/`prometheus.rules.enabled` split to stay useful for
local dev. The Grafana dashboard has no `config/` mirror: no `config/
grafana/` directory existed before this unit, nothing else in `config/`
wires up Grafana's sidecar-discovery convention, and kustomize's audience
(contributors rendering `config/default` for local iteration) has no
Grafana to discover it. Verified empirically, not just by reading the
templates: temporarily uncommented `../prometheus` in `config/default/
kustomization.yaml`, ran `make build-installer`, confirmed
`ichor-alert-rules` and `ichor-ichor-ui-metrics-monitor` both render with
the names this ADR states (the double `ichor-ichor-` prefix on the UI one
matches the pre-existing convention every other `config/manager-ui`
resource already has, not a new defect), then reverted.

## Left to another unit: Talos control-plane metrics

architecture.md's "expose the control-plane metrics that Talos binds
locally by default (controller-manager, scheduler, etcd) and restrict
scraping to a CIDR with Talos's ingress firewall" is Talos machine
configuration -- a `machine.controlPlane`-level binding (or equivalent) plus
`machine.network.kubespan`/ingress-firewall-style CIDR rules -- that belongs
in `pkg/render`'s system layer, not in the chart or `internal/uiapi`. This
unit's own scope (`dist/chart/`, `config/`, `internal/uiapi/`, `cmd/`,
`docs/`) does not include `pkg/render`, and another agent is actively
working there, so nothing was implemented. What that unit still needs to
decide and build:

- Which of `controller-manager`/`scheduler`/`etcd`'s locally-bound metrics
  ports get exposed, and on which typed `TalosConfigPatch` field(s)
  (ADR 0003's typed-mirror types, not raw YAML -- CLAUDE.md's "Machine
  config is typed only").
- The ingress-firewall CIDR restriction that scopes who can reach those
  ports once exposed -- without it, "expose the metrics" and "open the
  control plane to the whole pod network" are the same change.
- Whether exposing these needs its own chart/config option (this unit's
  `prometheus.enabled` only covers the controller and ichor-ui's own
  `/metrics`, served from this Kubernetes cluster, not a Talos node's).

## Consequences

- `internal/uiapi/metrics.go` adds one histogram (`ichor_uiapi_rpc_
  duration_seconds`) and one `connect.Interceptor`
  (`MetricsInterceptor`), wired into every `*ServiceHandler` constructor in
  `cmd/ichor-ui/main.go` via `connect.WithInterceptors`.
- `cmd/ichor-ui/main.go` serves `/metrics` (`promhttp.Handler()`) on its
  existing mux and port -- no new listener, no new chart Service port.
- The controller (`internal/controller/`, `cmd/main.go`'s manager wiring)
  is untouched: zero new metrics, per the reasoning above. Anyone wanting
  cluster-state gauges (pending approvals, lock status, `TalosOperation`
  progress) points `kube-state-metrics`'s CRS config at the CR fields and
  `Lease` object that already carry them.
- `dist/chart/templates/prometheus/ichor-ui-metrics-monitor.yaml`,
  `alert-rules.yaml`, and `dist/chart/templates/grafana/
  dashboard-configmap.yaml` are new, each gated by its own `values.yaml`
  flag (`prometheus.enabled`, `prometheus.rules.enabled`,
  `grafana.dashboards.enabled`), all defaulting `false`.
- `dist/chart/templates/ui/ui-service.yaml` and `config/manager-ui/
  service.yaml` both name their port `http` so the new `ServiceMonitor`s
  can target it by name, mirroring `metrics/controller-manager-metrics-
  service.yaml`'s own named port.
- `config/prometheus/` gained `ui-monitor.yaml` and `alert-rules.yaml`,
  wired into the existing `[PROMETHEUS]` toggle; no `config/grafana/`
  directory was created (reasoning above).
- `dist/chart/chart_test.go` gained `checkMonitoringOptionsOffByDefault`
  (wired into `TestChartInvariants`) and `TestMonitoringOptionsEnabled`.
- `go.mod`: `github.com/prometheus/client_golang` moved from an indirect to
  a direct dependency; no new dependency was added.
- Talos control-plane metrics binding and ingress-firewall CIDR scoping
  remain unimplemented, left to the unit already working in `pkg/render`'s
  system layer (see above).
- Update: the Talos control-plane metrics binding and the ingress-firewall CIDR restriction were built in ADR 0069 (`spec.controlPlaneMetrics`).