# 0116. Patch upgrades can be applied automatically when a setting allows it

Status: Accepted · Date: 2026-11-04 · Design section: Upgrades · Touches: 0043, 0057, 0059, 0095, 0099, 0105, 0114, 0115

Decided: `IchorInstallation.spec.autoPatch` (and the chart's `autoPatch`, three controller flags) is a setting that is off by default; when it is on, a leader-only controller loop, once a minute, starts at most one upgrade `TalosOperation` per cluster for the newest patch of the running minor of Talos, then Cilium, then Kubernetes, taken only from the candidates the release check recorded in `status.upgradeTargets` and filtered by the compatibility table, only inside the configured daily UTC window, only while every approved machine is Ready and nothing else holds the cluster; the operation carries the annotation `ichor.maesh.dev/auto-patch: "true"` and `started-by: ichor-auto-patch` and runs through the normal executor; a failed or paused automatic operation halts the patcher (condition `AutoPatchHalted`, `status.autoPatch.haltedOperation`) until a person writes `spec.autoPatch.acknowledgedHalt`, or the operation completes or is deleted, or a later manual upgrade of that cluster completes; the patcher never retries. The controller itself never writes a `TalosCluster` spec: the version of a Talos or Kubernetes patch, and the operation, are written by a separate identity, the ServiceAccount `ichor-auto-patch`, which the controller impersonates (a namespaced Role that names that one ServiceAccount); the account holds only `get` and `update` on `talosclusters` and `create` on `talosoperations`, and the admission webhook refuses any write of it except a newer patch of the same minor in those two fields, and any spec write by the controller's own username. The audit log shows the writes as `system:serviceaccount:<namespace>:ichor-auto-patch`.

## Context

The user decided on 2026-11-04 that patch upgrades may be applied automatically when a setting allows it. ADR 0114 made the controller list releases, ADR 0115 made each component upgrade on its own against one compatibility table and put the candidate lists in `status.upgradeTargets` (Talos, Kubernetes, and the Cilium patches whose images verified). Today a person starts each of those operations in the UI or CLI. The architecture says the controller never mutates `spec` and that the UI and CLI change the world by writing CRs. This ADR keeps the rule in as many places as it can and names the one place it cannot.

## Decision

### 1. The setting

```yaml
spec:
  autoPatch:
    enabled: true                         # unset falls back to --auto-patch (chart autoPatch.enabled), default false
    components: [talos, cilium, kubernetes]  # unset or empty means all three
    window: "02:00-05:00"                 # daily UTC window, may wrap midnight (22:00-02:00); unset means any time
    acknowledgedHalt: ""                  # see 5
```

All fields are optional and append-only. A set `spec.autoPatch` wins field by field over the flags (`--auto-patch`, `--auto-patch-components`, `--auto-patch-window`, from the chart's `autoPatch.enabled`, `components`, `window`); `--settings-read-only` ignores the whole spec group exactly as for the other settings (ADR 0105, 0114). `status.effective` gains `autoPatch`, `autoPatchComponents` and `autoPatchWindow`. Changing the first three needs `update` on `ichorinstallations/settings`, as the other settings do, and leaves the usual `SettingsChanged` Event; `ichor settings set autoPatch.enabled=true autoPatch.components=talos,cilium autoPatch.window=02:00-05:00` and the Settings page (a Switch, one Checkbox per component, an Input for the window) write it through SettingsService. The webhook and the CRD both reject a malformed window or an unknown component.

### 2. What it may start

Only a patch: the same minor as what runs, newer, a final release. The candidate is the newest of `release.TalosChoices`, `release.KubernetesChoices` or `release.CiliumChoices` (the choices the UI offers, built from `status.upgradeTargets` and the table of ADR 0115) that has the running minor. So it is never a minor, never a version only `--override` would pass, never a version the release check did not list, and for Cilium only an image set whose signatures the check verified. Kubernetes needs the Cilium the cluster runs to be known (nothing is compatible with an unknown one), exactly as in the UI. The unmanaged CNI preset has no Cilium patch.

The order is Talos, Cilium, Kubernetes, one component per decision. Kubernetes comes last because its supported range is limited by the other two; a patch never changes a minor, so no component's patch makes another's candidate invalid. After one operation completes, the next minute's decision picks the next component.

### 3. When

Every guard is a field of the pure function `autopatch.Decide`, with a table test each:

1. the setting is on, and not halted;
2. no Ichor upgrade is in progress (`status.installedVersion` is `spec.version`);
3. now (UTC) is inside the window;
4. the release lists exist and are younger than 48 hours (the check runs every six hours, so older means it is failing or off);
5. no TalosOperation of the cluster is unfinished (running, paused or pending; a node shutdown resting in `ShutDown` does not count), and the cluster lock (the Lease of ADR 0043) is not held;
6. the cluster has approved machines and every one is Ready (a machine with a pending configuration change is not Ready, so a pending apply blocks it);
7. no operation was started by the patcher on this cluster in the last two minutes (the guard for reads that have not caught up; the patcher reads without the cache as well).

### 4. How it runs: its own loop

`controller.AutoPatcher` is a leader-only Runnable that ticks every minute with jitter. It does not reuse the release check's loop: that one runs every six hours and the window is a time of day, so reusing it would miss most windows or force the check to run every minute. It does not hang off the TalosCluster reconcile either: that reconcile is per cluster and event-driven, while the halt state is global and the setting is read once. The release check stays the only thing that reaches a registry; the patcher only reads what it wrote. While the setting is off and nothing is halted the tick returns after reading the settings.

A start, for Talos or Kubernetes: the operation's creation is dry-run first, then the version is written into the cluster's spec as ichor-auto-patch (compare-and-set: only if the field still holds what was decided on, so a person's edit in between is not overwritten), then the operation is created, also as ichor-auto-patch. This is the order the UI uses because the executors read the target from the spec when a machine's turn comes. For Cilium the operation carries the target itself (`spec.cni.chartVersion` and `imageVersion`, as StartCniUpgrade does). The operation gets `ichor.maesh.dev/from-version`, `ichor.maesh.dev/started-by: ichor-auto-patch` and `ichor.maesh.dev/auto-patch: "true"`; the UI's operation page shows "Started by: Automatic patching". There is no spec field for the starter: the annotations are what the audit trail and the UI read, as for `started-by` today. Each start emits a Normal Event `AutoPatchStarted` on the TalosCluster and records `status.autoPatch.lastOperation`. If the version was written but the operation could not be created in three tries, the patcher halts and says so; it does not leave the half-start unseen.

### 5. Halt and resume

An automatic operation (by the annotation) in phase Paused, which is how a failed step and a person's pause both show (ADR 0095), halts the patcher: `status.conditions[AutoPatchHalted]=True` with the operation and its failure reason, `status.autoPatch.haltedOperation: <namespace>/<name>`, a Warning Event `AutoPatchHalted`, and an item in the Overview's attention list. While halted nothing starts on any cluster, and nothing is ever retried by the patcher: running a failed step again stays `ichor operation resume` (ADR 0095).

The halt ends when:

- a person writes `spec.autoPatch.acknowledgedHalt` equal to `haltedOperation` (the Settings page's "Clear the halt" button, `ichor settings set autoPatch.acknowledgeHalt=<namespace>/<name>`). The controller never edits spec, so it only compares; the value is not removed, and a later halt names another operation, so an old acknowledgement does not clear it. This is allowed under `--settings-read-only` too (still needing `update` on `ichorinstallations/settings`): it clears a state Ichor wrote, it is not one of the deployment's settings;
- the halting operation completed (a person resumed it) or no longer exists (deleted);
- a later operation of the same cluster that the patcher did not start, of kind talos, kubernetes or cni, completed.

An acknowledged halt does not unblock a cluster: a paused operation still holds it (guard 5) until a person resumes or deletes it.

### 6. Who writes the version: a delegated identity, not the controller

The target of a Talos or Kubernetes upgrade is `TalosCluster.spec.talosVersion` / `kubernetesVersion`, which the executors (`talos_upgrade.go`, `kubernetes_upgrade.go`) read and the renderer (`render.go`) uses for every node's machine configuration. An operation that carried its own version would leave the spec behind the nodes, and the next reconcile would render the old version back. So the spec has to change, and the architecture says the controller never writes `spec` (principle 5). The rule is kept whole: the write is made by another identity acting for the user who turned the setting on.

- **`ichor-auto-patch`** is a ServiceAccount in the controller's namespace with a ClusterRole `ichor-auto-patch-role` of three rules: `get` and `update` on `talosclusters`, `create` on `talosoperations`. Nothing else, and nobody else is bound to it. It writes the version and creates the operation, so both are attributed to it. Config (`config/rbac/ichor-auto-patch/`) and chart (`templates/rbac/ichor-auto-patch.yaml`) are identical and a chart test compares them. Its objects carry the installer's label like every other installed object, so the prune of ADR 0111 treats them as any other; the installer's `bind`/`escalate` lists name the ClusterRole and the Role.
- **The controller impersonates it.** `AutoPatcher.Actor` is a client built with `Impersonate: system:serviceaccount:<ns>:ichor-auto-patch`. The controller's right is `impersonate` on `serviceaccounts` with `resourceNames: [ichor-auto-patch]` in a namespaced Role, so it can act as that one account in its own namespace and no other. Its own ClusterRole holds no `update` or `patch` on `talosclusters` (a chart test asserts it). Cilium takes the same path for uniformity, though its operation carries the target itself.
- **The webhook limits both.** `TalosClusterCustomValidator` refuses any update of a `TalosCluster` spec by the controller's own username, and limits the `ichor-auto-patch` username to one change: `spec.talosVersion` or `spec.kubernetesVersion` to a newer patch of the same minor, with labels, annotations, finalizers and every other spec field unchanged. The compatibility webhook and both executors check the step again. A bug in the patcher therefore cannot edit a cluster's configuration.
- **A GitOps tool that owns the TalosCluster** will see the version drift and revert it, so a deployment that keeps clusters in Git should leave auto-patch off (the default).

### 7. Not decided here

- A separate maintenance window per cluster, or a per-cluster switch: the setting is installation-wide. A cluster wanting to opt out keeps the hold annotation on every machine (ADR 0043), which the executors honour.
- Unsetting a field of `spec.autoPatch` from the UI or CLI (the same gap as the other settings, ADR 0105). An empty window written through the UI falls back to the chart's.
- Auto-applying anything else: minors, Ichor itself, and Cilium chart upgrades stay manual.

## Consequences

- Off by default; nothing changes for an installation that does not turn it on. The CRD gains optional fields only (append-only), the chart a value group and three flags, the proto optional fields.
- A new ServiceAccount, ClusterRole and binding `ichor-auto-patch`, and a namespaced Role that lets the controller impersonate that account only; the controller's own ClusterRole is unchanged (`manager-role` equals `config/rbac/role.yaml` as before). The installer's escalation lists name the new Role and ClusterRole.
- The release check must be on for there to be anything to apply; with it off `status.upgradeTargets` is cleared and the patcher starts nothing.
- `ichor settings get` and the Settings page show the effective switch, components and window, and the halted operation.
- Tests: table tests of `autopatch.Decide` for every guard and of `autopatch.Resume`; an envtest of `AutoPatcher` for start (annotations, spec bump, no second start), the guards that need a cluster, halt on a paused operation, resume by acknowledgement and by a later manual operation; the webhook tests of the automatic patcher's limited write and of the controller's refused one, the chart test of the identity and of the settings gate (including read-only and the acknowledgement); the uiapi tests of the new fields; the UI's pure tests of the form.

## Amendment 2026-11-04: Cilium has no automatic patch

ADR 0115, amendment 2026-11-04 (3), removed the Cilium patches that run on the embedded chart, so there is nothing for the patcher to apply for Cilium: `autopatch.Candidate("cilium", ...)` returns nothing and `Cluster.CiliumChart` is gone. `cilium` remains an accepted value of `components` (stored settings and the chart defaults keep working) and selects no operation. The patcher's order is Talos, then Kubernetes. A Cilium patch reaches a cluster with an Ichor release; the chart upgrade stays a manual operation.