0043. The Talos upgrade executor is one per-node function behind ADR 0026's seam

Status: Accepted · Date: 2026-10-26 · Design section: Upgrades, Storage Decided: Implement Talos upgrades as a single AdvanceMachineFunc (evacuation hook, Upgrade API, boot-ID-based reboot detection, version/readiness/etcd verification, A/B fallback is a failure) with all resumability held in TalosMachine conditions, and no Ichor-side cordon, drain or Preserve handling

Context

ADR 0026 already supplies the lock, the status-held state machine, pause-on-failure, maxUnavailable, hold annotations, label ordering and one-control-plane-at-a-time. It leaves one seam, AdvanceMachineFunc. This ADR is what fills it for kind: Talos.

Decision

• Per call, state is re-derived from the node (Version, BootID) and two TalosMachine conditions: TalosUpgrade (True from the acknowledged Upgrade call; its message holds the boot ID read just before the call) and StorageEvacuation (True from the evacuation write; its transition time is the timeout clock). A restarted controller resumes from those.
• Not yet upgraded: check pkg/upgrade path and Kubernetes pairing, node Ready, then evacuate, then Upgrade API with the factory installer image. etcd health is not re-checked: Talos 1.14.3's Server.Upgrade (internal/app/machined/internal/server/v1alpha1/v1alpha1_server.go) calls etcd.Client.ValidateForUpgrade for non-worker nodes unless force or preserve is set, and we set neither.
• After the call: dial or RPC errors are "still rebooting". An unchanged boot ID means not rebooted yet. A changed boot ID with a version that is not the target is the A/B slot having fallen back: an error, which ADR 0026 turns into a paused operation. Otherwise wait for Ready and, on a control plane, etcd membership, then end evacuation and mark converged.
• Evacuation timeout is a rollout pause (error, scheduling re-enabled), never "proceed". No TalosStorage selecting the node means nothing to evacuate. On success scheduling is re-enabled only after the node is verified.
• No cordon, drain or uncordon: Talos's Sequencer.Upgrade does them. UpgradeRequest.Preserve is deprecated and ignored by the installer, so it is not handled.
• The version contract is never bumped automatically.
• Kubernetes and Ichor kinds are not wired; cmd/main.go returns an error for them rather than treating them as Talos upgrades.

Consequences

pkg/storage has its first caller here. Node removal (removal.go) now runs the same evacuation before its drain: the decision lives once in evacuateMachine / endMachineEvacuation (evacuation.go), which both the upgrade and removal call, with the same StorageEvacuation condition as the resume marker and timeout clock. A removal-side timeout stops the removal (scheduling re-enabled, Warning Event) and retries evacuation; it never drains.

E2E observation (2026-10-26)

Run on the 3-control-plane cluster (Talos v1.14.3, Cilium, no storage), TalosCluster.spec.talosVersion set to v1.14.4 (factory schematic 376567988a..., the harness's), then a TalosOperation kind: talos. The controller ran in a pod on the outer cluster with webhooks off. The pkg/render vv image bug (7dcc455) was not hit: the harness builds the node config with the machinery generator directly, and the controller only dry-ran it (RebootRequired, not applied).

• First attempt failed, a real bug: Upgrade ran under the 15 s etcdDialTimeout context. The RPC blocks while the node pulls and installs the installer image, so it returned DeadlineExceeded, the operation went Paused, and the node was untouched (still v1.14.3). Fixed with a separate 10-minute bound for that call (upgradeCallTimeout).
• Second attempt (created 14:02:53 JST, finished 14:12:13): the operation went Paused at once with no machines (a cluster lock still held from the deleted first operation), then started at about 14:05. Order: ha-0 TalosUpgrade=True/UpgradeRequested, then Converged (about 14:06:35); ha-1 NotReady,SchedulingDisabled during the reboot, Converged (about 14:09:54); ha-2, Converged; operation Completed at 14:12:13. Never two nodes at once.
• Afterwards talosctl version on all three: v1.14.4; etcd members: three voters, none learner; get machinestatus: running, ready; Kubernetes nodes Ready. No A/B fallback occurred.
• Rough edges seen: while a control plane rebooted the operation briefly went Paused (clusterlock: held by "ichor-controller" and API connection errors in the controller log, because the test controller's kubeconfig names one node's apiserver) and resumed on its own; the status showed Paused with ha-1 InProgress for about a minute. The evacuation hook was not exercised (no TalosStorage), only the pass-through path.
• Drain/cordon is Talos's own; the node showed SchedulingDisabled during its turn.

Amendment (real hardware, Talos v1.14.3)

The drain assumption above holds for Upgrade and is re-checked against the source: Sequencer.Upgrade (internal/app/machined/pkg/runtime/v1alpha1/v1alpha1_sequencer.go:440-455) runs the drain phase (CordonAndDrainNode, which drains through pkg/kubernetes/kubernetes.go:242, a five-minute bound, DaemonSet and mirror pods skipped) unless node registration is skipped, and the upgrading stage cordons the node (controllers/k8s/node_cordoned_spec.go:73-76). It does not hold for Reboot, which ADR 0080 now drains itself. The executor also no longer requires spec.hostname: the node name comes from the node's Nodename resource when the field is unset (ADR 0080, "Observed on hardware").

• Amendment (ADR 0113): the target of a Talos upgrade is the release set's Talos, and the admission webhook and the operation's planning refuse any other. A Talos step of the cluster rollout also writes the release set's version contract in the same write. The executor is unchanged.