0085. Node actions are controller-run operations of kind node

Status: Accepted · Date: 2026-10-27 · Design section: Upgrades, Nodes (removal), UI backend, Talos access Decided: items 1, 2, 4, 6, 7 and 8 of ADR 0082's proposal, plus a per-node shutdown, are one new TalosOperation kind node with spec.node {machineRef, action, service, allowLeader, confirm} and one action per operation: reboot, shutdown, restart-service (kubelet, containerd, cri only), etcd-defrag, etcd-forfeit-leadership and reset; "Remove node" is the existing deletion of the TalosMachine through the removal flow (a new RemoveMachine RPC and ichor node remove), not an operation, and etcd member removal stays inside it (ADR 0032); the UI and CLI write only these CRs, the controller is the only writer to the node

Context

ADR 0082 listed what the Talos client enables and ranked it, and implemented none of it. The operation backbone (ADR 0026) already gives a lock, a status-held state machine, pause-on-failure and refusal while another disruptive operation or a shutdown or CA rotation runs; ADR 0068 gives Plan and Create with the plan text shown by the CLI and the UI; ADR 0080 gave a reboot a drain, a marker and a boot ID; ADR 0067 gave a stop its resting phase. This ADR adds a kind on that backbone rather than a second mechanism.

Decision

CRD. spec.kind=node requires spec.node and nothing else allows it (CEL), and spec.node is immutable. action is an enum; service is an enum of the three allowed services and is required for restart-service only; confirm must equal machineRef.name for reset (CEL, so even a hand-written CR cannot skip it); allowLeader goes with etcd-defrag only. status.node {step, requestedAt, message} is the state machine; the machine's entry in status.machines holds the boot ID read just before the action was sent (as for shutdown, ADR 0067). One machine, one action, one operation.

Executor (internal/controller/node_action.go), dispatched by TalosOperationReconciler after the cluster lock like shutdown. Every step is written to the status before the next call, so a restarted controller never sends an action twice. This is ADR 0080's marker approach with the operation's status as the marker: it is a CR the controller can always read back, including after its own node rebooted. A failed step pauses the operation with failureReason and is never retried by itself; a cordon made for it is lifted.

• reboot: the node must be Ready; for a control plane etcd must list every approved control plane (the check config rollout uses, now one function etcdAllMembersPresent); storage evacuation; cordon and drain with drainForReboot (Talos 1.14.3's Reboot does not drain, so Ichor does, with the helper the config rollout uses, PDB-respecting and bounded); record boot ID; Reboot; wait for a new boot ID, a Ready Node and, for a control plane, the member back in etcd, for at most 20 minutes; uncordon (only if Ichor cordoned) and end the evacuation. The controller's own node is not special-cased: if the drain evicts the controller or the reboot stops it, the next controller (or the same one after the reboot) resumes from status.node.step; an unreachable node during Requested is waiting, not failure.
• shutdown: no Ready check (a sick node may be stopped); the quorum guard as for reboot; storage evacuation; record boot ID; Shutdown(force=false), so Talos drains the node itself (it does, ADR 0067). The cluster's only control plane is stopped with force=true: there is no API left to drain against, and the plan says the Kubernetes API goes away. Refused for a control plane when the cluster has two control planes (one left is below quorum) or an etcd member is missing. When the node stops answering the operation releases the lock and rests in phase ShutDown (the phase and awaitPowerOn of ADR 0067); it completes when the node answers on a new boot. A node that is off holds no lock and does not block the next operation.
• restart-service: the allowlist is checked in the CRD enum, the plan, and the executor, so three independent places refuse etcd, apid, trustd and machined. After the call the node must be Ready again (not before 15 s, since the condition lags).
• etcd-defrag: control plane only. On a multi-member etcd it is refused on the leader unless allowLeader is set (the plan says writes pause for the whole cluster then) and refused while a member is missing. The lock is renewed while the call runs (bounded to 10 minutes). Afterwards the alarm list must be empty, otherwise the operation fails; Ichor does not disarm alarms.
• etcd-forfeit-leadership: control plane only, only on the member that currently leads, with at least two members all present. Completes when the node reports a different leader.
• reset: the existing removal path, not a second one. The executor sets the reset-on-delete opt-in (ADR 0042) and allow-delete, adds the removal finalizer and deletes the TalosMachine; the removal flow (lock, quorum and declared-count checks, storage evacuation, drain, etcd member removal for a control plane, Reset of STATE and EPHEMERAL, finalizer) does everything else. The removal takes the cluster lock itself, so the operation releases its lock first and only watches the machine go (step: Removing, the removal condition's message is relayed). The removal flow now also runs for a worker that carries the finalizer (the drain and the reset; no etcd step and no control-plane counts); a worker only gets the finalizer from a reset or a RemoveMachine, so deleting a worker by hand behaves as before.

Override and confirmation for reset. CreateOperation requires confirm equal to the machine name and an override reason, and asks Kubernetes whether the caller holds the override verb on that TalosMachine (ADR 0065's SelfSubjectAccessReview as the user). With OIDC off it is refused, as override is. The reason goes to the audit record (create-override) and a Warning Event on the machine. Reset is not a GitOps write, but it is the one action that destroys a node's data, so it uses the one permission Ichor already has for "do it now, outside the normal path". A reset of a TalosMachine that Argo CD tracks is refused in the plan: Argo would create it again.

Remove node. MachineService.RemoveMachine plans like the other actions (same plan RPC, action REMOVE), needs the name typed, refuses what the webhook and removal flow refuse (last control plane, quorum, declared count) up front in the plan, refuses a tracked machine, sets allow-delete and the finalizer and deletes the machine. The node is not wiped and keeps the cluster's certificates (ADR 0032's Event still fires); reset is the action that wipes it. The ichor-ui ClusterRole gains delete on talosmachines (split from the talosclusters rule; config/rbac/ichor-ui/role.yaml and the chart template change together).

Plan, Create, entry points. PlanOperation and CreateOperation take NodeActionRequest and a node plan carries plain-English steps, a warning, the name to type (confirm_name) and whether an override reason is needed. The plan's blockers are what the API can know: not approved, being removed, service not allowed, etcd action on a worker, two control planes, quorum and count for reset and remove, tracked, another unfinished operation. What needs the node (is it the leader, is a member missing) is the executor's and pauses the operation with the reason. CLI: ichor node reboot|shutdown|reset|remove NODE, ichor node restart-service NODE SERVICE, ichor etcd defrag|forfeit-leadership --node NODE, each showing the plan and asking [y/N] (--yes skips it); reset and remove need the name typed or --confirm NAME; reset needs --override REASON; --follow. UI: an Actions menu (preactcn dropdown-menu) in the node page header; each item opens a preactcn alert-dialog that shows the plan, a destructive warning, the typed name where needed (shutdown, remove, reset) and the override reason (reset), and a notice with a link to the operation after it starts. Actions that cannot apply (a node that is not approved or is being removed, etcd actions on a worker) are not offered; what a user's own RBAC forbids is not probed and is shown as the server's refusal.

Not decided here

Support bundle, etcd snapshot and packet capture (ADR 0082 items 3, 5, 9) are not implemented. Override for the last-worker refusal: a node operation carries no override field and the webhook checks it for reset only, so the refusal has no override yet.

Amended: a reboot or shutdown of a worker is refused (the operation pauses with the reason) when no other schedulable, Ready, non-control-plane Node exists. restart-service also requires the Talos service to report Running (and healthy, when it has a health check) through the Talos ServiceList API, in addition to the Node being Ready; a client that cannot read service state gets the Node check alone.

Consequences

• One more operation kind; TalosOperation gains spec.node and status.node; TalosClient gains ServiceRestart, EtcdStatus, EtcdAlarms, EtcdDefragment, EtcdForfeitLeadership (all plain calls from machinery v1.14.3).
• The operation reconciler gets KubeClient and Evacuator (the same as the config rollout).
• Not verified on hardware: any of the actions against a real Talos node or cluster (the drain eviction of the controller's own pod, Talos's drain on Shutdown(force=false), EtcdDefragment and EtcdForfeitLeadership responses, the removal flow of a worker with a reset). Tests use a fake Talos client and a fake clientset.
• A node reset is also guarded server-side: the override webhook (ADR 0065) now covers CREATE of talosoperations and requires the override verb on the target talosmachines for action: reset, so a hand-written TalosOperation cannot skip the CLI/UI check; the typed-name confirm was already a CRD rule.

Amendment: the Kubernetes Node stays after a remove

A remove deletes the TalosMachine but not the Kubernetes Node. The node is not wiped, so its kubelet keeps running with the cluster's credentials and would register a deleted Node again; the controller therefore never deletes the Node of a machine that may still run. Once the machine is powered off the Node goes NotReady and DaemonSets keep creating Pending pods for it until someone deletes it. The remove plan (CLI and UI) now ends with that step ("power it off or reset it, then kubectl delete node <name>"), ichor node remove prints it after it succeeds, and the NodeStillHasClusterCredentials Event names the stale Node. Not done: the controller deleting the Node of a removed machine after it has been NotReady for a while; it needs a record of which Nodes Ichor removed (new state), so it stays manual.

Amendment 2026-11-05 (ADR 0130): a reset needs the typed confirm and the override, and no override reason; the plan says whether the override is needed, and the dialog shows only the checkbox. spec.node.overrideReason is deprecated and unused.

Amendment 2026-11-06: --confirm NAME is the typed name given up front and counts as the confirmation, as typing it at the prompt does; ichor node reset|remove NODE --confirm NODE shows the plan and does not also ask [y/N]. --yes alone still does not replace the name. This matches ichor restore --live --confirm and ichor machine reclaim --confirm.

Amendment 2026-11-06 (ADR 0135): remove and reset are one action. A remove now also wipes STATE and EPHEMERAL, powers the machine off (or reboots it) and deletes the Kubernetes Node; the split into reset and remove and the paragraph above describe the earlier behaviour.

Amendment 2026-11-07: the typed confirmation is one rule everywhere: the TalosMachine name (the UUID the UI dialog shows with a copy button), which is what the CRD rule (confirm equals machineRef.name) and RemoveMachine check. The 2026-11-06 amendment's --confirm NODE is corrected to --confirm MACHINE: the CLI prompt and refusal read "type the machine name (node )", and neither the CLI nor the server accepts the hostname or Node name as the confirmation any more. The NODE argument is unchanged and still takes the machine name, hostname or Node name.

Amendment (ADR 0137)

An operation parked with ClusterTransitioning holds no lock and does not block the removal of a control plane; see ADR 0137.