0042. Reset is opt-in per removal, named partitions only, and idempotent by recording after the ack

Status: Accepted · Date: 2026-10-26 · Design section: Nodes (removal), Roadmap phase 3 Decided: Add an opt-in ichor.maesh.dev/reset-on-delete annotation that, after a removal's etcd member is gone, wipes only STATE and EPHEMERAL by label and reboots the node to maintenance mode -- holding the finalizer if Reset fails, and recording the request only after Talos has acknowledged it, so a resumed reconcile never asks twice

Context

ADR 0032 completed TalosMachine removal as far as etcd: drain, leave etcd, clear the finalizer. What it left unresolved, and said so out loud with a Warning Event (NodeStillHasClusterCredentials): the physical node was never reset, so it still holds the secrets bundle's PKI even after Ichor has forgotten about it entirely. That gap is this ADR's reason to exist.

Architecture.md's "Nodes" section already investigated Talos's Reset call and recorded two facts, not guesses:

The reset must name the partitions to wipe. With its default mode and no partitions named, Reset wipes the entire system disk, boot and EFI included, which means a reinstall rather than a return to maintenance mode. Passing STATE and EPHEMERAL explicitly is what leaves the installed system in place and brings the node back up with no configuration.

Reset is fire-and-forget. It returns an actor id immediately and the sequence runs asynchronously, so status has to record that a reset was requested before the controller can crash -- otherwise a resumed reconcile fires a second concurrent reset at the same node.

Both are design-level constraints this unit has to satisfy, not implement around.

Decision

Reset is irreversible, so it is opt-in per removal, not automatic

A new annotation, ichor.maesh.dev/reset-on-delete (value "true"), gates whether reconcileRemoval ever calls Reset at all -- the same shape deleteguard.AllowDeleteAnnotation already uses: a deliberate, per-object decision rather than a default, because an operator may legitimately want the disk left alone to investigate a failure, or intact to reuse for something else. Its absence changes nothing from ADR 0032: etcd member removed, finalizer cleared, NodeStillHasClusterCredentials fired exactly as before.

TalosClient grows one more method, verified against machinery v1.14.3's own source

Reset(ctx context.Context) error adapts *client.Client.ResetGeneric with a fixed, never-caller-configurable request:

&machineapi.ResetRequest{
	Graceful: false,
	Reboot:   true,
	SystemPartitionsToWipe: []*machineapi.ResetPartitionSpec{
		{Label: constants.StatePartitionLabel, Wipe: true},
		{Label: constants.EphemeralPartitionLabel, Wipe: true},
	},
}

• SystemPartitionsToWipe is never empty and never caller-supplied: ResetPartitionSpec's own doc comment says plainly that an empty list means "all the partitions are erased," and the machined server (internal/app/machined/internal/server/v1alpha1/v1alpha1_server.go) rejects Mode: USER_DISKS together with a non-empty SystemPartitionsToWipe -- confirming STATE+EPHEMERAL by label, Mode at its ALL zero value, is exactly talosctl reset --system-labels-to-wipe STATE,EPHEMERAL's own usage, not a novel combination. constants.StatePartitionLabel/EphemeralPartitionLabel are used rather than the literal strings, so a future machinery bump renaming either constant breaks this at compile time instead of silently wiping the wrong (or no) partition.
• Graceful: false: by the time this is ever called, removeEtcdMember has already drained the node and left etcd itself (ADR 0032). Talos's own graceful pre-reset routine exists to do exactly that when nothing else has, which is never true here -- architecture.md's own "belt rather than trousers" framing for Ichor's drain/leave applies the other way round to this flag: Ichor's own sequence is the trousers here, Talos's graceful-reset is the belt this call has no use for.
• Reboot: true: required for the "machine reappears as pending and can be reused" story architecture.md's removal section describes -- a halted node needs a human to power it back on before discovery can see it again, which defeats the point of resetting it automatically.

No second interface: this is the same TalosClient ADR 0032 widened, reusing the same r.TalosClient factory function a third time (self, survivor, and now self again for Reset).

Order: etcd leave, then Reset, then the finalizer -- Reset failure holds it

resetNode runs after removeEtcdMember succeeds and before completeRemoval. A failed Reset records reasonResetFailed on the Removal condition and returns without clearing the finalizer -- the same "stay, don't disappear" shape every other refusal in this file already has, and the one that matters most here: a TalosMachine that opted into Reset but whose Reset request never actually reached the node must not be allowed to vanish from Ichor while the physical hardware is untouched and still credentialed.

Idempotency: recorded after the RPC's own ack, not before

Reset is fire-and-forget -- the RPC returns once Talos has accepted the request and started the wipe-and-reboot sequence in a background goroutine (confirmed by reading v1alpha1_server.go's Reset handler: go func() { s.Controller.Run(resetCtx, runtime.SequenceReset, &opts) }() runs after the response is already built), not once the sequence finishes. resetNode therefore checks conditionRemoval's own status before calling Reset, and writes reasonResetRequested on it immediately after Reset's RPC returns without error -- the earliest point at which "a reset was requested" is actually true, and architecture.md's own fix for "a resumed reconcile fires a second concurrent reset": a crash after this write means the next reconcile sees the condition already True and returns without calling Reset again at all.

Two notes against calling Reset twice, read from Talos's own source (not inferred), for whoever next has to reason about how dangerous a near-simultaneous double-call actually would have been if this check were ever bypassed: Controller.Run takes a priorityLock before running any sequence, so a second SequenceReset arriving while the first is still in flight blocks (up to a minute) rather than racing the first on the same disk; and once a first Reset actually completes with Reboot: true, the node reboots into fresh maintenance mode, where the admin certificate any second attempt would authenticate with is no longer trusted at all -- the second call fails closed rather than repeating the wipe. Neither of these is why this ADR's own check exists (the architecture doc's instruction is unconditional, and this project does not rely on a dependency's internal locking as its own safety net), but both explain why no real-hardware attempt at forcing a genuine double Reset was expected to find anything more alarming than a clean refusal.

The Warning Event is wrong once Reset actually succeeded, so it is skipped

completeRemoval takes a reset bool. When true, the success message says the node was reset and is rebooting to maintenance, and NodeStillHasClusterCredentials is not emitted -- it would be false: Reset just wiped the STATE partition those credentials lived on. The Event fires exactly when it did before this ADR, in exactly the case this ADR did not change.

Consequences

• Removal has a real way to close the gap ADR 0032's own Warning Event named, but only when an operator explicitly asks for it per machine -- the default for every existing and future removal stays unchanged.
• resetNode reuses conditionRemoval rather than introducing a fourth condition type: IsStatusConditionTrue on it, checked before any destructive call in this file, now means "something terminal-shaped already happened here, do not redo it" uniformly across removeEtcdMember's sibling check, resetNode's own check, and completeRemoval's final write.
• A removal that opts into Reset and succeeds takes one more Talos RPC than one that does not, on the same node it was already dialing for removeEtcdMember's self-leave attempt.
• Nothing here touches forceRemove (reset on a node that cannot be dialed at all): opting into Reset when the node is unreachable simply fails at the dial, same as any other Talos call in this file, and holds the finalizer exactly like any other failed step.

Real-hardware verification

Run on 2026-10-26 against the 3-node HA cluster (Talos v1.14.3, ichor-e2e namespace, the real controller binary running as a pod, ADR 0007). One node, ichor-e2e-ha-1, was used for both cases in turn: it had already left etcd in an earlier run (etcd listed only ha-0), so deleting its TalosMachine exercised the drain, the etcd-already-gone path and, when opted in, Reset, without touching quorum. A TalosMachine is created approved with allow-delete, waited on until the removal finalizer is present, then deleted. State was read over the Talos API from a pod, with the cluster's own admin credentials.

• Without the annotation. Before: ha-1 machinestatus stage running, STATE and EPHEMERAL volumes ready. After the delete completed: a new NodeStillHasClusterCredentials Warning Event on the new object (new UID), and the node was unchanged -- stage running, STATE and EPHEMERAL still ready, the admin credentials still authenticating. This is what "opt-in" means: nothing is wiped and the Event keeps firing.
• With ichor.maesh.dev/reset-on-delete: "true". The delete completed and no NodeStillHasClusterCredentials Event was emitted for the new object (the Event list held exactly the two earlier ones). The node came back in stage maintenance with STATE missing (META still ready), and the admin credentials were refused with x509: certificate signed by unknown authority: it no longer holds the cluster's PKI, and the disk was not reinstalled (the node rebooted to maintenance rather than needing an install), as the named-partitions design intends. EPHEMERAL is not listed in maintenance; it was ready before.

Not observed: the Removal condition's ResetRequested reason (the object is gone once the finalizer clears and nobody was watching mid-flight), the ResetFailed hold (no failing node was available), and the second-call-is-a-no-op path against a live node. Those stay covered by internal/controller/reset_test.go only.

Amendment 2026-11-05: an optional full wipe

The default stays as decided here: STATE and EPHEMERAL by label, nothing else. A node that has to be installed afresh (a SecureBoot move, ADR 0123) cannot use it, because the installed boot partition survives and the node boots its installed system again. TalosOperation.spec.node.wipeMode (system-disk or all, only with the reset action; ichor node reset --wipe-mode) therefore asks for the other shape of ResetRequest: Mode SYSTEM_DISK or ALL and no systemPartitionsToWipe, which Talos reads as the whole system disk (and, for ALL, every user disk). The mode travels to the removal flow in the machine annotation ichor.maesh.dev/reset-wipe-mode, next to reset-on-delete; reclaim.ResetRequestWipe is the one place that builds the request, and MachineReclaim is unchanged (it always wipes STATE and EPHEMERAL). Needs the same override and typed name as any reset. Talos reference: talosctl reset --wipe-mode all|system-disk|user-disks (docs.siderolabs.com/talos/v1.14).