0032. Etcd member removal completes the removal sequence; Reset stays a separate unit

Status: Accepted · Date: 2026-10-26 · Design section: Nodes (removal) Decided: Complete a TalosMachine deletion by removing its etcd member -- self-leave first, a survivor removing it by ID as fallback -- clearing the finalizer only after that succeeds, and warning that the node keeps the cluster's credentials until it is reset

Context

ADR 0010 covers two of the removal sequence's three refusals (quorum, last-control-plane); pkg/deleteguard.CheckControlPlaneCount added the third (declared count). All three ran inside reconcileRemoval, but the step they gate — "remove the etcd member if it's a control plane" — was never implemented: the package comment on internal/controller/removal.go said plainly that etcd member removal and the Talos reset itself were "deliberately not implemented here," and a removal that passed every check and finished draining still held its finalizer forever. Confirmed for real against a live 3-node cluster (docs/adr/0007's appendix): a TalosMachine whose deletion is admission-allowed and controller-approved never actually disappears, and deleteguard.CountApprovedControlPlanes's List therefore never stops counting it either, since nothing ever clears its finalizer. That is not a "mostly working" removal path — it is a machine that drains and then gets permanently stuck, which Ponytail's "no half-working thing left behind" rule treats as a real problem, not a phase boundary to leave alone.

Finishing it needs a real Talos API call. TalosClient's doc comment calls itself "the one interface rule 0 allows," so the fix widens that one interface rather than adding a second — a second Talos-facing interface would double unit rule 2's fake-test-double burden for no reason the design asks for.

Decision

`TalosClient` grows three methods, verified against machinery v1.14.3's own client.go and api/machine/machine.pb.go

EtcdMemberList(ctx) ([]EtcdMember, error) — adapts *client.Client.EtcdMemberList (QueryLocal: true), flattening EtcdMemberListResponse.Messages[].Members[] down to {ID uint64, Hostname string}: enough to find a member by the Node hostname etcd itself reports, and the numeric ID EtcdRemoveMemberByID needs.
EtcdLeaveCluster(ctx) error — adapts *client.Client.EtcdLeaveCluster, called on the departing machine's own client. talosctl's own etcd leave command doc says plainly to "always prefer etcd leave over" removing a member by ID.
EtcdRemoveMemberByID(ctx, memberID uint64) error — adapts *client.Client.EtcdRemoveMemberByID, called on a surviving sibling's client. talosctl's own etcd remove-member command doc: "Use this command only if you want to remove a member which is in broken state. If there is no access to the node, or the node can't access etcd to call etcd leave."

No second interface, no new field on TalosMachineReconciler beyond reusing the existing TalosClient factory function a second time — once for the departing machine, and (only on the fallback path) once more for a surviving sibling found by listing approved control-plane TalosMachines in the same cluster via r.APIReader (the same direct-read requirement CountApprovedControlPlanes already has, for the same reason: a cached read could still show a sibling that started its own removal moments ago).

`removeEtcdMember`'s order: self-leave, then survivor-remove, then hold

Dial the departing machine itself. If reachable, list members. If the departing machine's hostname is not present, it is already gone (the resumption case below) — done, no further Talos call.
If present, call EtcdLeaveCluster on that same connection. Success — done.
Otherwise (unreachable, or leave itself failed) fall back to a surviving sibling: list members there, and if the hostname is still present, call EtcdRemoveMemberByID with its numeric ID.
If no survivor can be dialed either, return an error. reconcileRemoval records it as the Removal condition's EtcdMemberRemovalFailed reason and holds the finalizer — the same "stay, don't disappear" shape every other refusal in this file already has.

Only after this returns successfully does completeRemoval clear machineRemovalFinalizer — in that order, deliberately: clearing the finalizer first and removing the etcd member second would let the object vanish while etcd still has a stale member nobody will ever ask about again, exactly the ordering bug this ADR exists to not ship.

Idempotent by construction, not by string-matching an error

Architecture.md already names the invariant: "removing an already-removed etcd member reports not-found, which counts as success." Rather than parsing Talos's error text for that, removeEtcdMember checks the live member list before acting, on every call, and returns immediately if the departing hostname is already absent. That is what makes resumption after a crash correct in both directions:

Killed before the etcd call ever lands: the finalizer is still there, nothing changed in etcd, the next reconcile re-lists, finds the hostname still present, and retries EtcdLeaveCluster/EtcdRemoveMemberByID from scratch.
Killed after a successful removal but before the finalizer cleared: the next reconcile re-lists, finds the hostname already gone, and skips straight to completeRemoval — it never re-asks etcd to remove something that is not there, and never trusts any local memory of "did I already do this" the way a status flag written before the call would have to.

This is also why EtcdLeaveCluster/EtcdRemoveMemberByID need no fire-and-forget "record intent before acting" status write the way Reset does (next section): both return their result synchronously in the RPC, so there is nothing in flight to lose track of across a crash that the live member list wouldn't already answer on the next read.

Reset stays out of this interface entirely, and the roadmap still says so

Architecture.md: with no partitions named, Reset's default mode wipes the entire system disk rather than returning the node to maintenance mode — a reinstall, not the reuse story the design wants — and the call itself is fire-and-forget, needing its request recorded before a crash so a resumed reconcile cannot fire a second concurrent reset at the same node. Both are a materially different kind of work than everything else TalosClient does (every other method here is synchronous and side-effect-idempotent), and deciding the right partition-naming argument shape is a real design question this ADR is not going to answer as a side effect of fixing finalizers. Reset is left for whoever builds it, same as ADR 0010 left the declared-count refusal for whoever built that.

TestTalosClientHasNoResetMethod (internal/controller/etcd_removal_test.go) guards this with a reflect-based check over TalosClient's method set, failing loudly if a future change adds anything Reset-shaped without also bringing back the fire-and-forget handling that would need.

The consequence of deferring Reset: a removed machine still holds cluster PKI

Once completeRemoval clears the finalizer, Ichor has no record of the machine at all — but the physical node was never reset, so it still holds the secrets bundle's PKI (its Talos API client certificate, trusted by the cluster's CA). It cannot rejoin etcd (it is not a member anymore and has no path back in without being re-approved from scratch), but it is not safe to repurpose or discard either: whoever finds it still has valid credentials for this cluster's Talos API.

The TalosMachine this would normally live on disappears in the same reconcile that discovers it, so a status condition is not durable enough — the object carrying it is gone moments later. completeRemoval instead emits a Warning Kubernetes Event (NodeStillHasClusterCredentials) naming the machine before clearing the finalizer: Events are their own object with their own lifetime, independent of the TalosMachine's, which is exactly the durability this caveat needs and a condition on a soon-to-be-deleted object cannot provide. This is not silently accepted — it is the one thing this ADR requires getting said out loud every time a control plane leaves, until Reset exists to actually close it.

`CheckControlPlaneCount`'s refusal message now names both ways out

The message previously said only "approve a replacement control plane before removing this one." That is correct but incomplete: an operator who actually wants to shrink the cluster, not replace a member, has no way to read that out of the message. It now also says "or lower spec.controlPlane.count first (to 1, 3 or 5) if you mean to shrink the cluster" — deliberately not naming a specific target number, since the enum only allows 1, 3 or 5 and the arithmetically "obvious" next value (current - 1) is frequently not one of them.

The four failure modes, re-checked now that two of them are no longer hypothetical

Architecture.md's "Nodes" section names four ways to get this fatally wrong. Two are Reset's alone (partition naming; fire-and-forget double-reset) and still cannot be stepped into, because no code in this package calls Reset at all — TestTalosClientHasNoResetMethod is the guard that stays true. The other two are what this ADR's own tests exercise directly: Ichor's refusals happening before any Talos call (removal_test.go's "the four failure modes" Describe block — unchanged in spirit, updated in wording now that a passing removal really does call Talos), and the not-found-counts-as-success idempotency (etcd_removal_test.go's four removeEtcdMember specs: self-leave success, survivor fallback, already- absent on both paths, and held-finalizer-on-total-failure).

Consequences

A TalosMachine removal that passes every check now actually finishes: the object disappears, not just its finalizer's reason-for-staying.
Two Talos dials instead of one for a removal that needs the survivor fallback (list on self, then list-and-remove on a survivor) — accepted, since the alternative is never completing a removal of an unreachable node at all.
reconcileRemoval now needs a loaded secrets bundle to get past drain, where it previously needed none at all. A cluster without a working bundle Secret holds the finalizer at EtcdMemberRemovalFailed instead of reaching RemovalApproved — the same "refuse loudly, don't guess" shape loadBundle's callers already use elsewhere.
Operators now see a Warning Event, not nothing, telling them a decommissioned node still carries this cluster's credentials until it is reset. Reset itself remains unbuilt; this ADR only makes the gap visible instead of silent.

Real-hardware verification

Two real, previously-latent bugs found and fixed, neither anticipated going in:

secrets.Bundle.Clock carries yaml:"-" -- deliberately excluded from the exact serialization the bundle Secret decoder reads back. Nothing refills it on decode except populate(), which only runs while building a brand-new bundle. The first ever call from real production code to GenerateTalosAPIClientCertificate on a bundle loaded this way panicked on a nil-interface Clock.Now() -- NewMachineryTalosClient dialing with the fixture's real bundle, recovered by controller-runtime but looping forever on retry. No test anywhere had exercised this combination before: unit tests fake TalosClient entirely, and every previous real-hardware pass either had no bundle at all or never got a working one. Fixed in internal/secretsbundle.Load (ADR 0033 had already centralized the decode this ADR's own fix first landed in): fill Clock with secrets.NewClock() if nil after decode, the same fallback populate() itself uses. Centralizing it there means internal/uiapi's own read-only Talos client gets the same fix for free, not just this package's.
No context timeout anywhere in removeEtcdMember/ removeEtcdMemberFromSurvivor. Confirmed for real: the control plane that had just left etcd (ichor-e2e-ha-0) went unreachable on its own Talos API moments later -- its local kube-apiserver stopped answering too, consistent with Talos no longer running the control-plane stack on a node that is not an etcd member -- and the next reconcile's self-dial attempt hung with no deadline, wedging this controller's one talosmachine worker on every other TalosMachine behind it for as long as the process stayed up. Fixed with etcdDialTimeout (15s), wrapping both the self-attempt (trySelfLeave) and the survivor-fallback attempt.

The removal itself, with etcd's own before/after member list:

# before, queried from ichor-e2e-ha-0 itself (10.244.1.40):
NODE          ID                 HOSTNAME         ...
10.244.1.40   221caadec20f5672   ichor-e2e-ha-2
10.244.1.40   5085f640db370298   ichor-e2e-ha-1
10.244.1.40   74b4db31286a41e2   ichor-e2e-ha-0

Lowered TalosCluster.spec.controlPlane.count from 3 to 1 (the operator action CheckControlPlaneCount's own refusal message now names), annotated ichor-e2e-ha-0's TalosMachine with allow-delete=true and deleted it.

# after, queried from a survivor (10.244.1.208, since ha-0's own endpoint
# was no longer answering by this point -- see bug 2 above):
NODE           ID                 HOSTNAME
10.244.1.208   221caadec20f5672   ichor-e2e-ha-2
10.244.1.208   5085f640db370298   ichor-e2e-ha-1

Two members, both voters, ichor-e2e-ha-0 gone -- a real graceful EtcdLeaveCluster, not the quorum math being taken on faith.

Idempotency, both sides of the one real crash:

Killed before the Talos call lands: annotated and deleted ichor-e2e-ha-0's TalosMachine, then kill -9'd the manager process immediately after the delete request returned. Confirmed before restarting: finalizer still present, no Removal condition written at all, etcd still the same 3 members as the "before" snapshot above. On restart, the manager re-acquired its cluster lock under its own stable identity (no HeldError) and retried from scratch.
Interrupted after the etcd call succeeds, before the finalizer clears: this is bug 2 above, and it is also the "after" case -- ichor-e2e-ha-0 left etcd for real, then went unreachable before completeRemoval's status write could land (its own apiserver had just stopped answering). Once the controller's own client was pointed at a surviving node instead and restarted, the resumed reconcile's self-dial to ichor-e2e-ha-0 timed out (bug 2's fix), fell back to the survivor path, found the hostname already absent from that survivor's own member list, and went straight to completeRemoval -- confirmed by the NodeStillHasClusterCredentials Warning Event firing and the TalosMachine object actually disappearing, with no second EtcdLeaveCluster or EtcdRemoveMemberByID call attempted (nothing to call it on: the member was already gone). This is the scenario this ADR calls "the most fragile point," found not by engineering a precise kill-timing race but by the environment providing the exact same window on its own -- which is arguably stronger evidence than a deliberately timed kill would have been, since nothing about it was staged.

Deleting an unreachable node: attempted, not cleanly completed, and the reason is itself informative. The delete request itself never reached the inner apiserver: with only one of two etcd members still reachable, etcd on the inner cluster lost its leader entirely (etcdserver: no leader, confirmed directly) -- exactly what architecture.md's "Topology" section already says about two-member clusters ("with two members etcd needs both to be up, so it is strictly less available than one") and exactly why controlPlane.count's enum rejects 2, now observed rather than taken on faith. removeEtcdMemberFromSurvivor's EtcdRemoveMemberByID path (as opposed to the EtcdMemberList calls already exercised repeatedly above) was consequently never exercised against a real, still-a-member, unreachable node this pass -- reaching that scenario cleanly needs a fourth real control plane (so removing one unreachable member still has a reachable quorum left to ask), which this pass's two remaining real nodes could not provide without engineering a second full HA boot. Left for a future pass; the machinery-client call itself is the same verified client.Client wrapper EtcdLeaveCluster already proved out, and etcd_removal_test.go's fake-backed test already covers the control flow.

Update: removal now evacuates storage before the drain (shared with the Talos upgrade, see ADR 0043), so the order is evacuation, drain, etcd member, reset.

Addendum: removal resolves the node name with resolveNodeName (an empty spec.hostname no longer skips the drain, the evacuation or the etcd member); a control plane whose name cannot be resolved is refused with NodeNameUnresolved and retried, and the credentials event says "removed from etcd" only for a control plane.