0136. An orphan of a managed cluster is wiped with the cluster's own credentials

Status: Accepted · Date: 2026-11-07 · Design section: Nodes, Trust · Amends: 0128 (reclaim refusals, the --add-machine refusal), 0135 (risk "machines removed before this change were not wiped"), 0107, 0103 (section 2.2)

Decided: a machine that carries the configuration of a cluster Ichor manages but has no approved TalosMachine (an orphan: removed before ADR 0135, so never wiped) is no longer refused by reclaim. ichor machine reclaim ADDRESS --cluster NAME without --talosconfig, and the UI's "Reset and add" with "use this cluster's own credentials", create a MachineReclaim with the new spec.clusterCredentials: true and spec.clusterRef; the controller dials the node with an os:admin credential built from the cluster's own secrets bundle, applies the same assessment (typed UUID, quorum and last-control-plane refusals, graceful reset of STATE and EPHEMERAL, reboot into maintenance mode, the reclaim verb), and, once the machine is back in maintenance mode, deletes its Kubernetes Node as Remove does. A machine that still has an approved TalosMachine is refused with a pointer to Remove node. adopt --add-machine does not derive the hostname and link choice from the node: when it blocks a machine it adds the line "to start over, wipe it with ichor machine reclaim ADDRESS --cluster NAME (or the UI Reset and add)".

Context

The hardware run of ADR 0135 found a dead end. A machine removed with the pre-0135 remove (the TalosMachine deleted, the node not wiped, still carrying the managed cluster's configuration) could not be brought back or wiped through Ichor. Reclaim refused it ("belongs to cluster X, which Ichor already manages; a reset is not needed, add it with ichor adopt --add-machine"), and adopt --add-machine refused it too: the HostnameConfig, LinkAliasConfig and DHCPv4Config documents were rendered by Ichor from the deleted TalosMachine's spec (hostname and NIC chosen at approval) and differ from a fresh render of a machine that has no spec yet, so adopt asked for a hand-written TalosConfigPatch. The two refusals pointed at each other; the only way out was talosctl reset. ADR 0128 and ADR 0135 both named --add-machine as the way back.

Decisions

What an orphan is. A node whose cluster ID is one of a managed cluster's and whose SMBIOS UUID is not that of an approved TalosMachine. A pending or rejected TalosMachine of the UUID does not count (ADR 0128 already ignores them). reclaim.Assess therefore drops the "belongs to a managed cluster" refusal; the UUID refusal stays and now says "use Remove node". For an orphan it adds a Warning that names the alternative which keeps the configuration (ichor adopt --cluster NAME --add-machine ADDRESS). A machine that joined with the user's own talosctl apply-config is also an orphan by this definition; it can be wiped, and the warning says what keeps it instead. Rejected: keeping the refusal for a machine that never had a TalosMachine. Ichor cannot tell it from a removed one (the TalosMachine is gone), and the typed UUID is the consent either way.
Credentials. A MachineReclaim has spec.clusterCredentials (bool) and spec.credentialSecretRef is now optional; a CRD rule says exactly one of "credentialSecretRef" or "clusterCredentials with clusterRef" is set, and neither can change after creation. The field is added, nothing is removed (CRD fields are append-only). The controller builds a talosconfig for the one node from the cluster's secrets bundle (reclaim.ClusterCredential: an os:admin client certificate signed by the bundle's Talos API CA, no endpoints) and uses the existing reclaim.Dial, so the inspection, assessment, confirmation and reset code is the same as for a pasted foreign talosconfig. No Secret is written for such a request, so the person needs no talosconfig and nothing sensitive is stored beyond what Ichor already holds. The controller already reads the bundle Secret in its own namespace (ADR 0011) and gains no new Secret rights. Who may ask is unchanged: the reclaim verb on machinereclaims, judged by the webhook (ADR 0107); the credential is only used for the one address of the request, and the controller dials the node only after the cluster named by clusterRef was found. A node that does not belong to that cluster cannot be dialled (its certificate is not signed by that CA), and one that is another managed cluster's would be refused by the typed UUID and the same assessment. Rejected: asking the person for the cluster's talosconfig (ichor talosconfig needs an approved os:admin request and the person may not have one); a new TalosOperation (reclaim already is the one-shot reset request with the audit trail).
Quorum. Unchanged: an orphan control plane still in etcd is assessed like any other: its members are read, a removal that would break quorum or remove the last control plane is refused unless --force and the typed cluster name, and the reset is graceful when the cluster answers (leave etcd). A control plane whose cluster does not answer needs --force-ungraceful, as before.
The Kubernetes Node. When the machine is back in maintenance mode (Done), the controller deletes the Node named by the machine's hostname if its status.nodeInfo.systemUUID equals the machine's UUID; a Node of that name with another UUID is left and the result says kubectl delete node NAME. A machine in maintenance mode cannot register the Node again, so no wait for NotReady is needed (ADR 0135 waits because it deletes right after the reset is accepted). This applies to any reclaim whose inspected cluster ID is a managed cluster's, whichever credential dialled it. The controller already has delete on Nodes (ADR 0135).
UI API, UI, CLI. CreateReclaimRequest.use_cluster_credentials (field 7, appended) takes the cluster and no talosconfig and writes no Secret. The Reset and add dialog shows a checkbox "It was removed from this cluster" when the person may update the cluster; ticking it hides the talosconfig fields. The CLI, without --talosconfig, needs --cluster NAME, creates the request as the user (the webhook checks the audit annotations), prints the controller's inspection, asks for the UUID, patches confirmUUID, waits for Done and registers the address. The cluster must live in the Ichor namespace (where the controller reads it), as with the UI.
adopt --add-machine. Not changed to derive spec fields. Hostname, link alias and DHCP documents are rendered from TalosMachine spec fields that adopt does not model (the NIC choice is a hardware selection made at approval), so a derivation would be a guess of the person's earlier choice; ADR 0128's class C rule (what the typed spec cannot represent blocks the machine) stays. What changed is the text: a blocked --add-machine machine now also says "to start over instead, wipe it with ichor machine reclaim ADDRESS --cluster NAME (or the UI Reset and add)", and, when every differing path is one of those three documents, that they are what Ichor rendered from the deleted TalosMachine. The hand-written patch line stays. Rejected for now: reading hostname and NIC back into spec.hostname and spec.network (revisit if wiping is too costly for a node with user data on other disks).
No loop. Reclaim never refuses an orphan with a pointer to adopt; adopt's blocked line points to reclaim only as the wipe. The refusal for an approved machine points to Remove node, which neither command refuses. Tests: TestAssess_OrphanOfManagedClusterWarnsAndDoesNotLoop (the orphan is not refused, the approved machine's refusal does not mention adopt), the MachineReclaim of an orphan envtest specs (the same orphan is reclaimed, the approved machine is refused with Remove node, the control plane quorum refusal, the Node is deleted only when the UUID matches), TestAdoptRefusalOffersTheWipe.

Consequences

A machine removed before ADR 0135 comes back as a pending machine with one command; the guide's table row for it moves from adopt to reclaim, with adopt as the alternative that keeps its configuration.
Wiping an orphan destroys its STATE and EPHEMERAL; the typed UUID is the consent, and the inspection lists the volumes that are not wiped.
The credential Ichor holds for the cluster is used from the controller for a node of that cluster only; the request, its inspection and its audit stay in MachineReclaim status and Events as before.

Risks

Not verified on hardware: the wipe of a real orphan (reset of a node whose cluster Ichor still manages, its Node deleted afterwards), the system UUID match on a real kubelet Node, and the CLI against a real controller.
A hostname that differs from the Kubernetes Node name (a custom nodeName) leaves the Node; the result names kubectl delete node.
An orphan that is still a working member of its cluster (a machine that was never removed from etcd) is reset gracefully; the person typed its UUID, and the warning says what keeps it.

Amendment (2026-11-07): discovery reports an orphan as Orphan

Hardware verification found that discovery never flagged an orphan: knownHosts skipped the address of every Kubernetes Node (a member whose machine endpoint is stale), and an orphan still runs as a Node, so it was never probed and the Add node page had no entry for it. Decided:

A Node whose status.nodeInfo.systemUUID is not the UUID (name or spec.identity.uuid, case-insensitive) of any TalosMachine in the namespace, and whose address no machine's spec.endpoint or status.observedEndpoint names, is an orphan candidate. Its address is probed instead of skipped. A Node without a system UUID is still treated as a member.
A probe of a candidate that answers "already configured" records the Event reason Orphan (same endpoint annotation) instead of AlreadyConfigured; any other outcome goes through the normal registration. An address with no such Node, a foreign device, stays AlreadyConfigured. Neither Event wipes or changes anything; the unauthenticated probe cannot show the node is this cluster's, the wipe can (only the cluster's CA dials it), and the typed UUID is still the consent. Nothing in this change can reset a foreign machine.
Cluster.orphan_addresses (field 25) lists orphans next to already_configured_addresses. Both are derived from the latest Event per address and drop an address that a non-rejected TalosMachine of the cluster now has, so a machine that became pending again, or was approved, no longer shows a stale link. The Add node page offers Reset and add with the cluster's own credentials preset for an orphan; the Nodes page passes the cluster to the dialog as the Add node link does.
ReclaimInspection.warnings (field 9) carries the assessment's warnings (the orphan warning naming adopt --add-machine) to the UI inspection step.

Risk: a member whose Node reports a system UUID that differs from its SMBIOS UUID would be reported as an orphan; it is a warning only and the reclaim would be refused for an approved machine's UUID only if the UUIDs matched, so the typed-UUID step still shows what the machine is.