0107. Reset and add replaces the override reclaim switch

Status: Accepted; built · Date: 2026-10-30 · Design section: Trust, Nodes, Deployment · Amends: 0103 (section 2.2), 0105 · Touches: 0094

Decided: there is no installation-wide switch for resetting a machine that another Talos cluster configured and adding it here; who may do it is decided by RBAC alone, the custom verb reclaim on machinereclaims, checked by the webhook with a SubjectAccessReview as the requesting user (as before); ichor-admin (group ichor:admin) holds that verb by default, ichor-machine-reclaimer stays for granting it to anyone else, editors and viewers do not get it; IchorInstallation.spec.reclaim.enabled and the chart value reclaim.enabled are deprecated and ignored (the CRD field stays, append-only; the admission webhook warns when it is set; ichor settings set reclaim.enabled=... and UpdateSettings refuse it with a pointer to this ADR; status.effective stops reporting it; the SettingsGate no longer has a ichorinstallations/reclaim check); the UI calls the feature "Reset and add" (dialog "Add a machine from another cluster"), offers it on the Add node page next to the AlreadyConfigured message and on the Machines page to users who hold the verb, and drops the reclaim card from Settings; the kind MachineReclaim and the command ichor machine reclaim keep their names.

Context

ADR 0103 section 2.2 made machine reclaim an override feature, off unless reclaim.enabled (chart value, then IchorInstallation.spec.reclaim.enabled with a confirmation annotation, ADR 0105). The user found the name "override reclaim" wrong for what is an ordinary step of adding a machine that already runs Talos, and its on/off switch out of place on the Settings page; it belongs with adding nodes.

Decisions

1. No installation switch. The request already needs the foreign cluster's admin talosconfig: whoever holds it can reset the node with talosctl anyway. On top of that Ichor requires a typed UUID, refuses a reset that would break quorum or remove the last control plane, keeps the credential in a TTL'd single-use Secret, and writes audit Events and status.audit. A second, global key adds a forgotten-toggle problem, not safety. The controller reconciler, the MachineReclaim webhook and the UI API (ReclaimService) therefore no longer read a switch; the webhook's SubjectAccessReview and the UI API's own SAR for the verb are the only gates. CanReclaimResponse.enabled stays in the proto, always true.
2. Roles. ichor_admin_role (config/rbac) and ichor-admin (chart ichor-user-roles) list create, get, list, update and reclaim on machinereclaims, as a plain rule like their other explicit verbs (these roles are not built by aggregation). ichor-machine-reclaimer has the same rules and is now installed whenever rbac.helpers.enabled is set (before: only with reclaim.enabled). ichor-operator and ichor-viewer do not get the verb.
3. Users off. With Ichor's users off the UI API writes as its own ServiceAccount, like it does for its other writes (the talosoperations verbs, settings). That identity is judged by the same webhook and holds no reclaim verb by default, so the button is not offered until the operator binds ichor-machine-reclaimer to ichor-system:ichor-ichor-ui. This is the behaviour of ADR 0103 C8, unchanged; the ServiceAccount's own Role gains nothing.
4. Deprecation. spec.reclaim is kept in the schema because CRD fields are append-only and installed objects may carry it; its doc comment says Deprecated. The IchorInstallation admission webhook adds a Warning when it is set. The chart value reclaim.enabled stays in values.yaml with a deprecation note; the manager no longer gets --reclaim and ichor-ui no longer gets ICHOR_RECLAIM_ENABLED. status.effective.reclaimEnabled and the proto fields reclaim_enabled, can_edit_reclaim and confirm_reclaim stay (wire compatibility) but are never filled or read. The annotation ichor.maesh.dev/confirm-reclaim has no meaning any more. The RBAC subresource ichorinstallations/reclaim is no longer checked; it is removed from the shipped roles (a role losing a verb that nothing asks for breaks no installed binding, since bindings refer to roles by name).
5. UI. The Settings page has no reclaim card. On the Add node page an AlreadyConfigured address shows "Reset and add" when the user holds the verb; it opens the dialog prefilled with the address and the target cluster. Without the verb the CLI hint stays. The dialog keeps the stored-credential warning and the typed-UUID step; the "Override" banner wording is gone. The Machines list button has the same name.

Consequences

• A fresh install lets ichor:admin use Reset and add with no step to take; anyone else needs a RoleBinding to ichor-machine-reclaimer.

• An installation that had the switch off and relied on it as a guard now has the verb in ichor-admin; an operator who does not want it there must not grant ichor:admin or must edit the role.

• Existing IchorInstallation objects with spec.reclaim keep working; the Warning tells the person writing them.

• The removed flags --reclaim (manager) and --reclaim-enabled (ichor-ui) stay accepted as deprecated no-ops for one release, as --http-addr in ADR 0105 unit 10, so a hand-edited Deployment that still passes them starts. Delete them in the release after.

Amendment (2026-10-30): findings of the hardware run

• Maintenance deadline. A request in WaitingForMaintenance fails exactly reclaimMaintenanceWait (10 minutes) after the reset was sent. The wait was measured from a timestamp taken at the start of each reconcile and every reconcile held one of the controller's single worker, so any slow call (a node call has no timeout of its own) delayed all later checks; the hardware run saw 14.5 minutes. Now each reconcile has a 2 minute timeout, the controller runs four workers, no probe runs past the deadline, and the requeue is the smaller of 10 s and the time left, so the failure comes at the deadline with no more slack than the probe in flight.
• TTL. spec.ttl (default 15 minutes, at most 60) bounds only the part before the reset: the credential Secret, and the request while it waits for the typed UUID. The reset deletes the credential; from then on the TTL is not consulted and the request ends Done, Failed (10 minutes after the reset) or is kept for seven days after its expiry time.
• Discovery addresses. The controller still never writes spec. The UI API, which adds the address as the user, records each address it appended in the annotation ichor.maesh.dev/reclaim-added-addresses on the TalosCluster (metadata; an address the user had listed is never recorded). CreateReclaim (before it appends its own address) and ConfirmReclaim then take an address back out, as the caller and best effort, when no Done or running request of that cluster names it and no TalosMachine of the cluster is at it: that covers failed, expired and deleted requests. Reads (GetReclaim, ListReclaims) never write a cluster. There is no DeleteReclaim RPC (it would need a new delete grant in the admin roles and a UI path); a request deleted with kubectl leaves its address until the next create or confirm. Rejected: an annotation on the MachineReclaim (a deleted request would take its record along) and a "remove address" button (the user cannot tell which addresses Ichor added). A user who lists the same address by hand after Ichor added it loses it again with the request; they should add it before.
• Time to a pending machine. A Done request enqueues its target cluster, so discovery probes the new maintenance address right away (no write; the TalosCluster controller watches MachineReclaim). Before, a cluster whose list already held the address waited for its 5 minute discovery pass. Expected now: a few seconds after Done.

Hardware result 2026-10-30 (build v0.0.1850000002-gc549005e, the tree at c549005e): deadline, address cleanup and wake-up

Setup (prefix ichor-e2e-z, 4 nodes): an ichor init cluster ichor-e2e-z (1 control plane + 1 worker, users off, upgraded twice, see ADR 0105) and a foreign Talos v1.14.3 cluster ichor-e2e-zf (talosctl gen config with a KubeNetworkConfig outside 10.244.0.0/16, control plane bootstrapped, worker joined; Kubernetes flannel). ichor-machine-reclaimer bound to ichor-system:ichor-ichor-ui (reclaimui can: enabled, allowed, clusterAllowed), ichor-ui reached through an inner NodePort 30080. spec.discovery.addresses of ichor-e2e-z held the user's 10.244.1.243 (the Ichor worker) and 10.244.9.9 (added by hand). No Go tests were run.

• (b) A machine that never returns: Failed exactly 10 minutes after the reset. Phase WaitingForMaintenance until Failed at 03:13:10Z (Event ReclaimFailed 03:13:10Z): delta 10 min 0 s (the earlier run saw 14.5 min). status.result: 10.244.2.191 did not return to maintenance mode with UUID 71c86230-...: It may have taken a new address from DHCP: scan for the machine and add its address to a cluster's spec.discovery.addresses. The credential Secret was deleted at 03:03:10Z (CredentialDeleted). A reading of the cluster after Failed (via get) still listed 10.244.2.191 and the annotation ichor.maesh.dev/reclaim-added-addresses: 10.244.2.191: reads write nothing. As in ADR 0103's C7 run, the stop within a second of the reset interrupted the wipe: after the node was started again (new address 10.244.2.117) the worker answered an authenticated call with the foreign talosconfig as a member of ichor-e2e-zf.
• (c) Address cleanup passed. Creating reclaim-flbc5 for the new address 10.244.2.117 took 10.244.2.191 out and put 10.244.2.117 in: spec.discovery.addresses went from ["10.244.1.243","10.244.9.9","10.244.2.191"] to ["10.244.1.243","10.244.9.9","10.244.2.117"], and the annotation to 10.244.2.117. The user's two addresses stayed. The confirm path was not separately tested (the create already pruned).
• (a) Done to a pending machine: under 1 s. reclaim-flbc5 confirmed 03:14:09Z (reset sent 03:14:09Z), Done seen at 03:14:21.2Z (12 s, node back in maintenance), the pending TalosMachine 71c86230-... (endpoint 10.244.2.117:50000) has creationTimestamp 03:14:21Z and was seen by polling at 03:14:22.1Z. The address was already in the list (added at create), so this is the wake-up: before it waited for the 5 minute discovery pass.
• Failing tests: none run. Everything with prefix ichor-e2e-z and ichor-e2e-zf was deleted; kubectl get shows none left and the cached ISO disk is untouched.