# 0107. Reset and add replaces the override reclaim switch

Status: Accepted; built · Date: 2026-10-30 · Design section: Trust, Nodes, Deployment · Amends: 0103 (section 2.2), 0105 · Touches: 0094

Decided: there is no installation-wide switch for resetting a machine that another Talos cluster configured and adding it here; who may do it is decided by RBAC alone, the custom verb `reclaim` on `machinereclaims`, checked by the webhook with a SubjectAccessReview as the requesting user (as before); `ichor-admin` (group `ichor:admin`) holds that verb by default, `ichor-machine-reclaimer` stays for granting it to anyone else, editors and viewers do not get it; `IchorInstallation.spec.reclaim.enabled` and the chart value `reclaim.enabled` are deprecated and ignored (the CRD field stays, append-only; the admission webhook warns when it is set; `ichor settings set reclaim.enabled=...` and `UpdateSettings` refuse it with a pointer to this ADR; `status.effective` stops reporting it; the SettingsGate no longer has a `ichorinstallations/reclaim` check); the UI calls the feature "Reset and add" (dialog "Add a machine from another cluster"), offers it on the Add node page next to the AlreadyConfigured message and on the Machines page to users who hold the verb, and drops the reclaim card from Settings; the kind `MachineReclaim` and the command `ichor machine reclaim` keep their names.

## Context

ADR 0103 section 2.2 made machine reclaim an override feature, off unless `reclaim.enabled` (chart value, then `IchorInstallation.spec.reclaim.enabled` with a confirmation annotation, ADR 0105). The user found the name "override reclaim" wrong for what is an ordinary step of adding a machine that already runs Talos, and its on/off switch out of place on the Settings page; it belongs with adding nodes.

## Decisions

1. **No installation switch.** The request already needs the foreign cluster's admin talosconfig: whoever holds it can reset the node with `talosctl` anyway. On top of that Ichor requires a typed UUID, refuses a reset that would break quorum or remove the last control plane, keeps the credential in a TTL'd single-use Secret, and writes audit Events and `status.audit`. A second, global key adds a forgotten-toggle problem, not safety. The controller reconciler, the MachineReclaim webhook and the UI API (`ReclaimService`) therefore no longer read a switch; the webhook's SubjectAccessReview and the UI API's own SAR for the verb are the only gates. `CanReclaimResponse.enabled` stays in the proto, always true.
2. **Roles.** `ichor_admin_role` (config/rbac) and `ichor-admin` (chart `ichor-user-roles`) list `create`, `get`, `list`, `update` and `reclaim` on `machinereclaims`, as a plain rule like their other explicit verbs (these roles are not built by aggregation). `ichor-machine-reclaimer` has the same rules and is now installed whenever `rbac.helpers.enabled` is set (before: only with `reclaim.enabled`). `ichor-operator` and `ichor-viewer` do not get the verb.
3. **Users off.** With Ichor's users off the UI API writes as its own ServiceAccount, like it does for its other writes (the talosoperations verbs, settings). That identity is judged by the same webhook and holds no `reclaim` verb by default, so the button is not offered until the operator binds `ichor-machine-reclaimer` to `ichor-system:ichor-ichor-ui`. This is the behaviour of ADR 0103 C8, unchanged; the ServiceAccount's own Role gains nothing.
4. **Deprecation.** `spec.reclaim` is kept in the schema because CRD fields are append-only and installed objects may carry it; its doc comment says Deprecated. The `IchorInstallation` admission webhook adds a Warning when it is set. The chart value `reclaim.enabled` stays in `values.yaml` with a deprecation note; the manager no longer gets `--reclaim` and ichor-ui no longer gets `ICHOR_RECLAIM_ENABLED`. `status.effective.reclaimEnabled` and the proto fields `reclaim_enabled`, `can_edit_reclaim` and `confirm_reclaim` stay (wire compatibility) but are never filled or read. The annotation `ichor.maesh.dev/confirm-reclaim` has no meaning any more. The RBAC subresource `ichorinstallations/reclaim` is no longer checked; it is removed from the shipped roles (a role losing a verb that nothing asks for breaks no installed binding, since bindings refer to roles by name).
5. **UI.** The Settings page has no reclaim card. On the Add node page an AlreadyConfigured address shows "Reset and add" when the user holds the verb; it opens the dialog prefilled with the address and the target cluster. Without the verb the CLI hint stays. The dialog keeps the stored-credential warning and the typed-UUID step; the "Override" banner wording is gone. The Machines list button has the same name.

## Consequences

- A fresh install lets ichor:admin use Reset and add with no step to take; anyone else needs a RoleBinding to `ichor-machine-reclaimer`.
- An installation that had the switch off and relied on it as a guard now has the verb in `ichor-admin`; an operator who does not want it there must not grant `ichor:admin` or must edit the role.
- Existing `IchorInstallation` objects with `spec.reclaim` keep working; the Warning tells the person writing them.

- The removed flags `--reclaim` (manager) and `--reclaim-enabled` (ichor-ui) stay accepted as deprecated no-ops for one release, as `--http-addr` in ADR 0105 unit 10, so a hand-edited Deployment that still passes them starts. Delete them in the release after.

## Amendment (2026-10-30): findings of the hardware run

- **Maintenance deadline.** A request in `WaitingForMaintenance` fails exactly `reclaimMaintenanceWait` (10 minutes) after the reset was sent. The wait was measured from a timestamp taken at the start of each reconcile and every reconcile held one of the controller's single worker, so any slow call (a node call has no timeout of its own) delayed all later checks; the hardware run saw 14.5 minutes. Now each reconcile has a 2 minute timeout, the controller runs four workers, no probe runs past the deadline, and the requeue is the smaller of 10 s and the time left, so the failure comes at the deadline with no more slack than the probe in flight.
- **TTL.** `spec.ttl` (default 15 minutes, at most 60) bounds only the part before the reset: the credential Secret, and the request while it waits for the typed UUID. The reset deletes the credential; from then on the TTL is not consulted and the request ends Done, Failed (10 minutes after the reset) or is kept for seven days after its expiry time.
- **Discovery addresses.** The controller still never writes `spec`. The UI API, which adds the address as the user, records each address it appended in the annotation `ichor.maesh.dev/reclaim-added-addresses` on the TalosCluster (metadata; an address the user had listed is never recorded). `CreateReclaim` (before it appends its own address) and `ConfirmReclaim` then take an address back out, as the caller and best effort, when no Done or running request of that cluster names it and no TalosMachine of the cluster is at it: that covers failed, expired and deleted requests. Reads (`GetReclaim`, `ListReclaims`) never write a cluster. There is no `DeleteReclaim` RPC (it would need a new `delete` grant in the admin roles and a UI path); a request deleted with kubectl leaves its address until the next create or confirm. Rejected: an annotation on the MachineReclaim (a deleted request would take its record along) and a "remove address" button (the user cannot tell which addresses Ichor added). A user who lists the same address by hand after Ichor added it loses it again with the request; they should add it before.
- **Time to a pending machine.** A Done request enqueues its target cluster, so discovery probes the new maintenance address right away (no write; the TalosCluster controller watches MachineReclaim). Before, a cluster whose list already held the address waited for its 5 minute discovery pass. Expected now: a few seconds after Done.

## Hardware result 2026-10-30 (build `v0.0.1850000002-gc549005e`, the tree at c549005e): deadline, address cleanup and wake-up

Setup (prefix `ichor-e2e-z`, 4 nodes): an `ichor init` cluster `ichor-e2e-z` (1 control plane + 1 worker, users off, upgraded twice, see ADR 0105) and a foreign Talos v1.14.3 cluster `ichor-e2e-zf` (`talosctl gen config` with a `KubeNetworkConfig` outside 10.244.0.0/16, control plane bootstrapped, worker joined; Kubernetes flannel). `ichor-machine-reclaimer` bound to `ichor-system:ichor-ichor-ui` (`reclaimui can`: enabled, allowed, clusterAllowed), ichor-ui reached through an inner NodePort 30080. `spec.discovery.addresses` of `ichor-e2e-z` held the user's `10.244.1.243` (the Ichor worker) and `10.244.9.9` (added by hand). No Go tests were run.

- **(b) A machine that never returns: Failed exactly 10 minutes after the reset.** Phase `WaitingForMaintenance` until `Failed` at 03:13:10Z (Event `ReclaimFailed` 03:13:10Z): delta 10 min 0 s (the earlier run saw 14.5 min). `status.result`: `10.244.2.191 did not return to maintenance mode with UUID 71c86230-...: It may have taken a new address from DHCP: scan for the machine and add its address to a cluster's spec.discovery.addresses`. The credential Secret was deleted at 03:03:10Z (`CredentialDeleted`). A reading of the cluster after `Failed` (via `get`) still listed `10.244.2.191` and the annotation `ichor.maesh.dev/reclaim-added-addresses: 10.244.2.191`: reads write nothing. As in ADR 0103's C7 run, the stop within a second of the reset interrupted the wipe: after the node was started again (new address `10.244.2.117`) the worker answered an authenticated call with the foreign talosconfig as a member of `ichor-e2e-zf`.
- **(c) Address cleanup passed.** Creating `reclaim-flbc5` for the new address `10.244.2.117` took `10.244.2.191` out and put `10.244.2.117` in: `spec.discovery.addresses` went from `["10.244.1.243","10.244.9.9","10.244.2.191"]` to `["10.244.1.243","10.244.9.9","10.244.2.117"]`, and the annotation to `10.244.2.117`. The user's two addresses stayed. The confirm path was not separately tested (the create already pruned).
- **(a) Done to a pending machine: under 1 s.** `reclaim-flbc5` confirmed 03:14:09Z (reset sent 03:14:09Z), Done seen at 03:14:21.2Z (12 s, node back in maintenance), the pending `TalosMachine` `71c86230-...` (endpoint `10.244.2.117:50000`) has `creationTimestamp` 03:14:21Z and was seen by polling at 03:14:22.1Z. The address was already in the list (added at create), so this is the wake-up: before it waited for the 5 minute discovery pass.
- Failing tests: none run. Everything with prefix `ichor-e2e-z` and `ichor-e2e-zf` was deleted; `kubectl get` shows none left and the cached ISO disk is untouched.