# 0103. Deploy onto machines that already run Talos

Status: Accepted; built; hardware-verified including A8 (its two gaps were closed by 8ebee126 and 26da779b; rerun passed 2026-10-30, see "Results, A3, A8, D4, D6 (2026-10-30)"; the second kill window remains unhit); Updated: 2026-10-30 (C6 and C7 hardware-verified, see C6 and C7 below) · Date: 2026-10-29 · Design section: Bootstrap (`ichor adopt`), Nodes, Cilium · Builds on: 0014, 0042, 0066, 0075, 0076, 0077, 0099, 0101

Amended by [ADR 0104](https://ichor.maesh.dev/next/adr/0104-a-cluster-with-no-cni-gets-ichors-cilium-before-ichor-and-ichor-is-installable-with-helm/): a cluster with no CNI is adopted as managed Cilium (Ichor installs it first), not unmanaged `talosCNI: none`; the network-policy gate follows live detection; test D5 and open question 3 are superseded.

Decided: Ichor adopts a running Talos cluster on Talos v1.12 or later (older is refused with "upgrade Talos first"); a Cilium cluster is taken over as Ichor's own Helm release through a diff-checked `TakeOwnership` Job, while any other CNI (and a Cilium the user wants left alone) is adopted only on the operator's explicit request in a new CNI-unmanaged mode, `spec.cni.preset: unmanaged`, in which Ichor neither installs, operates nor renders the CNI and skips its CiliumNetworkPolicies unless Cilium is present; it recovers the cluster's secrets bundle from a control plane's machine config, holds every adopted machine until zero drift is verified, only advises (never starts) a CA rotation afterwards, and refuses to join an already-configured node unless that node is reclaimed, either by `ichor machine reclaim` with the operator's own talosconfig or, as an override path that is off by default, from the UI through a short-lived, single-node, auto-deleted credential Secret, a typed-UUID confirmation and an audit Event; reclaim resets only STATE and EPHEMERAL

## Decisions by the user (2026-10-29)

These answered the questions this ADR first raised; the sections below are rewritten to match.

1. **Non-Cilium CNI on adoption**: not refused. A CNI-unmanaged mode is designed (1.1a). Fresh `ichor init` stays Cilium-only; unmanaged exists only for adoption.
2. **Reclaim**: both paths, the CLI with the operator's own credential and a UI/controller path with a short-lived Secret (section 2).
3. **CA rotation after adoption**: advice only, printed and shown, never started automatically (1.4).
4. **Minimum version**: Talos v1.12 or later; older is refused with a message to upgrade Talos first (1.9).

## Context

The user asked to think about deploying onto machines that already have Talos installed. Two situations are meant, and Ichor handles neither well today.

1. **A running cluster Ichor did not create**, brought under management (`ichor adopt`).
2. **One machine that already runs Talos, configured, added to an Ichor cluster.** Ichor's join flow starts from maintenance mode (ADR 0076); a configured node never reaches it.

Constraints from the repo: Cilium is the only CNI and Ichor's own Helm release (ADR 0099, kept by 0101); Argo CD, Kokumi and every GitOps tool are out of scope (0101); machine config is typed only, so a difference that is not a `SysctlConfig`/`KernelModuleConfig` document cannot become a patch (0014, 0075, 0077); the controller is the only writer to nodes, except that `init`, `adopt` and full-loss recovery predate the cluster and run in the CLI (architecture, "CLI and UI"); a reset names `STATE` and `EPHEMERAL` explicitly (0042).

### What `ichor adopt` does today (read from `internal/cli/adopt*.go` at this commit)

- It needs `--cluster`, `--secrets-bundle FILE` (a `talosctl gen secrets` file; the flag is required) and the operator's talosconfig. It dials nodes read-only.
- It **creates nothing but `TalosConfigPatch` objects**. `runAdopt` Gets the `TalosCluster`, lists `TalosMachine`s and patches, and Creates patches. The `TalosCluster`, the bundle Secret, the `TalosMachine`s, the approvals and Ichor itself must already exist. No command creates them for a cluster Ichor did not build.
- Per approved machine it reads the live config, renders a baseline, and diffs (`controller.ChangedPaths`). Only `SysctlConfig` and `KernelModuleConfig` are captured; a patch is returned only if render plus patch equals the live bytes. Everything else is reported by path with a hint (`specFieldHints`, ADR 0077). Unapproved machines are skipped (0075). A machine with JWT authenticators is refused (0094).
- It knows nothing about the CNI beyond a hint that flannel is unsupported. It does not read Kubernetes state, kube-proxy, KubePrism or inline manifests, and it never touches Cilium.
- The render drops only an inline manifest named `cni` (`pkg/cni.InlineManifestName`, `dropHandedOverInlineManifest`). An existing cluster's Cilium entry has whatever name its author chose.

## Findings

### Helm SDK v3.21.4 (`helm.sh/helm/v3`, the version in `go.mod`, read from the module cache)

- `Install` with `TakeOwnership` calls `requireAdoption` instead of `existingResourceConflict` (`pkg/action/install.go:353`, `validate.go:41`). `requireAdoption` only checks that each object exists. `checkOwnership` (the `app.kubernetes.io/managed-by: Helm` label and the two `meta.helm.sh/release-*` annotations) is skipped. So **`TakeOwnership` works on objects with no Helm metadata (Talos inline manifest, `helm template | kubectl apply`) and on objects owned by a different release name or namespace.** Without it those installs fail with "invalid ownership metadata".
- Before that, `setMetadataVisitor(..., force=true)` rewrites the ownership label and annotations on the rendered objects, so they carry Ichor's release afterwards (`install.go:339`).
- The objects are then updated with `UpdateThreeWayMerge(toBeAdopted, resources)` (`install.go:461`). In `requireAdoption` the infos appended are the *rendered* ones (the live object is read and thrown away), so "original" equals "target". Reading `createPatch` (`pkg/kube/client.go:640`) the consequence is: fields that differ are overwritten by the chart's values, but **live-only fields are kept and live-only objects are never deleted**, because nothing in the new release's manifest records them. Adoption converges the rendered fields and silently keeps the rest. This is read from source, not yet observed; hardware test A5 checks it.
- `availableName` looks for a release of that name **in the storage backend `cfg` points at** (`install.go:533`). Ichor stores records in `ichor-system` (ADR 0099, departure 2). A user's `sh.helm.release.v1.cilium.vN` in `kube-system` is therefore invisible: Install does not fail on a name clash, it creates a second, independent revision 1. Two Helm histories would then describe one set of objects, and a later `helm upgrade` by the user would fight Ichor.
- `Upgrade` also has `TakeOwnership` (`upgrade.go:121,347`), but Ichor's `Apply` only runs it after an Ichor-side record exists.
- Chart `crds/` handling (`installCRDs`) runs on Install; the pinned Cilium 1.18 chart is checked for a `crds/` directory in unit 5.

### Talos v1.14 (docs.siderolabs.com via Context7, and ADR 0099's earlier reading)

- Inline manifests are created and never updated or deleted by Talos; a values change needs `upgrade-k8s`. Therefore an inline-manifest Cilium cannot be left in the machine config once Ichor owns the release (ADR 0099, decision 2): the entry has to go, or a Kubernetes upgrade can re-apply it over the Helm release.
- `talosctl reset` accepts `--system-labels-to-wipe`, `--graceful` (cordon, drain, leave etcd; default true) and `--reboot`; ADR 0042 already fixed the request Ichor sends.
- `secrets.NewBundleFromConfig(clock, config.Config)` exists in machinery v1.14.3 (`config/generate/secrets/bundle.go:203`). It rebuilds the bundle (OS, Kubernetes, aggregator and etcd CAs, service-account key, cluster ID and secret, trustd token, bootstrap token, secretbox secret) from a control plane's loaded config, tolerating the v1.14 multi-document form.

## Decision

### 1. Adopting a running cluster

**1.1 The CNI decides the mode; Ichor never migrates one.** `ichor adopt` first runs a read-only inspection and prints a report. Provenance is classified from the node config and the Kubernetes API, not from names alone:

| Found | Result |
|---|---|
| Cilium, Ichor-bootstrapped | Already handled; adopt only checks the release exists. |
| Cilium from a Helm release (`helm`, `cilium install` of the Cilium CLI, which also writes a Helm release) | Take over (1.2, case B), or adopt unmanaged with `--cni-unmanaged`. |
| Cilium from a Talos inline manifest or an applied `helm template` | Take over (1.2, case C), or adopt unmanaged with `--cni-unmanaged`. |
| Talos's own flannel (`KubeFlannelCNIConfig`, `cluster.network.cni.name: flannel` or unset) | **Unmanaged only**: refused with the explanation below unless `--cni-unmanaged` is given. |
| Calico, another CNI, or `cni.name: custom` without manifest URLs | **Unmanaged only**, same. |
| `cluster.network.cni.name: custom` with `urls` (Talos applies the manifests) | Those URLs are an unrepresentable config path (class C, 1.7); the machine blocks. |
| Mixed (Cilium DaemonSet plus a second CNI's DaemonSet, or two CNI configs under `/etc/cni/net.d`) | **Refuse**, naming both, even with `--cni-unmanaged`: Ichor cannot say what it is leaving alone. |
| No CNI installed (`cluster.network.cni.name: none`, nodes NotReady) | Unmanaged with `talosCNI: none`. Installing Cilium onto it is an open item. |

The mode is never chosen for the user. Without `--cni-unmanaged`, a non-Cilium cluster is refused with nothing written (no Secret, no CR, no node call beyond reads), and the message names the CNI found and says: "Ichor will not install or operate it; to adopt the cluster with the CNI left exactly as it is, run again with --cni-unmanaged." Where the CNI is Cilium, the report offers both outcomes. The inspection is identical in both modes.

**1.1a The CNI-unmanaged mode.** The user asked whether CNI management can be turned off, so that a cluster with any CNI is adopted. It can, as an explicit mode with these properties.

*What it means.* The adopted cluster keeps whatever CNI it runs. Ichor does not install it, upgrade it, tune it or hand it over. Everything else keeps working: Talos and Kubernetes upgrades, CA rotation, config rollout, etcd backup and restore, node add/remove, reclaim, drift reporting.

*API choice.* A new enum value `unmanaged` on the existing `spec.cni.preset`, plus one new optional struct, `spec.cni.unmanaged`, with a single field `talosCNI` (`default|none`, default `default`):

- `default`: the machine config has no CNI selection (Talos runs its own flannel, or whatever `cluster.network.cni` selects by default). Render emits nothing about the CNI.
- `none`: render emits `cluster.network.cni.name: none` (the live value), because the baseline otherwise defaults to flannel.

Nothing is reused from retired values: `flannel` and `custom` stay refused (ADR 0099 decision 8) and keep meaning "removed", so a stored `preset: flannel` object is not silently reinterpreted as hands-off. The enum gains a value (append-only: the installer's additive check does not read `enum`, ADR 0099); `spec.cni.unmanaged` is a new property. `spec.cni.kubeProxy` keeps its field with a defined meaning in this mode: it records the Talos kube-proxy setting the node already has (`enabled`: Talos runs kube-proxy; `replaced`: it is disabled in Talos) so the render reproduces it, and it is no longer something Ichor switches (see below).

*Why a preset value and not `spec.cni.managed: false`.* (a) The preset is already immutable by CEL (`self.preset == oldSelf.preset`) and already the one switch every render, webhook and controller branch reads (`cniPresetFor`, `networkinstall.go`, `CNIUnsupported`); an unmanaged value is one more branch of the same switch and inherits immutability, which matters because turning management on or off changes the machine config (flannel document, kube-proxy, inline entry) of a running network. A mutable boolean would be a way to flip that by editing a field. (b) A boolean is orthogonal to the preset and so admits meaningless pairs (`preset: cilium, managed: false, cilium.values: {...}`, or `managed: false` with `tunnel` set) that each need a CEL rule and a runtime check; with a preset value the existing "only applies to the cilium preset" rules (`tunnel`, `cilium.values`) already forbid them. (c) It cannot be confused with the deprecated values: those say the cluster is unsupported; this one says it is supported and hands-off. The cost: a cluster cannot move between managed and unmanaged without being recreated, which matches ADR 0099's stance that a CNI change is not an in-place operation. Rejected: a new top-level `spec.networking.mode` (a second place describing the same thing).

*What is disabled* (each by a check, not only by hiding):

- The `ichor-cni` Job is never created: `CNIHandoverReconciler` returns early for `unmanaged`; no `CNIHandover`, `CNIValuesApplied` or `CiliumUpgradeAvailable` conditions are set, and `cniHandoverPending` is false so a Kubernetes upgrade is never blocked on a hand-over that cannot happen.
- `spec.cni.cilium` is refused by CEL for this preset, so there is no values page to write to; `ichor cni values|upgrade` and their `CniService` calls return "the CNI is not managed by Ichor (spec.cni.preset: unmanaged)".
- A `TalosOperation` of kind `cni` (chart upgrade, kube-proxy switch) is refused by the webhook for an unmanaged cluster, and the executor re-checks.
- The kube-proxy switch is disabled. The operator who changes Service routing owns that change; Ichor's consistency check (ADR 0066) between Talos kube-proxy and Cilium's replacement is skipped, since there is no Cilium contract to compare with.
- Render leaves the node's CNI configuration as the live node has it: no `KubeFlannelCNIConfig` `$patch: delete` document, no `cni.name` change, no inline `cni` entry, no KubePrism requirement or injection for Cilium (KubePrism is left as the node has it). `injectKubeProxy` writes the recorded `spec.cni.kubeProxy` value and nothing else. A render test asserts byte-equality for the CNI-relevant paths against the live fixture.
- The adopt report never lists class B CNI items (Cilium inline entry, flannel removal) for an unmanaged cluster; they are not Ichor-owned.

*What still works that touches the CNI:* Ichor keeps reading (never writing) CNI facts for display: `status.cni.detected` (`cilium|flannel|calico|other|none`, plus image tag where a DaemonSet exposes it), refreshed on a slow requeue, used by the UI/CLI and by the network-policy gate below.

*NetworkPolicies (ADR 0100).* Ichor's own policies are `CiliumNetworkPolicy` and need Cilium. They are applied only when `status.cni.detected` is `cilium` and the `CiliumNetworkPolicy` CRD exists (the installer already tests the CRD; the gate adds the detection). Otherwise the install step and the installer Job skip them and the chart renders `networkPolicy.cilium.enabled: false`; the `ichor-system` namespace is then not isolated by Ichor, and the UI and `ichor status` say so ("Ichor's network policies are not applied: the CNI is flannel"). Ichor does not write standard `NetworkPolicy` objects in this ADR (open item). Because the gate follows live detection, a hands-off Cilium cluster gets the policies, and a user who dislikes that sets `networkPolicy.cilium.enabled=false`.

*A Cilium cluster whose user wants Ichor hands-off* is allowed with the same mode: `ichor adopt --cni-unmanaged`. Nothing is taken over, no Helm record is touched, nothing is removed from the machine config (including a Cilium inline entry, which stays and becomes the user's drift problem: render cannot express it, so it is class C and blocks the machine, see 1.7; the user who wants to keep that entry should keep it in the node's own tooling). The user may later want Ichor to own it; that needs a new cluster, since the preset is immutable.

*How the UI and CLI show it.*

- Cluster header/list: a badge "CNI: unmanaged (flannel)" using `status.cni.detected`.
- The Cilium page is replaced by a card with the title and text "CNI not managed by Ichor", the detected CNI and version, and a one-line explanation ("Ichor does not install, upgrade or configure this CNI"). No values editor, no upgrade, no kube-proxy switch; those routes return a not-found page with the same text.
- The Operations "New" dialog omits `cni` for such a cluster.
- `ichor cluster status` prints `cni: unmanaged (flannel, not managed by Ichor)`; `ichor cni ...` commands exit non-zero with the message above; `ichor adopt` prints the same line in its plan and the Ichor-owned class B list omits all CNI entries.
- `ichor init` has no `--cni-unmanaged` flag and the webhook refuses `preset: unmanaged` on create unless the object carries the adoption marker the adopt write phase sets (annotation `ichor.maesh.dev/adopted: "true"`, 1.8). It is not a general way to build a cluster without a CNI.

*Why refuse migration, still.* A CNI swap takes pod networking down; the `TalosCluster` CRD already says so and makes `cni.preset` and the tunnel immutable for that reason. A migration cannot be rolled back by Ichor once the old datapath is gone (the new CNI owns the pod CIDR and node routes), cannot be verified before it is done, and needs per-workload knowledge Ichor does not have: NetworkPolicy semantics differ between CNIs, Service routing changes if kube-proxy is also switched, and the pod CIDR and IPAM mode of the old CNI constrain Cilium's (`ipam.mode` is Ichor-owned, `kubernetes`). Cilium's own documented migration is a node-by-node dual-CNI procedure with a different overlay CIDR; wrapping it would make Ichor the operator of a rolling production network outage, which is what ADR 0099's "Ichor bootstraps CNI once" avoided. The unmanaged mode removes the cost of that stance: a flannel cluster (Talos's default) is adopted and left alone. A migration ADR can still follow, and nothing here forecloses it.

**1.2 Taking over an existing Cilium** (managed mode only; none of this runs with `--cni-unmanaged`).

Preconditions, checked before anything is written:

1. **Version.** The live Cilium minor equals the embedded chart's minor, or is exactly one lower (the existing one-minor-at-a-time rule for `cni` operations, architecture "Upgrades"). Otherwise refuse and say to upgrade with the tool that installed it first. Adoption of an older minor is an upgrade; it is announced as one.
2. **Ichor-owned keys.** The live configuration is compared with the keys Ichor owns (ADR 0099, decision 3): `ipam.mode` (must be `kubernetes`; a `cluster-pool` Cilium is **refused**, changing IPAM renumbers pods), `kubeProxyReplacement` (maps to `spec.cni.kubeProxy`, 1.5), `tunnelProtocol` (maps to `spec.cni.tunnel`; immutable afterwards), `k8sServiceHost`/`k8sServicePort`, the Talos capability and cgroup keys. A value that Ichor would change on a running data plane (other than the capability and cgroup keys, which Ichor makes match the Talos guide) is reported and requires `--accept-cilium-changes`; IPAM and tunnel protocol are never accepted.
3. **Values.** Case B: the user values are decoded from the newest release record (`sh.helm.release.v1.cilium.vN`, base64, gzip, JSON, field `config`) and become `spec.cni.cilium.values` after removing owned keys, which are then checked as in step 2. Case C has no record, so the operator supplies `--cilium-values FILE` (what they gave `helm template`); without it adopt uses the Ichor defaults and the diff in step 4 will show what is missing.
4. **Diff gate.** Ichor renders the pinned chart with the effective values (the same `EffectiveValuesFor` path as bootstrap) and compares every rendered object with the live one, with the volatile fields removed. An empty difference proceeds. A non-empty one is printed per object and path and needs `--accept-cilium-diff`. This is the Cilium counterpart of adopt's zero-drift rule. It is needed because of the live-only-fields finding above: `TakeOwnership` would otherwise leave an unreviewed mix of chart and user state that Ichor's later `Upgrade` (`ResetValues`) interprets differently.
5. **Health.** The DaemonSet and operator are Ready on all nodes, and KubePrism answers on every node (1.6), because the Job reaches the API through `localhost:7445` while agents restart.

The takeover itself:

- It reuses the `ichor-cni` Job and `cilium.Apply` (ADR 0099 unit 7). With no Ichor record, `Apply` runs `action.Install` with `TakeOwnership: true`, which is correct for both cases: B (objects carry another release's metadata, possibly another name or namespace) and C (no Helm metadata). Only the diff gate and the foreign-record handling are new.
- **Foreign release record.** After the Ichor record is written and the rollout is Ready, a foreign `cilium` record in `kube-system` (or elsewhere) is copied into a Secret `ichor-cilium-prior-release` in `ichor-system`, then deleted (the record Secrets only; never `helm uninstall`, which would delete the CNI). Deleting it is announced in the plan and part of the consent. Leaving it would give two histories for one object set (see the `availableName` finding). Rejected alternative: point Ichor's storage at the foreign record's namespace and `Upgrade`; it keeps the user's history but breaks ADR 0099's namespaced-Role decision and the hand-over reader.
- **Inline entry.** In case C and in any cluster whose Cilium also sits in `cluster.inlineManifests`, the entry is found by content (a manifest that contains the `cilium` DaemonSet in `kube-system`), not by name, because `dropHandedOverInlineManifest` matches only the name `cni`. Its name is recorded (additive status field on the TalosCluster, unit 6) so the render drops it. **Ordering:** the entry is removed from the machine config only after the Helm record is `deployed` and the DaemonSet Ready (`CNIHandover`, unchanged in shape). If takeover fails nothing on the nodes has changed. Other inline or external manifests are not touched and make the machine's drift unrepresentable (1.7).
  *As built (unit 6):* `ichor adopt` finds the entry with `cilium.FindInlineManifest` and writes its name in the annotation `ichor.maesh.dev/cni-inline-manifest` at create time; the CNI hand-over controller copies it to `status.cni.takenOverInlineManifest` in the same status update that sets `CNIHandover=True`, and render reads only the status. The `ichor-cni` Job gets `--takeover` for every Job of an adopted cluster until the hand-over completes (a retry after a failed first run still retires the foreign record; the retirement is idempotent), and it needs get/list/delete on Secrets in kube-system only (Role `ichor-cni-records-role`). `cilium.Preflight` runs the preconditions read-only for the adopt plan.
- No rollback of an adoption (Atomic would uninstall a running CNI, ADR 0099). A failed Install leaves a `failed` record; the next run goes to `Upgrade` (as `Apply` already does) and Helm's `pending-install` state needs the explicit handling listed in Risks.

**1.3 The cluster's secrets bundle.** The `--secrets-bundle` flag becomes optional. If a file is given it is used (as today). Otherwise `--recover-secrets` reads one control plane's machine config over the Talos API with the operator's talosconfig and builds the bundle with `secrets.NewBundleFromConfig`, then `Validate`s it against the chosen version contract. Recovery is read-only on the node. The result is written under `~/.ichor/clusters/<name>/` with mode 0600, as `init` does, and the operator is told to back it up; nothing is printed. A wrong or partial bundle cannot get through: the render uses it, so any secret field that differs from the live config appears in the zero-drift diff (1.7) and no patch is produced. The three-CA PKI (Talos, Kubernetes, etcd) is unchanged; Ichor does not rotate anything during adoption.

**1.4 The talosconfig.** It is the operator's own and is used only by the CLI, read-only, never stored, never sent to the controller (as ADR 0014 decided). `--talosconfig` gains `--context` (the file may hold several clusters). The node list comes from the control plane's discovery membership, not from typing every address. After adoption the controller uses a certificate minted from the recovered bundle (its short-lived `os:admin` identity, architecture "Trust"). The operator's old client certificate stays valid until it expires or a CA rotation (ADR 0063, 0091). **CA rotation is advice only.** When adoption finishes, `ichor adopt` prints a recommendation ("the Talos and Kubernetes client certificates of the previous administrators, and the CA keys held by whoever created this cluster, remain valid; consider `ichor cluster rotate-ca`"), the UI shows the same text as a dismissible notice on the cluster page, and the cluster carries a condition `CARotationAdvised` (True, reason `Adopted`, cleared by the first completed rotation or by dismissal, which is an annotation written by the user). Nothing starts a rotation: no flag chains it to adopt, no controller creates the `TalosOperation`. The recommendation is the same in unmanaged-CNI mode.

**1.5 kube-proxy mode detection.** In unmanaged mode only the Talos side is read, and the result is written to `spec.cni.kubeProxy` as the recorded setting (`enabled` when Talos runs kube-proxy, `replaced` when disabled); there is no consistency check. In managed (Cilium) mode both sides are read and must agree:

- Talos side: `KubeProxyConfig.enabled` (v1.14) or `cluster.proxy.disabled` (legacy), plus whether the `kube-proxy` DaemonSet exists.
- Cilium side: the `kube-proxy-replacement` key of the `cilium-config` ConfigMap.

Mapping to `spec.cni.kubeProxy`: Talos proxy enabled and replacement off gives `enabled`; Talos proxy disabled and replacement on gives `replaced`. Disabled with replacement off (no Service routing) and enabled with replacement on (works, two owners) are **refused** with the reason, because Ichor's consistency check (ADR 0066) would mark the cluster unhealthy from its first minute and the fix is a switch the operator should make deliberately. The CRD default is `enabled`, so adopt always writes the detected value; it never relies on the default.

**1.6 KubePrism** (managed mode only; unmanaged leaves it as the node has it). Ichor's Cilium Job and its replaced-mode values require KubePrism on port 7445 on every node (ADR 0099). Detected from `machine.features.kubePrism` or the `KubePrismConfig` document. Disabled, or on another port: reported as an Ichor-owned difference (1.7, class B) that the first render will change, with consent; in `replaced` mode with KubePrism off, adoption is **refused**, since turning it on and moving Cilium onto it cannot be ordered safely in one step.

**1.7 Machine config drift on adoption.** The single "zero drift or nothing" rule of ADR 0014 stays for what patches can express. Three classes replace the two (patch, unrepresentable):

- **A. Representable**: sysctl and kernel-module documents, and values that now have spec fields (subnets, install disk, ADR 0077). Captured or mapped as today.
- **B. Ichor-owned** (managed mode only; empty of CNI items when unmanaged): the Cilium inline entry (1.2), KubePrism, kube-proxy (when `replaced`), flannel removal. The render will change these on its first apply. They are listed, need the plan's consent, and do not block.
- **C. Unrepresentable**: everything else in the legacy `Config` document and any other inline or extra manifest. Blocks that machine, as today, with the path list and hint.

Two additions make this workable:

- **Version contract probing.** The `TalosCluster.spec.versionContract` is chosen by rendering at each contract from the nodes' running version downwards, never below v1.12 (1.9), and taking the one with the fewest class C paths, not assumed equal to the running Talos version. A cluster generated on an old `talosctl` and upgraded in place has a legacy-form config; forcing v1.14 turns every field into drift (the ADR 0075 failure). The result is shown and can be overridden.
- **Hold.** The controller applies drift on every approved machine as soon as it exists. So `adopt` creates each `TalosMachine` with the annotation `ichor.maesh.dev/hold-apply: "true"`, which the machine reconciler reads: it computes and reports drift and never applies. Adopt removes the annotation, machine by machine, only after the live zero-drift check passes and the operator confirmed class B. No CRD field is added.

**1.8 Order and what is written.** `ichor adopt` becomes four phases; each ends at a state in which nothing has been applied to a node.

1. `--plan` (default without `--yes`): inspection and report, writes nothing anywhere.
2. Write: the bundle Secret, the `TalosCluster` (contract, subnets, `cni.preset: cilium` or `unmanaged` with `unmanaged.talosCNI`, `kubeProxy`, annotation `ichor.maesh.dev/adopted: "true"`), `TalosMachine`s (identity read over the authenticated API, role, install, held), `TalosConfigPatch`es, Ichor's install manifest if Ichor is not yet running (the same step `init` does; the network-policy gate of 1.1a decides whether the Cilium policies are included). All held.
3. Cilium takeover (1.2); skipped in unmanaged mode.
4. Release the hold per machine: first render is applied by the controller, control planes one at a time under the cluster lock like any config rollout (ADR 0080).

**1.9 Minimum version: Talos v1.12.** `ichor adopt` reads the Talos version of every node it will adopt (and the Kubernetes version) before anything else, and refuses a cluster with any node below v1.12.0 with nothing written and the message "Talos X.Y.Z is older than the oldest version Ichor adopts (v1.12). Upgrade Talos first, with talosctl or your usual tool, and run adopt again." A mixed cluster (some nodes at or above v1.12, some below) is refused the same way, naming the nodes. Contract probing (1.7) stops at v1.12, so the render path to be supported is v1.12 (legacy `Config` form), v1.13 and v1.14 (multi-document). The Kubernetes minimum is whatever Talos v1.12 supports and the pinned machinery validates; adopt does not add a second number. Each supported version has a harness case (test group D).

### 2. Adding a machine that already runs Talos

Four states must be told apart: (i) maintenance (existing flow, ADR 0076); (ii) configured by a foreign cluster; (iii) configured by this Ichor cluster (a returning machine; recognised by identity and cluster ID, handled by the authenticated path); (iv) installed but STATE wiped, which is maintenance.

For (ii), discovery cannot read identity (a configured node's API requires a client certificate), so today it fails with a TLS error and, per ADR 0011, registers nothing. Discovery now reports **`AlreadyConfigured`** as an Event and in the add-by-IP response when it sees a Talos API that completes a TLS handshake but rejects an insecure call. Nothing is created.

**Decision: refuse by default; reclaim only on an explicit request, by one of two paths that share one implementation.** Path 1 is the CLI with the operator's own credential (2.1). Path 2 is a UI/controller path fed by a short-lived Secret (2.2), an override feature that is off by default. Both run the same checks, build the same reset request and require the same typed UUID.

#### 2.1 CLI path

```sh
ichor machine reclaim ADDRESS --talosconfig FILE [--context C] [--confirm UUID]
```

- It is an in-process Talos write by the CLI, the same kind of exception as `init` and `adopt` (the node is not yet part of any Ichor cluster, and the credential is a foreign admin certificate that never leaves the operator's machine).
- Steps: authenticated read of identity, hostname, role, the foreign cluster's name and ID, disks, user volumes and, for a control plane, etcd members and health. It prints this, then:
  - **Refuses** if the node's identity or cluster ID is already an Ichor `TalosMachine` or cluster (state iii).
  - **Refuses** a control plane whose removal would break its cluster's etcd quorum, or the last control plane of a reachable cluster, unless `--force` and a second typed confirmation.
  - Uses `Graceful: true` (cordon, drain, leave etcd) when the foreign cluster is healthy and reachable from the node; `Graceful: false` only with `--force-ungraceful` when it is dead.
  - Sends exactly the request of ADR 0042 (`STATE` and `EPHEMERAL`, `Reboot: true`), built by the same code, so one place defines what a reset wipes. User disks and non-system volumes are not touched, but they are listed as orphaned, and encrypted ones are called out: their keys derived from STATE or META are gone, so the data is unreadable.
  - Requires **typed consent**: the operator types the node's UUID (or passes `--confirm UUID`; `--yes` does not satisfy it). Consent is the credential plus the typed identity, shown beside the cluster name that will lose the node.
  - Waits for the node to answer the maintenance API with the same UUID, then registers it through the normal add-by-IP path as a pending `TalosMachine`. A changed DHCP address after the reboot is reported with the scan hint instead of guessed.
- Without the node's credential there is no API route; the documented out-of-band routes are reinstall from the ISO or a wipe by kernel argument. The kernel-argument route works on Talos v1.14.3 with a rebuilt boot asset, then a boot from the disk (test C6, below).

#### 2.2 UI/controller path (override, off by default)

The user wants reclaim available without a shell. It is the one place the controller writes to a node of a cluster that is not Ichor's, using a credential that is not Ichor's, so the design is built around that credential never living longer, going wider or showing up in more places than one reset needs. The shared code is a new package `pkg/reclaim` (inspection, the refusals of 2.1, the request builder of ADR 0042); the CLI and the controller both call it.

**Objects.** One new namespaced kind, `MachineReclaim` (append-only: a new CRD, nothing existing changes), in `ichor-system`:

- `spec.address` (one IP or host:port), `spec.credentialSecretRef` (name only, same namespace), `spec.confirmUUID` (empty until the user types it), `spec.force` and `spec.forceUngraceful` (as 2.1), `spec.ttl` (default 15 minutes, hard maximum 60).
- `status.phase` (`Inspecting`, `AwaitingConfirmation`, `Resetting`, `WaitingForMaintenance`, `Done`, `Failed`, `Expired`), `status.inspection` (UUID, hostname, role, foreign cluster name and ID, disks, volumes and which are encrypted, etcd members, the refusals that apply), `status.expiresAt`, `status.result` (the new maintenance address, or the scan hint).
- The controller never writes `spec`; the typed UUID is written by the UI on behalf of the user, as every other change is.

**Flow.**

1. The user opens Machines, "Reclaim a configured machine" (the page `AlreadyConfigured` from unit 16 links here), enters the address and pastes or uploads the foreign talosconfig. The UI API creates the Secret and then the `MachineReclaim` as the caller.
2. The controller reads the Secret, dials **only** `spec.address` (endpoints and nodes in the talosconfig are ignored, so a multi-node config cannot widen the scope; a talosconfig with several contexts needs the one to use named in the request), runs the read-only inspection and the 2.1 refusals (a node that is already an Ichor machine or cluster, an etcd quorum that would break, the last control plane of a reachable cluster), and stops at `AwaitingConfirmation`. Nothing has been written to the node.
3. The UI shows the inspection with the foreign cluster's name, the orphaned and encrypted volumes, and the typed-UUID box. Confirmation is the user typing the node's UUID exactly (pasting is allowed; the dialog does not pre-fill it, and `Enter` does not submit an empty or mismatching box). The UI writes `spec.confirmUUID`. The webhook rejects a `confirmUUID` that differs from `status.inspection.uuid`.
4. Only when `spec.confirmUUID` equals the freshly re-read UUID, the TTL has not passed and the phase is `AwaitingConfirmation`, the controller sends the ADR 0042 request (`STATE`, `EPHEMERAL`, `Reboot: true`, `Graceful` per the inspection). It waits for the maintenance API to answer with the same UUID and then registers the machine as a pending `TalosMachine` through the add-by-IP path, as in 2.1.
5. The Secret is deleted when the phase becomes `Done`, `Failed` or `Expired`, whichever comes first, and in any case when the TTL passes; the `MachineReclaim` itself is kept for 7 days (no credential in it) as the record and then removed by the controller.

**The credential Secret.**

- Created by the UI API in `ichor-system`, type `ichor.maesh.dev/foreign-talosconfig`, with an owner reference to the `MachineReclaim` (so deleting the CR deletes it), the labels `ichor.maesh.dev/reclaim` and the expiry in an annotation. Its only key is `talosconfig`.
- **Scoped to one node.** It is bound to `spec.address` by the controller as described above, usable once (the controller records the use in status and refuses a second reset from the same request), and never reused for another request: a new request needs a new paste.
- **Short-lived and deleted.** Deleted after use (point 5) or at `expiresAt`. A sweeper in the controller, independent of the reconcile of any one CR, deletes every Secret of that type older than the maximum TTL, so a lost CR cannot leave a credential behind. The Secret lifetime is a requeue, not a side effect of a user returning to the page.
- **Never logged or echoed.** The credential lives in a type with redacting `String`, `GoString` and `MarshalJSON` (the Talos client config is parsed into it and nothing prints the parsed struct); it is not in `status`, in Events, in error text (wrapped Talos errors are passed through a filter that keeps the address and code, not the transport detail), in the audit Event, in gRPC/HTTP access logs (the UI API's logging interceptor drops the request body for the reclaim calls by method name, and a test asserts it) or in metrics labels. A unit test builds a distinctive marker into a fake talosconfig and asserts it appears in none of the log output, Events, status or panics from a full reconcile.
- The Secret is subject to the cluster's encryption at rest like any Secret; the UI warns that the pasted credential is stored in the cluster for up to the TTL, and that nothing prevents an `ichor-system` reader from seeing it in that window (hence the RBAC below).

**RBAC and override.**

- (Amended by ADR 0107: there is no installation switch; RBAC alone decides, and `ichor-admin` holds the verb. The UI calls it "Reset and add".) The feature is off unless the installation enables it (`IchorInstallation` Helm value `reclaim.enabled`, default `false`; the controller and the UI API refuse the calls and the Secret create when off). The controller's own permission is the namespaced `get;list;delete` on Secrets in `ichor-system` it already has for other purposes; the sweeper selects by the `ichor.maesh.dev/reclaim` label.
- A new ClusterRole `ichor-machine-reclaimer` grants `create`, `get`, `list` and `update` on `machinereclaims` (the UI API reads and confirms as the user) and the custom verb `reclaim` on `machinereclaims`; the webhook performs a `SubjectAccessReview` for that verb on every create and on every `confirmUUID` update. It is not part of any default role in the chart, including the admin role: it is bound only by an explicit RoleBinding, which is the override step. Reading Secrets of the credential type is not granted to anyone by the chart (the UI API creates them with a service identity limited to `create` of that type; it cannot read them back).
- The UI shows a banner on the dialog ("Override: this resets a machine that belongs to another cluster") and does not offer the page to a user without the verb, checked by the same SAR.
- Implementation notes (unit 14). Registration is not the controller's: it never writes `spec`, so `spec.clusterRef` (optional, additive) names the target cluster and the UI API adds the address to that cluster's `spec.discovery.addresses` as the user when it creates the request; the controller waits for maintenance mode and reports `status.result` (and whether the cluster already discovers the address). `status.audit.requestedBy`/`confirmedBy` are copied from the annotations `ichor.maesh.dev/reclaim-requested-by` and `-confirmed-by`, which the writer sets and the validating webhook insists equal the authenticated user (a validating webhook cannot stamp them itself). The UI API creates the request first and the Secret after (owner reference at creation); the controller requeues until the Secret exists and expires the request at its TTL if it never does. Because ichor-ui therefore holds `create` on Secrets in `ichor-system`, a ValidatingAdmissionPolicy (`ichor-ui-reclaim-credential`) limits what that ServiceAccount may create to the credential type, the `ichor.maesh.dev/reclaim` label and the single key `talosconfig`. A finished request is kept for 7 days after its `status.expiresAt`. The etcd health check judges the dialled member by its own etcd status and the other members by whether their peer port answers; `ichor init` has no switch for `reclaim.enabled` yet (the chart has).
- GitOps is not part of Ichor (ADR 0101), so no rule about tracked objects is needed.

**Audit.** The controller emits an Event on the `MachineReclaim` and on the `IchorInstallation` (so it survives the CR's removal in the namespace-level view) for: `ReclaimRequested` (user, address), `ReclaimInspected` (UUID, foreign cluster name and ID), `ReclaimConfirmed` (user who wrote `confirmUUID`, UUID), `ReclaimReset` (request sent, graceful or not), `ReclaimCompleted`, `ReclaimFailed` and `ReclaimExpired`, and `CredentialDeleted` (Secret name only). The identity is the caller recorded by the UI API (Kubernetes user name from the authenticated request) and kept in `status.audit`, not only in an Event that the cluster discards in an hour. Events never carry contents of the Secret; the same events exist for the CLI path as printed lines, since the CLI has no cluster object to attach them to.

*Why this is acceptable against "the controller never holds a credential for another cluster's nodes".* The statement protected a standing credential. This is a scoped, user-supplied, expiring one with a human typing the node's identity; the architecture's "Trust" section gets an exception paragraph, and the credential is the one thing in Ichor that is deleted by time rather than by an object's lifecycle.

Rejected: *joining without a reset.* Applying a new config with new secrets to a node that has the old cluster's etcd data and state does not give a clean member and the outcome differs by role; Ichor would also have to hold the old admin credential in the cluster. Rejected: *automatic reset by the controller* on approval; ADR 0042 made reset opt-in per object because it is irreversible, and a foreign node's data is the one thing Ichor has no record of. Both reclaim paths need the typed UUID; the UI path does not weaken that for convenience. Rejected: *storing the foreign talosconfig for reuse* (a "saved clusters" list): a standing foreign credential.

## What must change (work units)

Sizes: S under a day, M one to two days, L three or more. Units on the same row can run in parallel. Each lists files; a unit adds its own tests (names noted in the brief, per the rule on failing test names).

| # | Unit | Files | Size | Needs |
|---|---|---|---|---|
| 1 | Talos-side facts: Talos >= 1.12 gate (1.9), contract probing down to v1.12, KubePrism, kube-proxy, CNI document, inline manifests, install, roles from membership; `--context` | new `internal/cli/adopt_talos_facts.go`; `adopt.go`, `talosdial.go`, `adopt_plan.go` (contract loop) | M | none |
| 2 | Kubernetes-side facts: CNI detection and provenance (DaemonSets, `cilium-config`, CiliumNode CRD, Helm release Secrets, kube-proxy DaemonSet), the table of 1.1 (mixed refusal, `--cni-unmanaged` required for non-Cilium), 1.5 | new `internal/cli/adopt_k8s_facts.go`, `adopt_cni.go`; `adopt_run.go` (replace the flannel hint and refusal) | M | none |
| 3 | Secrets recovery: `--recover-secrets`, optional `--secrets-bundle`, validate, 0600 store | new `internal/cli/adopt_secrets.go`; `adopt.go`, `bundlefile.go` | S | none |
| 4 | Hold: `ichor.maesh.dev/hold-apply` annotation constant, reconciler reports drift and never applies, condition `AdoptionHeld` | `api/v1alpha1/talosmachine_types.go` (constant), `internal/controller/talosmachine_controller.go`, `talosmachine_controller_test.go` | S | none |
| 5 | Cilium takeover library: release-record decode, version rule, owned-key conflicts, live-object diff gate, foreign-record backup and removal, `crds/` check | new `pkg/cni/cilium/takeover.go`, `takeover_test.go`; `helm.go`, `validate.go` | L | none |
| 6 | Takeover wiring: Job flag and plan input, content-based inline-entry detection, recorded entry name (additive `TalosCluster.status.cni` field), render drops it, handover ordering | `internal/cli/cni_helm.go`, `internal/controller/cni_helm.go`, `cnihandover_controller.go`, `api/v1alpha1/taloscluster_types.go`, `pkg/render/render.go`, regenerate CRDs (`make manifests generate`) | L | 5, 4; edits the same CRD file as 7, so lands after 7 or rebases onto it |
| 7 | Unmanaged CNI API: enum value `unmanaged`, `spec.cni.unmanaged.talosCNI`, `status.cni.detected`, CEL rules (forbid `cilium`, `tunnel`; require the adoption annotation on create), webhook, condition `CARotationAdvised` constant, regenerate CRDs | `api/v1alpha1/taloscluster_types.go`, `internal/webhook/v1alpha1/taloscluster_webhook.go`, `config/crd/bases/*`, `dist/chart`, installer additive-check test | M | none (first to land among CRD edits) |
| 8 | Unmanaged render and controller guards: no flannel removal, no `cni.name` change, no inline entry, `injectKubeProxy` records the stored value; `CNIHandoverReconciler` early return; `cniHandoverPending` false; `cni` operations refused (webhook and executor); kube-proxy switch disabled; `CNIUnsupported` unchanged for flannel/custom; render byte-equality test | `pkg/render/networkinstall.go`, `pkg/render/render.go`, `internal/controller/cnihandover_controller.go`, `kubernetes_upgrade.go`, `cni_kubeproxy_switch.go`, `internal/webhook/v1alpha1/talosoperation_webhook.go`, tests | M | 7 (the constant only; can start on a stub) |
| 9 | Network-policy gate: `status.cni.detected` refresher, installer and chart skip Cilium policies unless Cilium is detected and the CRD exists, install-step input for adopt, UI/CLI notice | `internal/controller` (new `cni_detect.go`), `internal/installer`, `dist/chart` (`networkPolicy.cilium.enabled`), `config/cilium-network-policy` | M | 7 |
| 10 | Unmanaged UI/CLI: badge, "CNI not managed by Ichor" card and routes, Operations dialog, `CniService` and `ichor cni ...` refusals, `ichor cluster status` line, `ichor adopt --cni-unmanaged`, CA-rotation advice notice | `ui/` (cluster header, Cilium page, operations dialog), `internal/uiapi/cni_service.go`, `internal/cli/cni.go`, `cluster_status.go`, `adopt.go` | M | 7 |
| 11 | Three-class drift (A/B/C), consent list, release of the hold, the four phases, CA-rotation advice print | `internal/cli/adopt_plan.go`, `adopt_run.go`, `adopt_test.go` | M | 1, 2, 4, 7 |
| 12 | Write phase: create bundle Secret, `TalosCluster` (adoption annotation, preset), held `TalosMachine`s (authenticated `Identity`), Ichor install step | new `internal/cli/adopt_write.go`; `internal/controller/talosclient.go`, `talosclient_machinery.go` (an `Identity` read, fakes updated) | M | 3, 4, 7 |
| 13 | Reclaim core and CLI: `pkg/reclaim` (inspection, refusals, request builder, redacting credential type), `ichor machine reclaim` | new `pkg/reclaim/`, `internal/cli/machine_reclaim.go`; `internal/controller/removal.go` (export the request builder), `talosclient_machinery.go` (etcd members, volumes read) | M | none |
| 14 | Reclaim controller path: `MachineReclaim` CRD and types, reconciler (inspect, confirm, reset, wait, register), credential Secret lifecycle and sweeper, TTL, webhook with `SubjectAccessReview` for verb `reclaim`, RBAC and ClusterRole `ichor-machine-reclaimer`, `reclaim.enabled` chart value, audit Events and `status.audit`, no-credential-in-logs test | new `api/v1alpha1/machinereclaim_types.go`, `internal/controller/machinereclaim_controller.go`, `internal/webhook/v1alpha1/machinereclaim_webhook.go`, `config/rbac`, `config/crd`, `dist/chart`, regenerate (`kubebuilder create api`, `make manifests generate`) | L | 13 (the library; can start on its interface) |
| 15 | Reclaim UI: `ReclaimService` (create Secret and CR, set `confirmUUID`), page and dialog with inspection, orphaned/encrypted volumes, typed-UUID box, override banner, SAR-gated visibility; access-log body redaction | `proto/ichor/v1alpha1`, `internal/uiapi/reclaim_service.go`, `ui/` (machines page, dialog), logging interceptor | M | 14 (types only; can start on a stub) |
| 16 | Discovery `AlreadyConfigured` classification, add-by-IP response, UI message with a link to the reclaim page or the CLI command | `internal/controller/discovery.go`, `maintenanceclient_machinery.go`, `internal/uiapi/discovery_add.go`, `ui/` machine list message | S | none |
| 17 | Docs: architecture Bootstrap, Nodes, CNI and Trust (the reclaim exception) paragraphs, CLI help, notes in ADRs 0099/0100 (unmanaged preset; policy gate), ADR status; `make adr-index` | `docs/design/architecture.md`, `docs/adr/` | S | all |
| 18 | Hardware harness cases and cluster generators (below) | `test/e2e` | L | 1-16 as they land |

Parallelism: three tracks touch disjoint files and run concurrently. **Adopt track:** 1, 2, 3, 4, 5 start together; 6 starts on a stub of 5 and rebases onto 7; 11 and 12 follow 1/2/3/4/7. **Unmanaged track:** 7 first (small, only CRD and webhook), then 8, 9, 10 together. **Reclaim track:** 13 and 16 start immediately; 14 starts on 13's interface; 15 on 14's types. Units 6 and 7 both edit `taloscluster_types.go` and the generated CRDs and must not regenerate them in separate checkouts at the same time, so 7 lands first. Unit 18 starts with generators and the refusal cases immediately.

## Risks

- **Data-plane disturbance on takeover.** The pod template changes (checksums, image, args) and agents restart on every node; eBPF programs persist but a version jump or changed values can drop connections. Mitigated by the minor rule, the diff gate and the continuity measurement in test A4; the amount of loss is not yet measured.
- **Wrong diff semantics.** The live-only-fields conclusion comes from reading `createPatch`. If a later Helm version changes it, the diff gate still protects, but the gate itself needs live-object normalisation that can miss defaulted fields (false positives are safe, false negatives are not).
- **Helm half-states.** A killed Job can leave `pending-install` or a failed record; Upgrade then refuses ("another operation is in progress"). Needs an explicit repair path in unit 5.
- **Config applied too early.** If the hold is missing the controller would apply a render that differs from the live config. Unit 4 is a hard prerequisite of unit 8; a test must assert no apply call while held.
- **Contract probing picks a wrong contract** and hides drift as class C. It is shown, overridable, and class C blocks.
- **Reclaim wipes the wrong node or a live quorum.** Mitigated by typed UUID, cluster name on screen, the etcd checks and the Ichor-cluster refusal; the residual risk is a stale or copied talosconfig pointing at a different node than the operator believes, which the identity print is there to catch.
- **Deleting the foreign release record** removes a user's `helm rollback` path. Backed up to `ichor-system`; part of the consent.
- **Recovered bundle is incomplete** (for example, an etcd-encryption config in a non-standard shape: machinery states it extracts "our best" of the secretbox key). Caught by the zero-drift diff and by a Secret-decryption check in test B3.
- **Version floor v1.12**: the v1.12 legacy-form render path must be tested end to end (group D); a patch release below the floor in a mixed cluster is refused, which can surprise an operator mid-upgrade (the message says so).
- **Unmanaged mode hides a broken network.** Ichor does not manage the CNI and cannot fix it; it reports node readiness and `status.cni.detected` only. The UI says what Ichor does not do so that a CNI incident is not blamed on Ichor, and not the reverse.
- **Unmanaged render drifts from the node.** If the render emitted a CNI or kube-proxy setting the node does not have, the first apply would change a running network. Mitigated by the byte-equality test (unit 8), the hold, and class C for anything the typed fields cannot express (for example a Cilium or other inline manifest, or `cni.name: custom` with URLs).
- **Override credential exposure.** The foreign talosconfig sits in a Secret for up to the TTL. Mitigated by off-by-default, the explicit verb, scoping to one address, single use, TTL and sweeper, no logging, and encryption at rest being the cluster's; residual risk is a cluster-admin reading the Secret in that window, which is true of every Secret.
- **Reclaim UI path used on the wrong machine.** Same residual risk as the CLI (a stale talosconfig); the inspection shown before the typed UUID is the control, and the foreign cluster name sits next to the box.

## Open questions (still need the user)

Answered on 2026-10-29: non-Cilium CNI (unmanaged mode), reclaim (both paths), CA rotation (advise only), minimum version (Talos v1.12). Remaining:

1. Delete the foreign Helm release record after takeover (recommended, with a backup to `ichor-system`, part of the consent) or keep two histories? Applies to managed Cilium only.
2. Non-Cilium inline or external manifests in a node's config: refuse the machine as class C (this ADR) or add a patch kind for `KubeInlineManifestConfig` so they can be carried? This also decides whether an unmanaged cluster that carries its CNI as an inline manifest, or a hands-off Cilium installed that way, can be adopted at all.
3. A cluster with `cni.name: none` and no CNI: this ADR adopts it unmanaged (`talosCNI: none`). Should adopt also be able to install Cilium onto it?
4. Does `ichor adopt` install Ichor into the foreign cluster itself, or do we require `ichor install` first? This ADR assumes it does, as `init` does.
5. Should Ichor write standard Kubernetes `NetworkPolicy` objects for `ichor-system` when the CNI enforces them but is not Cilium (for example Calico)? This ADR skips policies there.
6. Reclaim UI path: default TTL 15 minutes, maximum 60, `reclaim.enabled` default off, 7-day retention of the `MachineReclaim` record. Are these the right numbers?

## As built (2026-10-29)

Units 1-16 landed with these differences from the text above. Where a section above disagrees with this list, this list is what the code does.

- **Reclaim registration is the UI API's write, not the controller's** (2.2, unit 14 note). The controller never writes `spec`, so `MachineReclaim.spec.clusterRef` names the target cluster and the UI API adds the address to that cluster's `spec.discovery.addresses` as the user. The CLI does the same with `--cluster`. Audit identity travels in the annotations `ichor.maesh.dev/reclaim-requested-by` and `-confirmed-by`, which the webhook insists equal the authenticated user, and is copied into `status.audit`.
- **KubePrism is a precondition in every managed mode** (1.6, 1.7, 2026-10-30; hardware B4). Adopt refuses up front, nothing written, when KubePrism is off (or not on 7445) on any machine and the adoption ends in an Ichor-managed Cilium (takeover, or a bootstrap on a no-CNI cluster), in `enabled` as well as `replaced` mode. The `ichor-cni` Job reaches the API through `localhost:7445`, and enabling KubePrism needs a reboot that adopt's hold release never orders, so the earlier "class B, Ichor will enable it" plan deadlocked: the Job failed and the holds were only released after it. KubePrism is therefore not a class B item; unmanaged adoption still leaves it as the node has it.
- **GitOps is out of scope** (ADR 0101), so the GitOps-tracking rule that earlier drafts had for the UI write path does not exist.
- **Reclaimer role.** `ichor-machine-reclaimer` grants `create`, `get`, `list`, `update` and the custom verb `reclaim` on `machinereclaims`. It is installed with `reclaim.enabled=true`, bound to nobody, and in no default role. The UI API creates the request first and the Secret after; a ValidatingAdmissionPolicy (`ichor-ui-reclaim-credential`) limits ichor-ui's Secret creates to the credential type, the `ichor.maesh.dev/reclaim` label and the key `talosconfig`. `CreateReclaim` checks get and update on the target cluster before writing anything, and the dialog registers with the cluster only when `CanReclaim` says the caller may.
- **Foreign Helm record (open question 1, decided: delete after backup).** `RetireForeignRecord` runs only when the operator consented (`RemoveForeignReleaseRecord`, set by adopt's consent; the plan states the deletion). Each record Secret is deleted only after the backup in `ichor-cilium-prior-release` is read back and found byte-identical, and only if it is still the Helm `cilium` record with the same UID. The Job gets `--takeover` for every Job of an adopted cluster until hand-over completes; its Role `ichor-cni-records-role` is get/list/delete on Secrets in `kube-system` only.
- **Flags of `ichor adopt`** (1.3, 1.8, 1.2): `--secrets-bundle FILE` or `--recover-secrets` is required, one of the two and not both (not "optional"); `--recover-from ADDRESS` is required with `--recover-secrets`; `--node ADDRESS` seeds discovery of a cluster that has no `TalosCluster` yet; `--plan` prints the plan and writes nothing (without it adopt shows the plan and asks for consent, which `--yes` skips; there is no "plan unless `--yes`" default); `--version-contract` overrides the probed contract; `--accept-cilium-diff` and `--accept-cilium-changes` are as in 1.2; `--context` and `--cni-unmanaged` as designed. **`--cilium-values FILE` (1.2, step 3) was not built.** A Cilium with no release record (case C) is taken over with Ichor's defaults and the diff gate shows what is missing.
- **Flags of `ichor machine reclaim`**: `--talosconfig`, `--context`, `--confirm UUID`, `--force` with `--confirm-cluster NAME` as the second typed confirmation, `--force-ungraceful`, `--cluster NAME` (registers the address in that cluster's discovery addresses), `--wait` (default 10m). The etcd health check judges the dialled member by its own etcd status and the others by whether their peer port answers.
- **`ichor init` has no switch for `reclaim.enabled`**; set it in the Helm chart values.
- **Hold.** The annotation is `ichor.maesh.dev/hold-apply`; a held machine reports `AdoptionHeld`. Release is one machine at a time, control planes first, each waiting for Ready without `AdoptionHeld`.
- **Inline entry.** Found by content (`cilium.FindInlineManifest`); adopt writes its name in the annotation `ichor.maesh.dev/cni-inline-manifest`, and the hand-over controller copies it to `status.cni.takenOverInlineManifest` in the same update that sets `CNIHandover=True` (unit 6 note in 1.2).
- **CA rotation.** The restart order after a rotation now puts the CNI first and an unmanaged cluster's CNI is not restarted (amendment in ADR 0063). A dismissal annotation or a completed rotation clears `CARotationAdvised`.
- **Help text.** The `Long` help of `ichor adopt` describes the four phases as built.
- **Class A on a fresh adopt (install, registry mirrors, cert SANs).** A talosctl-made cluster states an installer image and disk, usually a registry mirror and extra cert SANs. `ichor adopt` maps them instead of blocking every machine as class C, and on a fresh adopt it writes them itself, so the rerun plan has zero drift.
  - *Install.* `UnattendedInstallConfig` (v1.14) and `machine.install` (v1.12/v1.13) map to `TalosMachine.spec.install`: `disk` from `installDisk` or from a disk selector that is exactly one of the five equalities Ichor renders (wwid, serial, by-id symlink, bus path, device path; any other selector, such as a size range, has no field and stays class C), `image`, and `wipe`. `wipe` is a new additive field, default false. A live `wipe: true` is recorded, not ignored: the install-time-only classification would otherwise leave that machine's drift unapplied for ever, and Talos reads the flag only at install, so on a running node it does nothing. Extra kernel arguments, extensions and the bootloader options still have no field.
  - *Registry mirrors.* `RegistryMirrorConfig` documents and `machine.registries.mirrors` map to `TalosCluster.spec.registries` (host, endpoints), with two new additive fields, `skipFallback` and `overridePath` (one flag for all of a host's endpoints; a host with mixed per-endpoint values is not mapped). Every node must state the same mirrors, else nothing is mapped and the difference is class C. Render now emits the mirrors as `RegistryMirrorConfig` documents from contract v1.12, the form Talos's own generator uses there (below v1.12 it keeps `machine.registries.mirrors`); credentials stay in the legacy `machine.registries.config`, which Talos folds with the documents host by host. That changes the rendered bytes of an existing v1.12+ cluster that has `spec.registries`; both forms are hot-applied (fieldmeta classes them safe). The drift comparison (`controller.ChangedPaths`, used by the reconciler and by adopt alike) normalises both forms into one representation on both sides before diffing, so the same mirrors in either form are not drift and a node whose mirrors are in the legacy form (generated before v1.12, upgraded in place, or configured by an earlier Ichor build) is not class C and stays Ready after an Ichor upgrade; different endpoints, `skipFallback` or `overridePath` still are drift. The node moves to the document form only when it is applied for another reason. **Credentials and TLS are never mapped**: `RegistryAuthConfig`, `RegistryTLSConfig` and `machine.registries.config` stay class C, with the hint to create the TalosCluster first with `spec.registries[].authSecretRef`, so no password reaches a spec or a plan line.
  - *Cert SANs.* No spec field existed, and a TalosConfigPatch cannot carry them: `KubeAPIServerConfig` is not a patch document kind, and a `Config` patch must restate the secret `machine.token`. So `TalosCluster.spec.certSANs` is a new additive field, handed to the generator's `WithAdditionalSubjectAltNames`, which writes `machine.certSANs` and the kube-apiserver SANs (`KubeAPIServerConfig.certExtraSANs`, `cluster.apiServer.certSANs` below v1.14) from one list, as `talosctl gen config --additional-sans` does. It is mapped only when every node states the same `machine.certSANs` and every control plane the same kube-apiserver SANs; otherwise the paths stay class C. The render adds no SAN of its own (the endpoint host is not added).
  - *Hints.* Class C paths that share a hint are printed on one line, and the hints name `spec.install`, `spec.registries` and `spec.certSANs`. Tests: `internal/cli/adopt_class_a_test.go` with fixtures from `talosctl` v1.14.3 and v1.13.0 under `internal/cli/testdata/adopt-class-a/`.
- Open questions 2 to 6 are unchanged: non-Cilium inline manifests stay class C, `cni.name: none` is adopted unmanaged only, adopt installs Ichor itself, no standard `NetworkPolicy` objects are written, and the reclaim TTL numbers are as designed.

The user guide is `docs/guides/adopt-and-reclaim.md`. The design text is in the architecture's Bootstrap, Cilium, Nodes, CNI and Trust sections.

## Hardware test plan

**Status: group C run (lane 2, 2026-10-29/30; Talos v1.14.3, Cilium chart 1.18.0, Kubernetes v1.37.1).** Groups A, B, D are run, see "Results, groups A, B, D". Tests: `test/e2e/foreign/reclaim_test.go`, `reclaim_ui_test.go`, `reclaim_edges_test.go`; UI client `test/harness/reclaimui`.

- C1 passed: worker reset in 11 s, back in maintenance with the same UUID, pending machine registered, approved, Node Ready; 5m56s from reclaim to joined (the pending machine appears at the next discovery probe, about 5 minutes, when the address is already in `spec.discovery.addresses`).
- C2 passed: graceful reset of 1 of 3 control planes left 2 etcd members, healthy and consistent; the Kubernetes Node object stays (NotReady, cordoned) in the old cluster, so `talosctl health` fails on it. A control plane of the resulting 2-member cluster: `cluster "..." has 2 etcd members; removing one loses quorum; pass --force to override`, nothing sent.
- C3 passed: only control plane refused (`this is the last control plane of cluster "..."; pass --force to override`); `--force --force-ungraceful` without `--confirm-cluster` fails at the prompt, with it the node reset and returned to maintenance.
- C4 passed: Ichor's own control plane refused with `--force --confirm-cluster --force-ungraceful` and the right UUID (`is already an Ichor machine`); a wrong UUID prints `the typed UUID does not match the node's UUID; nothing was sent` and the node was unchanged.
- C5 passed: a plain user volume and a LUKS2 (node ID key) user volume plus the disk are listed, the encrypted one marked; partitions and LUKS still present after the join.
- C6 passed (2026-10-30): the kernel-argument wipe works on Talos v1.14.3 with a rebuilt boot asset; see "C6" below. C7 passed (2026-10-30): a stop and start of the node during the reset changes the pod IP and the CLI prints the DHCP hint; see "C7" below.
- C8 passed with ichor-ui users off (the caller is the ichor-ui ServiceAccount): reclaim off -> `failed_precondition`; no `reclaim` verb -> webhook `permission_denied`; with a RoleBinding to `ichor-ichor-machine-reclaimer` the request is held at `AwaitingConfirmation` with the inspection, a wrong UUID is rejected by the webhook, the right one reaches `Done` and the node is in maintenance. Enabling: webhook refuses `spec.reclaim.enabled=true` without `ichor.maesh.dev/confirm-reclaim=enable`; `ichor settings set reclaim.enabled=true` needs `--confirm-reclaim`. Not run: a signed-in user without the verb (users were off) and the UI page itself.
- C9 passed: key and certificate fragments and a comment marker from the talosconfig are in no controller or ichor-ui log, Event or `MachineReclaim`; the Secret is deleted at `Done`, `Failed` (unreachable node), `Expired` (ttl 1m) and with an early-deleted request (owner reference); a controller restart in `AwaitingConfirmation` loses nothing; a second confirm after `Done` produces no second `ReclaimReset`; an orphan credential Secret was deleted by the sweeper at 60 minutes. A talosconfig whose endpoint is another node still inspected only the dialled address.
- C10 passed: `status.audit` has requestedBy, confirmedBy, resetSentAt and credentialDeletedAt; the inspection has UUID and foreign cluster; after the Events expired (1 hour) these remain.
- Findings: the webhook message names `ichor-machine-reclaimer` but the chart installs `ichor-ichor-machine-reclaimer`; `status.audit` has no outcome or UUID of its own (phase and inspection hold them); `ichor init` gave up after 2 minutes without a pod network (now 10 minutes) and created the `IchorInstallation` without waiting for the webhook (now waits).

Run on the e2e harness (ADR 0007, 0102) within the homelab budget: control plane 2560Mi-3Gi, worker 1536Mi, at most 10 nodes. Build fixtures with `talosctl` in `test/harness`, not by Ichor, so the clusters are foreign. One run per integrated batch, in its own worktree; failing test names are recorded even when a rerun passes. Cases below are the minimum; the first of each group is 1 control plane plus 1 worker, 3 control planes only where stated.

**A. Cilium takeover** (one cluster per row)
- A1 Cilium in a Talos inline manifest, values from the Talos guide, same minor. Expect: diff gate empty or only listed items, takeover, inline entry removed, `CNIHandover` true, a Kubernetes patch upgrade does not roll Cilium back.
- A2 `helm install cilium` in `kube-system`, user values. Expect: values lifted, foreign record backed up and removed.
- A3 Cilium CLI install. Same expectations as A2 (release created by the CLI).
- A4 Data-plane continuity: continuous pod-to-pod and pod-to-Service traffic across A1 and A2; record loss and connection resets; the pass criterion is set after the first run.
- A5 Live-only residue: add an extra annotation, an extra arg and an extra ConfigMap before takeover; assert the diff gate reports the first two and that the extra object stays after takeover (verifies the source reading).
- A6 Older minor (1.17) is refused with no write; one lower than the embedded minor is accepted if the embedded chart is a minor above it; a newer minor is refused.
- A7 `ipam.mode: cluster-pool` and native-routing/tunnel variants are refused or mapped as specified.
- A8 Kill the Job after the update and before the record write; re-run; assert idempotent. Kill after a `pending-install` record; assert the repair path.

**B. Whole-cluster adoption**
- B1 Flannel cluster (Talos default) and a Calico cluster, without `--cni-unmanaged`: refused, asserted by identical machine-config hashes and no new objects in `ichor-system`. A cluster with Cilium plus a second CNI: refused even with the flag.
- B8 Talos older than v1.12 (one node, and a mixed cluster): refused with the upgrade message, nothing written.

**D. Unmanaged CNI** (one cluster per row, each also at Talos v1.12, v1.13 and v1.14 across the group)
- D1 Flannel cluster adopted with `--cni-unmanaged`: `preset: unmanaged`, hold released, first render applied; assert the CNI-relevant machine-config paths are byte-identical before and after, pods keep their connectivity, and no `ichor-cni` Job, `CNIHandover` or Cilium policy exists.
- D2 Calico cluster, same assertions, plus `status.cni.detected: calico` and the "policies not applied" notice.
- D3 Cilium cluster adopted unmanaged: no Helm record touched, policies applied (Cilium detected), `ichor cni values set` refused, a `cni` TalosOperation refused.
- D4 Talos upgrade, Kubernetes upgrade and CA rotation on a D1 cluster succeed; the Kubernetes upgrade is not blocked by a hand-over.
- D5 `cni.name: none` cluster adopted with `talosCNI: none`; render keeps `none`.
- D6 UI/CLI: the "CNI not managed by Ichor" card and the CLI messages; `preset: unmanaged` refused on create without the adoption annotation, and `ichor init` has no way to make one.
- D7 CA rotation advice: printed by adopt, shown in the UI, and no `TalosOperation` is created.
- B2 kube-proxy matrix: enabled with replacement off; disabled with replacement on; the two inconsistent combinations are refused.
- B3 Secrets: cluster created without a saved bundle; recover; zero-drift diff clean; a deliberately wrong bundle produces secret-field drift and no patch; the recovered bundle mints a certificate the nodes accept and decrypts a Secret created before adoption.
- B4 KubePrism disabled in kube-proxy-enabled and replaced modes.
- B5 Config generated at an older contract and upgraded in place: contract probing result and drift class counts.
- B6 Hold: assert no `ApplyConfiguration` while the annotation is set; assert control planes roll one at a time on release.
- B7 3 control planes, one worker, with a user inline manifest of a non-Cilium app: that machine is blocked class C, nothing applied.

**C. Reclaim**
- C1 Worker of another cluster: reset, reappears in maintenance with the same UUID, approve into an Ichor cluster, joins.
- C2 Control plane of a 3-member cluster (graceful: the old cluster's etcd has 2 healthy members afterwards) and of a 2-member cluster (refused).
- C3 Single-node cluster's only control plane: refused; with `--force` ungraceful path.
- C4 Machine already in the Ichor cluster: refused. Wrong typed UUID: nothing sent.
- C5 Node with a user volume and an encrypted volume: both listed; user disks survive, encrypted data is unreadable.
- C6 Without a credential: record whether the kernel-argument wipe works on this Talos version.
- C7 Address changes after the reboot: the hint, not a guess.
- C8 UI path: with `reclaim.enabled` off nothing can be created; a user without the `reclaim` verb gets no page and a webhook denial; with it, a worker is inspected, held at `AwaitingConfirmation`, a wrong UUID is rejected, the right one resets it, and it appears in maintenance.
- C9 Credential hygiene: a marker in the talosconfig is absent from controller and UI logs, Events, status, the `MachineReclaim` and error text; the Secret is deleted at `Done`, at `Failed`, at TTL expiry with the CR deleted early, and by the sweeper after the controller is restarted mid-request; a talosconfig listing other endpoints sends no call to them; the same request cannot reset twice.
- C10 Audit: `status.audit` and Events carry user, UUID, foreign cluster and outcome after the Events have expired.

### Results, groups A, B, D (2026-10-30; Talos v1.14.3, Cilium chart 1.18.0, Kubernetes v1.37.1)

- A1 passed after fix 4f9c033c (inline-manifest takeover). A2 passed: the foreign Helm record was backed up byte-identical and removed, 0 traffic failures. A4 continuity: 0 failures in 1950 and 1750 probes. A5 passed. A6: 1.16 refused, 1.17 accepted as an upgrade. A7: `cluster-pool` refused. A3 and A8: see "Results, A3, A8, D4, D6".
- B1, B2, B3, B5, B6, B7 passed. B4: `replaced` refused; `enabled` with KubePrism off is now refused up front (fix 3c9eca32, see "As built"). B8: v1.11.5 refused.
- D1, D2, D3 passed. D5 is replaced by ADR 0104 E1. D7 passed. D4 and D6: see "Results, A3, A8, D4, D6".
- Fixed after these runs: tunnel geneve takeover (be69d2a7); KubePrism precondition (3c9eca32). Fixed and not yet re-verified on hardware: adopt re-run after an interruption (pending-install repair 9ae973ca, bundle accept 2acd843b, `--tunnel` 908de05a).

### Results, A3, A8, D4, D6 (2026-10-30; Talos v1.14.3 on 1 control plane + 1 worker, Cilium chart 1.18.0, Kubernetes v1.37.0 or v1.37.1, build `v0.0.103-gb9e4cbb4`; tests `test/e2e/foreign/a8_test.go`, `d_test.go`, `TestA3Adopt` in `cilium_test.go`)

- **A3 passed** (Cilium CLI 0.19.7, now in `flake.nix` as `cilium-cli`; `cilium install --version 1.18.0 --wait=false --namespace kube-system` with the Talos-guide values, `tunnelProtocol=geneve`, `operator.replicas=1`, `debug.enabled=true`; it wrote one record `kube-system/sh.helm.release.v1.cilium.v1`, status deployed; cluster up in 8m16s). `adopt --plan` found the record, live 1.18.0 = chart 1.18.0, and the diff gate listed 21 differences (cilium-secrets Namespace finalizers and label, the `hubble-generate-certs` objects, three `cilium-config` keys, AppArmor annotations, `envoy-sockets` `readOnly`, ...): it required `--accept-cilium-diff`, and with it adopt wrote Ichor's objects and `CNIHandover` was True within seconds of the TalosCluster. Lifted `spec.cni.cilium.values`: `{"cluster":{"name":"ichor-e2e-fn"},"debug":{"enabled":true},"operator":{"replicas":1},"routingMode":"tunnel"}` (the CLI adds `cluster.name` and `routingMode`). The CLI's record was backed up byte-identical (599864 bytes base64) in `ichor-system/ichor-cilium-prior-release`, and no record Secret was left in `kube-system`; Ichor's own record `ichor-system/sh.helm.release.v1.cilium.v1` was deployed. The DaemonSet kept generation 1 and both pod UIDs (no restart). Traffic (A4 probes, service and pod IP, every 0.2 s): 5350 ok, 0 failures. Test time 515 s including the Ichor install, the takeover and 30 s of traffic after it.
  - `adopt --tunnel geneve` on this cluster is refused: `adopt: --tunnel only applies when adopt installs Cilium, which it does only on a cluster with no CNI`. A takeover keeps the live tunnel, so the harness passes no flag; the brief's "use tunnel geneve" was met by installing the CLI Cilium with `tunnelProtocol=geneve`.
- **A8 partly passed; two product gaps.** The test deletes the `ichor-cni` Job pod (in `kube-system`, not `ichor-system`; Ichor's release record is in `ichor-system`) as soon as Ichor's record exists, that is during Helm's rollout wait.
  - *Kill 1 (after the Helm update started, before the record write)*: the pod died; the Job failed at once (`BackoffLimit` is 0 by design, "one attempt", `newCNIHelmJob`), the TalosCluster showed `CNIHandover=False reason JobFailed` ("Cilium Helm Job ichor-cni-3dc4eb59b2 failed: BackoffLimitExceeded Job has reached the specified backoff limit"), Ichor's record stayed `pending-install` (revision 1), the foreign record was untouched, and the cluster kept running (nodes Ready, Cilium DaemonSet unchanged). **Gap 1: nothing retries.** The controller does not watch Jobs and does not requeue after `JobFailed`; deleting the failed Job started no new one, a new Job needed an event on the TalosCluster (I annotated it). The brief's "let it retry" is therefore a manual retry; the first test run waited 14 minutes for a retry and timed out.
  - *Retry with a `pending-install` record*: a new Job started 8 minutes after the kill failed in 4 s with `cilium: upgrading Helm release cilium: another operation (install/upgrade/rollback) is in progress` (twice). **Gap 2: the pending-* repair of 9ae973ca does not apply to the Job.** `repairPending` runs only when `ApplyOptions.Exclusive` is set or the record is older than the 10-minute wait timeout, and no caller sets `Exclusive` (the `ichor-cni` Job holds the CNI lock, so it could). The third retry, 10 min 20 s after the record was created, repaired the record and finished in 20 s (`Helm release kube-system/cilium upgraded: revision 2, status deployed, chart 1.18.0`, then `foreign Helm release records kube-system/sh.helm.release.v1.cilium.v1 backed up in ichor-system/ichor-cilium-prior-release and removed`). Final state: `CNIHandover` True; Ichor records `cilium.v1` superseded and `cilium.v2` deployed (one deployed, no pending-*); the backup Secret present; no record in `kube-system`; both nodes Ready. Byte-identity of that backup was not compared (the test run ended on its timeout first; the uninterrupted A2/A3 comparisons passed).
  - *Not run*: the second kill (Ichor's record deployed, foreign record still there); the window is under a second and the poller never hit it. The test is in place (`TestA8KillJob`, kill 2 is best effort).
- **D4 passed** on a D1 cluster (flannel 0.28.9, `--cni-unmanaged`, built with `--k8s v1.37.0`; D1 itself re-passed in 270 s). `ichor upgrade talos v1.14.4 --yes` (the image factory has v1.14.4 and v1.14.5; v1.14.4 is the next patch): the operation Completed in 7m56s, both nodes report server tag v1.14.4, nodes Ready. `ichor upgrade kubernetes v1.37.1 --yes`: Completed in 5m50s, both kubelets v1.37.1. `ichor cluster rotate-ca --yes`: the hand-off returned after 20 s (credential files written, "the new kubeconfig authenticates"), the operation Completed 5m24s later (about 5m45s in all); afterwards both nodes Ready, `/readyz` ok with the rotated kubeconfig, `CARotationAdvised=False/Rotated`, `BundleBackupStale=True/CARotated`. Throughout there was no `CNIHandover` condition and no `ichor-cni` Job, so the Kubernetes upgrade was not blocked by a hand-over; the flannel pods ran undisturbed.
- **D6 passed** (41 s): `ichor cluster status ichor-e2e-fn` prints `cni: unmanaged (flannel, not managed by Ichor)`; `ichor cni values get` exits 1 with `Error: the CNI is not managed by Ichor (spec.cni.preset: unmanaged)`; through `ichor ui --open-browser=false` the UI API `GetCluster` returns `"cniPreset":"unmanaged"`, `"cniDetected":"flannel"`, `"cniDetectedVersion":"0.28.9"` and the `NetworkPoliciesApplied=False/CNINotCilium` condition, while `GetCiliumValues`, `SetCiliumValues` and `StartCniUpgrade` each answer `failed_precondition: the CNI is not managed by Ichor (spec.cni.preset: unmanaged)` (client: `test/harness/cniui`). A copy of the TalosCluster with `preset: unmanaged` and without the adoption annotation is refused: `admission webhook "vtaloscluster-v1alpha1.kb.io" denied the request: spec.cni.preset "unmanaged" is only for clusters brought in by `ichor adopt` (ADR 0103): the object must carry the annotation ichor.maesh.dev/adopted="true"`. `ichor init --help` has no unmanaged option. The UI card itself was not looked at.
- **A8 rerun, 2026-10-30, passed** (`TestA8KillJob`, 384s; build `v0.0.1296-g7c5065a`, with 26da779b; cluster from `foreign up --cni cilium-cli --cilium-set operator.replicas=1,debug.enabled=true,tunnelProtocol=geneve`, up in 7m05s; takeover adopt without `--tunnel`, the diff gate needed `--accept-cilium-diff`, adopt took 6m21s). Kill 1 at 06:17:06 (pod `ichor-cni-0996e1b29a-cqvng`, log tail `helm: checking 25 resources for changes`, Ichor record `cilium.v1=pending-install`, 1 foreign record left). The Job then retried by itself: the Job object shown at the end was 3m18s old and ran 103s, i.e. a new Job started about 1m after the kill (the 1 min retry), and its log shows the repair at once, with no 10 min wait: `Helm release kube-system/cilium upgraded: revision 2, status deployed, chart 1.18.0` then `foreign Helm release records kube-system/sh.helm.release.v1.cilium.v1 backed up in ichor-system/ichor-cilium-prior-release and removed`. Final: Ichor records `cilium.v1 superseded; cilium.v2 deployed`, backup of the foreign record byte-identical (599656 bytes b64), no foreign record left, `CNIHandover` True, nodes Ready. Both gaps (no retry, no repair of a young pending record) are closed. Kill 2 (Ichor record deployed, foreign still there) was again not hit; the window is under a second. No failing tests in this run.
- Failing test names seen: `TestA3Adopt` (first run: `--tunnel` refusal above, a harness mistake, fixed), `TestA8KillJob` (first run: pod looked for in the wrong namespace so it was never killed; second run: timed out waiting for a retry that does not come, see Gap 1). Every other test passed on its first run.

Record, per case, the time taken, the Talos and Cilium versions and the exact refusal text, and write down what worked, not only conclusions.

## Consequences

- `adopt` stops being a patch-only step and becomes the four-phase adoption of 1.8; its read-only default plan is the first thing a user sees.
- Flannel and other-CNI clusters are adopted in the unmanaged mode; Ichor still does not migrate a CNI, and fresh `ichor init` stays Cilium-only.
- The CRD gains additive API only, and no field is removed: the `unmanaged` enum value and `spec.cni.unmanaged` on `TalosCluster`, `status.cni.detected` and the recorded inline-manifest name, and a new kind `MachineReclaim`. The hold is an annotation. `flannel` and `custom` stay refused.
- Ichor's Cilium network policies become conditional on a detected Cilium (ADR 0100 gains a gate), and a cluster without Cilium has no Ichor network isolation.
- The controller gains an override, off-by-default path that uses a user-supplied foreign credential for one reset; the architecture's "Trust" text records the exception.
- Adoption needs Talos v1.12 or later.
- `TalosClient` grows an authenticated `Identity` read; the reset request moves to a shared builder.
- ADR 0014's statement that adopt "never writes to a node" still holds: reclaim is a different command, and phase 4 is the controller's.

## Amendment (2026-10-29): ADR 0104

A cluster with no CNI is no longer adopted as unmanaged `talosCNI: none`: ADR 0104 has adopt install Cilium first and adopt it managed (the "No CNI installed" row of 1.1, test D5 and open question 3 are superseded), fixes the network-policy wiring found by D3, and adds the Helm install path.

### C6, credential-less wipe by kernel argument (2026-10-30): passed

The earlier note here (not feasible) was wrong about the harness: no kernel-boot is needed. The argument is baked into an Image Factory boot asset and the node boots that ISO.

- **Asset.** Schematic `customization.extraKernelArgs: [talos.experimental.wipe=system:EPHEMERAL,STATE]` posted to `https://factory.talos.dev/schematics` (the same service the harness already uses): id `1ba3a3c502bbb6fc012a1f4b9b6f252c7f5e77b9c7709cc1163e8405a01e1764`; ISO `https://factory.talos.dev/image/<id>/v1.14.3/metal-amd64.iso`, imported as its own disk (`ichor-e2e-fr-wipe-iso`, 3 min 22 s to Ready; the cached ISO disk was not touched). The factory accepted it and the ISO booted Talos v1.14.3.
- **node edit.** Stop and start gives the node a new pod IP (10.244.0.112 -> 10.244.0.87).
- **Wipe boot.** Configured foreign worker (UUID `05921e5a-a95f-4ef5-9792-348d4e6d593b`), ISO first. The Talos console showed `task unmountEphemeralPartition`, the STATE volume going `waiting -> closed`, then `error running phase 6 in initialize sequence: task 1/1: failed, failed to get volume status with label "EPHEMERAL"`, `META: loading from /dev/vda4`, `rebooting in 10 seconds`. It is a reboot loop, about 12 s per cycle, for as long as the wipe ISO stays first: the API never answers (connection refused for 3 minutes) and the node never reaches maintenance. The wipe argument is acted on every boot, so it must not stay in the boot order.
- **Plain ISO first is not enough.** With the plain ISO first again the node halts: `task haltIfInstalled: Talos is already installed to disk but booted from another media and talos.halt_if_installed kernel parameter is set. Please reboot from the disk.` (the boot partitions are still on the disk).
- **Disk first.** Boot order swapped (system disk 1, ISO 2): the node was in maintenance mode 27 s after the start (07:39:39 start, 07:40:06 `machinestatus` stage `maintenance`, new pod IP 10.244.0.208), same UUID `05921e5a-a95f-4ef5-9792-348d4e6d593b`. Discovered volumes: EFI, BIOS, BOOT, META (`talosmeta`) remain; the STATE (vda5) and EPHEMERAL (vda6) partitions are gone.
- **Join.** That maintenance node was then given the foreign worker config again (needed for C7), so the join was shown after the C7 reset of the same node (same disk state, STATE and EPHEMERAL gone): address added to `spec.discovery.addresses`, pending `TalosMachine` after 1 s, `ichor machine approve ... --role worker --install-disk /dev/vda --hostname ichor-e2e-fr-w0`, Node Ready 91 s after approve. The join straight from the C6 maintenance state was not run separately.
- **Meaning for the doc.** The kernel-argument wipe is a real credential-less route, but not a console edit: it needs a rebuilt boot asset (Image Factory schematic), booted once, and then the machine must be started from its disk (or the plain ISO after the install media's `halt_if_installed` is considered). Ichor does not offer it; the guide keeps naming it as an out-of-band route. `wipe=system:EPHEMERAL,STATE` removes the two partitions; `wipe=system` (whole disk, which would also let the plain ISO reach maintenance) was not run.

### C7, address changes after the reset (2026-10-30): passed (CLI); UI and controller path in the section after this one

On the e2e harness the guest reset keeps the pod IP, but stopping and starting the node gives it a new one, which is the same effect as a DHCP address change. Result after the 3 minute wait:

```text
Reset accepted (graceful: true). Waiting for 10.244.0.179 to return to maintenance mode.

waiting for 10.244.0.179 to return to maintenance mode: context deadline exceeded (last probe: rpc error: code = DeadlineExceeded desc = context deadline exceeded while waiting for connections to become ready)
The machine may have taken a new address from DHCP after the reboot. Find it (its UUID is 05921e5a-a95f-4ef5-9792-348d4e6d593b) and add it to a cluster's spec.discovery.addresses or cidrs.
Error: machine reclaim: the reset was sent but the machine did not come back at 10.244.0.179
```

Exit code non-zero, nothing registered, no guess at a new address. The reset itself had completed (the STATE and EPHEMERAL wipe finished in the 2 to 3 s before the halt). Adding the new address to `spec.discovery.addresses` produced the pending `TalosMachine` after 1 s.

- A first attempt did not trigger the hint: the node was stopped 17 s after `Reset accepted`, but by then the node had already come back in maintenance at the old address, and the CLI reported success. The reset-to-maintenance time on this harness is about 11 s, so the stop has to come within a few seconds.
- **Finding (product):** a second `ichor machine reclaim` of the same machine was refused (`node <UUID> is already an Ichor machine`) while the first run's pending `TalosMachine` for that UUID still existed (the machine had been registered as pending by `--cluster`, then given its foreign config again). Deleting that pending `TalosMachine` needs the `ichor.maesh.dev/allow-delete="true"` annotation. A pending (not approved) machine counts as "already an Ichor machine" in the check.
- The controller path (`MachineReclaim`, `waitForMaintenance`) fails the request with `... did not return to maintenance mode with UUID ... It may have taken a new address from DHCP: scan for the machine ...` after `reclaimMaintenanceWait`; it was not run on hardware here (no ichor-ui users or reclaim chart value in this Ichor cluster).

### C7 through ichor-ui and the controller (2026-10-30): passed for the outcome; the wipe was interrupted by the stop

Setup (prefix `ichor-e2e-y`, 3 nodes, no Go tests, all by hand): an `ichor init` cluster `ichor-e2e-y` (1 control plane, users off, build `v0.0.1840000001-g0f09c67a` after the 0105 upgrade), a foreign Talos cluster `ichor-e2e-yf` made with `talosctl gen config` (1 control plane `-cp4` bootstrapped, 1 worker `-w` joined; ADR 0103 section 1.1a style) and the ichor-ui NodePort 30080 (`test/harness/ui-demo/inner-nodeport.yaml`). `kubectl create rolebinding reclaim-y --clusterrole=ichor-machine-reclaimer --serviceaccount=ichor-system:ichor-ichor-ui` in `ichor-system`; `reclaimui can` answered `enabled: true, allowed: true, clusterAllowed: true`. Requests were made with `reclaimui create/confirm` against `https://<control plane>:30080` from the tools pod, with the foreign talosconfig and the worker's pod IP as the address.

Harness lessons (not product): (1) a worker alone does not work as the foreign machine: its apid gets its server certificate from the control plane's trustd, so without a reachable control plane `apid` rejects every client ("certificate required"); the foreign cluster needs its control plane node. (2) The foreign cluster needs `KubeNetworkConfig` pod and service subnets outside the pod network (10.244.0.0/16), otherwise Talos reports `address-overlap` and etcd never starts after `bootstrap` (this is what `test/internal/foreign` already patches); with v1.14 the subnets must be a `KubeNetworkConfig` document, the v1alpha1 `cluster.network` form is refused together with the default Flannel document.

Run A, address change (`MachineReclaim reclaim-7wnqc`, worker `10.244.0.165`, UUID `3580980a-4cde-4887-bebe-cb9455a2564e`):

- 01:44:28Z `ReclaimRequested`, 01:44:29Z `ReclaimInspected` (cluster `ichor-e2e-yf`), phase `AwaitingConfirmation`; confirmed through ichor-ui at 01:44:47Z (`ReclaimConfirmed`), `ReclaimReset` at 01:44:47Z (`STATE and EPHEMERAL, reboot`, graceful), `CredentialDeleted` (Secret `reclaim-5fz89ggb`) at 01:44:48Z, so the Secret is gone one second after the reset is sent, long before the outcome.
- The node came back **not** in maintenance but still running with the foreign configuration at the new address: the stop interrupted the graceful reset before the wipe took effect. So the wiping and the address change did not both happen in this run, unlike C7 through the CLI. A second run that stopped the node 0.4 s after the `ReclaimReset` Event (reset sent at 11:00:06 local, patch returned 0.1 s later, new IP `10.244.0.62`) had the same effect; I deleted that request after 4 minutes and did not wait for its end.
- Outcome of run A: phase `Failed` at 01:59:16Z with `status.result`: `10.244.0.165 did not return to maintenance mode with UUID 3580980a-4cde-4887-bebe-cb9455a2564e. It may have taken a new address from DHCP: scan for the machine and add its address to a cluster's spec.discovery.addresses`; a Warning Event `ReclaimFailed` with the same text; `status.audit` with `confirmedBy`, `requestedBy`, `resetSentAt`, `credentialDeletedAt` (`system:serviceaccount:ichor-system:ichor-ichor-ui`); the credential Secret absent; no `TalosMachine` other than the control plane. The failure came 14.5 minutes after the reset, not after `reclaimMaintenanceWait` (10 minutes); the 4.5 minutes of extra delay were not explained (the relay to the API server dropped for a short time around minute 10, but the controller runs inside the cluster).
- `spec.discovery.addresses` of the TalosCluster ended with all four addresses (`10.244.0.182`, `.165`, `.28`, `.62`): creating a reclaim with a cluster adds the request's address to the discovery list, including the addresses of requests that failed or were deleted. Nothing was registered from them (the failed request's addresses never answered in maintenance).

Run B, no address change (`reclaim-mvspt`, worker `10.244.0.62`, same UUID, created 02:06:44Z): `AwaitingConfirmation`, confirmed 02:06:51Z, `ReclaimReset` 02:06:52Z, `CredentialDeleted` 02:06:52Z, `ReclaimCompleted` 02:07:03Z (11 s after the reset), phase `Done`, `status.result`: `10.244.0.62 is in maintenance mode; cluster ichor-e2e-y discovers it and will show it as a pending machine`. The Secret `reclaim-c2x7zj9g` was gone, and an insecure (maintenance) `machinestatus` showed `stage: maintenance` on that address. About 90 s later no pending `TalosMachine` had appeared yet (the control plane had logged `AlreadyConfigured` for that address before the reset); I tore the cluster down before the next discovery pass and did not see it appear.

Not verified: a reset that completes and a changed address in the same run (the node stop kills the guest before the wipe is durable; stopping later than the wipe but before the reboot needs a stop timed inside a window of a few seconds I could not hit with `kubectl patch`), and a pending `TalosMachine` after Done.

## Hardware result 2026-10-30 (rerun at c549005e): C7 through ichor-ui, the controller deadline

Details in ADR 0107 ("Hardware result 2026-10-30 (build `v0.0.1850000002-gc549005e` ...)") and ADR 0105 (plaintext upgrade). For C7 through ichor-ui (foreign cluster with its control plane, Ichor controller reset deadline fix): a request whose machine never returned at the address (node stopped within a second of the reset) went `Failed` exactly 10 min 0 s after `status.audit.resetSentAt` (03:03:10Z to 03:13:10Z) with the DHCP hint in `status.result`; the 14.5 minutes of the earlier run are gone. The stop interrupts the graceful wipe (the node came back still foreign at the new address), so C7's address change plus wipe was again not shown through the controller path; through the CLI it was (above).

## Hardware result 2026-10-30 (C7 combined): the reset completes and the address changes, through ichor-ui and the controller

Passed. One request ends with STATE and EPHEMERAL wiped, the machine in maintenance mode with the same UUID at a new address, `Failed` with the DHCP hint 10 min 0 s after `resetSentAt`, and the machine then joined by adding the new address. Build `v0.0.1793331257-gf3fc27f3` (this tree), Talos v1.14.3, Kubernetes v1.37.1, prefix `ichor-e2e-q7`, 3 nodes, no Go tests, all by hand. The graceful reset was used; `forceUngraceful` was not needed.

**Setup.** Ichor cluster `ichor-e2e-q7` from `ichor init` (1 control plane, preset `single`, users off, Cilium geneve; the worker is the reclaimed machine, so no second Ichor node), `ichor-ui` NodePort 30080 (`test/harness/ui-demo/inner-nodeport.yaml`), `kubectl create rolebinding reclaim-q7 --clusterrole=ichor-machine-reclaimer --serviceaccount=ichor-system:ichor-ichor-ui` (`reclaimui can`: `enabled: true, allowed: true`). The tools pod ran static copies of `ichor`, `talosctl` (extracted from `ghcr.io/siderolabs/talosctl:v1.14.3` with `crane export`), `kubectl` and `reclaimui`. The request was made with `reclaimui create` (foreign talosconfig, worker address, `-cluster ichor-e2e-q7`) and confirmed with `reclaimui confirm` through ichor-ui.

**Why the earlier stops failed: the Talos API goes down before the wipe.** Attempt 1 polled port 50000 and killed the node 0.1 s after it stopped answering. A `talosctl dmesg -f` stream of the same reset shows why that is too early: the sequence stops `cri`, unmounts `STATE` and `EPHEMERAL` (`phase volumeFinalize (13/18)`, 03:55:28.7 to 29.1Z), then `phase stopEverything (14/18)` stops `apid` at 03:55:29.129Z, and the log stream ends there; the wipe (`reset`) and the reboot come after that, so the API at 50000 is down about 1 s before the wipe, not after. Apid down was 7.5 s after the confirm (03:55:22.6 to 03:55:30.0), the node was deleted 0.1 s later, and the node came back at the new address (`10.244.0.249`) still configured (`tls: certificate required` on the maintenance call, not in maintenance). The request ended `Failed` with the DHCP hint, with no wipe. This is the same outcome as the two earlier ichor-ui runs; the stop time of 7 s was never "after the wipe".

**What works: stop when the guest restarts.** The guest keeps answering ICMP until the kernel actually restarts, which happens after the wipe. A rehearsal with a direct `talosctl reset --reboot --graceful=true --system-labels-to-wipe STATE --system-labels-to-wipe EPHEMERAL` on the foreign worker (reset sent 04:02:51.8Z, `PINGDOWN` 04:02:56.0Z, node delete 04:02:56.1Z, new IP `10.244.0.206`) gave a node in maintenance mode with the same UUID and no STATE/EPHEMERAL; the foreign worker config was then applied again to it (running in 1 to 2 min, node `ichor-e2e-q7f-w` Ready with the new address).

**The run (attempt 2, `MachineReclaim reclaim-zsb9q`, worker `10.244.0.206`, UUID `892fccfb-2c9e-4f98-9480-5bd2cfaa08f2`, cluster `ichor-e2e-q7f`).**

- 04:08:35Z created through ichor-ui, `AwaitingConfirmation` after inspection (UUID, hostname `ichor-e2e-q7f-w`, role `worker`, cluster `ichor-e2e-q7f`). Confirmed at 04:08:42.97Z (`ReclaimConfirmed`, `ReclaimReset`: `reset sent to 10.244.0.206 (STATE and EPHEMERAL, reboot), graceful: true`, `CredentialDeleted` Secret `reclaim-jqqhxpcx`, all within 04:08:42 to 04:08:43Z).
- `PINGDOWN` at 04:08:47.150Z (4.2 s after the confirm, after 59 pings); launcher pod force-deleted, call returned at 04:08:47.244Z (0.09 s later). New pod IP `10.244.0.233`; the node was in maintenance mode at 04:09:05Z (18 s after the kill).
- Evidence on the node (insecure maintenance API at `10.244.0.233`): `machinestatus` stage `maintenance`, ready true; `systeminformation` UUID `892fccfb-2c9e-4f98-9480-5bd2cfaa08f2` (same); discovered volumes: EFI (vda1), BIOS (vda2), BOOT (vda3), META (vda4) and the install ISO; **no STATE and no EPHEMERAL partition** (as in C6).
- The request stayed `WaitingForMaintenance` and ended `Failed` at 04:18:43Z, i.e. `resetSentAt` (04:08:42Z) plus 10 min 0 s, with `status.result`: `10.244.0.206 did not return to maintenance mode with UUID 892fccfb-2c9e-4f98-9480-5bd2cfaa08f2. It may have taken a new address from DHCP: scan for the machine and add its address to a cluster's spec.discovery.addresses` and a Warning Event `ReclaimFailed` with the same text. `status.audit`: `requestedBy` and `confirmedBy` `system:serviceaccount:ichor-system:ichor-ichor-ui`, `confirmedAt` 04:08:42Z, `resetSentAt` 04:08:42Z, `credentialDeletedAt` 04:08:43Z. The Secret was gone from `ichor-system`, and the only `TalosMachine` was the control plane: nothing was registered.
- Closing the loop: `spec.discovery.addresses` got `10.244.0.233` added by hand at 04:18:58.44Z; the pending `TalosMachine` named by the UUID existed 0.19 s later (`APPROVAL pending`, endpoint `10.244.0.233:50000`); `ichor machine approve 892fccfb-... --role worker --install-disk /dev/vda --hostname ichor-e2e-q7-w --yes --local` at 04:19:03Z; the node `ichor-e2e-q7-w` was Ready about 70 to 90 s later (`get nodes` at 04:20:30Z showed it 68 s old, Ready; `TalosMachine` `approved`, READY True).

**Attempt 1 request** (`reclaim-dnvjz`, worker `10.244.0.251`, confirmed 03:55:22Z, apid down 03:55:30.0Z, node deleted 03:55:30.1Z, new IP `10.244.0.249`): `Failed` at `resetSentAt` plus 10 min with the same hint, but the node was not wiped (see above). Counted as one honest attempt; the second hit the window.

**Findings.** (1) Nothing in the product changed or was found broken: controller deadline, hint, audit, credential deletion and the join all behave as designed. (2) For tests, "the Talos API stopped answering" is not "the wipe is done"; "the guest stopped answering ICMP" is, and a hard kill at that moment leaves the disk wiped and the address changed. (3) `spec.discovery.addresses` of the cluster contained only `10.244.0.206` after the run (the earlier request's `.251` was not in the list), then the hand-added `.233`; the list behaviour from the first ichor-ui run (all request addresses kept) was not seen again. (4) No failing Go tests (none were run).

**Addendum (ADR 0108).** A machine written by `ichor init` or `ichor adopt` at an address outside `spec.discovery` needs the user to hold `update` on `talosmachines/endpoint` as well as `approve` on `talosmachines/approval` (both in `ichor-admin`); add by IP and reclaim already put the address in `spec.discovery.addresses` first. `MachineReclaim.spec.address` may not be loopback, link-local, unspecified or multicast.

## Amendment 2026-11-03: the foreign cluster starts on Cilium 1.19.10 and Kubernetes 1.34.12

Decided by the user (2026-11-03). Adopt accepts a live Cilium of the embedded minor or one lower (ADR 0113), and refuses 1.18 now that the embedded chart is 1.20.4. The foreign-cluster cases therefore install a one-minor-older Cilium and take it over. Talos is v1.14.5 (the release set's; its ISO is the cached `ichor-e2e-talos-v1-14-5-37656798-base`).

**What changed (commits on `worktree-agent-aec301966d7d2b29e`):**

- `make foreign-cilium-chart` fetches `pkg/cni/cilium/chart/cilium-1.19.10.tgz` (latest 1.19 patch, per helm.cilium.io/index.yaml) and checks its SHA-256 against the index digest `b7252afc…`. `cilium-1.18.0.tgz` is removed; nothing embeds or installs it (`go:embed` names 1.20.4 only).
- `test/internal/foreign`: `foreignCiliumVersion` 1.19.10 (the `cilium-cli` variant's `--version` and `CiliumChartPath`); `foreignK8sVersion` v1.34.12; `cachedVersion` v1.14.5.
- The Kubernetes start has to be a minor below the release set for the upgrade cases. `ichor upgrade kubernetes` targets only the pinned v1.36.6 (`internal/cli/upgrade.go`) and the executor refuses a skipped minor (`pkg/upgrade/kubernetes_skew.go`). So TestA1Upgrade and TestD4K8s start on v1.35.10 (`foreignK8sUpgradeStart`), not on the default v1.34.12. Both are unchanged in what they assert.
- `TestA6Accepted117` is `TestA6Accepted119`: one minor below the embedded 1.20.4, taken over and upgraded to `cilium:v1.20.4`. `TestA6Refused` keeps the 1.16 chart (more than one minor behind, refused). The probe images in `adr0104_e2`/`adr0104_e6b` are `cilium:v1.19.10`.

**Compatibility, corrected.** The brief gave Cilium 1.19 as tested with Kubernetes 1.31 to 1.34. Cilium's own `Documentation/network/kubernetes/compatibility.rst` at tag v1.19.10 lists **1.32, 1.33, 1.34, 1.35**. 1.34 is in that list and in Talos v1.14.5's 1.33 to 1.37, so the default stands. 1.35 is also in both, which is why the upgrade cases can start there.

**Hardware runs (2026-11-03; images `v0.0.1310-g257e46e2` pushed to a foreign registry on localhost:5024; Talos v1.14.5; Kubernetes v1.34.12; 1 control plane + 1 worker; `--cilium-set tunnelProtocol=geneve,operator.replicas=1`; prefix `ichor-e2e-fk`).**

- `TestA6Accepted119` (chart 1.19.10 installed with `cilium-helm`, takeover adopt): **failed** after 698 s. The plan and the diff gate ran; `CNIHandover` stayed `False` with `JobFailed` for 10 minutes (`adopt: the Cilium takeover (CNIHandover) did not finish within 10m0s: not finished; the machines stay held`). Cilium DaemonSet 2/2 on 1.19.10 throughout.
- `TestA1Adopt` (chart 1.19.10 as an inline manifest, managed takeover): **failed** after 704 s, same message. The first adopt refused with "114 differences between the live objects and the chart" (`--accept-cilium-diff` needed); the rerun wrote nothing further and the hand-over never finished.
- **Root cause, the same in both (product bug, not a harness bug).** The Cilium Helm Job (`ichor-cni`, `kube-system`, ADR 0099) fails in Helm's upgrade: `failed to create resource: roles.rbac.authorization.k8s.io "hubble-generate-certs" is forbidden … attempting to grant RBAC permissions not currently held: {APIGroups:[""], Resources:["secrets"], Verbs:["create"]}` plus `update` on the hubble and `cilium-ca` secrets. The chart's Role for `hubble-generate-certs` grants those secret verbs; the Job's ServiceAccount holds none in kube-system (`config/rbac/ichor-cni/role.yaml`, `records_role.yaml`: "The Job needs no create, update or patch on Secrets here"). Helm's escalation check refuses to create that Role, the atomic rollback then has no prior release, and each Job retry fails the same way (ten failed revisions in `ichor-system`, the Job's BackoffLimit exhausted three times). The 1.18 takeovers passed on hardware, so the 1.19 chart renders a Role the Job cannot grant. The fix (a Job RBAC rule, or a chart change) is a security decision (ADR 0108) and was not made in this run.
- Stopped after two identical failures, per the brief. Not run: `TestA6Refused`, `TestA3Adopt`, `TestA1Upgrade`, `TestD4K8s`, and the rest of groups A to D. Teardown verified: no `ichor-e2e-fk` node, disk, pod, Service or Deployment remains; the registry port-forward is stopped; the registry Deployment and Service were removed by `foreign down`. The nodes `ichor-e2e-rs2-*` and `ichor-ui-demo5-*` were not touched.

### Amendment 2 (2026-11-03): the RBAC fix and the reruns

**The fix.** `a36b0f5b` gives the Cilium Job's ServiceAccount, in kube-system only, the secret rights of the chart's `hubble-generate-certs` Role, rule for rule from the embedded chart: `create` on secrets (not name-scoped), `update` on the five hubble secrets, and `get` and `update` on `cilium-ca`. No `escalate` verb. It is `config/rbac/ichor-cni/hubble_role.yaml` (with its RoleBinding), mirrored in `dist/chart/templates/rbac/ichor-cni.yaml`; `config/installer` puts the Role in kube-system and the installer's bind/escalate names list it (ADR 0108). `TestCNIRoleHoldsEveryRuleTheCiliumChartGrants` (`dist/chart/admission_hardening_test.go`) fails when the Job does not hold a rule of that Role; removing the chart's create rule makes it fail.

**Why 1.18 takeovers passed and 1.20 fails: not explained by the Role.** The `hubble-generate-certs` rules are the same in the 1.18.0, 1.19.10 and 1.20.4 charts: diffing `templates/hubble/tls-cronjob/role.yaml` in 1.18.0, 1.19.10 and 1.20.4 shows only a `helm.sh/chart` label added in 1.20.4. The Job's ServiceAccount held none of these rules before the fix either, so an earlier 1.18 pass must come from the live state: whether the Role already existed in kube-system when Helm ran (Helm's escalation check applies to a create or an update, and an unchanged existing object needs neither). The 1.18 runs did not record that state, so the cause is unknown here.

**Not covered by the fix, and not enforced by the test.** The same class of gap exists for other Cilium Roles in kube-system (for example `cilium-config-agent` needs `configmaps:watch`, `cilium-operator-ztunnel` needs `serviceaccounts:watch`, and the `cilium` and `cilium-operator` Roles hold many rules the Job does not hold) and for the `cilium` ClusterRole. A takeover does not create or change those objects when they already exist, so they did not fail on hardware; a fresh install, or a chart bump that changes them, would. Covering them is an RBAC decision for the user (it widens the Job in kube-system), so the test is scoped to the Role that failed.

**Reruns on hardware** (images `v0.0.1311-ga36b0f5b`, prefix `ichor-e2e-fk`, Talos v1.14.5, one control plane, plus one worker where the case needs it; the foreign registry on port 5024 was torn down at the end):

- **TestA6Accepted119 passed** (248 s): chart 1.19.10 taken over, `cilium:v1.20.4` afterwards. The diff gate refused first with 114 differences as designed, then the retry with `--accept-cilium-diff` passed.
- **TestA1Adopt passed** (327 s, inline manifest, 1.19.10 taken over).
- **TestA6Refused passed** (18 s): Cilium 1.16.19 refused with "more than one minor behind" and nothing written.
- **TestA3Adopt passed** (357 s): a Cilium 1.19.10 record written by the Cilium CLI taken over.
- **TestA1Upgrade failed** (887 s; cluster at v1.35.10 with Cilium 1.20.4 after the takeover, then `ichor upgrade kubernetes v1.36.6`). Three findings:
  1. The gate refused the upgrade: `cilium 1.19.10 is tested with kubernetes 1.31 to 1.34`. The status `status.cni.detectedVersion` still said 1.19.10 although the DaemonSet ran 1.20.4; the CNI detection refreshes every 10 minutes (`internal/controller/cni_detect.go`). The test now waits for the status to show 1.20.4 (`fd0de1e9`). The refusal's advice, "upgrade Cilium to 1.20.4 first", was wrong for that state.
  2. After the wait the operation started but stayed `InProgress` at stage `Kubelets` with the single control plane still on v1.35.10: condition `RemovedAPIsInUse` is `CheckFailed`, "fetching an admin kubeconfig from the Talos API: kubeconfig is only available on control plane nodes". The check asks a worker's Talos API for the kubeconfig. **Product bug.**
  3. The test's 8-minute deadline passed before the upgrade finished; the Cilium DaemonSet pods changed UIDs across the upgrade (the test asserts they do not), and the pod-IP probe leg failed 41 times.
- **TestD4K8s failed** (9 s; flannel cluster on v1.35.10, prerequisite TestD1FlannelAdoptedUnmanaged passed in 256 s): the gate refused `cilium 0.28.9 is not known`. `installedCilium` (`internal/uiapi/operation_service.go`) returns `status.cni.detectedVersion` whether or not the detected CNI is Cilium, so flannel's version 0.28.9 is read as Cilium's. **Product bug.** This is a different root cause from TestA1Upgrade, so the two-identical-failures rule did not stop the run.

**Compatibility sources disagree.** Cilium's `Documentation/network/kubernetes/requirements.rst` at tag v1.19.10 lists Kubernetes 1.31 to 1.34, which is what `pkg/release/compat.go` encodes; `compatibility.rst` at the same tag lists 1.32 to 1.35 (see the first amendment). 1.34 is in both. Not resolved here.

**Teardown.** Every `ichor-e2e-fk` node, disk, pod, Service and Deployment is deleted, the registry port-forward is stopped and the registry is removed. The `ichor-e2e-rs2-*` and `ichor-ui-demo5-*` objects were not touched.

**Open for the user.** (1) The RemovedAPIsInUse kubeconfig source and the upgrade gate's CNI-kind and stale-status handling are product fixes (not made here). (2) Whether the Cilium Job may hold the rules of the other kube-system Cilium Roles (the gap list above). (3) The Cilium 1.19 compatibility range (1.31 to 1.34 in `requirements.rst` and the table, 1.32 to 1.35 in `compatibility.rst`).

## Amendment 3 (2026-11-03): removed-API check, Cilium applicability, stale detection, and the RBAC change that was reverted

The foreign hardware run of amendment 2 found four product bugs. Each fix was made test first (a spec or unit test that failed before the change) and committed separately.

1. **RemovedAPIsInUse on a worker (132d2bdf).** The check dialed the node being upgraded and a worker refuses the admin kubeconfig, so the operation stayed InProgress at Kubelets. It now asks every control plane and reports the union; a control plane that cannot answer makes the check Unknown. Spec `asks every control plane, never the worker being upgraded, for removed APIs` (failed before with the worker's client, passed after).
2. **Cilium applicability (b2578fbb, ADR 0115 amendment 3).** The Cilium compatibility rule is applied only when the observed CNI is Cilium. A flannel cluster was refused with `cilium 0.28.9 is not known`. Specs: the uiapi plan (`TestPlanKubernetesSkipsTheCiliumConstraintOfAnUnmanagedFlannel`) and the webhook update (`TestTalosClusterDoesNotCheckCiliumOfAnUnmanagedFlannel`) failed before with that exact refusal; the release (`TestKubernetesChoicesSkipTheCiliumConstraintOfANonCiliumCNI`) and adopt (`TestCheckAdoptCompatJudgesNoCiliumOfAnUnmanagedAdopt`) specs were compile-red against the old signatures.
3. **Stale detection (e1d4e998).** The CNI detection also runs when CNIHandover turns True or a cni operation completes (`cniDetectTrigger`); the interval stays ten minutes. The e2e wait for the stale version was removed. Unit spec `TestCNIDetectRefreshesWhenAHandoverOrACNIUpgradeCompletes` passes; the live wiring is covered only by the hardware run below.
4. **RBAC by name (82f769b3, reverted in 88b0cefa).** Dropping the hubble-generate-certs mirror and granting bind and escalate on the Cilium Role and ClusterRole names failed on hardware: the Helm Job logged `roles.rbac.authorization.k8s.io "hubble-generate-certs" is forbidden: user "system:serviceaccount:kube-system:ichor-cni" ... attempting to grant RBAC permissions not currently held`. Escalation by resourceNames does not cover creating the Role, and create on Secrets cannot be name-scoped. The mirror is restored; the name-restricted rules and the two-way check of the names (`TestCNIEscalationNamesOnlyCiliumsRoles`, now checking rendered names both ways) are kept. ADR 0108 records this.
5. **Cilium 1.19 Kubernetes range (9bce2bd0, ADR 0115 amendment 3).** From the CI matrices at the tags: 1.18.16 runs 1.30 to 1.33, 1.19.10 runs 1.32 to 1.35, 1.20.4 runs 1.33 to 1.36. The table had 1.31 to 1.34 for 1.19.

Verification (commits 88b0cefa and 9fcf8a4d): `go build ./...`, `go vet ./...`, `go vet -tags e2e ./test/...`, `make build-installer`, and `bin/golangci-lint run` on the touched packages (0 issues, built inside `nix develop`). Under the gate lock: controller suite ok (240 s); uiapi, webhook, cli, pkg/release ok; installer ok (envtest assets from bin/k8s); dist/chart ok. Tests that failed before a fix were recorded by name above; no other test failed in these runs.

**Hardware (prefix `ichor-e2e-fk2`, Talos v1.14.5, registry on localhost:5034, one control plane and one worker, Kubernetes v1.34.12 unless a case says otherwise):**

- **TestA6Accepted119 at 9fcf8a4d (images v0.0.1400-g9fcf8a4d): FAILED** after 1274 s, and again with the cluster kept (1066 s). CNIHandover stayed False; the Helm Job `ichor-cni` failed on the `hubble-generate-certs` Role create (the RBAC message above). This was fix 4. Stopped after two identical failures.
- **TestA6Accepted119 at 88b0cefa (images v0.0.1401-g88b0cefa): PASSED** (799.7 s). Chart 1.19.10 taken over, the diff gate refused first as designed, the retry passed.
- **TestA1Adopt at 88b0cefa (cluster kept): PASSED** (634.9 s), default Kubernetes v1.34.12.
- **TestA1Upgrade: NOT PASSED, test setup problem.** Run alone it has no takeover, so it failed on a missing TalosCluster. Run attached to the kept v1.34.12 cluster, it reached the upgrade with `status.cni.detectedVersion` already v1.20.4 (fix 3 observed: the status refreshed at the hand-over, with no wait), and the upgrade was refused with `version skew policy allows at most one minor version at a time: v1.34.12 to v1.36.6`. TestA1Adopt builds the default Kubernetes, not the v1.35.10 that TestA1Upgrade expects. Fix 1 (RemovedAPIsInUse) is not verified on hardware.
- **TestA1Adopt attached to a v1.35.10 cluster (harness up, kept): FAILED** after 904 s. The adopt was killed (`signal: killed`) while Ichor's webhook service refused connections (`ichor-webhook-service ... connect: connection refused`) for about 15 minutes. Not diagnosed; TestA1Upgrade was not run on it.
- **TestD1FlannelAdoptedUnmanaged and TestD4K8s: not run.** Fix 2 is verified by the uiapi and webhook specs only.

**Teardown:** every `ichor-e2e-fk2` node, disk, pod, Service, Deployment and the registry is deleted and checked; the registry port-forward is gone. The `ichor-e2e-rs2-*` and `ichor-ui-demo5-*` objects were not touched.

**Open for the user.** (1) The RBAC decision (ADR 0108): the Job needs the hubble mirror for the create; a name-restricted escalate cannot replace it. (2) TestA1Upgrade needs an adopt at v1.35.10, which the adopt test does not set; the test or its setup must change. (3) The adopt webhook connection refusal in the v1.35.10 attempt. (4) Hardware for TestA1Upgrade, TestD1 and TestD4 (fixes 1 and 2) is still to be run.

## Amendment 4 (2026-11-04): TestA1Upgrade standalone, the removed-API check and the flannel cases on hardware

**Test change (commit 6944c68d).** `TestA1Upgrade` builds its own cluster at v1.35.10 (`foreignK8sUpgradeStart`, a per-test option, no global default change), takes the inline Cilium over with the same adopt as `TestA1Adopt`, waits for `CNIHandover`, and only then upgrades. Its upgrade wait is 20 minutes, and a TalosOperation in `Failed` stops the wait. It no longer depends on `TestA1Adopt` having run first.

**Hardware (prefix `ichor-e2e-fk3`, Talos v1.14.5, image `v0.0.1420-g6944c68d`, registry on localhost:5044, relay on 5045, `--cilium-set` none: the inline manifest is the chart 1.19.10 with the Talos guide values; one control plane and one worker).**

- **TestA1Upgrade: PASSED** (836 s). The cluster came up at v1.35.10 with `cilium:v1.19.10` in the inline manifest. The adopt gate refused without `--accept-cilium-diff` (114 differences, as designed), the retry passed, and `CNIHandover` went True about two minutes after the hand-over started. The inline entry was absent from both nodes. Then `ichor upgrade kubernetes v1.36.6` completed: the operation was `Completed` with `RemovedAPIsInUse=False` (`NoneRequested`), so the check ran against the control plane without the CheckFailed that amendment 3 fixed. Both kubelets reported v1.36.6. The Cilium DaemonSet generation, image, pod UIDs and the Helm records were unchanged across the upgrade, which the test asserts. The client tally was `ok=236 fail=64`. There were 0 failures on the Service leg and 77 on the pod-IP leg, which is the leg that runs to the server pod on the control plane being upgraded. No failing test name.
- **Adopt on v1.35.10 webhook refusal (amendment 3, item 3): not reproduced.** In the single run above, the controller's webhook refused connections (`dial tcp ...:443: connect: connection refused`) from 23:29:48 to 23:30:09, about 21 seconds, not 15 minutes. The adopt then continued and finished. The controller pod (the first replica) had a readiness/liveness timeout in the kubelet probes during the upgrade, not during the takeover. The data collected: the endpoints of `ichor-webhook-service` are present once the controller is Ready. The CiliumNetworkPolicies were not dumped in this run, so the apiserver-to-webhook path after a hand-over is still unexamined. No product fix was made because no failure reproduced. The 15-minute refusal's cause remains open.
- **TestD1FlannelAdoptedUnmanaged: PASSED** (254 s) on a flannel cluster at v1.35.10 (attached). `adopt --cni-unmanaged` took the cluster with the CNI left as it was, and the holds were released.
- **TestD4K8s: PASSED** (304 s) on the same cluster. The Kubernetes upgrade to v1.36.6 took 5m04s and both kubelets reported v1.36.6. The CNI-kind fix holds: the gate did not judge flannel's 0.28.9 as Cilium (`cilium 0.28.9 is not known` did not appear), and no `ichor-cni` Job or Cilium policy was created.

**Harness pitfalls found on the way (not committed as code).** `ICHOR_E2E_F_ICHOR_BIN` is resolved against the repository root with `repoPath`. A value of `../../../bin/...` resolves outside the worktree to the main checkout's stale binary (no `adopt --node` flag), so the first attempt failed with `unknown flag: --node`. Use `bin/img/ichor/ichor`. The ichor relay on 127.0.0.1:5045 returned one `EOF` during the upgrade, and the next call worked.

**Verification.** `go vet -tags e2e ./test/...` passed. `golangci-lint` with `--build-tags e2e` reports 92 issues on `test/e2e/foreign` (goconst, lll, modernize, unparam, and one gofmt on `d_test.go`). None are on the lines this change touched; the baseline was already failing. `go build` was not needed (no non-test code changed).

**Teardown.** The Cilium cluster and the flannel cluster were each torn down with the harness; the registry was removed by the final `down`. Every `ichor-e2e-fk3` node, disk, pod, Service and Deployment is gone, and both port-forwards are stopped. The `ichor-e2e-rs2-*` and `ichor-ui-demo5-*` objects were not touched.

**Open for the user.** (1) The 15-minute webhook refusal from amendment 3 is not reproduced; the next time it appears, capture the controller's events, logs, and `CiliumNetworkPolicy` objects in `ichor-system` during the refusal window. (2) The golangci-lint baseline on `test/e2e/foreign` (92 issues) is pre-existing and would fail a lint run with the e2e tag.

## Amendment 5 (2026-11-04): the 77 and 239 pod-IP failures of TestA1Upgrade are the probe's fixed target, not the datapath

**Question.** Amendment 4's `TestA1Upgrade` run logged 77 pod-IP-leg failures and 0 Service-leg failures during the control-plane upgrade. Was it a datapath fault (Cilium on the control plane, drain or kube-proxy replacement), or the probe?

**Probe.** The client on the worker (`fa-client`) loops over the Service IP and over `POD_IP:8080`, where `POD_IP` is the server pod's IP captured once at `trafficStart`. The server is a one-replica Deployment that prefers the control plane. Nothing refreshes `POD_IP`.

**Hardware (prefix `ichor-e2e-fpip`, one control plane and one worker, Talos v1.14.5, Kubernetes v1.35.10 to v1.36.6, Cilium 1.19.10 inline, image `v0.0.1421-g04ef3f70`, registry on localhost:5084, relay on 5085).** `go test -run TestA1Upgrade` against the attached cluster, with per-snapshot logging added in `cilium_test.go` (`probeDiag`, `ciliumDrops`, `eventsTail`).

- **Result: PASS** (1390 s, one run, no failing test name). The client tally was `ok=234 fail=216` at 16:13:06. Failure lines: 239, all on the pod-IP leg, 0 on the Service leg. Amendment 4's 77 came from a different run and is not comparable: this run's probe ran longer after the eviction.
- **Timeline (UTC).** Worker `w0` was cordoned at 16:00:38 and its client moved to the control plane. The control plane `cp0` was cordoned at 16:01:52. At 16:01:51 the drain killed the server pod `fa-srv-…-qj5fm` (UID `3856440e…`, IP 10.248.0.197, on `cp0`). The Deployment created `fa-srv-…-qrnwq` (UID `931a819f…`) on `w0` with IP 10.248.1.188, and it was Ready at 16:02:17. The Service leg had an endpoint throughout. The pod-IP leg kept 10.248.0.197, which no longer existed, so its first failure was 16:02:21 and it failed until the end of the run.
- **Cilium.** Both agents kept `restarts=0` for the whole window, and the DaemonSet UIDs were unchanged (the test asserts this and it passed). `w0`'s agent was briefly `ready=false` once during the worker's cordon. The drop counters after the upgrade increased on `cilium-4ggbh` (`cp0`): `Policy denied` egress by 210 and `Stale or unroutable IP` ingress by 468. The counters are aggregates and do not name the flow, so Hubble or a live drop monitor is needed to tie them to the probe IP.
- **Drain.** `drainForReboot` uses `IgnoreAllDaemonSets: true`, so no Cilium pod was evicted, which matches the observed UIDs.

**Conclusion: hypothesis (a).** The pod-IP leg fails because its fixed target pod was evicted by the normal kubelet-roll drain and recreated on another node. This is test accounting, not a product fault. Hypotheses (b) and (c) are not supported: no agent restarted, the DaemonSet did not change, and no Service-leg failure occurred.

**Not changed in this amendment.** The test's pass criteria are the Cilium DaemonSet and Helm record assertions and the kubelet versions. None depends on the pod-IP failure count, so the test still passes. The count in the report is the misleading part. I did not add a per-probe liveness classifier. The follow-up is to make the pod-IP leg follow the server's current IP (or classify failures after the target pod's deletion timestamp) so the count means something. Placing the target on the worker does not avoid the problem, because the worker is drained too.

**Harness deviations.** The brief's prefix `ichor-e2e-pip` is not valid: `foreign.PrefixRoot` must start with `ichor-e2e-f`, so `ichor-e2e-fpip` was used. The first attempt on registry port 5064 and relay 5065 collided with another lane's port-forwards (`ichor-e2e-fwh`), so that attempt was torn down and its images went into that lane's registry. Both have been reported to the user.

**Teardown.** `foreign down --prefix ichor-e2e-fpip` removed the registry, relay, tools pod, both nodes and their disks. No `ichor-e2e-fpip` object remains in `ichor-e2e`. The `ichor-ui-demo5` nodes, `ichor-e2e-talos-v1-14-5-…-base` and the `fwh` port-forwards were not touched.

## Amendment 6 (2026-11-04): the A4 pod-IP leg follows the server's Ready endpoint

**Change (test only, no product change).** The follow-up to amendment 5. The client's pod-IP leg in `test/e2e/foreign/cilium_test.go` no longer probes a fixed IP. Before each pod-IP probe it reads the server's Endpoints from the API, using a read-only Role on the `fa-srv` endpoints. It takes the first address in `addresses` and ignores `notReadyAddresses`. With no Ready address it skips that round (`pskip`). When the address changes it logs a `RETARGET` line. It counts a pod-IP failure only when a Ready server pod was listed at probe time. The Service leg is unchanged. The tally classification is `ParsePodLeg` and `PodLeg.Check` in `test/internal/foreign/podleg.go`, unit-tested with `go test ./test/internal/foreign/...`.

**Pass criterion.** `TestA1Upgrade` asserts `PodLeg.Check(PodLegBound)` over the window since `trafficStart`, which covers the takeover and the upgrade: at least one successful pod-IP probe, and at most `PodLegBound = 10` pod-IP failures. Why 10: a failed probe takes up to the 2 s wget timeout plus the 0.2 s round, so a stale window of up to about 10 s yields at most 5 failures. The windows that can produce them are a dead IP still listed after its pod was deleted, and a Ready pod that Cilium has not programmed yet. The server moves once in this test (one drain), so two such windows fit. This is a judgement, and the first run's counts tune it.

**No hardware run for this change.** The next foreign run exercises it. The client's shell logic was checked locally against a fake `wget` (20 failed probes of a dead IP, 20 rounds with no Ready address, then a retarget to the new IP), and its COUNT line is pinned in the parser test. Two things stay unverified until that run: that busybox 1.37 `wget` reaches the API server over TLS with `--no-check-certificate` and the service-account token, and the Endpoints timing against Cilium. If the lookup never succeeds, the leg records no successful probe and `Check` fails, so a broken lookup cannot pass as a clean leg.

**Known gaps.** The tally lives in the client pod's log, so a client pod replaced during the window loses its earlier counts, as before. The report sums the pods present now, so the bound can under-count across a client move.

## Amendment 7 (2026-11-04): the webhook refusal during a takeover is the controller's start-up, not a policy

The 21-second and 15-minute refusals of amendments 3 and 4 were investigated on three fresh clusters: `ichor-e2e-fwh` (the harness rejects prefixes outside `ichor-e2e-f`, so `ichor-e2e-wh` was not usable), Talos v1.14.5, Kubernetes v1.35.10, one control plane and one worker, inline Cilium 1.19.10 from the chart's Talos-guide values, `--registry-mirror` on localhost:5064, image `v0.0.1430-g04ef3f70`. Each run is `TestWHAdopt` (`test/e2e/foreign/wh_test.go`, commit c3d15eff): the takeover of TestA1Upgrade with a sampler alongside, every 5 s, of the controller pods (phase, ready, restarts, waiting reason), the `ichor-webhook-service` endpoints, the CiliumNetworkPolicies, the Cilium agents, the `cilium-dbg monitor --type drop` lines for 9443 and 8081 of each agent, and (when the controller is not Ready) the `ichor-system` events. The log is `<cluster dir>/wh.log`.

**Results (all three PASSED as tests; the adopt finished in each):**

| run | refusal (`connection refused` from the apiserver) | controller pod | policies applied | adopt |
|---|---|---|---|---|
| 1 | 00:39:45 to about 00:40:15, 30 s | created 00:39:44, Ready about 00:40:15 | 00:40:18, after Ready | passed (295 s) |
| 2 | 00:57:47 to 00:58:20, 33 s | created 00:57:46, Ready about 00:58:16 | 00:58:20, after Ready | the test's 580 s limit ended the run during the wait for CNIHandover; CNIHandover became True at 16:06:12Z, about 8 min after the TalosCluster was written (15:58:20Z) |
| 3 | 01:18:21 to 01:18:52, 31 s | created 01:18:20, Ready about 01:18:51 | 01:18:52, after Ready | passed (301 s) |

**Root cause of the window.** The install manifest (99 objects, including the `ValidatingWebhookConfiguration`) is applied before the controller pod exists. The apiserver then calls `ichor-webhook-service` for the `IchorInstallation` write, and the Service has no ready endpoints until the controller is Ready. A Service with no endpoints is answered with a reject, which the apiserver reports as `connect: connection refused` rather than a timeout (a policy drop would time out). That the reject comes from the Service path is inferred from the error text and the empty endpoints, not checked in `cilium-dbg`. The sampler shows the controller in `ContainerCreating` while the refusal lasts, and in run 3 the controller's events show `Pulled ... in 14.975s` for `ghcr.io/maeshinshin/ichor-controller:v0.0.1430-g04ef3f70` through the in-cluster registry mirror. Readiness then needs the container start, the 5 s initial readiness delay and the webhook server (`cmd/main.go` gates `webhook` readiness on the server starting). The window is therefore the controller's start-up: image pull, start, and the first readiness probe. Its length (30 to 33 s in three runs) is the pull time plus up to one readiness period.

**Hypotheses checked.**
- (a) Policy. Not the cause of the window. No CiliumNetworkPolicy existed in `ichor-system` during any refusal: `init` applies the four policies only after the controller is Ready (ADR 0100 order), and the sampler's policy list is empty until then. The drop monitors of runs 1 and 3 saw no 9443 or 8081 drop (run 2's sampler log was overwritten by run 3 and was not checked for drops). After the policies were applied, the webhook admitted every write (`TalosCluster`, both `TalosMachine` objects and the `Secret`) with the controller on the control plane's node, which is the apiserver's host-network node; the remote-node case (controller on the worker) was not exercised here.
- (b) Controller not Ready. Confirmed as the mechanism: the endpoints are empty from the pod's creation until it is Ready, then populated (the sampler's `webhook` column shows `ready=` at the same tick the pod turns Ready).
- (c) Certificate or CA timing. Not observed. The readiness gate on the webhook server did not hold the pod past its start-up in any run.

**The 15-minute refusal is not reproduced.** Three runs gave 30 to 33 s. The same mechanism would produce a long window only if the controller pod stayed not Ready for minutes, for example an image pull that backed off or a pod that could not get a sandbox, and the adopt has no bound on its wait for the webhook (it waits until the harness context ends, 14 minutes). The amendment 3 run was on a cluster kept from an earlier run, and its pod events were not captured, so which of these it was is unknown. The sampler now records the events while the controller is not Ready, so the next occurrence will show the waiting reason.

**Variance of the takeover.** The CNIHandover took about 2 minutes in amendment 4's run and about 8 minutes in run 2 (TalosCluster written 15:58:20Z, True at 16:06:12Z); runs 1 and 3 finished the whole adopt within their 5-minute test runs, so their handover time was not read. Run 2's Cilium DaemonSet was replaced with the chart 1.20.4 image on fresh nodes, so an image pull through the mirror is the likely cost; not measured.

**Product change.** None. The adopt's behaviour (retry the write until the webhook answers) is correct, and the window it covers is the controller's own start-up. A bounded wait would not have changed the outcome of any run here. A bound that fails the adopt after the controller's start-up time could be added if the 15-minute case recurs, with the controller's waiting reason in the message; that is deferred until the events are in hand.

**Test.** `TestWHAdopt` is kept as the diagnostic; run it attached (`ICHOR_E2E_F_ATTACH=1`, `ICHOR_E2E_F_KEEP=1`) to a cluster `foreign up --cps 1 --workers 1 --cni cilium-inline --talos v1.14.5 --k8s v1.35.10`. Its 580-second Bash-call limit can end a slow takeover before CNIHandover (run 2); the log is complete up to that point. `go vet -tags e2e` passes on the package; the golangci-lint run reports nothing on the new file (the 92 baseline issues are elsewhere).

**Teardown.** Every `ichor-e2e-fwh` node, disk, pod, Service, Secret and the registry was deleted; the port-forwards on 5064 and 5065 are gone. The `ichor-ui-demo5-*` objects and the cached ISO disks were not touched.

## Amendment 8 (2026-11-04): init and adopt wait for the controller to be ready before writing, instead of retrying the write

Amendment 7 measured the refusal at about 20 to 30 seconds after `ichor init` or the adopt install step: the API server calls the validating webhooks of the controller that was started moments earlier. The retry-the-write loop covered it but printed one refusal per attempt and bounded the wait at 30 minutes (`init`) and 15 (`adopt`). It is replaced.

**Decision.** `createWhenWebhooksUp` (every CR write of `init` and `adopt`, and `createIchorInstallation`) first waits until the controller can serve the webhook, then writes. Ready means both:

- the Deployment `ichor-controller-manager` in `ichor-system` is rolled out (its generation observed, all replicas updated, no old replica left) with at least one available replica;
- the Service `ichor-webhook-service` has an EndpointSlice endpoint that is ready (a missing `ready` condition counts as ready, per the API).

The wait polls on the existing pod diagnostics (`waitDiag`), so a cluster with no pod network still fails early. After it, the write goes through `createRetrying`, which now also retries a webhook refusal (an InternalError naming a webhook) but only inside `initRetryWindow` (3 minutes): that covers the short gap between a ready endpoint and the API server picking up the webhook's CA bundle. There is no second long loop.

**Bound.** The defaults are unchanged: 30 minutes for `init` and 15 for `adopt` (ADR 0104 kept 30 for `init` because a first install pulls images over a slow link, and a fresh install must not fail on a shorter default). `--wait-timeout` sets the bound; `adopt` already had it and `init` gets the same flag. A timeout fails with: `the controller Deployment ichor-controller-manager in ichor-system is not Ready after 30m0s (<pod reasons>): <what is missing>; check "kubectl -n ichor-system get pods"`.

**Not covered.** The names are the manifest's and the chart's defaults (release `ichor`, namespace `ichor-system`). A Helm release with another name or namespace is not found by label and the wait times out with the message above. An operator who may not read EndpointSlices is judged on the Deployment alone.

**Tests.** `internal/cli/webhook_ready_test.go`: ready, becoming ready after polls, four timeout shapes (no Deployment, no available replica, no endpoint, endpoint not ready), and no `Create` before the controller is ready.

## Amendment: ADR 0128 (2026-11-04)

[ADR 0128](https://ichor.maesh.dev/next/adr/0128-adopting-installed-talos-machines-and-clusters/) maps this ADR against the cases a user has and closes four gaps: a machine that already carries an Ichor cluster's CAs but has no `TalosMachine` is registered with `ichor adopt --add-machine` (section 2, state iii, was not reachable before); reclaim no longer counts a stale pending `TalosMachine` as an Ichor machine and points a managed cluster's node at `--add-machine`; `--cilium-values` (planned in 1.2, step 3, "not built" above) now carries the operator's values into a takeover of a Cilium with no release record; the non-Cilium refusal says Ichor does not migrate a CNI.