# 0120. The CA rotation also replaces the etcd CA and the Kubernetes aggregator CA

Status: Accepted; built; amended after hardware 2026-11-05 (aggregator rotation verified; the etcd roll by reboot awaits its hardware re-run) · Date: 2026-11-04 · Design section: Backup, recovery, certificates · Touches: 0063, 0091, 0093, 0108

Decided: the `ca-rotation` operation can also replace the etcd CA and the Kubernetes aggregator (front-proxy) CA, chosen with the new, optional `spec.caRotation.targets` (`talos`, `kubernetes`, `etcd`, `kubernetes-aggregator`; absent means `talos` and `kubernetes`, as before). It is the same state machine, lock, pause/continue and finalizer, with the same three passes. The aggregator CA uses Talos's own accepted-CA list (`KubeAggregatorCAConfig`, `acceptedCAs`). Talos has no accepted-CA list for etcd, so Ichor puts both etcd CA certificates into `cluster.etcd.ca.crt` (a PEM bundle, the new key signing from Issuing on) and restarts the etcd members one at a time after each pass; the credential hand-off (AwaitingCredentials) happens only when the Talos or Kubernetes CA is among the targets. The etcd backup key does not change. Before the first hardware run the etcd bundle is a reading of Talos v1.14.5 source, not of its documentation; ADR section "Hardware verification" says what must be seen before it is trusted.

## Context

ADR 0108 (S6) left the etcd CA and the aggregator CA out of the rotation: "the etcd CA rotation needs every member restarted with both CAs trusted in a defined order". The user decided on 2026-11-04 that both ship in v0.1.0. The service-account signing key and the secretbox key are a different mechanism (ADR 0121).

### What Talos v1.14 offers (version in go.mod: `siderolabs/talos/pkg/machinery` v1.14.5)

- `talosctl rotate-ca` rotates the Talos API CA (`--talos`) and the Kubernetes API CA (`--kubernetes`) only. [Talos CA rotation](https://docs.siderolabs.com/talos/v1.14/security/ca-rotation) lists the general flow (generate, accept, issue, refresh, drop) and covers those two; `cmd/talosctl/cmd/talos/rotate-ca.go` and `pkg/rotate/pki/{talos,kubernetes}` contain nothing for etcd or the aggregator. The same page says: "The aggregator CA is configured separately through `KubeAggregatorCAConfig`" and gives no procedure for it.
- **Aggregator CA.** The typed document `KubeAggregatorCAConfig` (`pkg/machinery/config/types/k8s/aggregator_ca.go`) has `issuingCA` and `acceptedCAs`; its documentation says "The issuing CA certificate is automatically added to the list of accepted CAs". Talos renders `acceptedCAs` into `aggregator-ca.crt`, which is kube-apiserver's `--requestheader-client-ca-file` (`controllers/k8s/render_secrets_static_pod.go`, `control_plane_final.go`), and signs the front-proxy client certificate (`--proxy-client-cert-file`) with `issuingCA` (`controllers/secrets/kubernetes_dynamic_certs.go`). kube-apiserver publishes the accepted bundle in the ConfigMap `kube-system/extension-apiserver-authentication` for extension API servers ([Configure the Aggregation Layer](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/)). The document exists for control planes only and only with a version contract that has the multi-document Kubernetes configuration (`MultidocKubernetesConfigSupported`, v1.14); earlier contracts use `cluster.aggregatorCA`, a single CA with no accepted list, and the two forms conflict (`V1Alpha1ConflictValidate`).
- **etcd CA.** The only field is `cluster.etcd.ca` ([config reference](https://docs.siderolabs.com/talos/v1.14/reference/configuration/v1alpha1/config): "The `ca` is the root certificate authority of the PKI"). There is no accepted-CA list. In the source: `EtcdConfig.CA()` feeds `secrets.EtcdRoot` (`controllers/secrets/root.go`); `etcd/pki.go` writes `EtcdCA.Crt` verbatim to `/system/secrets/etcd/ca.crt`; etcd runs with `trusted-ca-file` and `peer-trusted-ca-file` pointing at that file (`system/services/etcd.go`); `secrets/etcd.go` makes the server, peer, admin and kube-apiserver client certificates from the CA with `x509.NewCertificateAuthorityFromCertificateAndKey`, which parses only the first PEM block of `Crt` (`pem.Decode`) and the key; kube-apiserver gets the same file as `etcd-client-ca.crt` (`--etcd-cafile`). Nothing restarts etcd when the files change, and nothing validates the CA's content beyond "present". Consequence: with a plain replacement of `cluster.etcd.ca` a member on the new CA and a member on the old one cannot talk to each other, so rotating the CA member by member loses quorum at the middle restart.

## Decision

### 1. Targets (API)

Append-only: `TalosOperation.spec.caRotation` (`TalosOperationCARotationSpec`, optional) with `targets []string` (enum `talos`, `kubernetes`, `etcd`, `kubernetes-aggregator`; unique; at least one). Absent or empty means `talos` and `kubernetes`, so every existing operation, script and UI call is unchanged. The names are the ones `status.certificates` of the cluster already uses for expiry (ADR 0063). `status.caRotation.certificates` gets entries named like the targets, and two new status fields: `etcdRestarted` and `etcdRestartedAt` (below). The stage enum is unchanged. The wire gets `repeated string ca_targets` on `CreateOperationRequest` and `PlanOperationRequest` (the choice), `PlanOperationResponse` (the effective list, the default pair spelled out) and `Operation` (for the UI).

A separate operation kind was rejected: it would need its own lock handling, finalizer, machine-controller quiescing, pause/continue, `rotation.yaml` and bundle replacement, which are exactly what the existing machine provides, and a second kind could run beside a first one. One operation per cluster at a time stays true.

The targets are decided once, when the operation starts: the new CAs are generated for the targets and written to `rotation.yaml` (optional keys `talos`, `kubernetes`, `etcd`, `aggregator`; the set of keys present is the set of targets). A restarted controller, or a rotation started by an older Ichor, reads the targets from `rotation.yaml`, not from the spec.

### 2. Stages and rendered configuration

The stages and their meaning stay: Accepting (the new CA is trusted, the old one issues), Issuing (the new one issues, the old is trusted), Dropping (only the new one), Finalizing. Per target, in the typed documents:

| target | Accepting | Issuing | Dropping, Finalizing |
| --- | --- | --- | --- |
| talos | `machine.acceptedCAs` + new | `machine.ca` new, accepted old | `machine.ca` new |
| kubernetes | `KubeAPIServerCAConfig` accepted + new | issuing new, accepted old | issuing new |
| kubernetes-aggregator (control planes) | `KubeAggregatorCAConfig.acceptedCAs` + new | `issuingCA` new, `acceptedCAs` old | `issuingCA` new |
| etcd (control planes) | `cluster.etcd.ca`: certificate = old then new, key = old key | certificate = new then old, key = new key | certificate = new, key = new key |

The etcd rows rely on two facts read in Talos's source (above): the certificate field is written verbatim into the trust file, which etcd and kube-apiserver read as a PEM bundle, and the signing CA is the first block with the key. The first row therefore keeps the old CA signing while both are trusted; the second changes the signer while both are still trusted; the third drops the old one. Dropping and Finalizing render byte for byte what a bundle holding the new CAs renders (a test pins it), so the machine controller sees no drift afterwards. A target that is not rotated renders exactly as before.

Refused at the start (plan `blocked` and executor preflight): `kubernetes-aggregator` on a cluster whose version contract is older than v1.14 (no accepted list exists there). A two-member etcd, which would lose its quorum while one member restarts, needs no check of its own: the cluster lock already refuses every disruptive operation on an even number of approved control planes (`ClusterTransitioning`). An etcd rotation also refuses to start unless etcd is whole: every control plane reports etcd Running and healthy, one member per control plane, no alarm, kube-apiserver ready.

### 3. etcd: restart the members, one at a time, after each pass

Applying the configuration changes the files and the leaf certificates, but the running etcd keeps what it loaded, so after each of the three passes the controller restarts the etcd service of every control plane, one at a time, in the order the control planes are visited:

1. The restart starts only when every machine already runs the stage's configuration (the apply and the restart are separate reconciles) and only when the cluster is healthy: every control plane's `etcd` service is Running and healthy, its member list has one member per approved control plane, no alarm is set, its kube-apiserver is ready, and the node's etcd PKI is the stage's: the root CA the node reports (COSI `secrets.EtcdRoot`) holds the expected certificates in the expected order and its server certificate (`secrets.Etcd`) is signed by the expected CA. Otherwise the operation pauses (`Blocked`) and tries again every 30 s; nothing is restarted.
2. If the member leads, it forfeits leadership first (`EtcdForfeitLeadership`), then `ServiceRestart("etcd")`.
3. The name goes into `status.caRotation.etcdRestarted` and the time into `etcdRestartedAt`; the next member is restarted only after `etcdRestartSettle` (15 s) and a new health check. `etcdRestarted` is emptied when the stage changes. A controller killed after the restart but before the status write restarts that member once more, which is harmless.
4. The pass is verified (as before: Ichor reaches each node, every Node is Ready) only after the last member has restarted and the final health check passes.

Because both CAs are trusted by every member from the first pass on, and every member presents a certificate either CA vouches for, a member on an earlier or later step of the roll can always talk to the others: during each roll at most one member is down, which is an ordinary rolling restart, quorum holds with three or more members, and a single control plane merely has its API unavailable while its only member restarts (the plan says so: `kubernetes_api_unavailable`). Three rolls are needed, not one: after Accepting every member trusts both CAs; after Issuing every member presents a certificate of the new CA (kube-apiserver then trusts both); after Dropping only the new CA is trusted. kube-apiserver reads its etcd client CA and certificate from the same Talos secrets, and Talos restarts its static pod on every change, so each pass also restarts kube-apiserver once per control plane; the existing Node-Ready verification covers it.

### 4. Aggregator: wait for the extension API servers, never restart them

kube-apiserver restarts on its own (Talos re-renders the static pod). Before Issuing, the controller reads `extension-apiserver-authentication` through the admin kubeconfig a Talos node issues (as the `kube-root-ca.crt` check does) and waits until its `requestheader-client-ca-file` holds the new CA, then waits `ClientSettleDelay` (2 minutes; the clock is a condition transition time, so it survives a restart) so that extension API servers that reload the ConfigMap have the new CA before the front-proxy certificate is signed by it. Ichor restarts none of them (they are workloads such as metrics-server; ADR 0101): the plan and the completion message say that an extension API server that read the CA only once must be restarted by the owner. The wait is Verifying-stage of Accepting, not a new stage.

### 5. Credentials hand-off, secrets bundle, backups

- **AwaitingCredentials** exists for the operator's talosconfig and kubeconfig (ADR 0091). When neither `talos` nor `kubernetes` is a target nothing the operator holds changes, so the stage is skipped (Accepting goes to Issuing) and the CLI/UI do not ask for a hand-off. `CARotationHandoffToken` is unchanged in form; it also names etcd and aggregator entries when present.
- **Bundle.** `Certs.Etcd` and `Certs.K8sAggregator` are replaced in the one Secret update that replaces the others (`Rotation.Apply` replaces what `rotation.yaml` holds and nothing else). `rotation.yaml` holds the keys of the new CAs from before any node is told anything, as in ADR 0063. A key that is not in `rotation.yaml` is never touched.
- **BundleBackupStale.** Raised as before. `BundleFingerprint` (Talos and Kubernetes CAs) is unchanged and is still what the credentials code compares; a new `ExportFingerprint` adds the etcd and aggregator CAs, and `bundle-export` and the cert-expiry reconciler use it, so an export made before an etcd-only rotation does not count as current. Cost: an export annotation written by an earlier Ichor no longer matches, so a cluster that has BundleBackupStale up at the moment of the upgrade needs one more `bundle-export`.
- **etcd backups.** The backup key derives from `Cluster.Secret` and `Cluster.ID` (ADR 0063, ADR 0108 section 4), which a CA rotation does not touch; snapshots contain no PKI. Snapshots taken before the rotation stay readable and restorable (a restore renders the configuration with the then-current CAs). `TestPassphraseSurvivesCARotation` still pins it, and the controller test asserts the cluster ID and secret are unchanged. The secretbox key stays out of scope (ADR 0121).
- **A fresh snapshot first.** The plan warning for an etcd rotation tells the user to take an etcd snapshot before starting (`ichor etcd backup now NAME`). Ichor does not require one: a missing backup is the user's decision for their cluster, not something Ichor forces (the "warn, do not force" rule for the managed cluster).

### 6. Failure and rollback

Every step is idempotent and re-derived from the nodes and from status (ADR 0063): a failed apply, a node that does not answer, a Node not Ready, an etcd member that does not come back, a refused dry run: the operation is Paused with `CARotation=False/Blocked` and a message, the next stage is never started, and it retries every 30 s. No member is restarted while any is unhealthy, so at most one member is down at any time. A rotation is never rolled back by Ichor automatically; it only moves forward. Until Dropping has started, every member and kube-apiserver still trust the old CA, so a rotation that cannot proceed can be abandoned by hand: remove `rotation.yaml` from the bundle Secret and the operation's finalizer (ADR 0063), after which the machine controller renders the old CAs again; the members pick that up at their next restart (expected, not exercised on hardware). After Dropping started, the old etcd CA is gone from the nodes that were already rolled: continue, do not revert.

### 7. UI and CLI

- CLI: `ichor cluster rotate-ca NAME [--ca talos,kubernetes,etcd,kubernetes-aggregator] [--all]`; default unchanged. `--yes`, `--follow` as before. The plan lists the targets, the stages, the etcd restarts and the warnings; the credential hand-off runs only when `talos` or `kubernetes` is a target.
- UI: the start dialog for a CA rotation shows four checkboxes (Talos API and Kubernetes API checked, etcd and aggregator unchecked) and sends `ca_targets`; the plan view names the CAs, the etcd restarts and the warnings and shows the credentials alert only when `talos` or `kubernetes` is a target; the operation page lists the CAs and the etcd members restarted in the stage (`Operation.ca_targets` and the details), and the credentials panel (`CARotationCredentials`) appears only for rotations that include `talos` or `kubernetes`.
- The cert-expiry message no longer says the two CAs "are not covered".

## Rejected

- **Replacing `cluster.etcd.ca` in one pass (all members at once, or one by one).** At once is an outage of the whole control plane; one by one, with a single CA per member, splits the cluster by CA at the middle restart.
- **Issuing etcd leaf certificates from the new CA and keeping the old trust through a separate file.** Talos has no field for it and Ichor may not write raw files (machine config is typed only, no raw escape hatch).
- **Treating the etcd CA as unrotatable and documenting it.** The user decided it ships in v0.1.0.
- **A restart of every member at the same time per pass, or no restart in Accepting.** The certificates are loaded at start; a member that never restarted after Accepting would not trust the new CA when Issuing makes the others present it.
- **A hard requirement for a recent etcd snapshot.** Warned about, not enforced (above).
- **Restarting extension API servers.** Not Ichor's workloads (ADR 0101).

## Risks

- The etcd bundle depends on Talos behaviour that is not in its documentation (first PEM block signs, whole field is the trust file, etcd and kube-apiserver read bundles). If a Talos release changes it, the etcd target would split the cluster; Ichor's own check (the node's `secrets.Etcd` is signed by the expected CA, the trusted list is the expected one) pauses the rotation before any restart if the node does not behave as read. The target is therefore also the one to run first on a disposable cluster.
- Three etcd rolls and three kube-apiserver restarts per control plane; a one-control-plane cluster has its Kubernetes API down during each member restart.
- Extension API servers that read the aggregator CA once need a restart by their owner.
- `ClientSettleDelay` adds two minutes to a rotation that includes the aggregator CA.

## Test

envtest (`internal/controller`, a fake node per machine as in ADR 0063): the stage sequence, rendered documents per stage and role for etcd and aggregator (typed: `cluster.etcd.ca` certificate order and key, `KubeAggregatorCAConfig`), Dropping equal to the replaced bundle's render, rotation of only etcd skipping AwaitingCredentials, etcd members restarted in order with the health gate and leader forfeit, each member restarted exactly once per stage with a new reconciler at every step, an unhealthy member holding the roll, a node that has not yet loaded the stage's etcd PKI holding it, a missing member and two control planes refusing the start, a legacy contract refused (render), the aggregator wait, `rotation.yaml` keys and the replaced bundle, backup key unchanged; `pkg/render` tests per stage and role; CLI flags and plan output; UI selection and card.

## Hardware verification

Not run. Steps (3 control planes and 1 worker, contract v1.14, a throwaway cluster; `kubectl --context admin@talos-cluster01` for the outer cluster only):

1. `ichor etcd backup now NAME` and wait for the snapshot. Note `talosctl etcd members` and a ConfigMap written before the rotation.
2. `ichor cluster rotate-ca NAME --ca etcd --yes --follow`. Expect: no AwaitingCredentials; after Accepting, `talosctl -n <cp> read /system/secrets/etcd/ca.crt` holds two certificates (old first); after Issuing, new first; after Dropping, one (the new). Each roll: only one `etcd` service restarts at a time (`talosctl service etcd` on each), `talosctl etcd members` never loses a member, `kubectl get --raw /readyz` recovers within a minute per control plane.
3. After Done: `talosctl etcd status` on every control plane, check `talosctl read /system/secrets/etcd/etcd.crt | openssl x509 -noout -issuer -fingerprint` against the new CA and `talosctl get etcdrootsecret -o yaml` (public part). The ConfigMap from step 1 is still readable; an etcd backup taken after the rotation succeeds; an older snapshot is listed and its key still decrypts it.
4. `ichor cluster rotate-ca NAME --ca kubernetes-aggregator --yes --follow` on a cluster with metrics-server: `kubectl top nodes` works before, after Accepting and the settle wait, after Issuing and after Done; `kubectl get cm -n kube-system extension-apiserver-authentication -o jsonpath='{.data.requestheader-client-ca-file}'` holds two certificates, then one.
5. `ichor cluster rotate-ca NAME --all --yes --follow` once on a fresh cluster, then `ichor cluster bundle-export NAME` clears BundleBackupStale.
6. Negative: kill the controller pod during an etcd roll; the roll resumes, no member is restarted twice in a row without a health check. Power off one control plane before starting; the operation refuses to start.
7. If step 2 shows a member that does not trust a certificate signed by the other CA, stop: the etcd bundle assumption is wrong for that Talos build; do not use the `etcd` target and record the Talos version.

## Amendment (hardware, 2026-11-05)

Lane A ran the rotation on a compact HA cluster (3 control planes and 1 worker, Talos v1.14.5, Kubernetes v1.36.6, etcd 3.7.3, Ichor main `ecb01c33`). The aggregator rotation worked; the etcd rotation did not, and three smaller defects showed. This amendment replaces what it names and leaves the rest of the ADR as written.

### 1. etcd members load the CA with a reboot, not a service restart (replaces section 3)

**Observed.** `--ca etcd` paused in Accepting: `restarting etcd on "<machine>" failed: rpc error: code = Unknown desc = service "etcd" doesn't support restart operation via API`. `talosctl service etcd restart` and `stop` fail the same way on v1.14.5. The Accepting configuration itself applied without a reboot (`/system/secrets/etcd/ca.crt` went from one to two certificates on all three control planes and etcd kept running), and afterwards Talos reported `RebootRequired` for `cluster.etcd.ca.crt`.

**Documentation (Talos v1.14).** [Edit machine configuration, "Changes which might need a reboot"](https://docs.siderolabs.com/talos/v1.14/configure-your-talos-cluster/system-configuration/editing-machine-configuration): ".cluster.etcd: the etcd service reads its settings ... when it starts. Talos does not restart etcd on a configuration change, since restarting a control plane member is disruptive and is not something Talos does implicitly. Reboot the node to pick up the new settings." The same page says machine configuration "is always applied to the running node without a reboot" in v1.14, that `--mode=reboot` was removed in Talos 1.14, and that a change which needs a reboot is applied first and the node then rebooted explicitly "so it can be sequenced with cordoning and draining". [etcd maintenance](https://docs.siderolabs.com/talos/v1.14/build-and-extend-talos/cluster-operations-and-maintenance/etcd-maintenance) repeats it for the etcd image: "Talos does not restart it on configuration change, so reboot the node." So a reboot is the only supported way for a running member to load a new `cluster.etcd.ca`, and there is no apply mode that applies and reboots in one call: the existing `NO_REBOOT` apply of the pass followed by an explicit, gated reboot is the design the documentation describes.

**Decision.** After each of the three passes, once every machine runs the stage's configuration, the controller reboots the control planes one at a time. The roll is the node action's reboot (ADR 0085) run inside the rotation, which holds the cluster lock (a child `TalosOperation` could not take it):

1. Before any reboot, and again before each next one: every control plane's etcd Running and healthy, a leader seen, one member per approved control plane, no alarm, kube-apiserver ready, the node's etcd PKI the stage's (the same checks as before), and every Kubernetes Node Ready. Otherwise the operation pauses (`Blocked`) and nothing is rebooted.
2. For the next control plane in order: the storage evacuation hook (nil when none is wired), then cordon and drain with the helper the config rollout uses (`KubeClient`, skipped when none is wired), a leader forfeits leadership (best effort), the boot ID it leaves is written to `status.caRotation.etcdRebootBootID` and its name to `etcdRebooting`, and only then `Reboot` is called. A controller killed between the status write and the call finds the marker and waits; if the node still reports that boot after `etcdRebootRetry` (5 minutes) the marker is cleared and the request made again. A failed `Reboot` call clears the marker, uncordons and retries.
3. While `etcdRebooting` is set the pass does nothing else (it does not dial the node to read its configuration) and waits, `InProgress` with reason `RebootingEtcd`, for: the node to answer on a different boot ID, every Node Ready, and the same health and PKI gate as in 1 on all control planes. Past `nodeActionTimeout` (20 minutes) the operation pauses. Then the node is uncordoned (storage evacuation ended), its name is appended to `etcdRestarted` (the name of the field is kept: it is append-only; it now lists the control planes rebooted in the stage) and `etcdRebooting` is cleared. `etcdRestartedAt` records when the reboot was requested.
4. `etcdRestartSettle` is gone: the gate after a reboot replaces a fixed wait.

New status fields, append-only: `status.caRotation.etcdRebooting` and `status.caRotation.etcdRebootBootID`. The reason `RestartingEtcd` became `RebootingEtcd`.

**Cost and risk (replaces the corresponding lines of Risks).** Three reboots of every control plane per etcd rotation (one per pass), each with a drain, one node at a time; a rotation of etcd alone on three control planes is nine reboots. The plan (CLI and UI) and `ichor cluster rotate-ca --help` say so. A cluster with one control plane has no Kubernetes API during each of its three reboots, and the drain of its only control plane evicts what runs there. A reboot is longer than the restart this replaced, and the node comes back with the stage's configuration loaded from disk (the pass applied it with `NO_REBOOT`, which writes it to the STATE partition).

**Still unverified.** The assumption that both CA certificates in one `cluster.etcd.ca.crt` are trusted, with the first signing, was not reached (no Issuing stage ran). The check that stops before any reboot unless the node reports the stage's PKI (`secrets.EtcdRoot` and `secrets.Etcd`) stays, and the hardware run below must still end the question. A member that does not trust a certificate signed by the other CA now shows as a rebooted node whose etcd does not become healthy: the roll stops at that node, pauses, and reboots no further one.

### 2. Deleting a rotation before its point of no return abandons it

**Observed.** `ichor operation delete` on a Paused ca-rotation set `deletionTimestamp`, but the operation stayed Paused with its finalizer, the machines stayed `Ready=Unknown` (`CARotationInProgress`) and the cluster lock stayed held; the agent removed `rotation.yaml` and the finalizer by hand. The finalizer step only released an operation that was no longer under way, and an operation paused mid-rotation is under way for ever.

**Decision** (ca-rotation and key-rotation, which share the pattern). A delete of a rotation that is under way is decided by `RotationAbandonable` (`api/v1alpha1`):

| rotation | abandonable in | not in | why |
| --- | --- | --- | --- |
| ca-rotation | Accepting, AwaitingCredentials; Issuing when neither the Kubernetes nor the etcd CA is a target | Issuing with the Kubernetes or etcd CA; Dropping; Finalizing | After Issuing the pods have been given the new Kubernetes CA alone (`kube-root-ca.crt`), and an etcd member that went back to the old CA would not trust its neighbours' new certificates; Dropping and Finalizing remove the old CA. Talos and aggregator CAs survive Issuing: the old one is still accepted. |
| key-rotation | Accepting | Issuing and later | From Issuing the new secretbox key encrypts and the new service-account key signs: going back leaves Secrets and tokens nobody can read. |

For an abandonable rotation the finalizer step, instead of waiting for a rotation that will never settle: removes `rotation.yaml` (`keyrotation.yaml`) from the cluster's bundle Secret (the one object a rotation writes; `secrets.yaml` was not touched before Finalizing), uncordons the control plane that was being rebooted (named by `spec.hostname` or `status.nodeName`, as the node may be down), releases the cluster lock if the operation holds it (`Release` is a no-op for another holder), records an Event on the cluster, and removes the finalizer. Every step is idempotent and retried as a whole. The machine controller then renders the old material again: the nodes still carry the extra accepted CAs (and for etcd the two-certificate field), which Talos classifies as reboot-requiring, so the machines report `Ready=False/RebootRequired` and `ichor config apply` finishes the revert (3 minutes for three control planes on hardware, as the hand-made abandon did). Ichor does not apply it by itself: it never reboots a node on its own.

For a rotation past the point: the controller sets the operation condition `DeletionHeld=True` (reason `PastPointOfNoReturn`, the stage in the message) once and goes on; the finalizer is released when the rotation is Done. `DeleteOperation` in the UI API (CLI and UI) refuses it up front with `FailedPrecondition` and the stage. A webhook on DELETE was rejected: it would put every TalosOperation delete behind a `failurePolicy: Fail` admission call, and the condition plus the API refusal reach the same people (a raw `kubectl delete` still ends in a held, explained delete).

### 3. The old aggregator CA leaves `extension-apiserver-authentication`

**Observed.** After Done the ConfigMap's `requestheader-client-ca-file` still held both CAs although `aggregator-ca.crt` on every control plane held one: kube-apiserver merges and never removes. Deleting the ConfigMap made it come back with one certificate and `kubectl top` kept working.

**Decision.** In Finalizing, after every machine is verified against the replaced bundle, `AggregatorAuthReset` reads the ConfigMap through the admin kubeconfig a Talos node issues for the call (the credential the other CA checks use; there is no standing credential and no RBAC of Ichor's own for the managed cluster, so no `ClusterRole` rule is added and the chart is unchanged). If it holds exactly the new CA the step is settled; if not, it deletes the ConfigMap by name with a UID precondition (a ConfigMap kube-apiserver has recreated meanwhile is not deleted again) and reports not settled; a missing ConfigMap is not settled either. The operation stays `InProgress` (`SettlingAggregator`) and the check repeats every 30 s; it completes only when the ConfigMap holds the new CA alone. Extension API servers that read the ConfigMap once still need a restart by their owner (unchanged).

### 4. The credential hand-off waits for the Kubernetes API

**Observed.** With a single control plane endpoint the hand-off failed once with `connection refused` while kube-apiserver restarted; the operation kept waiting and `ichor cluster credentials --acknowledge` fixed it. Only the final annotation patch retried transient errors.

**Decision.** Every API call of the hand-off (reading the secrets bundle, listing the operations, verifying the new kubeconfig) goes through `retryHandoff`, the CLI's existing transient-error retry (`retryTransient`: backoff from 1 s to 15 s for up to 10 minutes), and prints once that it is waiting for the Kubernetes API. A non-transient error (a certificate that does not authenticate) still stops at once and the operation is not acknowledged.

### Tests

envtest: the roll by reboot (order, one reboot per control plane per stage, no `ServiceRestart` ever, unhealthy member holds the roll, a node that does not answer is waited for, a boot ID that does not change is waited for and re-requested after `etcdRebootRetry`, the 20-minute pause, leader forfeits before its reboot, a Node not Ready holds the roll, cordon and uncordon around the reboot); abandon (ca-rotation paused in AwaitingCredentials, Issuing of the Talos CA, mid-reboot uncordon, held past Dropping with `DeletionHeld` and released at Done; key-rotation in Accepting abandoned and in Issuing held); the aggregator reset in Finalizing and the reset step itself against a fake ConfigMap client (by name, UID precondition, missing, settled). `api/v1alpha1`: `RotationAbandonable`. `uiapi`: the delete refusal. `cli`: the hand-off retries and gives up after the window.

### Hardware verification (replaces steps 2 and 6 above; the rest stands)

2. `ichor cluster rotate-ca NAME --ca etcd --yes --follow` on three control planes: no AwaitingCredentials; each pass reboots the control planes one after the other (`kubectl get nodes` shows one cordoned node at a time), `talosctl etcd members` never loses a member, `kubectl get --raw /readyz` recovers per node; after Accepting `ca.crt` holds two certificates (old first), after Issuing two (new first), after Dropping one. The first Issuing reboot answers the open question (a member that trusts a certificate signed by the other CA): if a rebooted node's etcd does not become healthy the roll stops there; record the Talos version and do not use the `etcd` target.
6. Kill the controller pod between the status write and the reboot, and during a reboot: the roll resumes without rebooting a second control plane meanwhile. `ichor operation delete` on a rotation Paused in Accepting: the operation disappears, `rotation.yaml` is gone from the Secret, the lock lease is gone, the machines show `RebootRequired`, `ichor config apply` finishes. The same in Dropping is refused by the CLI.
8. After an aggregator rotation: `kubectl get cm -n kube-system extension-apiserver-authentication -o jsonpath='{.data.requestheader-client-ca-file}'` holds one certificate when the operation reports Completed.
9. With one control plane endpoint, start `rotate-ca` (default CAs) from the terminal: a refused connection during the hand-off prints the waiting line and the hand-off completes by itself.

### Amendment (lane E re-verification, 2026-11-05): webhook failures in the hand-off, the etcd plan wording, and the aggregator restart

1. The hand-off retry (point 4) treated a refused webhook connection as transient but ended on `failed calling webhook ...: context deadline exceeded`, which arrives as a 500 InternalError whose text does not mention a refused connection. Any `failed calling webhook` error that is not a Forbidden or Invalid refusal (a webhook that answered "no" stays an answer) is now retried for the whole 10-minute window.
2. The CLI plan of an etcd rotation no longer says "without a reboot": it says the control planes reboot in each pass (the workers do not).
3. A rotation of the aggregator CA turned Completed about 13 s before the last kube-apiserver restart on the endpoint control plane ended: a pass applied the stage to every machine and verified only the Talos API and the Nodes, but Talos restarts kube-apiserver some seconds after the apply. A pass now records, per control plane it applied to, the kube-apiserver container id read just before the apply (`status.caRotation.apiServerInstances`, with `appliedAt`; both additive and emptied on every stage change) and does not verify the stage until each of them runs in another container and answers `/readyz`. A container that did not change within a minute of the apply (the stage changed nothing kube-apiserver loads) is taken as not restarted; without a recorded id (the controller died between the apply and the status write) only `/readyz` is asked. `/readyz` is not asked when the Kubernetes CA is rotated too, because the probe verifies against the bundle's old Kubernetes CA. Rotations that do not target the aggregator CA are unchanged.

### Amendment (lane F, 2026-11-05): discovery failures in the hand-off retry

On hardware the hand-off retry still ended after 11 s with `unable to retrieve the complete list of server APIs: ichor.maesh.dev/v1alpha1: no matches for ichor.maesh.dev/v1alpha1, Resource=` while the endpoint control plane's kube-apiserver came back and did not yet serve the CRD group: the controller-runtime client's REST mapper failed discovery, an error `isTransient` did not know. `isTransient` now matches by type `meta.IsNoMatchError`, `*apiutil.ErrResourceDiscoveryFailed` (what the lazy REST mapper returns) and `discovery.IsGroupDiscoveryFailedError`, so these are retried for the whole 10-minute window; Forbidden, NotFound, Invalid and webhook refusals stay final. The same client may be used again: the lazy REST mapper keeps no negative cache (a failed lookup leaves the group and version out of its cache, and the next call asks the server again), so the client is not rebuilt per attempt; the existing `redial` after a rotated kubeconfig is unchanged.

### Amendment (2026-11-05): an early hand-off acknowledgement does not skip the gate

`CredentialsHandoffAnnotation` holds `CARotationHandoffToken(status)`, the fingerprints of the new CAs, which `status.caRotation.certificates` shows from stage Accepting on. The controller read the annotation only in AwaitingCredentials, but an annotation set earlier (by hand, or by a client that computed the token from Accepting) already matched when the operation got there, so the gate was passed in the same reconcile and the operator never had a chance to hold working credentials. When the controller enters AwaitingCredentials (the pass that verifies Accepting) it now removes whatever annotation is on the operation, with a merge patch through a copy and the new resource version carried over, so a change made after the drop fails the status update and is dropped on the next pass. An annotation set while the operation waits is honoured as before; the CLI and the UI set it only after they see the stage. The remaining trust is unchanged (ADR 0129): the annotation is an attestation, and the controller checks only the token. Test: `TalosOperation ca-rotation ignores an acknowledgement given before the stage AwaitingCredentials`.