0120. The CA rotation also replaces the etcd CA and the Kubernetes aggregator CA

Status: Accepted; built; amended after hardware 2026-11-05 (aggregator rotation verified; the etcd roll by reboot awaits its hardware re-run) · Date: 2026-11-04 · Design section: Backup, recovery, certificates · Touches: 0063, 0091, 0093, 0108

Decided: the ca-rotation operation can also replace the etcd CA and the Kubernetes aggregator (front-proxy) CA, chosen with the new, optional spec.caRotation.targets (talos, kubernetes, etcd, kubernetes-aggregator; absent means talos and kubernetes, as before). It is the same state machine, lock, pause/continue and finalizer, with the same three passes. The aggregator CA uses Talos's own accepted-CA list (KubeAggregatorCAConfig, acceptedCAs). Talos has no accepted-CA list for etcd, so Ichor puts both etcd CA certificates into cluster.etcd.ca.crt (a PEM bundle, the new key signing from Issuing on) and restarts the etcd members one at a time after each pass; the credential hand-off (AwaitingCredentials) happens only when the Talos or Kubernetes CA is among the targets. The etcd backup key does not change. Before the first hardware run the etcd bundle is a reading of Talos v1.14.5 source, not of its documentation; ADR section "Hardware verification" says what must be seen before it is trusted.

Context

ADR 0108 (S6) left the etcd CA and the aggregator CA out of the rotation: "the etcd CA rotation needs every member restarted with both CAs trusted in a defined order". The user decided on 2026-11-04 that both ship in v0.1.0. The service-account signing key and the secretbox key are a different mechanism (ADR 0121).

What Talos v1.14 offers (version in go.mod: siderolabs/talos/pkg/machinery v1.14.5)

• talosctl rotate-ca rotates the Talos API CA (--talos) and the Kubernetes API CA (--kubernetes) only. Talos CA rotation lists the general flow (generate, accept, issue, refresh, drop) and covers those two; cmd/talosctl/cmd/talos/rotate-ca.go and pkg/rotate/pki/{talos,kubernetes} contain nothing for etcd or the aggregator. The same page says: "The aggregator CA is configured separately through KubeAggregatorCAConfig" and gives no procedure for it.
• Aggregator CA. The typed document KubeAggregatorCAConfig (pkg/machinery/config/types/k8s/aggregator_ca.go) has issuingCA and acceptedCAs; its documentation says "The issuing CA certificate is automatically added to the list of accepted CAs". Talos renders acceptedCAs into aggregator-ca.crt, which is kube-apiserver's --requestheader-client-ca-file (controllers/k8s/render_secrets_static_pod.go, control_plane_final.go), and signs the front-proxy client certificate (--proxy-client-cert-file) with issuingCA (controllers/secrets/kubernetes_dynamic_certs.go). kube-apiserver publishes the accepted bundle in the ConfigMap kube-system/extension-apiserver-authentication for extension API servers (Configure the Aggregation Layer). The document exists for control planes only and only with a version contract that has the multi-document Kubernetes configuration (MultidocKubernetesConfigSupported, v1.14); earlier contracts use cluster.aggregatorCA, a single CA with no accepted list, and the two forms conflict (V1Alpha1ConflictValidate).
• etcd CA. The only field is cluster.etcd.ca (config reference: "The ca is the root certificate authority of the PKI"). There is no accepted-CA list. In the source: EtcdConfig.CA() feeds secrets.EtcdRoot (controllers/secrets/root.go); etcd/pki.go writes EtcdCA.Crt verbatim to /system/secrets/etcd/ca.crt; etcd runs with trusted-ca-file and peer-trusted-ca-file pointing at that file (system/services/etcd.go); secrets/etcd.go makes the server, peer, admin and kube-apiserver client certificates from the CA with x509.NewCertificateAuthorityFromCertificateAndKey, which parses only the first PEM block of Crt (pem.Decode) and the key; kube-apiserver gets the same file as etcd-client-ca.crt (--etcd-cafile). Nothing restarts etcd when the files change, and nothing validates the CA's content beyond "present". Consequence: with a plain replacement of cluster.etcd.ca a member on the new CA and a member on the old one cannot talk to each other, so rotating the CA member by member loses quorum at the middle restart.

Decision

1. Targets (API)

Append-only: TalosOperation.spec.caRotation (TalosOperationCARotationSpec, optional) with targets []string (enum talos, kubernetes, etcd, kubernetes-aggregator; unique; at least one). Absent or empty means talos and kubernetes, so every existing operation, script and UI call is unchanged. The names are the ones status.certificates of the cluster already uses for expiry (ADR 0063). status.caRotation.certificates gets entries named like the targets, and two new status fields: etcdRestarted and etcdRestartedAt (below). The stage enum is unchanged. The wire gets repeated string ca_targets on CreateOperationRequest and PlanOperationRequest (the choice), PlanOperationResponse (the effective list, the default pair spelled out) and Operation (for the UI).

A separate operation kind was rejected: it would need its own lock handling, finalizer, machine-controller quiescing, pause/continue, rotation.yaml and bundle replacement, which are exactly what the existing machine provides, and a second kind could run beside a first one. One operation per cluster at a time stays true.

The targets are decided once, when the operation starts: the new CAs are generated for the targets and written to rotation.yaml (optional keys talos, kubernetes, etcd, aggregator; the set of keys present is the set of targets). A restarted controller, or a rotation started by an older Ichor, reads the targets from rotation.yaml, not from the spec.

2. Stages and rendered configuration

The stages and their meaning stay: Accepting (the new CA is trusted, the old one issues), Issuing (the new one issues, the old is trusted), Dropping (only the new one), Finalizing. Per target, in the typed documents:

The etcd rows rely on two facts read in Talos's source (above): the certificate field is written verbatim into the trust file, which etcd and kube-apiserver read as a PEM bundle, and the signing CA is the first block with the key. The first row therefore keeps the old CA signing while both are trusted; the second changes the signer while both are still trusted; the third drops the old one. Dropping and Finalizing render byte for byte what a bundle holding the new CAs renders (a test pins it), so the machine controller sees no drift afterwards. A target that is not rotated renders exactly as before.

Refused at the start (plan blocked and executor preflight): kubernetes-aggregator on a cluster whose version contract is older than v1.14 (no accepted list exists there). A two-member etcd, which would lose its quorum while one member restarts, needs no check of its own: the cluster lock already refuses every disruptive operation on an even number of approved control planes (ClusterTransitioning). An etcd rotation also refuses to start unless etcd is whole: every control plane reports etcd Running and healthy, one member per control plane, no alarm, kube-apiserver ready.

3. etcd: restart the members, one at a time, after each pass

Applying the configuration changes the files and the leaf certificates, but the running etcd keeps what it loaded, so after each of the three passes the controller restarts the etcd service of every control plane, one at a time, in the order the control planes are visited:

1. The restart starts only when every machine already runs the stage's configuration (the apply and the restart are separate reconciles) and only when the cluster is healthy: every control plane's etcd service is Running and healthy, its member list has one member per approved control plane, no alarm is set, its kube-apiserver is ready, and the node's etcd PKI is the stage's: the root CA the node reports (COSI secrets.EtcdRoot) holds the expected certificates in the expected order and its server certificate (secrets.Etcd) is signed by the expected CA. Otherwise the operation pauses (Blocked) and tries again every 30 s; nothing is restarted.
2. If the member leads, it forfeits leadership first (EtcdForfeitLeadership), then ServiceRestart("etcd").
3. The name goes into status.caRotation.etcdRestarted and the time into etcdRestartedAt; the next member is restarted only after etcdRestartSettle (15 s) and a new health check. etcdRestarted is emptied when the stage changes. A controller killed after the restart but before the status write restarts that member once more, which is harmless.
4. The pass is verified (as before: Ichor reaches each node, every Node is Ready) only after the last member has restarted and the final health check passes.

Because both CAs are trusted by every member from the first pass on, and every member presents a certificate either CA vouches for, a member on an earlier or later step of the roll can always talk to the others: during each roll at most one member is down, which is an ordinary rolling restart, quorum holds with three or more members, and a single control plane merely has its API unavailable while its only member restarts (the plan says so: kubernetes_api_unavailable). Three rolls are needed, not one: after Accepting every member trusts both CAs; after Issuing every member presents a certificate of the new CA (kube-apiserver then trusts both); after Dropping only the new CA is trusted. kube-apiserver reads its etcd client CA and certificate from the same Talos secrets, and Talos restarts its static pod on every change, so each pass also restarts kube-apiserver once per control plane; the existing Node-Ready verification covers it.

4. Aggregator: wait for the extension API servers, never restart them

kube-apiserver restarts on its own (Talos re-renders the static pod). Before Issuing, the controller reads extension-apiserver-authentication through the admin kubeconfig a Talos node issues (as the kube-root-ca.crt check does) and waits until its requestheader-client-ca-file holds the new CA, then waits ClientSettleDelay (2 minutes; the clock is a condition transition time, so it survives a restart) so that extension API servers that reload the ConfigMap have the new CA before the front-proxy certificate is signed by it. Ichor restarts none of them (they are workloads such as metrics-server; ADR 0101): the plan and the completion message say that an extension API server that read the CA only once must be restarted by the owner. The wait is Verifying-stage of Accepting, not a new stage.

5. Credentials hand-off, secrets bundle, backups

• AwaitingCredentials exists for the operator's talosconfig and kubeconfig (ADR 0091). When neither talos nor kubernetes is a target nothing the operator holds changes, so the stage is skipped (Accepting goes to Issuing) and the CLI/UI do not ask for a hand-off. CARotationHandoffToken is unchanged in form; it also names etcd and aggregator entries when present.
• Bundle. Certs.Etcd and Certs.K8sAggregator are replaced in the one Secret update that replaces the others (Rotation.Apply replaces what rotation.yaml holds and nothing else). rotation.yaml holds the keys of the new CAs from before any node is told anything, as in ADR 0063. A key that is not in rotation.yaml is never touched.
• BundleBackupStale. Raised as before. BundleFingerprint (Talos and Kubernetes CAs) is unchanged and is still what the credentials code compares; a new ExportFingerprint adds the etcd and aggregator CAs, and bundle-export and the cert-expiry reconciler use it, so an export made before an etcd-only rotation does not count as current. Cost: an export annotation written by an earlier Ichor no longer matches, so a cluster that has BundleBackupStale up at the moment of the upgrade needs one more bundle-export.
• etcd backups. The backup key derives from Cluster.Secret and Cluster.ID (ADR 0063, ADR 0108 section 4), which a CA rotation does not touch; snapshots contain no PKI. Snapshots taken before the rotation stay readable and restorable (a restore renders the configuration with the then-current CAs). TestPassphraseSurvivesCARotation still pins it, and the controller test asserts the cluster ID and secret are unchanged. The secretbox key stays out of scope (ADR 0121).
• A fresh snapshot first. The plan warning for an etcd rotation tells the user to take an etcd snapshot before starting (ichor etcd backup now NAME). Ichor does not require one: a missing backup is the user's decision for their cluster, not something Ichor forces (the "warn, do not force" rule for the managed cluster).

6. Failure and rollback

Every step is idempotent and re-derived from the nodes and from status (ADR 0063): a failed apply, a node that does not answer, a Node not Ready, an etcd member that does not come back, a refused dry run: the operation is Paused with CARotation=False/Blocked and a message, the next stage is never started, and it retries every 30 s. No member is restarted while any is unhealthy, so at most one member is down at any time. A rotation is never rolled back by Ichor automatically; it only moves forward. Until Dropping has started, every member and kube-apiserver still trust the old CA, so a rotation that cannot proceed can be abandoned by hand: remove rotation.yaml from the bundle Secret and the operation's finalizer (ADR 0063), after which the machine controller renders the old CAs again; the members pick that up at their next restart (expected, not exercised on hardware). After Dropping started, the old etcd CA is gone from the nodes that were already rolled: continue, do not revert.

7. UI and CLI

• CLI: ichor cluster rotate-ca NAME [--ca talos,kubernetes,etcd,kubernetes-aggregator] [--all]; default unchanged. --yes, --follow as before. The plan lists the targets, the stages, the etcd restarts and the warnings; the credential hand-off runs only when talos or kubernetes is a target.
• UI: the start dialog for a CA rotation shows four checkboxes (Talos API and Kubernetes API checked, etcd and aggregator unchecked) and sends ca_targets; the plan view names the CAs, the etcd restarts and the warnings and shows the credentials alert only when talos or kubernetes is a target; the operation page lists the CAs and the etcd members restarted in the stage (Operation.ca_targets and the details), and the credentials panel (CARotationCredentials) appears only for rotations that include talos or kubernetes.
• The cert-expiry message no longer says the two CAs "are not covered".

Rejected

• Replacing cluster.etcd.ca in one pass (all members at once, or one by one). At once is an outage of the whole control plane; one by one, with a single CA per member, splits the cluster by CA at the middle restart.
• Issuing etcd leaf certificates from the new CA and keeping the old trust through a separate file. Talos has no field for it and Ichor may not write raw files (machine config is typed only, no raw escape hatch).
• Treating the etcd CA as unrotatable and documenting it. The user decided it ships in v0.1.0.
• A restart of every member at the same time per pass, or no restart in Accepting. The certificates are loaded at start; a member that never restarted after Accepting would not trust the new CA when Issuing makes the others present it.
• A hard requirement for a recent etcd snapshot. Warned about, not enforced (above).
• Restarting extension API servers. Not Ichor's workloads (ADR 0101).

Risks

• The etcd bundle depends on Talos behaviour that is not in its documentation (first PEM block signs, whole field is the trust file, etcd and kube-apiserver read bundles). If a Talos release changes it, the etcd target would split the cluster; Ichor's own check (the node's secrets.Etcd is signed by the expected CA, the trusted list is the expected one) pauses the rotation before any restart if the node does not behave as read. The target is therefore also the one to run first on a disposable cluster.
• Three etcd rolls and three kube-apiserver restarts per control plane; a one-control-plane cluster has its Kubernetes API down during each member restart.
• Extension API servers that read the aggregator CA once need a restart by their owner.
• ClientSettleDelay adds two minutes to a rotation that includes the aggregator CA.

Test

envtest (internal/controller, a fake node per machine as in ADR 0063): the stage sequence, rendered documents per stage and role for etcd and aggregator (typed: cluster.etcd.ca certificate order and key, KubeAggregatorCAConfig), Dropping equal to the replaced bundle's render, rotation of only etcd skipping AwaitingCredentials, etcd members restarted in order with the health gate and leader forfeit, each member restarted exactly once per stage with a new reconciler at every step, an unhealthy member holding the roll, a node that has not yet loaded the stage's etcd PKI holding it, a missing member and two control planes refusing the start, a legacy contract refused (render), the aggregator wait, rotation.yaml keys and the replaced bundle, backup key unchanged; pkg/render tests per stage and role; CLI flags and plan output; UI selection and card.

Hardware verification

Not run. Steps (3 control planes and 1 worker, contract v1.14, a throwaway cluster; kubectl --context admin@talos-cluster01 for the outer cluster only):

1. ichor etcd backup now NAME and wait for the snapshot. Note talosctl etcd members and a ConfigMap written before the rotation.
2. ichor cluster rotate-ca NAME --ca etcd --yes --follow. Expect: no AwaitingCredentials; after Accepting, talosctl -n <cp> read /system/secrets/etcd/ca.crt holds two certificates (old first); after Issuing, new first; after Dropping, one (the new). Each roll: only one etcd service restarts at a time (talosctl service etcd on each), talosctl etcd members never loses a member, kubectl get --raw /readyz recovers within a minute per control plane.
3. After Done: talosctl etcd status on every control plane, check talosctl read /system/secrets/etcd/etcd.crt | openssl x509 -noout -issuer -fingerprint against the new CA and talosctl get etcdrootsecret -o yaml (public part). The ConfigMap from step 1 is still readable; an etcd backup taken after the rotation succeeds; an older snapshot is listed and its key still decrypts it.
4. ichor cluster rotate-ca NAME --ca kubernetes-aggregator --yes --follow on a cluster with metrics-server: kubectl top nodes works before, after Accepting and the settle wait, after Issuing and after Done; kubectl get cm -n kube-system extension-apiserver-authentication -o jsonpath='{.data.requestheader-client-ca-file}' holds two certificates, then one.
5. ichor cluster rotate-ca NAME --all --yes --follow once on a fresh cluster, then ichor cluster bundle-export NAME clears BundleBackupStale.
6. Negative: kill the controller pod during an etcd roll; the roll resumes, no member is restarted twice in a row without a health check. Power off one control plane before starting; the operation refuses to start.
7. If step 2 shows a member that does not trust a certificate signed by the other CA, stop: the etcd bundle assumption is wrong for that Talos build; do not use the etcd target and record the Talos version.

Amendment (hardware, 2026-11-05)

Lane A ran the rotation on a compact HA cluster (3 control planes and 1 worker, Talos v1.14.5, Kubernetes v1.36.6, etcd 3.7.3, Ichor main ecb01c33). The aggregator rotation worked; the etcd rotation did not, and three smaller defects showed. This amendment replaces what it names and leaves the rest of the ADR as written.

1. etcd members load the CA with a reboot, not a service restart (replaces section 3)

Observed. --ca etcd paused in Accepting: restarting etcd on "<machine>" failed: rpc error: code = Unknown desc = service "etcd" doesn't support restart operation via API. talosctl service etcd restart and stop fail the same way on v1.14.5. The Accepting configuration itself applied without a reboot (/system/secrets/etcd/ca.crt went from one to two certificates on all three control planes and etcd kept running), and afterwards Talos reported RebootRequired for cluster.etcd.ca.crt.

Documentation (Talos v1.14). Edit machine configuration, "Changes which might need a reboot": ".cluster.etcd: the etcd service reads its settings ... when it starts. Talos does not restart etcd on a configuration change, since restarting a control plane member is disruptive and is not something Talos does implicitly. Reboot the node to pick up the new settings." The same page says machine configuration "is always applied to the running node without a reboot" in v1.14, that --mode=reboot was removed in Talos 1.14, and that a change which needs a reboot is applied first and the node then rebooted explicitly "so it can be sequenced with cordoning and draining". etcd maintenance repeats it for the etcd image: "Talos does not restart it on configuration change, so reboot the node." So a reboot is the only supported way for a running member to load a new cluster.etcd.ca, and there is no apply mode that applies and reboots in one call: the existing NO_REBOOT apply of the pass followed by an explicit, gated reboot is the design the documentation describes.

Decision. After each of the three passes, once every machine runs the stage's configuration, the controller reboots the control planes one at a time. The roll is the node action's reboot (ADR 0085) run inside the rotation, which holds the cluster lock (a child TalosOperation could not take it):

1. Before any reboot, and again before each next one: every control plane's etcd Running and healthy, a leader seen, one member per approved control plane, no alarm, kube-apiserver ready, the node's etcd PKI the stage's (the same checks as before), and every Kubernetes Node Ready. Otherwise the operation pauses (Blocked) and nothing is rebooted.
2. For the next control plane in order: the storage evacuation hook (nil when none is wired), then cordon and drain with the helper the config rollout uses (KubeClient, skipped when none is wired), a leader forfeits leadership (best effort), the boot ID it leaves is written to status.caRotation.etcdRebootBootID and its name to etcdRebooting, and only then Reboot is called. A controller killed between the status write and the call finds the marker and waits; if the node still reports that boot after etcdRebootRetry (5 minutes) the marker is cleared and the request made again. A failed Reboot call clears the marker, uncordons and retries.
3. While etcdRebooting is set the pass does nothing else (it does not dial the node to read its configuration) and waits, InProgress with reason RebootingEtcd, for: the node to answer on a different boot ID, every Node Ready, and the same health and PKI gate as in 1 on all control planes. Past nodeActionTimeout (20 minutes) the operation pauses. Then the node is uncordoned (storage evacuation ended), its name is appended to etcdRestarted (the name of the field is kept: it is append-only; it now lists the control planes rebooted in the stage) and etcdRebooting is cleared. etcdRestartedAt records when the reboot was requested.
4. etcdRestartSettle is gone: the gate after a reboot replaces a fixed wait.

New status fields, append-only: status.caRotation.etcdRebooting and status.caRotation.etcdRebootBootID. The reason RestartingEtcd became RebootingEtcd.

Cost and risk (replaces the corresponding lines of Risks). Three reboots of every control plane per etcd rotation (one per pass), each with a drain, one node at a time; a rotation of etcd alone on three control planes is nine reboots. The plan (CLI and UI) and ichor cluster rotate-ca --help say so. A cluster with one control plane has no Kubernetes API during each of its three reboots, and the drain of its only control plane evicts what runs there. A reboot is longer than the restart this replaced, and the node comes back with the stage's configuration loaded from disk (the pass applied it with NO_REBOOT, which writes it to the STATE partition).

Still unverified. The assumption that both CA certificates in one cluster.etcd.ca.crt are trusted, with the first signing, was not reached (no Issuing stage ran). The check that stops before any reboot unless the node reports the stage's PKI (secrets.EtcdRoot and secrets.Etcd) stays, and the hardware run below must still end the question. A member that does not trust a certificate signed by the other CA now shows as a rebooted node whose etcd does not become healthy: the roll stops at that node, pauses, and reboots no further one.

2. Deleting a rotation before its point of no return abandons it

Observed. ichor operation delete on a Paused ca-rotation set deletionTimestamp, but the operation stayed Paused with its finalizer, the machines stayed Ready=Unknown (CARotationInProgress) and the cluster lock stayed held; the agent removed rotation.yaml and the finalizer by hand. The finalizer step only released an operation that was no longer under way, and an operation paused mid-rotation is under way for ever.

Decision (ca-rotation and key-rotation, which share the pattern). A delete of a rotation that is under way is decided by RotationAbandonable (api/v1alpha1):

For an abandonable rotation the finalizer step, instead of waiting for a rotation that will never settle: removes rotation.yaml (keyrotation.yaml) from the cluster's bundle Secret (the one object a rotation writes; secrets.yaml was not touched before Finalizing), uncordons the control plane that was being rebooted (named by spec.hostname or status.nodeName, as the node may be down), releases the cluster lock if the operation holds it (Release is a no-op for another holder), records an Event on the cluster, and removes the finalizer. Every step is idempotent and retried as a whole. The machine controller then renders the old material again: the nodes still carry the extra accepted CAs (and for etcd the two-certificate field), which Talos classifies as reboot-requiring, so the machines report Ready=False/RebootRequired and ichor config apply finishes the revert (3 minutes for three control planes on hardware, as the hand-made abandon did). Ichor does not apply it by itself: it never reboots a node on its own.

For a rotation past the point: the controller sets the operation condition DeletionHeld=True (reason PastPointOfNoReturn, the stage in the message) once and goes on; the finalizer is released when the rotation is Done. DeleteOperation in the UI API (CLI and UI) refuses it up front with FailedPrecondition and the stage. A webhook on DELETE was rejected: it would put every TalosOperation delete behind a failurePolicy: Fail admission call, and the condition plus the API refusal reach the same people (a raw kubectl delete still ends in a held, explained delete).

3. The old aggregator CA leaves extension-apiserver-authentication

Observed. After Done the ConfigMap's requestheader-client-ca-file still held both CAs although aggregator-ca.crt on every control plane held one: kube-apiserver merges and never removes. Deleting the ConfigMap made it come back with one certificate and kubectl top kept working.

Decision. In Finalizing, after every machine is verified against the replaced bundle, AggregatorAuthReset reads the ConfigMap through the admin kubeconfig a Talos node issues for the call (the credential the other CA checks use; there is no standing credential and no RBAC of Ichor's own for the managed cluster, so no ClusterRole rule is added and the chart is unchanged). If it holds exactly the new CA the step is settled; if not, it deletes the ConfigMap by name with a UID precondition (a ConfigMap kube-apiserver has recreated meanwhile is not deleted again) and reports not settled; a missing ConfigMap is not settled either. The operation stays InProgress (SettlingAggregator) and the check repeats every 30 s; it completes only when the ConfigMap holds the new CA alone. Extension API servers that read the ConfigMap once still need a restart by their owner (unchanged).

4. The credential hand-off waits for the Kubernetes API

Observed. With a single control plane endpoint the hand-off failed once with connection refused while kube-apiserver restarted; the operation kept waiting and ichor cluster credentials --acknowledge fixed it. Only the final annotation patch retried transient errors.

Decision. Every API call of the hand-off (reading the secrets bundle, listing the operations, verifying the new kubeconfig) goes through retryHandoff, the CLI's existing transient-error retry (retryTransient: backoff from 1 s to 15 s for up to 10 minutes), and prints once that it is waiting for the Kubernetes API. A non-transient error (a certificate that does not authenticate) still stops at once and the operation is not acknowledged.

Tests

envtest: the roll by reboot (order, one reboot per control plane per stage, no ServiceRestart ever, unhealthy member holds the roll, a node that does not answer is waited for, a boot ID that does not change is waited for and re-requested after etcdRebootRetry, the 20-minute pause, leader forfeits before its reboot, a Node not Ready holds the roll, cordon and uncordon around the reboot); abandon (ca-rotation paused in AwaitingCredentials, Issuing of the Talos CA, mid-reboot uncordon, held past Dropping with DeletionHeld and released at Done; key-rotation in Accepting abandoned and in Issuing held); the aggregator reset in Finalizing and the reset step itself against a fake ConfigMap client (by name, UID precondition, missing, settled). api/v1alpha1: RotationAbandonable. uiapi: the delete refusal. cli: the hand-off retries and gives up after the window.

Hardware verification (replaces steps 2 and 6 above; the rest stands)

2. ichor cluster rotate-ca NAME --ca etcd --yes --follow on three control planes: no AwaitingCredentials; each pass reboots the control planes one after the other (kubectl get nodes shows one cordoned node at a time), talosctl etcd members never loses a member, kubectl get --raw /readyz recovers per node; after Accepting ca.crt holds two certificates (old first), after Issuing two (new first), after Dropping one. The first Issuing reboot answers the open question (a member that trusts a certificate signed by the other CA): if a rebooted node's etcd does not become healthy the roll stops there; record the Talos version and do not use the etcd target.
3. Kill the controller pod between the status write and the reboot, and during a reboot: the roll resumes without rebooting a second control plane meanwhile. ichor operation delete on a rotation Paused in Accepting: the operation disappears, rotation.yaml is gone from the Secret, the lock lease is gone, the machines show RebootRequired, ichor config apply finishes. The same in Dropping is refused by the CLI.
4. After an aggregator rotation: kubectl get cm -n kube-system extension-apiserver-authentication -o jsonpath='{.data.requestheader-client-ca-file}' holds one certificate when the operation reports Completed.
5. With one control plane endpoint, start rotate-ca (default CAs) from the terminal: a refused connection during the hand-off prints the waiting line and the hand-off completes by itself.

Amendment (lane E re-verification, 2026-11-05): webhook failures in the hand-off, the etcd plan wording, and the aggregator restart

1. The hand-off retry (point 4) treated a refused webhook connection as transient but ended on failed calling webhook ...: context deadline exceeded, which arrives as a 500 InternalError whose text does not mention a refused connection. Any failed calling webhook error that is not a Forbidden or Invalid refusal (a webhook that answered "no" stays an answer) is now retried for the whole 10-minute window.
2. The CLI plan of an etcd rotation no longer says "without a reboot": it says the control planes reboot in each pass (the workers do not).
3. A rotation of the aggregator CA turned Completed about 13 s before the last kube-apiserver restart on the endpoint control plane ended: a pass applied the stage to every machine and verified only the Talos API and the Nodes, but Talos restarts kube-apiserver some seconds after the apply. A pass now records, per control plane it applied to, the kube-apiserver container id read just before the apply (status.caRotation.apiServerInstances, with appliedAt; both additive and emptied on every stage change) and does not verify the stage until each of them runs in another container and answers /readyz. A container that did not change within a minute of the apply (the stage changed nothing kube-apiserver loads) is taken as not restarted; without a recorded id (the controller died between the apply and the status write) only /readyz is asked. /readyz is not asked when the Kubernetes CA is rotated too, because the probe verifies against the bundle's old Kubernetes CA. Rotations that do not target the aggregator CA are unchanged.

Amendment (lane F, 2026-11-05): discovery failures in the hand-off retry

On hardware the hand-off retry still ended after 11 s with unable to retrieve the complete list of server APIs: ichor.maesh.dev/v1alpha1: no matches for ichor.maesh.dev/v1alpha1, Resource= while the endpoint control plane's kube-apiserver came back and did not yet serve the CRD group: the controller-runtime client's REST mapper failed discovery, an error isTransient did not know. isTransient now matches by type meta.IsNoMatchError, *apiutil.ErrResourceDiscoveryFailed (what the lazy REST mapper returns) and discovery.IsGroupDiscoveryFailedError, so these are retried for the whole 10-minute window; Forbidden, NotFound, Invalid and webhook refusals stay final. The same client may be used again: the lazy REST mapper keeps no negative cache (a failed lookup leaves the group and version out of its cache, and the next call asks the server again), so the client is not rebuilt per attempt; the existing redial after a rotated kubeconfig is unchanged.

Amendment (2026-11-05): an early hand-off acknowledgement does not skip the gate

CredentialsHandoffAnnotation holds CARotationHandoffToken(status), the fingerprints of the new CAs, which status.caRotation.certificates shows from stage Accepting on. The controller read the annotation only in AwaitingCredentials, but an annotation set earlier (by hand, or by a client that computed the token from Accepting) already matched when the operation got there, so the gate was passed in the same reconcile and the operator never had a chance to hold working credentials. When the controller enters AwaitingCredentials (the pass that verifies Accepting) it now removes whatever annotation is on the operation, with a merge patch through a copy and the new resource version carried over, so a change made after the drop fails the status update and is dropped on the next pass. An annotation set while the operation waits is honoured as before; the CLI and the UI set it only after they see the stage. The remaining trust is unchanged (ADR 0129): the annotation is an attestation, and the controller checks only the token. Test: TalosOperation ca-rotation ignores an acknowledgement given before the stage AwaitingCredentials.