0091. A CA rotation waits for the operator's new credentials before anything they hold stops working

Status: Accepted · Date: 2026-10-27 · Design section: Backup, recovery, certificates; Trust · Supersedes in part: 0063 (stage sequence, "How Ichor stays connected" covers Ichor only), 0068 Decision 7 (the CLI start) · Amended by: 0093 (the UI download)

Decided: the ca-rotation stages are Accepting, AwaitingCredentials, Issuing, Dropping, Finalizing, Done; AwaitingCredentials changes nothing on any node and the controller leaves it only when the operation carries the annotation ichor.maesh.dev/credentials-handoff with the value talos=<new talos CA fingerprint>,kubernetes=<new kubernetes CA fingerprint> (ichorv1alpha1.CARotationHandoffToken); the CLI sets it, the controller only reads it and never writes spec; ichor cluster rotate-ca waits for the gate through the Kubernetes API, then writes a kubeconfig and a talosconfig that trust both the old and the new CAs and are signed by the new ones, checks that the new kubeconfig authenticates to the cluster, and only then sets the annotation; ichor cluster credentials NAME writes the same files from the cluster (--acknowledge also sets the annotation) or, with --bundle FILE, offline from a bundle-export file; rotate-ca --follow follows through the Kubernetes API with the new kubeconfig, not through the ichor-ui tunnel; the UI offers no download here, which ADR 0093 reverses: it adds an admin-gated download and a continue button

Context

The first real rotation (3 control planes, 1 worker) locked the operator out. Within seconds of Issuing the kube-apiserver served the new CA and the operator's kubeconfig, which trusts the old one only, failed with x509: certificate signed by unknown authority. The nodes served certificates from the new Talos API CA from Issuing on, so the operator's talosconfig, which trusts the old Talos API CA only, was refused on every node from that pass on, not only after the old CA was dropped (about two and a half minutes later, at Dropping). Both files therefore stop working from the second pass (Issuing). Nothing the operator held reached the cluster afterwards, bundle-export needs a working kubeconfig, and nothing in Ichor handed the operator new credentials. talosctl rotate-ca does: it writes a new talosconfig and builds a new kubeconfig from the nodes during the rotation (Talos v1.14.3, pkg/rotate/pki/kubernetes fetches a client after the swap, cmd/talosctl/cmd/talos/rotate-ca.go saves the new talosconfig). ADR 0063 ran the rotation in the controller and left that part out. Its --follow also printed API unreachable, retrying (dial tcp [::1]:<port>: connection refused) every 15 s from the start: the port-forward to ichor-ui goes through the kube-apiserver, which restarted when it was given the new CA list, the tunnel died, and Reconnect (which does exist, follow_retry.go) failed every time without saying so (the old kubeconfig was refused by the same API server) so the failure looked like a hang.

Decision 1: the gate

After Accepting is verified on every machine, every node accepts both CAs for the Talos API and the Kubernetes API and the old CAs still issue, so every credential the operator holds still works and a credential signed by the new CAs works too. That is the only moment in the rotation where both are true, so it is where the operator gets the new ones. The stage AwaitingCredentials follows Accepting; the operation is Paused with the CARotation condition False/AwaitingCredentials (its message names ichor cluster credentials NAME --acknowledge), requeued every operationRequeueInterval like every other unverifiable step, and it applies nothing. There is no way to skip it: no flag, no annotation value that matches by accident (the token is the new CAs' fingerprints, so an acknowledgement left from another rotation does not release this one). A --skip-credential-handoff was considered and not built: an operator who does not want credentials still has to say so by running the acknowledge command, and nothing in the product needs the unattended path (no e2e covers rotation yet; a harness can set the annotation itself). If one is wanted later it must be an explicit field on the operation that the plan states; it must not be a default.

The wait is not a timeout and not a failure: it holds the cluster lock and the finalizer like any stage, the machine controller stands aside, and nothing is half-done, because a cluster in AwaitingCredentials is in the state Accepting left it.

Decision 2: the dual-CA files

ichor cluster credentials and rotate-ca write two files, mode 0600 (directory 0700 if created), through the atomic writer of bundlefile.go, never printing their contents:

• kubeconfig. The server, cluster, context, user and namespace names are copied from the current kubeconfig (so --kubeconfig, $KUBECONFIG and the default rules behave as for every other command); certificate-authority-data holds the new Kubernetes CA followed by the old one; the user is a client certificate with CN admin, O system:masters, signed by the new Kubernetes CA (key type follows the CA, one year). Only the current context is kept.
• talosconfig. One context named after the cluster; ca is the base64 PEM bundle of the new and the old Talos API CA; crt/key an os:admin certificate generated with Bundle.GenerateTalosAPIClientCertificate from a copy of the bundle whose OS CA is the new one. Endpoints are --talos-endpoints, else the control-plane machines' DialEndpoint(), else the current talosconfig's.

Default paths: the current kubeconfig's and talosconfig's path plus .ichor-new; --kubeconfig-out and --talosconfig-out choose others. The command refuses an output that is the current file, so a failure never costs the operator the credentials that work. Outside a rotation (no rotation.yaml) the same command writes files that trust the one current CA, which is what to run after the rotation completes.

Why this works with the Talos trust model. A kubeconfig may carry several CAs in certificate-authority-data; the client accepts a server certificate chaining to any of them, so it works against the old serving CA (before Issuing) and the new one (after). The Kubernetes API accepts client certificates signed by the CAs in cluster.acceptedCAs, which Accepting added; the Talos API does the same for machine.acceptedCAs. That is the claim the gate rests on, and Ichor does not take it on trust: before acknowledging, the CLI asks the API server who the new kubeconfig is (a SelfSubjectReview with the new files), and does not set the annotation if that fails, so a Talos build where an accepted CA does not authenticate clients leaves the rotation waiting, with the old credentials intact, rather than stranding the operator. The same check against a node's Talos API with the new talosconfig is made too, but only warns when it fails: the operator's machine often cannot reach the nodes except through the Kubernetes API, and the Kubernetes check is the one that guards the file the rotation cannot continue without. (ponytail: neither has run against real hardware yet; the first run on a disposable cluster should read the check's result.)

Decision 3: what sets the acknowledgement, and where the CLI reads the CAs

The CAs' keys are in the bundle Secret (rotation.yaml, written before Accepting, ADR 0063). Making credentials needs those keys, so the command reads the Secret with the operator's own kubeconfig, as bundle-export does, and the keys never travel through the ichor-ui API or the browser. (ADR 0093 reverses this for the UI: an admin-gated download and a continue button.) For the same reason the UI had no download (ADR 0084's downloads are diagnostics; these are cluster-admin private keys): the start dialog and the operation page say that the credentials stop working, that the operation waits, and which command to run. Choosing the CLI over a gated download is the simpler choice and the safer one: nothing new crosses the network, the admin gate and audit of ADR 0084 are not extended to key material, and the only party that can release the gate is one who holds a working kubeconfig at that moment. The acknowledgement is an annotation on the TalosOperation (metadata, not spec), which needs patch on talosoperations that the operator already has to start the rotation.

rotate-ca always performs the hand-off, with or without --follow: a rotation started without it would otherwise wait forever. It waits for the gate by reading the operation through the Kubernetes API with the old kubeconfig (retrying the transient errors of an API server restarting for the accepted-CA change), writes and checks the files, acknowledges, prints where the files are and how to use them (KUBECONFIG=… TALOSCONFIG=…), and returns; with --follow it then watches the operation to its end with a client built from the new kubeconfig. If it is interrupted while the operation waits, ichor cluster credentials NAME --acknowledge does the rest.

Decision 4: --follow and the tunnel

The tunnel to ichor-ui is a port-forward through the kube-apiserver. It dies when the apiserver restarts, and it cannot be rebuilt with a kubeconfig the apiserver no longer accepts, which after Issuing is the operator's old one. retryTransient called Reconnect and dropped its error, so the failure repeated silently. Two changes: the error is now printed (ichor: could not re-open the tunnel to ichor-ui (...)) for every --follow/--watch, and rotate-ca --follow does not use the tunnel after the hand-off: it polls the TalosOperation through the Kubernetes API with the new kubeconfig (watchCARotation), which is independent of ichor-ui's pod and of its endpoint, and retries transient errors with the shared helper. The plain operation get --watch keeps the tunnel and the generic reconnect, now with its reason shown.

Decision 5: offline credentials

ichor cluster credentials NAME --bundle FILE reads a bundle-export file and needs no cluster: it writes the files for the single CA in the file. It exists for the operator who holds a current bundle (exported after the last rotation) and has lost every kubeconfig and talosconfig; the Kubernetes server comes from --kubeconfig (any kubeconfig naming the server) and the Talos endpoints from --talos-endpoints. A bundle saved before a rotation names CAs no node accepts any more: that is the reason BundleBackupStale exists, and the command does not pretend otherwise.

Consequences

• The CRD's status.caRotation.stage enum gains AwaitingCredentials; the chart copy is regenerated. OperationService and the proto are unchanged apart from the comment on ca_rotation_stage; the UI's caStageProgress counts six stages.
• The plan text (CLI and UI) states that the operator's kubeconfig and talosconfig stop working, that the operation waits for the hand-off, and how it is done. The destructive alert in the dialog is shown for every ca-rotation plan.
• A rotation started from the UI waits at the gate until someone runs the CLI. It cannot stall the cluster: the lock is held, the machine controller stands aside, as in any stage; Paused/AwaitingCredentials is the visible state.
• rotate-ca now needs the operator's kubeconfig to read the Secret and the operation directly; bundle-export already did. Where the operator has only ichor-ui access, the rotation waits.
• Not built (the UI download was added by ADR 0093): --skip-credential-handoff, writing the new files over the current ones, refreshing a kubeconfig that names several clusters (only the current context is written), and credentials for the etcd and aggregator CAs (not rotated, ADR 0063).
• ADR 0063's statement that "Ichor's dial" is the only client it keeps connected stands: the controller's bundle-based dial is unaffected; this ADR is about every other client the operator owns.

Amendment 2026-11-06: ichor cluster rotate-ca without --follow still stays until the hand-off is done (Decision 3 stands: the files and the acknowledgement need the operator's own kubeconfig, and the CLI is the one party that has it); it returns once the operation has gone on, in seconds when the nodes already trust the new CAs. Its help text and the --follow usage now say so. To start a rotation without the CLI, create the operation with kubectl or the UI and hand it off later with ichor cluster credentials NAME --acknowledge.