Ichor architecture

Status: living document; Ichor is alpha. This document is the source of truth for the project.

Summary

Ichor manages Talos Linux clusters through Kubernetes custom resources. A controller renders machine configuration from typed resources and applies it to nodes, walks upgrades through the cluster, backs up etcd, and handles the lifecycle of nodes from maintenance mode to removal. A CLI and a web UI expose the same operations.

Ichor stops at the Talos layer, with one component it installs itself: Cilium is the only CNI and Ichor's own Helm release. Storage systems, monitoring and any GitOps tool are the user's business; Ichor only prepares the Talos side for them and coordinates with them around node operations.

Motivation

Running Talos with talosctl works well for one person and a handful of nodes. It gets awkward when machine configs live in a directory of YAML files, when several people need to see what's applied where, and when upgrades have to be sequenced by hand. Putting the configuration into the Kubernetes API gives you RBAC, audit, webhooks, and a place to hang a UI, without inventing a new control plane.

Goals

• Declarative machine configuration as Kubernetes resources, with drift detection and a view of the effective config per node.
• Node lifecycle: discover a node in maintenance mode, approve it in a UI, install, join, and later remove it safely (drain, etcd membership, reset).
• Rolling upgrades of Talos and Kubernetes, with the cluster lock, etcd quorum and storage evacuation taken care of.
• etcd backup to S3-compatible storage, and recovery from nothing but the secrets bundle and a snapshot.
• Bootstrap of Cilium for a brand-new cluster, with no Git repository needed.
• CLI and UI parity: anything you can do in one, you can do in the other.

Non-goals

• Getting machines to maintenance mode. Boot the ISO however you like.
• Storage or monitoring, and any CNI other than Cilium.
• Workloads. Ichor manages the cluster (Talos, Kubernetes, Cilium) and not what runs on it, and it does not sync its own resources with a Git repository; GitOps of workloads, or of Ichor's resources, is the user's tooling.
• Replacing the storage system's own controller. Volumes, replicas and disk state belong to Longhorn (or whatever you run).
• Multi-cluster management. One Ichor manages one cluster, and the UI is built for that.
• Air-gapped environments. Nothing is built or tested for an installation with no route to public registries; registry mirrors exist for pull-through caches and private registries.
• IPv6, PXE provisioning. Later, maybe.
• Know about GitOps tools. Ichor does not detect GitOps tools, does not open pull requests and does not refuse a write because a tool tracks the resource.

Overview

Ichor runs inside the cluster it manages; ichor-ui reads nodes and never writes to them, and the controller is the only writer.

Three deliverables come out of one repository:

• ichor, the CLI. Bootstraps the first node, adopts existing clusters, and offers every operation the UI has. It is also the recovery path that does not depend on a running controller.
• ichor-controller (plus ichor-installer, a Job image). The controller is the only component that writes to Talos nodes. Two replicas, leader election.
• ichor-ui, a Go backend that embeds the Preact SPA. It reads and writes CRs, reads logs and state from the Talos API, and never writes to nodes.

api/ (CRD types) and proto/ (Connect definitions) are the shared contracts. Deliverables don't import each other; shared code lives under pkg/. The controller's entry point stays at kubebuilder's cmd/main.go, which the generated Dockerfile and Makefile are tied to; the CLI and the UI backend are added beside it. The Kustomize base under config/ is the source of truth for manifests, and the Helm chart in dist/chart is generated from it.

Data flow

Every write, from the UI, the CLI or a user's own tooling, is a change to a custom resource. The controller is the only thing that turns those into Talos API calls. This keeps validation, RBAC and audit identical regardless of how a change arrived. Reads come from two places: CR status (via the API server) and raw node state and logs (via the Talos API, read-only).

Long-running work — upgrades, hand-overs, topology changes — keeps its state machine in CR status, not in memory. A controller restart or a leader change resumes from where it left off.

Trust

The secrets bundle (Talos, Kubernetes and etcd CA keys) is the root of trust. It is generated locally by ichor init, stored under ~/.ichor/clusters/<name>/ with 0600 permissions as the user's backup, encrypted as an age file under a passphrase, and pushed into a Secret that becomes the source of truth from then on. CRs reference it; they never contain it. The passphrase comes from a file, an environment variable or, on a terminal, a prompt or one generated and shown once; nothing stores it. A plaintext file from an earlier version still loads, with a warning, and ichor secrets encrypt migrates it.

Only the controller holds a CA private key and so can make any Talos or Kubernetes identity. The UI backend holds none, and its Kubernetes role reads no Secret that holds one: for its own reads of the nodes (an os:reader certificate) and for the three admin-gated downloads (support bundle, etcd snapshot, packet capture) it makes a key pair in memory and asks the controller to sign it by creating a TalosCredentialRequest. The validating webhook checks the Kubernetes permission of the requester (the signed-in user, for a download) and the controller signs for the least Talos role the call needs (os:reader, os:etcd:backup, os:operator) and for ten minutes; the private key never leaves the backend. The browser holds no Talos credentials, and the backend has no method that changes a node. A fourth admin-gated download hands the operator the new kubeconfig and talosconfig of a CA rotation that waits in AwaitingCredentials; it is signed the same way, needs the verb download on talosclusters/credentials, is granted to nobody by default, leaves a Warning Event first, and lasts a year unless the operator asks for less. The backend may impersonate only the users and groups it is given by name: the built-in ichor: groups, the Ichor users the controller lists, and what the chart values name; never a system: group. The restore of an etcd snapshot into a live cluster is the controller's, not the UI's: the UI backend only lists the snapshots a TalosEtcdBackup reports and writes the operation. In the first release the UI is reached through ichor ui (a port-forward), which needs no https URL.

ichor-ui serves HTTPS only, on 8443, never plain HTTP to the network. The certificate is the operator's (ichorUI.tls.secretName) or a leaf from the Ichor serving CA that the controller keeps for every install. The controller publishes the CA, public part only, as ichor-ui-ca (ConfigMap and Secret) so an edge or a client with --ca-file can verify the backend; certificates are reloaded when their files change. With an operator's certificate, a handshake for the Service name still gets the Ichor certificate, so in-cluster clients verify ichor-ui by that name (2026-11-03).

Ichor's runtime settings (the sign-in lifetimes, the default drift policy) are optional fields of the IchorInstallation singleton, read live by the controller, the webhooks and ichor-ui, and written by the UI Settings page or ichor settings as the signed-in user. The webhook splits write rights over the subresources ichorinstallations/version and /settings, and requires a confirmation annotation to enable reclaim. A deployment can make them read-only with --settings-read-only.

Users sign in to ichor-ui in one of two ways, and the kube-apiserver accepts both, so RBAC and audit apply per user.

• An external OIDC provider, configured with ichorUI.oidc.*. Its users are named oidc:<name> in Kubernetes.
• Ichor's own users, off until ichorUI.users.enabled is set. A user is an IchorUser CR with a display name, an e-mail, roles and a disabled flag. It has no password field. Password hashes (argon2id) live in one Secret, ichor-user-credentials. ichor-ui is a small stateless OIDC issuer for these users, and the controller renders it into the kube-apiserver's configuration beside the external provider. Its users are named ichor:<name>.

Roles are plain Kubernetes RBAC. Three built-in ClusterRoles, ichor-viewer, ichor-operator and ichor-admin, are bound to the groups ichor:viewer, ichor:operator and ichor:admin. None of them gets secrets, impersonate or cluster-admin. The destructive operation starts (shutdown, CA rotation, key rotation, etcd restore, node reset and shutdown) need their own verb, held by ichor-admin only. So does setting or changing a field that names a Secret (update on talosclusters/secret-references and the like): the controller reads those Secrets with its own privileges, so an operator who may edit a cluster must not be able to repoint one.

A user's kubeconfig holds no secret. It runs ichor auth token as a client-go exec plugin. That command logs in through the browser and keeps a 5 minute ID token and a 12 hour refresh token on the user's machine. The browser never holds a Kubernetes credential; its token is for ichor-ui only. The browser keeps its session going with a refresh token held in sessionStorage (rotated on use, reuse revokes the family), and signing out, in the UI or with ichor auth logout, revokes it at the issuer's RFC 7009 endpoint /oidc/revoke.

Users never get an admin talosconfig, because no per-user Talos identity exists. Two paths can hand Talos admin credentials to a person, and both stay admin-only: the CLI hand-off and the UI download. A third hands out much less: ichor cluster talosconfig and a UI download give an operator or admin an os:reader talosconfig valid for one hour (twelve at most), signed through a TalosCredentialRequest like the backend's own reads; it needs the verb download on talosclusters/talosconfig-reader, and each request is an Event with the user's name.

Override: the secrets bundle keeps the admin client certificate (system:masters) and the Talos os:admin identity in ~/.ichor/clusters/<name>/. Neither depends on any login. A broken issuer, a stopped ichor-ui or a deleted user cannot lock the operator out. Keep this bundle offline once users exist. The guide is ../guides/users-and-login.md.

Exception: reclaim. Ichor's controller never holds a credential for a node of another cluster, with one exception, "Reset and add". A user who holds the reclaim verb on machinereclaims can paste a foreign talosconfig into the UI to reset one machine; there is no installation switch. The credential is scoped (the controller dials only the request's address and ignores the endpoints in the file), user-supplied, and expiring: it is a Secret of type ichor.maesh.dev/foreign-talosconfig with a TTL (default 15 minutes, at most 60), used for one reset, and deleted at Done, Failed or Expired; a sweeper deletes any such Secret older than the maximum TTL. It is the one credential in Ichor that is deleted by time rather than by an object's lifecycle. It is redacted from logs, Events, status, error text and the UI API's access log. The ClusterRole ichor-machine-reclaimer (create, get, list, update and the verb reclaim on machinereclaims) is not bound by default, but ichor-admin (group ichor:admin) holds the same verbs by default, reclaim included, and the ClusterRole is there for granting them to anyone else; ichor-operator and ichor-viewer do not get the verb; the webhook checks the verb with a SubjectAccessReview on create and on every write of the typed UUID, and status.audit keeps who requested and confirmed it. A ValidatingAdmissionPolicy limits what ichor-ui's ServiceAccount may create in ichor-system to that Secret type.

Approval uses Talos's insecure apply, which does not authenticate the node. A separate provisioning network is assumed; the docs say so. Ichor pins what it can: the fingerprint of the maintenance certificate seen at discovery (or at the first provisioning probe) is recorded in the machine's status, and the controller refuses to send the first configuration, which carries the CA keys, to a node that presents another one. An operator who has checked a changed fingerprint accepts it with the annotation ichor.maesh.dev/trust-maintenance-fingerprint.

Deployment

The default is self-managed: Ichor runs inside the cluster it manages. The controller and UI run with two replicas, a PodDisruptionBudget and anti-affinity, except on clusters with a single schedulable node, where the installer drops to one replica and no PDB (otherwise a drain never finishes).

Ichor's own users and login are off by default. Turning them on needs three things a default install lacks: an https URL that browsers and the CLI can reach (ichorUI.users.externalURL, set by ichor init --ui-url), a Talos v1.14 contract or later (earlier contracts can carry only one issuer), and an ichor config apply rollout of the control planes. ichor-ui is the issuer. The kube-apiserver reads discovery and keys from a ClusterIP listener on port 8444, signed by an Ichor-made CA, so the external URL does not have to be reachable from the control planes. Ichor installs no ingress. Expose ichor-ui with an Ingress, a Gateway or a LoadBalancer. ichor-ui is HTTPS only on 8443, so an edge re-encrypts to it and verifies the backend against ichor-ui-ca, or a plain L4 load balancer passes through the certificate the operator supplied with ichorUI.tls.secretName.

A Helm install gets an IchorInstallation created once by the controller with spec.deployment.manager: helm; Helm then owns upgrades and ichor upgrade ichor refuses.

Network policy. Cilium is guaranteed, so ichor-system gets a CiliumNetworkPolicy default deny plus one allow per flow, using Cilium's entities (kube-apiserver, host, remote-node, world) for peers that run on the host network. They cannot ship in dist/install.yaml because Cilium's CRD does not exist yet, so ichor init applies Ichor's after Cilium is Ready, the installer Job re-applies them on every upgrade when the CRD exists, and the chart renders them behind networkPolicy.cilium.enabled. The webhook and OIDC ingress rules also admit remote-node, because the apiserver's calls to a pod on another node arrive from the control plane's cilium_host address. No policy allows world egress: what lies outside the cluster (Talos nodes, S3 stores, registries, ichor-ui's OIDC issuer) is allowed by policies the controller writes from the objects and the Deployment that name it, with toFQDNs for names, and the controller's metrics port is open to a namespace only when the operator names one.

Verified on hardware: a host-network kube-apiserver reaches a Service ClusterIP; Talos applies the issuer document without a reboot but the kube-apiserver restarts (15 to 45 s without /readyz per node), which is why the rollout is one control plane at a time. Talos does not validate the document, so Ichor validates it before rendering.

Principles

1. Machine config is never stored in a CR. CRs hold the ingredients (versions, patches, per-node values); the controller renders on every reconcile.
2. Secrets are referenced, never embedded.
3. Status holds hashes and state, never rendered config.
4. The Talos version contract is pinned in the CR, so a library bump cannot silently change what gets applied.
5. The controller never writes spec, so it never fights whatever tool the user keeps the CRs in. (Automatic patch upgrades, off by default, keep this true: a separate ServiceAccount, ichor-auto-patch, which the controller impersonates and the webhook limits to a newer patch of the running minor, writes the two version fields as a delegated user, and the audit log shows it)
6. Disruptive operations (upgrades, reboots, CA rotation, topology changes) take a per-cluster Lease. One at a time.
7. Progress lives in status and resumes idempotently. (A live etcd restore cannot keep a status, since it rolls the API back; its marker in the backup storage is the exception)
8. CLI and UI are equivalent. The only CLI-only operations are the ones that happen before a cluster exists.
9. There is always a recovery path that needs no working controller.
10. Dangerous-operation checks live in shared code and are enforced by the UI backend and by webhooks, never only in the frontend.

API

Group ichor.maesh.dev, version v1alpha1.

There are no CRDs for installing CNI, storage or monitoring. Their Talos-side needs are expressed on TalosCluster, TalosMachine and TalosStorage.

Typed configuration

TalosConfigPatch.spec.documents[] holds the Talos config documents as typed fields, restricted to an allow-list of kinds. There is no raw-YAML field. The reasons:

• Machinery's types are Talos's own definition; we don't maintain a parallel catalogue that drifts.
• A CRD schema keeps an unknown field out of storage — by default the API server prunes it (logs a warning, persists the object without it) rather than rejecting the request, but either way the value a field's absence was meant to keep out never reaches a node.
• A newer machinery renders config for older Talos versions when a version contract is set. The reverse doesn't hold, so supporting a newly released Talos means bumping machinery.

The types cannot be embedded directly. Machinery is YAML-first and carries no JSON tags, so controller-gen refuses them outright, and several field types (ByteSize, CEL expressions, the netip wrappers in the network documents) have no exported fields to describe. Instead the mirror types are generated from machinery with a go generate tool, together with conversion in both directions and a round-trip test per kind; machinery stays the only place where the field catalogue lives.

A handful of fields need the generator's type-keyed substitution and drop rules to be overridden per field instead, because the type alone can't say what to do with them:

• Eleven fields are meta.Unstructured — Talos's own escape hatch for a pass-through to another project's schema (kubelet's configuration, an audit policy, a resource list, a seccomp profile). A type-keyed rule can only drop all eleven or none; a field-level override table drops five (controller-gen errors, or no single schema to mirror) and mirrors six as their upstream type instead: five with no new dependency, plus machine.kubelet.credentialProviderConfig, which takes one (k8s.io/kubelet, an official k8s.io/* staging module versioned in lockstep with Kubernetes, the same class of dependency the other five come from). Under a typed-only API an absent field is an inexpressible one, so a commonly wanted field being absent is a product gap, not a saved dependency — that's why this one took the dependency where the other five didn't need to. machine.kubelet.extraConfig still gets nothing: what blocks it was never a dependency but a float32, which controller-gen only emits under allowDangerousTypes — and that flag applies to the whole API package, so one field would relax the type safety of every CRD in it.
• Nine more fields are plain strings or certificate pairs, not meta.Unstructured, but carry cluster secret material — join tokens, CA key pairs, encryption keys. These are dropped unconditionally, by a separate table, for a different reason than the eleven above: not because they can't be represented, but because a TalosConfigPatch must not be able to overwrite the real values the secrets bundle generates, however narrowly the patch claims to be scoped.

Both are the follow-up.

No validating admission webhook rejects a TalosConfigPatch carrying a secret field. Dropping the fields from the schema is what keeps the value out of storage, and that holds whatever the caller does, so a webhook adds nothing to the security property and only improves the error message. In exchange it would put failurePolicy on the write path: Fail means a controller outage blocks every configuration change, Ignore means the guard is absent exactly when it is invoked. Ichor's own clients ask for strict field validation instead, so the CLI and the UI backend report the field loudly, and kubectl is strict by default. A client that opts out of strict validation gets pruning, which is the safe outcome anyway.

Ichor adds what the types can't carry — whether a field needs a reboot, whether it can only be set at install time, which Talos version introduced it — in a table keyed by field path. Unknown paths are treated as reboot-required.

adopt reads existing node configs with the same types, so an existing cluster imports as-is. Kinds outside the allow-list are listed, and the user chooses between extending the list and dropping them. Nothing is dropped silently.

Typed hardening options for the clusters Ichor creates, all off by default and rendered only when set: spec.security.secureBoot (Off, Preferred, Required) picks the SecureBoot installer for approval and for each Talos upgrade and holds the first configuration of a node without SecureBoot under Required; spec.security.kernelLockdown is checked against what each node reports; both surface as per-node and cluster conditions, and a node's state is read from its SecurityState, never changed in place. A reset with spec.node.wipeMode system-disk or all (a TalosOperation of kind node; the CLI has no flag for it) wipes more than STATE and EPHEMERAL so a node can be installed afresh from the SecureBoot ISO. spec.security.kubeletServingCertificates renders serverTLSBootstrap and the kube-apiserver's kubelet CA, and the controller approves each kubelet's serving-certificate request itself, only for that node's own names and addresses, read from the node.

Patch layers

Patches apply bottom-up:

1. System layer, generated by Ichor: CNI settings, kube-proxy, KubePrism, volume configs derived from disk roles, kernel modules and sysctls for storage presets, metrics exposure, control-plane taints.
2. Cluster, 3. role, 4. selector, 5. machine (per-node values and machine-specific patches).

Within a layer, priority orders them. The effective-config view in the UI shows which layer each value came from.

Validation

Webhooks enforce, among other things: control-plane count in {1, 3, 5}; no approving more control planes than configured; never deleting the last control plane; no schedulable: false with zero workers; CNI preset is cilium and immutable; Cilium values are validated and may not set Ichor-owned keys; kube-proxy mode changes only with an explicit annotation; disk roles that require reinstall are immutable after install; TalosCluster and TalosMachine can only be deleted with ichor.maesh.dev/allow-delete: "true" (so a prune by the user's own tooling can't cascade into node resets); removing an Ichor finalizer, resetting a node and restoring etcd need the override verb; Talos/Kubernetes version pairs and upgrade paths are checked, and a version the webhook admits is never a downgrade, and never a pair the compatibility table refuses.

Design

Bootstrap

ichor init --preset standard [--cni-tunnel vxlan|geneve] [--ui-url URL]

1. Generate the secrets bundle locally; nag the user to back it up.
2. Settle the shape of the cluster: topology preset, endpoint (a VIP is recommended even for one node, because changing the endpoint later means re-issuing certificates), disk roles for the first node.
3. Apply the rendered config to the first control plane. Its inline manifest carries only Cilium, the one component Kubernetes cannot start without: the chart pinned in the binary, rendered with Helm's Go SDK, no network at init time and the same bytes every run. Everything else is applied later through the API server, with every write retried for a few minutes on a transient error.
4. Bootstrap etcd, fetch the kubeconfig, store the bundle as a Secret, apply Ichor's install manifest.
5. Create the CRs and the IchorInstallation, then apply Ichor's NetworkPolicies once Cilium is Ready.
6. Wait for the CNIHandover condition on the TalosCluster. init shows its steps as a short list with a spinner on a terminal (one line per change in a pipe or a log), and the last reason when the wait runs out; the controller keeps working either way.

There are no flags to choose a CNI. Adding later nodes goes through the normal approval flow; init is the special case of approving the first control plane.

ichor adopt brings a running Talos cluster that Ichor did not create under management. It needs Talos v1.12 or later on every node it adopts and refuses an older or mixed cluster with "upgrade Talos first", writing nothing. Above that floor, a cluster is adopted as it runs, whatever its version against this Ichor's set: a version the compatibility table does not know at all is refused, and an incompatible pair is reported and adopted. It works in four phases, and nothing is applied to a node until the last one:

1. Plan (--plan stops here and writes nothing). Read-only inspection with the operator's own talosconfig: Talos and Kubernetes versions, the CNI and where it came from, the kube-proxy setting, KubePrism, inline manifests and each approved machine's live config. The versionContract is probed from the running version down to v1.12 and the one with the fewest unrepresentable paths is chosen (--version-contract overrides it).
2. Write, after consent. The bundle Secret, the TalosCluster (annotation ichor.maesh.dev/adopted), the TalosMachines (identity read over the authenticated API), the TalosConfigPatches and, if Ichor is not running, Ichor's install manifest. Every TalosMachine carries the annotation ichor.maesh.dev/hold-apply: the reconciler computes and reports drift (AdoptionHeld) and never applies.
3. Cilium takeover (see "Cilium"), skipped for an unmanaged CNI. A cluster with no CNI at all is different: its nodes are NotReady, so Ichor's own pods could not start. Adopt then installs Ichor's pinned Cilium itself, in process, as Ichor's own Helm release, and waits for the nodes to be Ready before it installs Ichor or writes any CR; the order is Cilium, Ichor, CRs, hand-over, release. The cluster is adopted as the managed cilium preset and --cni-unmanaged is refused there. This needs KubePrism on port 7445 on every node and a pod CIDR on every node.
4. Release of the hold, one machine at a time, control planes first, only after a live zero-drift check passes. The controller then applies the first render under the cluster lock like any rollout.

The old "zero drift or nothing" rule now sorts differences into three classes. Class A is representable: sysctl and kernel-module documents, and values that have spec fields (subnets, install disk); adopt stores them as patches. Class B is Ichor-owned (the Cilium inline entry, KubePrism, kube-proxy when replaced, removal of Talos's flannel); the first render changes them, so they are listed and need consent, and they do not block. Class C is everything else, including other inline or extra manifests; it blocks that machine, with the path list and a hint, and adopt leaves it untouched.

Secrets. --secrets-bundle FILE takes a talosctl gen secrets file. With --recover-secrets (and --recover-from ADDRESS, a control plane) adopt rebuilds the bundle from that node's machine config, read-only, and stores it under ~/.ichor/clusters/<name>/ with mode 0600. A wrong or partial bundle cannot get through, because it shows up as secret-field drift and no patch is produced. After adoption the controller uses a certificate minted from the bundle. The previous administrators' client certificates stay valid, so adopt prints, and the UI shows as a dismissible notice (condition CARotationAdvised), the advice to run ichor cluster rotate-ca. Nothing starts a rotation.

Ichor can also be installed with Helm: helm install of the chart in dist/chart (published as the Helm repository https://ichor.maesh.dev/charts and as oci://ghcr.io/maeshinshin/charts/ichor once a release tag is pushed) installs Ichor, not a cluster. ichor adopt then detects the Helm release, skips its own install step, never creates an IchorInstallation, requires Ichor's version to equal the CLI's, and registers the cluster. A Helm-installed Ichor is upgraded with helm upgrade. The chart refuses to install on a cluster with no pod network (preflight.cni); that one case is covered by ichor cni bootstrap, which runs the same Cilium step as adopt. The chart never ships Cilium. See docs/guides/install-with-helm.md.

Cilium

Cilium is Ichor's own Helm release. It boots from the inline manifest; once the DaemonSet is Ready the ichor-cni Job (installer image, hidden ichor cni helm apply, host network via KubePrism) adopts the bootstrapped objects as release cilium with TakeOwnership, and the controller then removes the inline entry (CNIHandover). Removing it matters: Talos only server-side applies bootstrap manifests and re-applies them on a Kubernetes upgrade, so a stale entry would roll Cilium back. The Job needs cluster-admin-equivalent rights, so the controller does not run Helm itself.

Values. Cilium's live values are Ichor defaults, then the user's layer in spec.cni.cilium.values, then Ichor-owned keys (the Talos-required ones, the KubePrism address, kubeProxyReplacement, tunnelProtocol). They are edited in the UI (Cilium page) or with ichor cni values get|set|edit|diff, validated by one function in the webhook, the backend and the controller, and applied by the Job. Changing the layer needs update on talosclusters/cilium-values, held by ichor-admin only (the chart can run privileged pods on every node), and the *.secretsNamespace.name keys are Ichor-owned, pinned to cilium-secrets. The operator runs one replica while the cluster has one node and the chart's two from then on, counted from the Nodes that exist, and a Helm Job that keeps failing with the same values is retried with a growing backoff.

Upgrades. A new Ichor release carries new pinned charts. Cilium is upgraded only when the user starts a cni operation (ichor cni upgrade, or the UI), at most one minor at a time, rolled back by Helm on failure. The cluster rollout runs that operation last, as its Cilium step.

Takeover. ichor adopt can make a Cilium that someone else installed (a Helm release, the Cilium CLI, a Talos inline manifest or an applied helm template) Ichor's release. Before anything is written it checks that the live minor equals the pinned chart's minor or is one lower, that ipam.mode is kubernetes (a cluster-pool Cilium is refused), that the tunnel protocol is carried over, and that Cilium and KubePrism are healthy on every node. The user values come from the newest Helm release record; where there is none (a Talos inline manifest or an applied helm template) Ichor's defaults are used and the diff shows what is missing. Ichor then renders the pinned chart with the effective values and compares every object with the live one. A difference needs --accept-cilium-diff, and a change to an Ichor-owned key on the running data plane needs --accept-cilium-changes (IPAM and the tunnel protocol are never accepted). The ichor-cni Job runs cilium.Apply with --takeover, so Helm installs with TakeOwnership. Once the rollout is Ready the foreign release record Secrets are copied into ichor-cilium-prior-release in ichor-system, read back byte for byte, and only then deleted (RemoveForeignReleaseRecord; never helm uninstall). An inline manifest entry is found by content, its name is recorded in status.cni.takenOverInlineManifest, and it is removed from the machine config only after CNIHandover is true. There is no rollback of an adoption.

Git, override and recovery

A user may keep Ichor's CRs in their own Git and apply them with any tool. A write from the UI or CLI lands in the cluster and the user's tool decides what to do with the drift. Putting runtime-written state in Git (TalosOperations, unapproved machines, the labels and annotations Ichor writes on Nodes and on the storage system's node objects) makes that tool and Ichor fight, so keep those out. Ichor writes no repository skeleton.

Override is the Kubernetes RBAC verb override. It guards the dangerous operations: removing an Ichor finalizer from a deleting object, resetting a node, restoring etcd and overriding the last-worker guard. It is checked with a SubjectAccessReview made as the user.

Upgrades always run through the installer Job. Its CRD check lets only the retired fields disappear and refuses every other removed field.

The CLI keeps a way out of the corners: ichor upgrade ichor VERSION --local (no ichor-ui) and ichor restore --recover-cluster.

Nodes

Discovery starts from a node in maintenance mode: either registered by IP or found by scanning a subnet for the Talos API port. Scanning really is the only option — Talos offers no beacon, mDNS or announcement to listen for — and it is tractable because an unconfigured node brings up a DHCP client on every physical link by itself, so it has an address before anyone looks for it.

The controller reads hardware identity, disks and NICs over an insecure connection and creates a pending TalosMachine. The identity is the tuple of SMBIOS UUID, serial number and the permanent MAC addresses, not the UUID alone. Talos does not trust the UUID either: it ships two META keys to work around it, and the doc comment on one of them says plainly that the UUID "may repeat or be filled with zeros". Both need an admin-level connection to write, which a maintenance-mode connection never has — it is read-only by construction — so at the one moment Ichor keys a new resource off this identity, it has no way to detect or repair a collision from the node's side. The MACs are in the tuple because they are hardware-level, present when the UUID is not, and survive a reset.

So Ichor does not auto-register on a weak identity. An empty UUID, or one that already maps to a live TalosMachine, produces a pending entry that an operator has to resolve — the same machine returning, or a different one — with the address, MACs, serial and manufacturer shown to decide on. Writing a disambiguating token to the node is not a fix: it needs an authenticated connection Ichor only has after approval, and a reset wipes META anyway. The durable record is Ichor's own TalosMachine.spec.identity, which is consistent with the controller owning durable state rather than the node.

A machine that already runs a configured Talos cannot be discovered this way: its API demands a client certificate. Discovery reports it as AlreadyConfigured (an Event, and in the add-by-IP response) and creates nothing. Joining it needs a reset of STATE and EPHEMERAL, which is called a reclaim and is never automatic. Both paths run the same code (pkg/reclaim) and the same refusals: a machine that is already an Ichor machine or cluster, a control plane whose removal would break its cluster's etcd quorum, and the last control plane of a reachable cluster (--force overrides the last two with a second confirmation). The reset is graceful unless the foreign cluster is dead (--force-ungraceful). It sends the request, built by one shared function; user disks are not touched but are listed, and encrypted ones are called out. Consent is the node's UUID, typed. After the reboot the machine answers in maintenance mode with the same UUID and the normal add-by-IP flow registers it as pending.

A machine that already carries an Ichor cluster's own CAs (it joined with the user's tooling after an adoption, or its TalosMachine was deleted) needs no reset and is refused by reclaim. ichor adopt --cluster NAME --add-machine ADDRESS registers it: the adoption's read-only plan, three classes of drift, held TalosMachine and zero-drift release, for the listed addresses only, with a check that the machine's own configuration names this cluster's ID and Talos API CA. The UI has no button for it yet.

• CLI: ichor machine reclaim ADDRESS --talosconfig FILE. The CLI writes to the node itself, with the operator's own credential, which never leaves the operator's machine; like init and adopt it acts on a node that belongs to no Ichor cluster.
• UI: the Add node page and Machines, "Reset and add". Offered to users who hold the reclaim verb on machinereclaims. The UI API creates a MachineReclaim and a credential Secret as the signed-in user; the controller inspects the node, stops at AwaitingConfirmation, and resets only after the user has typed the UUID. See "Trust" for the credential.

Approval, in the UI or ichor machine approve, sets the role, labels, disk roles and basic networking (which NIC, DHCP or static, VLAN or bond if needed) through typed forms; no YAML required. "Which NIC" is expressed as a LinkAliasConfig whose CEL selector matches the permanent MAC discovered earlier, and the other documents refer to that alias — which is also the fix for kernel interface names changing across boots. Bond members take names or aliases rather than selectors, so a form that picks members by MAC emits one alias per member.

The controller then renders the config, applies it insecurely, and follows the node through install, reboot and join. "Follows" is load-bearing: the apply RPC returns before the install starts, because Talos's own controllers react to the stored config asynchronously. The approval flow therefore watches MachineStatus's stage move from maintenance through installing and rebooting to running, and treats the RPC returning as nothing more than "the config was stored".

An approved machine that has converged and whose node then answers only in maintenance mode with the approved identity (its disk was wiped or replaced) is sent its configuration again, on its own, with a Warning Event and a Ready message saying it re-installed the node. The identity checks, the SecureBoot requirement, the install disk and the control plane's cluster lock are the first provisioning's; a control plane's stale etcd member is removed through a surviving control plane first, and with no surviving control plane nothing is sent and the message points to ichor restore --recover-cluster.

Disks are identified by WWID, serial or by-id path, not /dev/sdX — but all three can be absent. Talos fills WWID and serial from sysfs, with a udev fallback only for ata and scsi transports, so NVMe is reliable while a plain virtio-blk disk usually has neither, and without one udev creates no by-id link either. So identification falls back through WWID, serial, by-id path and finally bus path, and "nothing stable to key on" is a state the approval form has to show rather than treat as an error.

Each disk gets a role: system (install target plus EPHEMERAL — the recommended layout is one dedicated disk for Talos), ephemeral (EPHEMERAL on its own disk), storage (prepared for a storage system via a preset), userVolume, or unused. A role becomes machine.install for system and a VolumeConfig/UserVolumeConfig/RawVolumeConfig document for the rest, each selecting its disk with a CEL expression over the disk's reported fields. All three documents are on the mirror allow-list, so a role is expressible today; TalosMachine.spec.disks sets ephemeral and userVolume at approval and spec.network the primary NIC, DHCP or static, and a VLAN; storage, unused and bonds are not done. The system disk is the exception to "a document per role": it is TalosMachine.spec.install.disk (WWID, serial, by-id path, bus path, or device path as a last resort), rendered as UnattendedInstallConfig from the v1.14 contract and as machine.install before it.

Two things about editing a disk role after the fact, both verified against Talos's volume manager. A changed selector on a volume that already exists does nothing, and nothing says so: Talos locates EPHEMERAL by its partition label before it ever consults the selector, and config validation accepts the edit regardless. Ichor has to detect that itself and report "accepted, no effect" rather than letting it look applied. Likewise a lowered maxSize is silently ignored — partitions only ever grow. (A negative maxSize is meaningful rather than invalid, and PartitionSpec.ResolveMaxSize in Talos v1.14.3 says exactly what it means: the value is space to leave free, so the volume gets available - value. A relative one is a percentage of available, not of the disk -- -25% of 200 GiB available leaves 50 GiB free and takes 150. If the reservation exceeds what is available the call fails with "partition size cannot be negative"; it does not clamp, so the volume simply does not provision. That is the mechanism for sharing a disk: EPHEMERAL reserves with a negative maxSize, and the user volume provisions in what is left.)

The confirmation before wiping a disk matters most for the install target, because machine.install.wipe defaults to true — Talos wipes the install disk unless told not to. For every other disk Talos is the opposite: it refuses to provision over any existing partition table or filesystem, so Ichor's confirmation there explains why provisioning will not happen rather than guarding against loss. Single-disk nodes can use systemDiskLayout: shared, which caps EPHEMERAL and leaves the rest for storage — Talos makes that work through partition waves, with EPHEMERAL claiming its region before user volumes, and EPHEMERAL's own minimum is 2 GiB; it is supported but discouraged and warned about, especially on control planes where etcd would share I/O with storage. Disk roles are fixed at install time because Talos ignores volume config changes after provisioning; changing system or ephemeral later is a reset-and- reinstall operation and is treated as such. Adding a storage role to an unused disk doesn't need a reset. Disks with existing partitions or filesystems require explicit confirmation.

Removal is a finalizer-driven sequence: storage evacuation hook (if configured) → cordon and drain with a timeout → remove the etcd member if it's a control plane (refused if quorum would break, if the count would drop below spec.controlPlane.count, or if it's the last one) → graceful Talos reset → delete the Node, in that order. The machine reappears as pending and can be reused, or, if it was removed with "power off", comes back when you power it on. Remove and reset are one action: node remove always wipes STATE and EPHEMERAL, and node remove --reboot reboots the node afterwards; the ichor node reset command was removed on 2026-11-07. A node that does not answer is removed without the wipe, its Node is deleted, and the Event says it was not wiped. forceRemove skips drain and reset for dead hardware. Control planes are replaced add-then-remove, never the other way round — a 3-node cluster replacing a member goes 3→4→3, never 3→2, because four members tolerate one failure while two tolerate none.

Four things about that sequence, each verified against Talos 1.14.3's source and each easy to get fatally wrong:

• The reset must name the partitions to wipe. With its default mode and no partitions named, Reset wipes the entire system disk, boot and EFI included, which means a reinstall rather than a return to maintenance mode. Passing STATE and EPHEMERAL explicitly is what leaves the installed system in place and brings the node back up with no configuration — which is the behaviour the reuse story depends on.
• Ichor's refusals are Ichor's alone. Talos's member-removal call requires only that a leader exists right now; it does not ask whether the remaining members can still reach quorum. The one quorum guard Talos ships refuses at exactly two members and only on the upgrade path. So the three refusals above live in shared code, as principle 10 requires, and are the thing most worth testing before any of this points at real hardware.
• A graceful reset already drains and already leaves etcd. Ichor's own drain and member removal are therefore belts rather than the trousers for a live node: they matter for forceRemove, where the node cannot run its own sequence, and for refusing before Talos is asked to do anything.
• Reset is fire-and-forget. It returns an actor id immediately and the sequence runs asynchronously, so status has to record that a reset was requested before the controller can crash — otherwise a resumed reconcile fires a second concurrent reset at the same node. Every other step is naturally idempotent: re-draining a cordoned node is harmless, and removing an already-removed etcd member reports not-found, which counts as success.

The finalizer cannot defend against one thing: a person removing it by hand mid-sequence. That deletes the resource immediately and orphans whatever state existed — a cordoned node, possibly still an etcd voter, with nothing left to resume it. Refusing finalizer removal except through override belongs with the other dangerous-operation checks.

Draining uses k8s.io/kubectl's own drain helper rather than a hand-written eviction loop: it already encodes the DaemonSet, mirror-pod, local-storage and PodDisruptionBudget rules, and it reports a timeout as an error, which is exactly the "pause, don't escalate" signal the design wants.

Topology. Control-plane count is 1, 3 or 5. Two is rejected: with two members etcd needs both to be up, so it is strictly less available than one. Four has the same fault tolerance as three. Even counts are allowed only transiently during a change and shown as "transitioning", during which the cluster lock blocks other disruptive work. Workers can be zero; control-plane- only clusters are a supported configuration, not a corner case.

controlPlane.schedulable is true or false, and required. It has no auto: nothing on the object can resolve one, since a cluster is created before any machine exists and the topology preset that carries the intent is a flag on ichor init, not a field. init sets it from the preset, and adding workers later never retaints the control planes underneath running pods. Making control planes unschedulable with zero workers is rejected. Presets: single (1+0, schedulable), compact-ha (3+0, schedulable), standard (3+n, tainted, recommended), large (5+n, tainted).

Small clusters get specific treatment: single-node upgrades can't drain anywhere, so the UI says there will be downtime and asks; a missing backup configuration is a warning; three-node clusters without workers get a capacity check before each drain.

CNI

Cilium is the only CNI; spec.cni.preset is cilium and immutable. spec.cni.tunnel (ichor init --cni-tunnel) picks vxlan or geneve and is immutable once set. Talos's own CNI is removed (the KubeFlannelCNIConfig document, or cni.name: none before v1.14). Pod and service subnets are spec.network.{podSubnets,serviceSubnets}, fixed once set. A cluster stored with another preset is refused, not guessed at: the controller sets CNIUnsupported and changes nothing.

A cluster adopted with no CNI is a managed cilium cluster: adopt installs the pinned Cilium with spec.cni.tunnel from --tunnel (or the default) and spec.cni.kubeProxy read from Talos (kube-proxy enabled gives enabled, disabled gives replaced); --cilium-values FILE sets the initial spec.cni.cilium.values. ichor init keeps its inline-manifest bootstrap.

The one exception is spec.cni.preset: unmanaged, which exists for adoption only. ichor init has no flag for it and the webhook refuses it on create unless the object carries the adoption annotation. The adopted cluster keeps whatever CNI it runs (flannel, Calico, another, or none); Ichor does not install, upgrade or configure it, and never migrates one. ichor adopt refuses a non-Cilium CNI unless --cni-unmanaged is given, and refuses a cluster with two CNIs even then. In this mode:

• spec.cni.cilium and spec.cni.tunnel are refused. spec.cni.unmanaged.talosCNI is default (render says nothing about the CNI) or none (render keeps cluster.network.cni.name: none). spec.cni.kubeProxy only records what the node has.
• The hand-over, the ichor-cni Job, ichor cni commands, cni operations and the kube-proxy switch are refused or skipped. A Kubernetes upgrade is never blocked on a hand-over. A CA rotation restarts no CNI workload.
• The preset is immutable. Moving between managed and unmanaged means a new cluster.
• status.cni.detected (cilium, flannel, calico, other, none) is read for display and refreshed on a slow requeue.

Network policy gate. Ichor's CiliumNetworkPolicy objects are applied only when the detected CNI is Cilium and the CRD exists. Otherwise ichor-system is not isolated by Ichor and the condition NetworkPoliciesApplied says so. A hands-off Cilium also gets them.

How Cilium is installed, owned and edited is described under "Cilium" above. The Talos-recommended values are built in; Hubble certificates come from a CronJob, not Helm, so a re-render does not change them.

spec.cni.kubeProxy is enabled or replaced. Replacing kube-proxy splits the setting across two owners, Talos (KubeProxyConfig, Ichor) and Cilium's kubeProxyReplacement (the Helm release, also Ichor), so the consistency check reads Cilium's live config and reports mismatches as health failures. Switching modes (ichor cni kube-proxy) is refused unless annotated (the server sets the annotation for that one write), and then runs in the only safe order: upgrade the Cilium release through the ichor-cni Job, verify on all nodes, then turn the Talos side off.

Turning the old side off is a deletion Ichor has to perform, not a consequence of the config change. Verified against Talos v1.14.3: ManifestController renders the kube-proxy objects only while proxying is enabled, but ManifestApplyController has no delete path, only a server-side Apply. Flipping the setting stops Talos managing kube-proxy and leaves the DaemonSet, ServiceAccount, ClusterRoleBinding and ConfigMap running, so an operator would get two proxies. The same controller stamps every object with an inventory annotation and refuses to apply over a foreign one.

Storage

The work splits into four layers; Ichor owns the first and the last.

TalosStorage stays thin: which nodes, which disk preset, how disks are handed to the storage system, and evacuation settings. The storage system's own configuration is yours. Presets start with longhorn-v1 (verify the iscsi-tools and util-linux-tools extensions are in the node's installer image, create the UserVolumeConfig Longhorn expects), followed by longhorn-v2, raw (for Rook/Ceph-style consumers) and filesystem. Missing extensions are reported by the informational ExtensionsPresent condition on TalosStorage, read from the Nodes' extensions.talos.dev/* labels; it never gates Ready. Adding one is an installer-image change: a new schematic, which the Talos upgrade treats as an upgrade even when the version does not move.

Disks are registered with the storage system through its own node label and annotation mechanism, not by writing its CRs. A consistency check confirms the storage system is configured to honour that.

The node-operation hook is shared by removal, upgrades and reboot-requiring config changes: mark the node unschedulable in the storage system, request evacuation, wait until no replica lives only there (with a timeout that pauses the rollout rather than proceeding), then drain; afterwards, re-enable scheduling. The hook is an interface so other integrations can use it.

For longhorn-v1 the write side of that hook is settled, from longhorn-manager v1.9.1's own type declarations rather than from the UI's labels: nodes.longhorn.io/v1beta2's NodeSpec carries allowScheduling (bool) and evictionRequested (bool), and DiskSpec carries the same two under spec.disks[<name>], so a hook can evict a whole node or one disk. The order is not a preference: Longhorn's own documentation states eviction only takes effect on a node or disk whose scheduling is already disabled, so allowScheduling: false has to land first.

The read side is now settled too, from longhorn-manager v1.9.1's own controller source rather than from the UI's replica count: there is no dedicated "eviction done" status field. DiskStatus.ScheduledReplica (the status map that looked like the obvious candidate) is the scheduler's capacity bookkeeping -- it is read for free space, not for residency, and does not reliably go to zero when eviction finishes. The actual mechanism is in VolumeController.cleanupEvictionRequestedReplicas: once a volume has a healthy, non-evicting replica elsewhere, it deletes the Replica object whose spec.evictionRequested is true. So "no replica lives only there" is "no replicas.longhorn.io object remains with spec.nodeID (whole-node eviction) or spec.diskID (one-disk eviction) still naming the target" -- an absence of objects, not a counter reaching zero. EvacuationStatus (pkg/storage/evacuation.go) takes that count already read by the caller, plus elapsed time against TalosStorage.spec.evacuationTimeout, and returns done, still-waiting, or paused -- paused, never proceeding to drain, once the timeout is reached with replicas still remaining.

TalosStorage reports health, so whatever installs the storage system can wait until the Talos side is ready.

Configuration reconcile

1. Load the bundle from the Secret and generate role-specific base configs with the pinned version contract.
2. Collect patches by layer and priority and apply them.
3. Validate, hash, and compare with the node's current config via COSI.
4. If different, apply. The mode (try, no-reboot, reboot, staged) is derived from the diff, because Talos will not derive it for us — its AUTO mode is a hardcoded alias for NO_REBOOT and never inspects the change. A dry run first is still worth it: it is side-effect free and returns validation warnings worth surfacing. Reboot-requiring changes take the cluster lock and go one node at a time through the operation hook and drain, control planes with etcd checks in between. They never start by themselves: the machine reports RebootRequired with the fields, and a config operation, started from ichor config apply or Apply in the Overview page's attention list after a plan, rolls them out. A third class, APIServerRestart (for example spec.authentication), needs no reboot but restarts the kube-apiserver; the machine controller never applies it to a control plane, and the same config operation does, one control plane at a time with no drain, waiting for that node's /readyz before the next. Talos's Reboot does not cordon or drain (unlike its Upgrade and Shutdown), so the rollout does both itself before the reboot and uncordons afterwards.
5. Watch COSI for state and drift. Drift handling is per cluster: warn only (default), or auto-correct non-reboot differences. A change in what the renderer emits (an Ichor upgrade, render.Version) is a desired-state change, not drift: an Ichor upgrade may apply configuration to nodes, and reboot-requiring changes wait for a config rollout.

Upgrades

The release set and the upgrades (amending 0113). Each Ichor build embeds one set of versions, pkg/release.Pinned: the Talos release and its version contract, the Kubernetes release, and the Cilium chart. It decides the versions of a first install (ichor init, the CLI defaults and the UI's proposed values). It does not bound an upgrade: Talos, Kubernetes and Cilium each upgrade on their own, to any version the rules below accept, and the set is not a limit on them. The Versions card shows what each component runs; it carries no state against the set.

An upgrade target is valid when it is a final semantic version newer than the running one, and:

• Talos: upgrade.TalosPath accepts the step (machinery's compatibility rule: the target's minor sets a minimum running version and denies listed versions; within those bounds Talos may skip minors, though its docs recommend adjacent minors; Talos upgrades reboot each machine).
• Kubernetes: at most one minor up within the same major (upgrade.KubernetesSkew, the version skew policy), and the cluster's Talos supports the target (upgrade.KubernetesPairing).
• Compatibility: the Talos and Kubernetes pair, and the Kubernetes and the installed Cilium, are checked against the one table pkg/release/compat.go. An incompatible pair is refused, and the override verb is the only way past the refusal; the override is recorded as a Warning Event on the cluster. Cilium moves only to the chart embedded in the running Ichor: there is no image-only patch, and a daily CI run opens a pull request when Cilium publishes a new patch of the pinned minor, so a patch reaches clusters with an Ichor release.

The upgrade dialog offers the Talos and Kubernetes targets as a searchable list of the candidates the controller found: the patches of the current minor and the next minor, grouped "Recommended" (the set's version, when newer), "Patch" and "Next minor", newest first, with the newest patch of the current minor as the default. A version typed in and not in the list can still be chosen; the server validates it the same way. ichor upgrade talos|kubernetes --list prints the same groups as a table (recommended, patch, next minor, the default marked). An Ichor upgrade does not move the cluster; each component is upgraded by its own operation.

A step that fails pauses its operation with a failureReason and is never retried by itself; ichor operation resume (or Resume in the UI) on such an operation re-runs the failed step through an annotation, not through spec.

Every operation takes the cluster lock under its own identity (ichor-controller/op/<namespace>/<name>), a control-plane join or removal under the machine's own, and the Cilium Helm Job under ichor-controller/cni-helm, so two disruptive pieces of work never run together, whoever created them; the one that waits pauses with ClusterLockHeld naming the holder, and a lease whose holder is gone, completed or paused on a failed step is taken over.

All three upgrade kinds share the lock, the status-held state machine, pause- on-failure (no automatic cluster-wide rollback), paused, maxUnavailable for workers, a per-machine hold annotation, label-based ordering, and a progress view. Outstanding hand-overs are flagged before a Kubernetes upgrade because of the bootstrap-manifest re-apply.

Talos: change spec.talosVersion or the schematic (a machine is upgraded when its running version or running schematic, read from its ImageFactorySchematic resource, differs from the target). Pre-checks cover etcd health, node readiness, the supported upgrade path and the Kubernetes pairing. Control planes go first, one at a time; workers follow within maxUnavailable. Per node: hook → Upgrade API → reboot → verify version, readiness and etcd membership. Cordon and drain are not Ichor's to do here: Talos's own Upgrade sequence already cordons (a controller does it reactively once the upgrade actor starts) and drains (via the kubelet's own kubeconfig, the standard eviction API) before it stops services and reboots, and uncordons the same way once the upgrade clears — confirmed in Talos 1.14.3's source (Sequencer.Upgrade's "drain" phase, CordonAndDrainNode, and NodeApplyController.ApplyCordoned). Doing it again on Ichor's side would only be redundant, not more correct, and would blur which side's cordon "owns" the uncordon afterwards; the one thing still Ichor's to do beforehand is the storage-evacuation hook, which Talos has no notion of. No etcd-preserving flag handling is needed either: UpgradeRequest.Preserve is explicitly deprecated in that same source ("only used for legacy upgrade path, to be removed in Talos 1.18"), and the installer ignores it in upgrade mode ("system disk wipe on upgrade is not supported anymore, option ignored") — a plain upgrade reboots in place and never touches the etcd data directory, for one member or many. A node that falls back to the previous A/B slot counts as a failure. The version contract is never bumped by the controller on its own: it changes only through the spec write of an operation the user starts.

Kubernetes: change spec.kubernetesVersion to a version one minor up at most, supported by the Talos the cluster runs and by the installed Cilium. Warn about removed APIs in use, pre-pull images, then in three stages, each finished on every machine before the next (no kubelet is ever newer than an API server): switch the control-plane components node by node through machine config; update kube-proxy and sync the bootstrap manifests (the rollout check is skipped when kube-proxy is replaced); roll the kubelets, control planes first. The intermediate configurations are renders with the kubelet and kube-proxy pinned to the version the node runs, never a write to spec. A minor upgrade of several control planes behind an endpoint of type node warns that the endpoint is one apiserver. This is upgrade-k8s's order, run on Ichor's own machinery.

Ichor: the controller only launches the installer Job for the target version and never upgrades itself. The Job runs with the RBAC of the release being replaced, so the installer applies its own RBAC first; its rights are cluster-scoped, plus a Role in Ichor's namespace. It then snapshots the current manifests, updates CRDs (additive, never downgraded), the controller, then the UI, applies the network policies when Cilium's CRD exists, runs post-checks, and rolls back from the snapshot on failure. The installer image never runs by tag: before the Job exists the controller resolves installerRepository:version to a digest (or takes the pin in spec.installerDigest), verifies the release workflow's keyless cosign signature offline against an embedded Sigstore root, and runs the Job as repository@digest; a failure pauses the operation, and the only bypass is a recorded spec.installerSignature: skip (--skip-signature-check) for builds nobody signs. The install manifest inside a release's installer names the controller and ichor-ui by digest too. The CLI refuses writes when more than one minor away from the server. A UI backend response header (X-Ichor-Version) that would make a stale SPA prompt a reload is not built; the top bar shows the installed Ichor version as a badge instead (ui/src/lib/ichorVersion.ts).

Shutdown: a TalosOperation of kind shutdown stops the whole cluster for planned downtime. Workers go first, in parallel and drained by Talos's own Shutdown sequence; then every control plane, and the controller's own node last, in one forced burst, so quorum is lost only in that last stage and a self-managed controller needs to live only until the requests are sent. A shutdown does not remove etcd members, so power-on loses no data. While the operation has not completed, the cluster lock refuses every other disruptive operation. Ichor cannot power machines on: the operation waits, and completes by itself once every machine answers on a new boot, which lifts the refusal.

Backup, recovery, certificates

TalosEtcdBackup takes snapshots on a schedule from a healthy non-leader control plane, verifies them, encrypts them (snapshots contain Secrets) and stores them in S3-compatible storage with retention. A snapshot can also be taken outside the schedule (ichor etcd backup now, Back up now on the Backups page), which sets an annotation the controller acts on. Failures and staleness surface as metrics and alerts.

Two recovery paths. Restoring into a live cluster stops etcd on all control planes and restores the snapshot; the UI and CLI both offer it, as a TalosOperation that needs the override right on the cluster and the cluster's name typed. Because the restore rolls the Kubernetes API back past the operation itself, its progress is kept in a marker next to the backups, not in status, and the operation is recreated from the marker. The backup store must therefore live outside the cluster being restored (not a Service such as minio.minio.svc, nor an address in the cluster's pod or service range): the marker is written to the store after the API is down, so a store inside the cluster deadlocks the restore. The plan refuses it, override does not override that, and the TalosEtcdBackup carries a StoreInCluster warning condition meanwhile. Full-loss recovery is CLI-only: ichor restore --recover-cluster rebuilds control planes from the local bundle and the snapshot, after which the controller re-reconciles every node the way adopt does and waits for confirmation before applying anything, since the restored CRs may not match the hardware. The goal is that bundle + snapshot is sufficient, and the e2e suite proves it regularly.

Ichor's own Talos client certificates are short-lived and renewed automatically; an expired one would lock Ichor out with no way back. CA rotation (Talos and Kubernetes by default, the etcd and aggregator CAs on request; the service-account key and the secretbox key by a separate key-rotation operation) follows the documented Talos procedure and runs through the same machinery as upgrades, except that it waits, after every node accepts the new CAs, until the operator has credentials that work with both CAs; otherwise the first stage that changes what the API servers present would lock the operator out. The new credentials are written by the CLI or downloaded by an administrator in the UI. In-cluster clients cannot hold both Kubernetes CAs (kube-root-ca.crt switches at Issuing), so before dropping the old CA the controller waits for the new one in every namespace and then a settle delay. Its own client trusts the new Kubernetes CA ahead of the switch through a process-wide trust pool, and its leader election outlives the API server restart. After the old CAs are dropped and every Node has been Ready for a while, it restarts, one at a time, the clients that keep the old CA (CoreDNS, kube-proxy, the CNI, ichor-ui); other in-cluster clients that load the CA once may still need a restart. Expiry dates are visible in status, the UI and metrics.

Monitoring and registry mirrors

Ichor does not install a monitoring stack. It can expose the control-plane metrics that Talos binds locally by default (controller-manager, scheduler, etcd) and restrict scraping to a CIDR with Talos's ingress firewall. The controller and UI backend export Prometheus metrics, and ServiceMonitors, alert rules and Grafana dashboards ship with the chart as an option. The monitoring stack itself is a documented set of Helm values, or nothing — the user's choice.

Registry mirrors are optional and off by default: they are pushed to every node through the system layer, alongside an override for the installer image source and a self-hosted Image Factory URL. They serve pull-through caches and private registries, not air-gapped installs. Ichor does not check that a mirror is reachable before an upgrade.

CLI and UI

The rule is parity. Both write CRs; both share the same Go packages for rendering, planning (diff, impact, pre-checks, upgrade paths) and dangerous-operation checks; both use the same Connect API, the CLI falling back to local computation when it can't reach the backend. The CLI prints for people by default (tables and aligned blocks) and offers -o yaml|json for scripts, --follow/--watch, and interactive confirmation with --yes. The exceptions are init, adopt and full-loss recovery, which predate the cluster.

The UI is a Vite + Preact + TypeScript SPA built with preactcn, embedded in the Go backend. The backend talks to the API server, read-only to the Talos API, and optionally to Prometheus for graphs; the browser talks only to the backend. It survives drains: stateless, two replicas where possible, graceful termination that asks clients to reconnect, and streams that resume from a token (resourceVersion, last log timestamp, progress sequence). ichor ui re-establishes its port-forward to another pod when the current one goes away.

The UI is built for one cluster: there is no cluster list, and a top bar shows the cluster's name, its Talos, Kubernetes and Ichor versions and a health badge. Screens: an Overview, built from stock preactcn components, with tiles for nodes (and the nodes waiting for approval), the running operation, backups and versions, a needs-attention list, recent operations, and a collapsible cluster details section (a config apply waiting for a reboot shows in the needs-attention list with its own Apply action), with Add node and cluster actions (rotating the CAs, shutting down) in the header; a Nodes list and an add-node (approval) page; a node page with tabs for Summary, Storage & network, Talos, Kubernetes & etcd, Configuration and Logs; Operations, with a detail page per operation (per-machine state, pause, resume, and the CA-rotation credentials hand-over); Backups, with each TalosEtcdBackup, its snapshots, Back up now and restore; Configuration, for editing patches by layer with an effective-config view annotated by origin and secrets masked, impact preview before applying, and schema-driven forms; Cilium, with the effective values read-only, a YAML editor for the user layer, validation and a diff preview, and an Upgrade card when the embedded chart is newer; Users, only when Ichor's own users are on. A merged multi-node log viewer is planned.

Testing

e2e runs against throwaway clusters from talosctl cluster create qemu from the first phase onwards. QEMU is not a preference but a requirement for most of it: the Docker provisioner is capped at one control plane, has no disks, and its own documentation excludes the upgrade and reset APIs, so install, disk roles, reboot-requiring patches, removal, topology changes and upgrades can only run under QEMU. Docker stays useful for the cheap half — config applies that need no reboot, and drift. Node discovery needs a node that is up but unconfigured, which --presets maintenance provides. Scenarios follow the features: bootstrap and join, patches with and without reboot, drift, removal and replacement, topology changes including the even-count rejection, CNI Cilium bootstrap, adoption and values edits, prune protection, storage preparation and evacuation, all three upgrade kinds with rollback, backup, restore and full-loss recovery, certificate rotation. Each scenario runs through both the CLI and the backend API. Chaos cases kill the controller or reboot a node mid-rollout and check that status-driven resumption works. A short set runs per PR, the full set nightly.

Roadmap

1. CRDs and the kubebuilder project, Helm chart and Kustomize base, deletion protection, e2e scaffolding.
2. Typed config (the mirror-type generator first), rendering and apply, drift, the cluster lock.
3. Discovery, approval, disk roles, removal, topology. The deletion guard phase 1 ships only checks for the allow-delete annotation; the quorum and last-control-plane rules land here, and an explicit test has to show that the annotation alone does not let the last control plane go.
4. init and adopt, Cilium.
5. Read-only UI, approval UI, backup and restore, client certificate renewal.
6. Config editing UI, TalosOperation, Talos and Kubernetes upgrades, Ichor self-upgrade.
7. Storage integration (longhorn-v1, operation hooks), CA rotation.
8. Override.
9. Monitoring settings, metrics and dashboards, optional components, OIDC and impersonation, Ichor's own users and issuer.
10. UI localisation.
11. Later: more storage presets, IPv6, replica-based scaling.

Alternatives considered

Raw YAML patches. Easier to start with, impossible to validate, preview or explain in a UI, and they tie user files to a Talos version. Rejected in favour of typed documents.

Ichor installing storage and monitoring itself. It would double the surface area, duplicate what the storage controllers already do, and keep us on the upgrade treadmill of every project we wrap. Ichor installs Cilium only.

Fewer CRDs. TalosEtcdBackup and TalosStorage could fold into TalosCluster. They are kept separate so their RBAC and Git diffs stay focused; merging later would need a migration, splitting later wouldn't.

Multiple repositories. The three deliverables share the CRD types and the Connect definitions; separate repositories would mean a fourth one for those and version skew to manage. One repository, three release artifacts.