Unreleased (main)
These are the development docs; they can describe changes no release has yet.
Decision records
One file per decision, NNNN-short-title.md, based on 0000-template.md.
Write one when you pick something the design doesn't pin down: a default, a
library, a metric name, a workaround. Keep it short. Don't rewrite accepted
records; supersede them.
The table below is generated from each file's Decided:/Status: header
line by internal/tools/adrindex (make adr-index) -- never hand-edit it.
Give a new ADR a Decided: line (0000-template.md shows where) and
regenerate; CI's generated-artifact drift check catches a stale table the
same way it catches a stale chart.
| # | Decided | Status |
|---|---|---|
| 0001 | Record decisions as one-file-per-decision ADRs, linked from the PR that makes them | Accepted |
| 0002 | Pinned nixpkgs to nixpkgs-unstable and committed flake.lock for a reproducible shell | Accepted, amended |
| 0003 | Generate the Talos machinery mirror types instead of hand-maintaining a parallel field catalogue | Accepted |
| 0004 | Phase 1 scaffolds only TalosCluster, TalosMachine and TalosConfigPatch, with no controllers yet | Accepted |
| 0005 | Moved controller-gen, kustomize, setup-envtest and golangci-lint out of flake.nix and into Makefile-managed ./bin | Accepted |
| 0006 | Run cert-controller in-process for the webhook serving certificate instead of depending on cert-manager | Accepted |
| 0007 | e2e runs Talos nodes on a real cluster, not talosctl cluster create qemu | Accepted |
| 0008 | controlPlane.schedulable takes a required true/false; dropped auto | Accepted |
| 0009 | Keep predicting reboot necessity from pkg/fieldmeta's field-path table; Talos itself won't answer the question | Accepted |
| 0010 | Added CheckControlPlaneQuorum as a pure guard stronger than the allow-delete annotation alone | Accepted |
| 0011 | Made TalosMachineSpec.Role optional; discovery never picks a role or auto-registers a weak identity | Accepted |
| 0012 | Mirror VolumeConfig/UserVolumeConfig/RawVolumeConfig as three more mirrorgen roots, for disk roles | Accepted |
| 0013 | Settled the cni.Preset interface, custom's required fields, Hubble's certificate handling, and where the two immutability guards live | Accepted; superseded in part by ADR 0099 |
| 0014 | Settled cmd/ichor's package layout and how ichor adopt guarantees zero drift | Accepted |
| 0015 | Built GitOps Application as a hand-rolled type (pkg/gitops) behind one constructor, plus a policy-free repo skeleton | Superseded by ADR 0101 |
| 0016 | Split the Connect read/approval API into ClusterService/MachineService/NodeService and fixed their message shapes | Accepted |
| 0017 | Settled ui/'s directory, cmd/ichor-ui as a second binary, and how the Go side builds without the UI present | Accepted |
| 0018 | One Server type implements all three handler interfaces over a single uncached client; settled the four approval refusals; fixed loadBundle resolving the TalosCluster's own namespace instead of the backend's, the same bug ADR 0011 had already fixed on the controller side; granted the nodes get/list findKubernetesNode had always read without it; added a chart-vs-config/ RBAC drift test after the chart's own copy of this ClusterRole was found silently narrower than config/rbac/ichor-ui/role.yaml; extended that drift test to the controller's manager-role and manager-secrets-role, which found and fixed missing talosoperations, Argo CD applications and metrics-cert Secret grants | Accepted, amended five times |
| 0019 | ichor init: VIP recommended not required, Ichor applied through the API server not inlined, topology presets carry count/schedulable | Accepted; superseded in part by ADR 0099 |
| 0020 | Gave ichor-ui a Dockerfile, a chart Deployment/Service reusing the controller's own mechanism, and RBAC wired into config/default | Accepted, amended |
| 0021 | Built the four read/approval UI screens on a committed, generated TS client, with the approval dialog disclosing identity/hardware/role before use | Accepted, amended twice |
| 0022 | Delegate the Talos/Kubernetes compatibility matrix to pkg/machinery/compatibility instead of reimplementing it | Accepted |
| 0023 | Keep TalosStorage to layer 1 only -- a longhorn-v1 preset emitting the Talos-side config a node needs, never the storage system itself | Accepted |
| 0024 | No CNI hand-over registry (one real member); removal happens at render time off a status condition; a separate controller; the generated Application syncs automatically (prune, selfHeal), amended 2026-10-26 | Accepted (amended 2026-10-26); superseded in part by ADR 0099 |
| 0025 | Layer 4's storage-system side: EvacuationHook (write side) plus a preset-independent EvacuationStatus function (read side), both verified against longhorn-manager v1.9.1's own controller source | Accepted |
| 0026 | TalosOperation: the backbone every upgrade kind shares (lock, status-held state machine, pause-on-failure, paused, maxUnavailable, hold annotation, ordering), with none of their execution | Accepted |
| 0027 | Private registry auth through TalosCluster.spec.registries[].authSecretRef, resolved and injected as the system layer by pkg/render; the auth-less Render/BuildConfig entry points were merged into one that always takes the resolved auth, after no production caller had ever been moved onto the auth-aware pair | Accepted, amended once |
| 0028 | ichor gitops export's allow-list of CR kinds and of the fields it writes for each (no status or server-assigned metadata), its Argo CD/Flux tracked-resource warning, and ichor gitops handoff's one-field TalosCluster patch | Superseded by ADR 0101 |
| 0029 | Generate docs/adr/README.md's index table from a Decided:/Status: line in each ADR's own header (internal/tools/adrindex) instead of a shared, hand-edited table | Accepted |
| 0030 | ichor ui discovers ichor-ui pods through the ichor-ui-service Service's own selector/port and reforwards to another ready pod (same local port) whenever the tunnel ends, instead of targeting one pod name | Accepted |
| 0031 | made config/samples/*.yaml minimal, valid and apply-tested instead of deleting them; gave the metrics endpoint a real, opt-in certificate named metricsCert.enabled (not certManager, and not the default) via a second cert-controller rotator instead of leaving insecureSkipVerify's documented escape hatch wired to nothing; confirmed the webhook certificate has the same cold-start race the metrics one does; and logged that race instead of papering over it with a bootstrap certificate, which would trade a loud failure for a silent one. | Accepted, amended twice |
| 0032 | Complete a TalosMachine deletion by removing its etcd member -- self-leave first, a survivor removing it by ID as fallback -- clearing the finalizer only after that succeeds, and warning that the node keeps the cluster's credentials until it is reset | Accepted |
| 0033 | moved the identity-tuple rule to TalosMachineIdentity.IsStrongIdentity in api/v1alpha1, and the secrets-bundle Secret read/decode to a new internal/secretsbundle package, so internal/controller and internal/uiapi each still call it from their own, separately-justified call site, but neither keeps its own copy of the rule. | Accepted |
| 0034 | built the one piece of phase 6's "config editing by layer ... effective-config view annotated by origin and secrets masked" that the current API surface actually supports — grouping GetEffectiveConfig's values by origin.layer in the design's own bottom-up order — and did not build layer editing, impact preview, or schema-driven forms, because none of them have a backend to call yet. | Accepted |
| 0035 | built PatchService (CRUD for TalosConfigPatch plus PreviewImpact) exactly as ADR 0034's gap report proposed it, with the opaque-JSON PatchDocument shape that avoids re-modeling Talos fields a third time, and with no new RPC for schema-driven forms since the CRD's own committed OpenAPI schema already serves that purpose. | Accepted |
| 0036 | ichor images list prints the five fixed Image Factory images (version-gated) plus an unconditional warning naming the two catalogs, instead of trying to resolve a schematic's extensions/overlays into images | Superseded by ADR 0090 (the command was removed) |
| 0037 | built pkg/prmode's ClassifySource/Locate/Patch/Diff -- finding a tracked CR's file via its Application's source and editing one field with go-yaml's yaml.Node, never a PR host API -- as its own host-independent, fully unit-testable unit, separate from opening a PR | Superseded by ADR 0101 |
| 0038 | pkg/prmode now clones a repository, pushes a branch via the git CLI (never go-git), and opens a PR/MR via one net/http POST per host (never a GitHub/GitLab/Gitea SDK) -- with the token carried only through git's own GIT_CONFIG_* environment-variable mechanism and an HTTP request header, never argv, a URL, or a log line | Superseded by ADR 0101 |
| 0039 | built the PatchService CRUD screens (list, create, edit, delete) and an explicit impact-preview action on ui/src/pages/PatchList.tsx/PatchDetail.tsx, editing each PatchDocument and LabelSelector as a JSON blob in a Textarea rather than through a generic schema-driven form renderer, and relying on the apiserver's own CRD validation (surfaced via describeError) instead of a second, hand-written validation layer. | Accepted |
| 0040 | added exactly one custom metric (ichor_uiapi_rpc_duration_seconds, via a single connectrpc.com/connect interceptor) because the UI backend exports nothing on its own; added zero custom metrics to the controller because controller-runtime already exports reconcile/workqueue health and kube-state-metrics already covers cluster-level state (pending approvals, the cluster lock, TalosOperation progress) from fields already in CR spec/status; shipped a ServiceMonitor for ichor-ui, one PrometheusRule with two alerts, and one Grafana dashboard, all as chart options defaulting off, the same opt-in shape metricsCert.enabled already established (ADR 0031). | Accepted |
| 0041 | redact bundle material from dry-run warnings unconditionally rather than relying on Talos v1.14.3's current (verified) behavior of never echoing one; renamed a registry-auth leak test to match what it actually exercises and documented its two genuinely unreachable paths; validate ichor init --cluster as a Kubernetes object name before any file write; added CEL format checks to TalosClusterRegistry.host/.endpoints since machinery validates neither. | Accepted |
| 0042 | Add an opt-in ichor.maesh.dev/reset-on-delete annotation that, after a removal's etcd member is gone, wipes only STATE and EPHEMERAL by label and reboots the node to maintenance mode -- holding the finalizer if Reset fails, and recording the request only after Talos has acknowledged it, so a resumed reconcile never asks twice | Accepted |
| 0043 | Implement Talos upgrades as a single AdvanceMachineFunc (evacuation hook, Upgrade API, boot-ID-based reboot detection, version/readiness/etcd verification, A/B fallback is a failure) with all resumability held in TalosMachine conditions, and no Ichor-side cordon, drain or Preserve handling | Accepted |
| 0044 | moved isTracked from internal/cli to a new pkg/gitopstracking package with Flux detection corrected to an inventory scan (the field manager is only a first signal; flux-client-side-apply was never Flux's), and added a pure pkg/prmode.Decide(trackedBy, hasRepoConnection) Mode so "tracked → don't write, open a PR instead" has one place to live rather than zero | Superseded by ADR 0101 |
| 0045 | Test the UI's dangerous decisions (may this approval proceed, may this patch be sent, how is an error shown) as pure functions in ui/src/lib/ under node --test, with no component test framework and no ui/src/gen import in those files; the page screens stay untested by unit tests | Accepted |
| 0046 | Ship Lua health checks for TalosCluster and TalosMachine only, as a chart ConfigMap (argocdHealth.enabled, default false) whose keys are exactly argocd-cm's resource.customizations.health.<group>_<kind> keys, tested by running the Lua under lua 5.1 | Superseded by ADR 0101 |
| 0047 | TalosEtcdBackup snapshots one healthy non-leader control plane over Talos's EtcdSnapshot stream, checks the etcd sha256 trailer, encrypts with age using a key derived from the secrets bundle, and stores <prefix>/etcd-<UTC>.snapshot.age in S3 via minio-go; restore reads exactly that | Accepted |
| 0048 | ichor init --argocd --kokumi prepares the OCI repo-creds Secret, installs neither Kokumi nor Argo CD, and leaves the repository skeleton (including the Cilium Application) unchanged; ichor images list --kokumi names Kokumi 0.17.0's images | Accepted (implemented 2026-10-27); superseded in part by ADR 0090 (ichor images list --kokumi removed) and in part by ADR 0099 |
| 0049 | `ichor machine | cluster |
| 0050 | Render a TalosStorage's Talos-side preparation as part of the system layer (below every patch), choose the governing TalosStorage with one storage.Select shared with the evacuation hook, hand Render a single *SystemInputs instead of a growing argument list, and report Ready from the convergence of the machines it governs | Accepted |
| 0051 | Every write to an existing CR in internal/uiapi (so the UI and the CLI, backend or fallback) goes through one Server.requireUntracked, which runs gitopstracking.IsTracked and prmode.Decide(trackedBy, false) and refuses anything but ModeDirectWrite with FailedPrecondition; a failed check refuses too; Create is not gated; the RBAC is two cluster-scoped rules, one narrowed by resourceNames | Superseded by ADR 0101 |
| 0052 | Render spec.nodeLabels into KubeNodeConfig (contract 1.14+) or machine.nodeLabels (older) right after layer 1, so the operator's label beats a TalosStorage preset label on the same key, and fail the render for labels the kubelet's NodeRestriction would refuse | Accepted |
| 0053 | ichor restore --recover-cluster runs standalone (no Kubernetes, no Ichor): it reads the local secrets bundle, takes S3 credentials from environment variables, picks the newest snapshot that decrypts and passes the etcd checksum (or the one named by --snapshot, which is refused if broken), asks for confirmation, then uploads it with EtcdRecover to one freshly installed control plane and calls Bootstrap{RecoverEtcd: true}; restoring into a live cluster is a separate controller-run operation (ADR 0089) | Accepted |
| 0054 | The TalosMachine webhook refuses an approval (update to approved, or create already approved) that would leave more than spec.controlPlane.count+1 approved control planes, counted by deleteguard.CountApprovedControlPlanes; the TalosCluster webhook refuses schedulable turning "false" with no approved worker, and the TalosMachine webhook refuses an approved worker leaving (delete, un-approve, re-role, move) when that would leave none; both webhooks now match CREATE/UPDATE as needed, not DELETE only; the race admission cannot close is closed by a controller-side hold of the newest surplus approvals (ControlPlaneOverCap) | Accepted |
| 0055 | The generated Cilium Application carries ignoreDifferences: [{group: cilium.io, kind: CiliumIdentity}] and ServerSideApply=true, without RespectIgnoreDifferences=true; Hubble certificates stay on hubble.tls.auto.method=cronJob; Kokumi cannot carry either setting today, so that case is an open question; the generated Cilium and root Applications use automated sync with prune and selfHeal (addendum 2026-10-26) | Accepted (amended 2026-10-26); superseded in part by ADR 0099 |
| 0056 | TalosCluster.spec.gitops carries mode (relaxed default, strict) and an optional repository connection (host, https API URL, owner/repo, https clone URL, base branch, a tokenSecretRef read in the UI backend's own namespace, machinesPath required for strict); Server.requireUntracked becomes planWrite, which sends a CR Argo CD tracks to a pull request when a connection exists and refuses otherwise, and sends a strict approval to a pull request that adds the TalosMachine file; changing mode rewrites nothing; the response carries pull_request_url and the audit action gets a -pr suffix; amended 2026-10-26: the PR branch is ichor/<verb>-<name> (since 2026-10-27 ichor/<verb>-<namespace>-<name>) and a request whose head branch already has an open PR returns that PR instead of opening another, and deleting a tracked TalosConfigPatch becomes a PR that removes its document (or its file) | Superseded by ADR 0101 |
| 0057 | Execute kind: Kubernetes as a per-node AdvanceMachineFunc that applies the freshly rendered machine configuration without a reboot (only the component image fields may differ), pre-pulls the images, verifies the node from the Kubernetes API, and syncs the bootstrap manifests once the last control plane runs the target; it refuses a skipped minor, an unsupported Talos pairing and an unfinished CNI hand-over before touching a node, and only warns about removed APIs in use | Accepted |
| 0058 | with ichorUI.oidc.enabled (default off) every ichor-ui RPC needs an OIDC ID token that github.com/coreos/go-oidc/v3 verifies (signature, issuer, audience = client ID, expiry), the backend calls the Kubernetes API as that user with Impersonate-User/Impersonate-Group so the API server alone decides what they may do, and AuditRecord.actor and the PR's Requested by: are that Kubernetes user name; the UI is a public client doing authorization code + PKCE in the browser (no client secret exists); the impersonate grant is on users and groups only, rendered only when OIDC is on and narrowable by name; with OIDC off nothing changes except a startup warning and a UI banner; the CLI does not forward credentials | Accepted |
| 0059 | Add the cluster-scoped singleton IchorInstallation (managementMode, target version, installer repository; installed version reported by the controller), execute kind: ichor by creating a one-shot installer Job that runs the target version's own image and embedded manifest (snapshot in ConfigMaps, additive-only CRD check, CRDs then controller then UI with post-checks, rollback from the snapshot), refuse CLI writes more than one minor from the server, and never launch the Job in external mode | Accepted; superseded in part by ADR 0090 (no air-gap requirement) and in part by ADR 0101 (decision 5 and the external mode) |
| 0060 | TalosMachineReconciler returns ctrl.Result{RequeueAfter: transientRecheck} (30s, one constant beside overCapRecheck) with a nil error after writing Ready=False for a cause that time can fix; a cause that waits for an operator returns an empty Result; the status write is the report and the interval is fixed, so no error is returned for backoff | Accepted |
| 0061 | TalosMachineReconciler watches TalosStorage, TalosConfigPatch and TalosCluster through one mapping, machinesOfCluster (every machine of the object's cluster), filtered to creates, deletes and spec changes (predicate.GenerationChangedPredicate); a Secret named by registries[].authSecretRef (controller namespace only) maps through an index on TalosCluster to the machines of the clusters that reference it, and a hash of the resolved credentials joins the desired-inputs hash (follow-up below) | Accepted |
| 0062 | a new GitOpsService backs the GitOps page with three read-only RPCs: ListGitOps (every TalosCluster, TalosMachine and TalosConfigPatch the caller can list, with its tracking Application's status.sync.status and status.health.status, read with get only, no new RBAC), FindPullRequests (open pull requests of one CR, asked of the Git host only when a row's button is pressed, answered from a 30 s per-replica cache, one host call per verb the kind can propose) and GetLinks; external tool URLs are registered in the chart's values (ichorUI.links.{argocd,grafana,hubble,longhorn}, env ICHOR_LINK_*), not in a CRD spec, and a tool with no URL gets no link; links are http(s) only, checked in the backend and again in the UI with the function pull request URLs already use; Argo CD links go to the CR's Application, Grafana to the bundled dashboard's fixed uid, Hubble and Longhorn to their front page | Superseded by ADR 0101 |
| 0063 | CertExpiryReconciler reads the CAs in the secrets bundle (talos, kubernetes, kubernetes-aggregator, etcd) and publishes each notAfter in TalosCluster.status.certificates and as ichor_certificate_expiry_timestamp_seconds; talosctl rotate-ca is not reused whole; rotating the CAs does not change the etcd backup key; the TalosOperation kind CARotation is decided below but is deliberately not implemented in this change; the ca-rotation TalosOperation kind is built as its own state machine on the ADR 0026 backbone: three verified passes, the new CAs saved to the Secret's rotation.yaml before any node is told, secrets.yaml replaced last, BundleBackupStale until ichor cluster bundle-export (see "Implementation") | Accepted (the stage sequence and the operator's credentials are amended by ADR 0091) |
| 0064 | The UI's fixed wording lives in two typed objects, en and ja, in ui/src/lib/i18n.ts, read through one t() function, with no i18n dependency; the language is the saved choice, else navigator.languages, else English, switched on screen and kept in localStorage with nothing on the server; names, field names, state words from Argo CD or the cluster, log lines and the body of apiserver or Talos errors are shown as received; dates and sizes go through Intl for the chosen language | Superseded by 0070 |
| 0065 | override is a request flag (Override{reason} on approve, reject, update-patch and delete-patch) that, only where the answer would otherwise be a pull request or a refusal because Argo CD tracks the resource, writes the CR directly and then opens the same pull request ADR 0056 would have opened; it is allowed only when Kubernetes RBAC says so, asked with a SelfSubjectAccessReview made as the user (verb override, group ichor.maesh.dev, resource talosmachines or talosconfigpatches, the object's name and namespace), so Ichor keeps no list of who may; with OIDC off (the backend on its own ServiceAccount) it is refused, and the CLI's --override always runs in-process on the caller's kubeconfig; the reason is optional (ADR 0110) and goes to AuditRecord (action <verb>-override), a Warning Event Override on the resource, and the PR title and description; Ichor never touches an Application's sync policy, so the response carries a notice that says the write will be reverted if the Application has selfHeal on, and to pause it in Argo CD until the PR merges; a validating webhook (/validate-ichor-maesh-dev-v1alpha1-override, UPDATE and DELETE of talosmachines and talosconfigpatches, fail-closed) refuses, unless the requesting user holds the same verb (a SubjectAccessReview on the admission request's userInfo), a direct update or delete of a CR that gitopstracking.IsTracked reports as tracked, and the removal of an ichor.maesh.dev/* finalizer from an object that is being deleted; Ichor's own controller, Argo CD's application controller, Flux's kustomize and helm controllers and the two kube-system controllers that cascade deletions are exempt by exact username; when tracking cannot be determined the write is refused; the UI offers the path after a pull request or a refusal, asks for a required reason and shows the response notice | Accepted (UI and webhook added 2026-10-26); superseded in part by ADR 0101 (the tracked-write part); amended by ADR 0110 (the reason is optional) |
| 0066 | Add TalosOperation kind cni whose per-machine step applies the Talos-side kube-proxy mode (rendered by pkg/render from status.cni.kubeProxy, falling back to spec.cni.kubeProxy); replacing waits for Cilium to serve Services before touching any node and deletes the kube-proxy objects last, enabling brings kube-proxy up and verifies it before asking Cilium to step aside; Ichor never writes the CNI's configuration (hand-over or not) and leaves node-level kube-proxy rules to the next reboot | Accepted |
| 0067 | a TalosOperation of kind shutdown stops a cluster: workers first (drained), then every control plane and the controller's own node in one forced burst; while any shutdown operation has not Completed, acquireDisruptiveLock (both of them) refuses everything else; the operation completes by itself once every machine answers on a new boot; no CLI command, no etcd snapshot taken by the shutdown | Accepted |
| 0068 | an OperationService (list, get, plan, create, pause/resume, delete) is the one entry point for TalosOperations, used by `ichor upgrade | cni |
| 0069 | TalosCluster.spec.controlPlaneMetrics {enabled, allowedCIDRs} (off by default; enabled needs at least one CIDR, /0 rejected) renders, on control-plane nodes only and in the system layer, bind-address: 0.0.0.0 for kube-controller-manager and kube-scheduler, listen-metrics-urls: http://0.0.0.0:2381 for etcd, and one NetworkRuleConfig for TCP 2381, 10257 and 10259 that drops every source outside allowedCIDRs, with Talos's default firewall action left at accept; the firewall kinds are no-reboot in pkg/fieldmeta, the etcd argument stays reboot-required; the chart gets no new monitoring resource. | Accepted |
| 0070 | The UI has one language, English. The ja dictionary, the language selector, and language detection and storage are removed; en stays in ui/src/lib/i18n.ts behind t() / useT(); dates and sizes use Intl with en, not the browser locale; the page header carries a light/dark switch that follows prefers-color-scheme until the user chooses, then keeps the choice in localStorage | Accepted |
| 0071 | The machine list drops its "Stage" column. Machine.stage is only filled by the node view (GetNode), which dials the machine; the list reads CRs only. The approved list shows the TalosMachine Ready condition (status and reason) instead; the pending list shows no status column. | Accepted |
| 0072 | The patch edit page opens with a short explanation of what a patch is, describes each layer and the priority rule next to the fields, shows each document as a card (kind and key fields, with Edit and Remove) whose editor says what the chosen kind configures and names its fields, and reminds the user to run the impact preview when it is missing or out of date, without ever blocking Save; the list adds what each patch targets, as written | Accepted |
| 0073 | NodeService.GetNodeTalosDetails returns five independently failing sections (system, network, storage, services, extensions) read from the node over the existing read-only Talos client; every field is copied by name, nothing from config, certificates, keys or tokens is carried, and kernel arguments that could hold a credential are masked; an unapproved machine is read over the insecure maintenance API and flagged maintenance_mode | Accepted |
| 0074 | The header wraps on phones, with the nav on its own row that scrolls inside itself; the modal scroll lock stops padding the body, and html gets scrollbar-gutter: stable | Accepted |
| 0075 | adopt reads approved machines only; for the v1.14 documents Ichor cannot yet express it says which document and why. Whether those documents become patch kinds or spec fields is left open (below). | Accepted |
| 0076 | an approved, never-converged machine that the bundle's identity cannot read is probed on the maintenance API; if it answers there and is the recorded machine and the render names an install disk, the render is applied once insecurely, and the node is then followed by requeue until MachineStatus says running. | Accepted |
| 0077 | TalosCluster.spec.network {podSubnets, serviceSubnets} (optional, fixed once set) and TalosMachine.spec.install {disk, image} carry the values ADR 0075 could not express; the system layer renders them as the v1.14 documents (KubeNetworkConfig, UnattendedInstallConfig) from that contract and as the legacy fields before it, and drops Talos's own CNI (the KubeFlannelCNIConfig document, or cluster.network.cni.name: none) for every preset but flannel; approval (ichor machine approve --install-disk, the UI form) sets the install disk, ichor init stops writing machine.install, hostnames are RFC 1123 checked, and adopt names the spec field for each such drift path. This is option C of ADR 0075 with the per-node half also as a spec field, not a patch kind. | Accepted |
| 0078 | The UI is a left sidebar (preactcn sidebar, icons-collapsible, a sheet on phones) plus a sticky top bar with a breadcrumb; detail pages have a header and card sections with stable ids, and long ones get a sticky right-hand "On this page" list instead of tabs | Accepted |
| 0079 | StreamNodeLogs without a resume token starts from the last tail_lines lines (default 500, cap 10000, negative for the whole history) instead of the beginning of the log; follow=false returns that snapshot and ends; the UI viewer keeps the newest 2000 lines, appends in 100 ms batches, shows at most 1000 characters of a message until expanded, follows the tail only while scrolled to the bottom, can pause, backs off between reconnects and shows stream errors inline. | Accepted |
| 0080 | a new TalosOperation kind config applies the pending reboot-requiring changes of the approved machines, one at a time (control planes first with an etcd membership check before each, then workers within maxUnavailable, the controller's own node last), staged and then rebooted; the machine reconciler only reports them, now naming the fields, and never reboots a node | Accepted |
| 0081 | The demo cluster's control plane is created by ichor init and its worker approved through ichor machine approve --install-disk, so every node's configuration is rendered by Ichor; images are pushed with crane to a registry:2 Deployment in ichor-e2e and reached through an anonymous spec.registries mirror of ghcr.io that ichor init --registry-mirror writes; init also gains --pod-subnets / --service-subnets, waits for the node, the API server and the webhooks, and applies the install manifest before the bundle Secret; the controller's shipped RBAC and cache are made consistent. | Accepted |
| 0082 | a second read-only RPC, NodeService.GetNodeTalosRuntime, returns six independently failing sections (etcd, top processes, containers, network state, kernel modules, the Kubernetes view of the node) under ADR 0073's rules; process command lines, container environments and arguments, WireGuard keys and machine configuration are never carried; the ichor-ui ClusterRole gains get;list on pods; actions the SDK enables are listed below as a proposal, none is implemented | Accepted |
| 0083 | The cluster page lists its nodes and links to an Add node page that explains how a node joins, shows spec.discovery and adds one address through a new ClusterService.AddDiscoveryAddress write that passes the same gate as every other write; every successful change shows a sonner toast; every dangerous action confirms in a preactcn alert-dialog; the node log lines render in a preactcn code-block; amended 2026-11-02: AddDiscoveryCidr, RemoveDiscoveryAddress and RemoveDiscoveryCidr, one Add node input for an address or a range, ichor cluster discovery, and the spec write itself starts the scan | Accepted |
| 0084 | three read-only downloads from a node's Talos API are served by the UI backend (DiagnosticsService, server-streaming byte chunks over the existing Connect transport): DownloadSupportBundle (one node, or every approved node of a cluster), DownloadEtcdSnapshot (a control plane node) and CapturePackets (one interface, 30 s by default and 5 min at most, 100 MiB at most, optional validated BPF filter); each needs the Kubernetes RBAC verb download on the subresource talosmachines/support-bundle, talosmachines/etcd-snapshot or talosmachines/packet-capture, checked with a SelfSubjectAccessReview as the caller, and granted to nobody by default, including the ichor-ui ServiceAccount; the chart grants it to named groups or, if the operator insists, to the ServiceAccount; every download records a Warning Event DiagnosticsDownload on the TalosMachine before any byte is read and is refused if that Event cannot be recorded; the backend mints a Talos client certificate per call with the least role the call needs (os:reader, os:etcd:backup, os:operator); nothing is stored; the UI adds an Actions menu on the node page, the CLI adds ichor node support-bundle, ichor node pcap and ichor etcd snapshot | Accepted |
| 0085 | items 1, 2, 4, 6, 7 and 8 of ADR 0082's proposal, plus a per-node shutdown, are one new TalosOperation kind node with spec.node {machineRef, action, service, allowLeader, confirm} and one action per operation: reboot, shutdown, restart-service (kubelet, containerd, cri only), etcd-defrag, etcd-forfeit-leadership and reset; "Remove node" is the existing deletion of the TalosMachine through the removal flow (a new RemoveMachine RPC and ichor node remove), not an operation, and etcd member removal stays inside it (ADR 0032); the UI and CLI write only these CRs, the controller is the only writer to the node | Accepted |
| 0086 | the controller never mutates TalosMachine.spec.endpoint; when a node answers somewhere else, the address is recorded in status.observedEndpoint (with status.observedEndpointFor, the spec.endpoint it replaces) by discovery, which matches the identity (ADR 0011), or by the machine reconciler, which reads the InternalIP of the node's Kubernetes Node; every dial of the node's Talos API uses DialEndpoint() (the observed address while it still supersedes spec.endpoint, else spec.endpoint); a converged machine is re-probed every two minutes so Ready turns False when the node is gone | Accepted |
| 0087 | the kubernetes executor visits the machines in four stages, recorded in status.kubernetes.stage: (1) APIServers — each control plane, one at a time, gets kube-apiserver at the target with controller-manager, scheduler, kubelet and kube-proxy pinned to the version its node runs; (2) ControlPlaneComponents — each control plane gets controller-manager and scheduler at the target; (3) KubeProxy — each control plane gets kube-proxy at the target, then the bootstrap manifests are synced and the DaemonSet's rollout awaited; (4) Kubelets — control planes, then workers within maxUnavailable, get the full rendered configuration, which is the cluster's desired state; pkg/render takes the pins as a system-layer input (SystemInputs.Staged) that only the executor sets, nothing is written to spec, status.machines is cleared at each stage change, an operation with no recorded stage starts at APIServers, and the plan names the stages and warns (does not refuse) about a minor upgrade of several control planes behind an endpoint of type node | Accepted |
| 0088 | render.Version (an integer in pkg/render/version.go) is part of every machine's input hash (desiredInputsHash), so an Ichor release whose renderer emits different output for the same CRs reads as a desired-state change and is applied like any other, while a node that changed on its own with the same version is still drift; the version is bumped by hand in any change that alters render output, and TestRenderVersionMatchesOutput fails when the output of a fixed set of inputs (secrets removed) changes without Version and the pinned fingerprint moving together | Accepted |
| 0089 | restoring a stored etcd snapshot into a running cluster is a new TalosOperation kind etcd-restore with spec.etcdRestore {backupRef, snapshot, confirm, reason} (confirm must equal the cluster name, reason is the override reason; CEL and the override webhook enforce both, and the verb is checked on the TalosCluster); the controller takes the cluster lock, decrypts and checksums the snapshot and keeps a pinned ciphertext copy of it in the bucket, writes a marker object next to the backups, wipes etcd on every control plane at once (Talos's own disaster-recovery procedure: non-graceful reset with reboot of EPHEMERAL), uploads the snapshot to one control plane and bootstraps it with RecoverEtcd, waits for the others to rejoin and the API to answer, asks every TalosMachine to reconcile again and waits until each is Ready; because the rollback deletes the operation itself, the progress that must survive lives in the marker, the wipe-and-recover runs as a job independent of the manager, and the operation is recreated from the marker; ichor restore --live and the cluster page's "Restore from backup…" create the CR | Accepted |
| 0090 | air-gapped environments are no longer a requirement, so ichor images list (with --kokumi) and the design's air-gap section and phase-10 "air-gap verification" are removed, while registry mirrors, registry auth, ichor init --registry-mirror, the installer-repository override and the self-hosted Image Factory URL stay because pull-through caches and private registries need them | Accepted |
| 0091 | the ca-rotation stages are Accepting, AwaitingCredentials, Issuing, Dropping, Finalizing, Done; AwaitingCredentials changes nothing on any node and the controller leaves it only when the operation carries the annotation ichor.maesh.dev/credentials-handoff with the value talos=<new talos CA fingerprint>,kubernetes=<new kubernetes CA fingerprint> (ichorv1alpha1.CARotationHandoffToken); the CLI sets it, the controller only reads it and never writes spec; ichor cluster rotate-ca waits for the gate through the Kubernetes API, then writes a kubeconfig and a talosconfig that trust both the old and the new CAs and are signed by the new ones, checks that the new kubeconfig authenticates to the cluster, and only then sets the annotation; ichor cluster credentials NAME writes the same files from the cluster (--acknowledge also sets the annotation) or, with --bundle FILE, offline from a bundle-export file; rotate-ca --follow follows through the Kubernetes API with the new kubeconfig, not through the ichor-ui tunnel; the UI offers no download here, which ADR 0093 reverses: it adds an admin-gated download and a continue button | Accepted |
| 0092 | TalosMachine.spec.network (one primary link chosen by permanent MAC, DHCP or static addressing, optional VLAN, DNS servers) and TalosMachine.spec.disks (roles ephemeral and userVolume on disks identified like spec.install.disk) are written at approval by the UI form and ichor machine approve flags; pkg/render emits a LinkAliasConfig alias net0 with a CEL selector on the permanent MAC and refers to the alias from DHCPv4Config, LinkConfig, VLANConfig and ResolverConfig, a VolumeConfig for EPHEMERAL and a UserVolumeConfig per volume, and the legacy machine.network.interfaces before the multi-document contract; link addressing documents are classified live in pkg/fieldmeta, aliases and everything else keep the reboot-required default; bonds, the storage role and unused are not done | Accepted |
| 0093 | OperationService gains DownloadCARotationCredentials (server-streaming, one zip with kubeconfig and talosconfig, the files ichor cluster credentials writes) and AcknowledgeCARotationCredentials (builds the new kubeconfig in memory, checks that it authenticates to the cluster's Kubernetes API, then sets ichor.maesh.dev/credentials-handoff with CARotationHandoffToken); the download follows ADR 0084's authorization model: the Kubernetes RBAC verb download on the subresource talosclusters/credentials, checked with a SelfSubjectAccessReview as the caller, granted to nobody by default (the chart's ichorUI.credentials.groups, or serviceAccount: true if the operator insists), a Warning Event CredentialsDownload on the TalosCluster before any byte is produced and a refusal if it cannot be recorded; the download is allowed from AwaitingCredentials to Finalizing, the acknowledgement only in AwaitingCredentials; the acknowledgement needs no download and no new RBAC (it is an update of the operation, like pausing); the operation page offers "Download new credentials" (only to a caller who holds the verb) and "Continue rotation" (behind a confirmation); the credential builder is the shared package internal/credentials, used by the CLI and the UI backend | Accepted |
| 0094 | Ichor keeps its own users as IchorUser CRs (namespace ichor-system, spec = display name, e-mail, roles[], disabled; never a password) with argon2id hashes in one Secret ichor-user-credentials, and ichor-ui becomes a minimal stateless OIDC issuer (hand-written on go-jose/v4, RS256, authorization code + PKCE for the browser and a loopback redirect for the CLI, 5 minute ID tokens, 12 hour non-rotating refresh tokens, signing keys kept and rotated by the controller in a Secret); the controller renders both Ichor's issuer (reached by the kube-apiserver through discoveryURL on a ClusterIP with an Ichor-made CA) and the existing external IdP as two jwt[] entries of Talos's KubeAuthenticationConfig document (Talos v1.14 contract and later), usernames ichor:<name> and oidc:<name>; permissions are plain Kubernetes RBAC: three built-in ClusterRoles ichor-viewer, ichor-operator, ichor-admin bound to the groups `ichor:viewer | operator |
| 0095 | ichor operation retry NAME (and a Retry button and the OperationService.RetryOperation RPC, next to pause and resume) sets the annotation ichor.maesh.dev/retry=<RFC 3339 time> on a TalosOperation that is Paused with a failureReason; the controller, when that time is newer than the new status.lastRetry, clears failureReason, records lastRetry and runs the failed machine's step again; resume stays as it was (it lifts spec.paused only) and says so when the operation has failed; ichor node actions also accept the machine's hostname or Kubernetes Node name instead of the TalosMachine name | Accepted |
| 0096 | any create or update that sets or changes a field naming a Secret needs the verb update on the subresource secret-references of that resource (talosclusters/secret-references, talosetcdbackups/secret-references), asked of the requesting user with a SubjectAccessReview in the validating webhook that already gates spec.authentication; the controller and the GitOps engines are exempt as there; writing the same value again is not gated and neither is removing a reference; ichor-admin holds the right, ichor-operator and ichor-viewer do not, and the ichor-ui ServiceAccount does not need it | Accepted |
| 0097 | Ichor manages exactly one cluster, now and later, and the UI is built for that: there is no Clusters list, the Overview is the cluster, every object URL drops the namespace, and a top bar shows the cluster's name, its Talos, Kubernetes and Ichor versions and a health badge (Ready, Degraded, Upgrading) | Accepted |
| 0098 | BootstrapManifest for the Cilium preset returns the pinned Cilium chart expanded with the preset values into plain manifests, rendered with Helm's Go SDK from a chart archive embedded in the binary, and the tunnel protocol is a preset option that defaults to Cilium's VXLAN | Accepted |
| 0099 | ichor init always installs Cilium, Kokumi and Argo CD with no opt-out; Cilium is the only CNI, bootstrapped through Talos inline manifests and then owned by Ichor as a Helm release run by an installer-style Job with user values stored on the TalosCluster; Kokumi and Argo CD are applied once, then managed by Argo CD from artifacts the Ichor controller publishes into Kokumi's in-cluster registry, so no Git repository is needed | Accepted; superseded in part by ADR 0101 |
| 0100 | ichor-system gets a default deny and one explicit allow per flow the controller, ichor-ui and the installer Job need, written as CiliumNetworkPolicy with the entities kube-apiserver, host, remote-node, world and cluster; the policies are built from config/cilium-network-policy into dist/network-policies.yaml, embedded next to the install manifest, applied by ichor init after Cilium is Ready and by the installer Job on an upgrade when Cilium's CRD exists, and rendered by the chart behind networkPolicy.cilium.enabled | Accepted |
| 0101 | Argo CD and Kokumi are not requirements and Ichor no longer installs, publishes or manages either; the whole standard stack (the stack reconciler, the ichor-stack Jobs and RBAC, pkg/stack, the hidden ichor stack apply, the stack wait and Kokumi password print in ichor init, the stack network policies) is deleted, while Cilium stays Ichor's own Helm release run by the ichor-cni Job, and the CRD fields it left unused (TalosCluster.spec.gitops, TalosCluster.spec.cni.handover, IchorInstallation.spec.managementMode, spec.stack, status.stack) are removed, not deprecated, through the installer's explicit retired-field list; GitOps tools, Flux included, are permanently out of scope | Accepted |
| 0102 | the e2e harness keeps one cached Talos ISO per Talos version and schematic (ichor-e2e-talos-<version>-<schematic8>-base, in ichor-e2e) that no down* deletes, and each node's ISO disk is cloned from it instead of importing the ISO again; ICHOR_E2E_BASE_CACHE=0 keeps the old per-node import; a pre-installed disk image is not cached. | Accepted |
| 0103 | Ichor adopts a running Talos cluster on Talos v1.12 or later (older is refused with "upgrade Talos first"); a Cilium cluster is taken over as Ichor's own Helm release through a diff-checked TakeOwnership Job, while any other CNI (and a Cilium the user wants left alone) is adopted only on the operator's explicit request in a new CNI-unmanaged mode, spec.cni.preset: unmanaged, in which Ichor neither installs, operates nor renders the CNI and skips its CiliumNetworkPolicies unless Cilium is present; it recovers the cluster's secrets bundle from a control plane's machine config, holds every adopted machine until zero drift is verified, only advises (never starts) a CA rotation afterwards, and refuses to join an already-configured node unless that node is reclaimed, either by ichor machine reclaim with the operator's own talosconfig or, as an override path that is off by default, from the UI through a short-lived, single-node, auto-deleted credential Secret, a typed-UUID confirmation and an audit Event; reclaim resets only STATE and EPHEMERAL | Accepted; built; hardware-verified including A8 (its two gaps were closed by 8ebee126 and 26da779b; rerun passed 2026-10-30, see "Results, A3, A8, D4, D6 (2026-10-30)"; the second kill window remains unhit); Updated: 2026-10-30 (C6 and C7 hardware-verified, see C6 and C7 below) |
| 0104 | ichor adopt on a Talos cluster with no CNI installs Ichor's pinned Cilium itself, in the adopt process with the Helm SDK as Ichor's own release record in ichor-system, before Ichor's own pods are installed, and adopts the cluster as the managed cilium preset (so ADR 0103's "no CNI becomes unmanaged talosCNI: none" row is replaced, and --cni-unmanaged is refused there); a cluster that already has a CNI keeps ADR 0103's rules and its network-policy gate follows live detection; every wait for Ichor in adopt and init is bounded and names why the pods are Pending; and Ichor can be installed with helm install of dist/chart (published as an OCI chart), after which ichor adopt detects the existing install, skips its own install step and registers the cluster, with a new ichor cni bootstrap for the one case Helm cannot cover, a cluster with no pod network. | Accepted; built; hardware-verified, E5 and E6 included (E5 passed on 2026-10-30 after the pending-record repair; see Hardware results) |
| 0105 | ichor-ui never serves plaintext to the network: it listens for HTTPS only on a new port 8443, with the certificate the operator supplies (ichorUI.tls.secretName, for example a cert-manager Secret) or, when none is supplied, a leaf from the Ichor CA that the controller already keeps for the 8444 issuer listener (ichor-oidc-ca, ichor-oidc-tls), issued for every install and not only with users enabled; the controller publishes that CA, public part only, as a ConfigMap and a Secret named ichor-ui-ca so an Ingress-NGINX, a Gateway API BackendTLSPolicy or a client holding --ca-file can verify the backend, while a plain L4 load balancer needs the operator's own certificate; an edge re-encrypts to ichor-ui; the kube-apiserver's 8444 fetch is unchanged and keeps the Ichor leaf even when the operator supplies a certificate for 8443; an existing install keeps a deprecated plaintext listener on 8080 for exactly one minor release (never on a fresh init or Helm install) and then loses it; certificate hot reload is the work of ADR 0094's amendment (branch cert-reload) and is used here, not redesigned; Ichor's runtime settings are new optional fields of the existing singleton IchorInstallation (not a new kind), live-read by the controller, the webhooks and ichor-ui, edited from a Settings tab and by ichor settings, guarded by RBAC verbs checked in the webhook; a Helm install gets an IchorInstallation too, created once by the controller if absent, and a new field spec.deployment.manager (installer or helm) is how ichor upgrade ichor, the UI, the webhook and ADR 0104's ichorPresent tell the installer Job path from Helm | Accepted; built (unit 10, the plaintext removal, done early by the user's decision on 2026-10-30); hardware-verified (T1 to T8) 2026-10-30 |
| 0106 | a TalosConfigPatch is dry-rendered by a validating webhook (create and update) against every approved machine of its cluster that it selects, with the controller's own render.Render, and refused with the render error when the render would fail and did not already fail without the patch; the generated Config mirror makes machine, cluster, machine.type and machine.certSANs optional (an additive loosening) so a sparse Config document no longer repeats them and no longer reads as "needs a reboot"; ichor restore --live and the TalosEtcdBackup condition StoreInCluster refuse or warn when the backup endpoint is inside the cluster being restored; a TalosMachine whose node cannot be read reports Ready=False NodeUnreachable with the endpoint and, when the Kubernetes Node shows another InternalIP, the likely new address (spec is never changed); ichor node remove of a control plane below the declared count is allowed when its etcd member is unhealthy or unreachable. | Accepted; hardware-verified (H2 and H7 re-verified 2026-10-30) |
| 0107 | there is no installation-wide switch for resetting a machine that another Talos cluster configured and adding it here; who may do it is decided by RBAC alone, the custom verb reclaim on machinereclaims, checked by the webhook with a SubjectAccessReview as the requesting user (as before); ichor-admin (group ichor:admin) holds that verb by default, ichor-machine-reclaimer stays for granting it to anyone else, editors and viewers do not get it; IchorInstallation.spec.reclaim.enabled and the chart value reclaim.enabled are deprecated and ignored (the CRD field stays, append-only; the admission webhook warns when it is set; ichor settings set reclaim.enabled=... and UpdateSettings refuse it with a pointer to this ADR; status.effective stops reporting it; the SettingsGate no longer has a ichorinstallations/reclaim check); the UI calls the feature "Reset and add" (dialog "Add a machine from another cluster"), offers it on the Add node page next to the AlreadyConfigured message and on the Machines page to users who hold the verb, and drops the reclaim card from Settings; the kind MachineReclaim and the command ichor machine reclaim keep their names. | Accepted; built |
| 0108 | Ichor is hardened area by area after five read-only audits (2026-11-02): ichor-ui and its auth, the controller's privileges and admission, workloads and the supply chain, the security of the clusters Ichor creates, and secrets and credentials. Fixes to Ichor itself are made directly. For the clusters Ichor creates, every new hardening is a typed option that is off by default, for new and existing clusters alike; disk encryption stays opt-in (the user's decision, 2026-11-02). Each area below records its decisions and what was built. | Accepted; built, hardware-verified (see the Hardware results sections); the five items it deferred as follow-ups are all built since (ADR 0120 to 0127, see the amendments after "Follow-ups") |
| 0109 | the permission-gated override of a safety check with a recorded reason (ADR 0065, amended by 0101, 0107 and 0108) is called "override" everywhere, and the earlier name is removed from code, manifests, the UI, the CLI and the docs. The mechanism is unchanged: the user writes a reason, Ichor asks Kubernetes with a SubjectAccessReview made as that user whether they hold the verb, and the reason goes into the audit record and an Event. The CLI flag is --override (the reason is --override-reason and is optional, ADR 0110), the RBAC custom verb is override, and the old verb is no longer accepted. | Accepted |
| 0110 | the override of a safety check is requested with a flag (--override, override on a node operation), and a reason is optional, recorded with --override-reason or overrideReason; the RBAC verb override stays the only gate, and an override with no reason is recorded as "no reason given". | Accepted |
| 0111 | after the new manifest is applied and checked, the installer deletes the objects that the last successful run recorded and the new manifest no longer ships, and the ClusterRole that ADR 0109 renamed when an earlier build still holds it. It deletes an object only when the object carries ichor.maesh.dev/installed-by=ichor-installer (for the renamed ClusterRole from before that label, app.kubernetes.io/name=ichor instead), is of a kind the installer may delete, and is cluster-scoped or in the installer's namespace. CRDs, Namespaces, Secrets, PriorityClasses, CiliumNetworkPolicies and unlabelled objects are never deleted. This is how a later version renames or drops an object. | Accepted |
| 0112 | discovery records the installable disks and the physical NICs with a permanent MAC that a node reports through its maintenance-mode API on TalosMachine.status.discoveredDisks and status.discoveredLinks; the approval form lets the person pick the install disk, the disk roles and the NIC from those lists, starting on the smallest disk of at least 10 GiB and on the first NIC with a link; a typed device path is offered only while neither list has anything; an approval whose pick is not in the recorded lists gets an admission warning, not a refusal. | Accepted |
| 0113 | the Talos release, its version contract, the Kubernetes release and the Cilium chart are one release set embedded in each Ichor build (pkg/release); init and adopt use it, every upgrade moves only to it, and ichor upgrade cluster (or the Versions card's one button) brings a cluster to it one step at a time. | Accepted · Superseded in part by 0115 |
| 0114 | the controller (leader only, at start and every 6 hours with jitter) lists the chart repository's tags on ghcr.io with Helm's registry client, takes the highest final semver release, and records it in IchorInstallation.status.latestRelease; the UI shows "vichor version names it, and the check is switched by spec.updateCheck.enabled or the chart's updateCheck.enabled (default on), with a controller-only egress policy to the registry host that exists only while the check is on | Accepted |
| 0115 | the release set in pkg/release (Pinned) decides the versions of a first install (ichor init, the CLI defaults, the UI's proposed values); after that Talos, Kubernetes and Cilium upgrade on their own, to any version the compatibility rules accept, and the set does not limit them. Each upgrade target must be a final semantic version newer than what runs, on a path the component's rule allows, and compatible with the other components in one table (pkg/release/compat.go) whose refusal only the override verb (ADR 0110) passes, recorded in the cluster's Events. | Accepted |
| 0116 | IchorInstallation.spec.autoPatch (and the chart's autoPatch, three controller flags) is a setting that is off by default; when it is on, a leader-only controller loop, once a minute, starts at most one upgrade TalosOperation per cluster for the newest patch of the running minor of Talos, then Cilium, then Kubernetes, taken only from the candidates the release check recorded in status.upgradeTargets and filtered by the compatibility table, only inside the configured daily UTC window, only while every approved machine is Ready and nothing else holds the cluster; the operation carries the annotation ichor.maesh.dev/auto-patch: "true" and started-by: ichor-auto-patch and runs through the normal executor; a failed or paused automatic operation halts the patcher (condition AutoPatchHalted, status.autoPatch.haltedOperation) until a person writes spec.autoPatch.acknowledgedHalt, or the operation completes or is deleted, or a later manual upgrade of that cluster completes; the patcher never retries. The controller itself never writes a TalosCluster spec: the version of a Talos or Kubernetes patch, and the operation, are written by a separate identity, the ServiceAccount ichor-auto-patch, which the controller impersonates (a namespaced Role that names that one ServiceAccount); the account holds only get and update on talosclusters and create on talosoperations, and the admission webhook refuses any write of it except a newer patch of the same minor in those two fields, and any spec write by the controller's own username. The audit log shows the writes as system:serviceaccount:<namespace>:ichor-auto-patch. | Accepted |
| 0117 | ichor init -f init.yaml reads the init settings from an InitConfig file (apiVersion: ichor.maesh.dev/v1alpha1) instead of a long flag line. The file is decoded strictly into one typed Go struct (unknown, duplicate and mistyped keys, and a wrong apiVersion or kind, are errors that name the file and the key) and every value is then set through the init flag it stands for, so the file adds no second code path: parsing, defaults and validation are the flags' own, and a file and the equivalent flags behave the same. A flag given on the command line overrides the file. The required flags (--cluster, --endpoint-address, --node-endpoint, --install-disk) are required unless the file gives them; the error names the flag and the file key. Relative paths in the file (ui.tls.certFile, ui.tls.keyFile) resolve against the file's directory. The file holds paths, never key material. --print-config-template prints the commented example, which a test parses. | Accepted |
| 0118 | ichor prints for a person unless -o yaml or -o json is given: a table or an aligned key/value block, enum names without their type prefix (control-plane, not MACHINE_ROLE_CONTROL_PLANE), the glyphs ✓ ! ✗ … for status and ages like 5m, with colour only on a terminal (NO_COLOR and TERM=dumb honoured, the layout identical in a pipe). The global -o defaults to unset instead of yaml. -o yaml and -o json print exactly what they printed before, so scripts opt in by naming the format. A command with no human view yet prints YAML when -o is unset (machineFormat, used by every writer of a machine format), so nothing prints nothing; every command that reports a result has a human view now, and ichor ui prints its status lines in the same style; what streams or prints raw content (node logs, cni values get, the auth credential plugin) stays as it is. The shared helpers are in internal/cli/human.go. Cluster and Machine in the Connect API gain created_at (for the AGE column) and Cluster gains endpoint_type, pod_subnets, service_subnets and cni_tunnel, so cluster get can show them. | Accepted |
| 0119 | Ichor pins the Cilium version, and the pin moves to a new upstream patch of the same minor through a pull request that CI opens: .github/workflows/cilium-patch.yml runs daily (and on demand) hack/bump-cilium-patch.sh (make bump-cilium-patch), which takes the newest final patch of the pinned minor from the Cilium Helm index, verifies the cosign signatures of the agent, operator-generic and hubble-relay images that chart pins against the Cilium release workflow of that tag (the identity internal/imagesig uses), checks that Cilium tests the minor with the same Kubernetes versions, and only then moves the pins (Makefile version and SHA-256, the go:embed line, ChartVersion, the chart archive, the bootstrap goldens, the version named in the README, the guides and the foreign hardware tests, the compat.go citation); the workflow runs make test and make lint and opens the pull request chore(cni): bump the embedded Cilium chart to X.Y.Z with peter-evans/create-pull-request. A merged bump reaches clusters only with the next Ichor release. The script never crosses a minor; a new minor is a person's decision. | Accepted; built |
| 0120 | the ca-rotation operation can also replace the etcd CA and the Kubernetes aggregator (front-proxy) CA, chosen with the new, optional spec.caRotation.targets (talos, kubernetes, etcd, kubernetes-aggregator; absent means talos and kubernetes, as before). It is the same state machine, lock, pause/continue and finalizer, with the same three passes. The aggregator CA uses Talos's own accepted-CA list (KubeAggregatorCAConfig, acceptedCAs). Talos has no accepted-CA list for etcd, so Ichor puts both etcd CA certificates into cluster.etcd.ca.crt (a PEM bundle, the new key signing from Issuing on) and restarts the etcd members one at a time after each pass; the credential hand-off (AwaitingCredentials) happens only when the Talos or Kubernetes CA is among the targets. The etcd backup key does not change. Before the first hardware run the etcd bundle is a reading of Talos v1.14.5 source, not of its documentation; ADR section "Hardware verification" says what must be seen before it is trusted. | Accepted; built; amended after hardware 2026-11-05 (aggregator rotation verified; the etcd roll by reboot awaits its hardware re-run) |
| 0121 | a new TalosOperation kind key-rotation (spec.keyRotation.targets: serviceAccount, secretbox, or both) rotates the Kubernetes service-account signing key and the kube-apiserver secretbox (encryption-at-rest) key through a status-held state machine of its own, Accepting, Issuing, Reencrypting, SettlingTokens, Dropping, Finalizing, Done: the new key is first only verified or decrypted with on every control plane, then made primary, every Secret is rewritten under it (by the controller, through the admin kubeconfig a Talos node issues, so Ichor's own RBAC does not grow), the old signing key is kept until bound tokens have been re-issued (default 90 minutes, spec.keyRotation.tokenSettle) and no legacy token Secret still depends on it, and only then is the old key dropped. The new keys are saved in the bundle Secret (keyrotation.yaml) before any node is told, and secrets.yaml is replaced last. The etcd backup key is versioned: the old secretbox secret is kept in a new bundle Secret key retired-keys.yaml (and in a file of the same name next to the exported bundle), etcdbackup.Key gains Retired passphrases derived from it, and Decrypt tries them, so backups sealed before a rotation stay readable. A restore of a snapshot older than a key rotation is refused, as for a CA rotation. It needs versionContract v1.14 or later. | Accepted; built; amended after hardware 2026-11-05 (minimum token settle; the rerun awaits) |
| 0122 | Before the controller creates the installer Job of an Ichor upgrade it pins the installer image to a digest and verifies that digest's keyless cosign signature, offline against an embedded Sigstore trusted root, for the identity https://github.com/maeshinshin/ichor/.github/workflows/release.yml@refs/tags/<spec.version> issued by GitHub Actions. The digest is spec.installerDigest when set, else what the tag <installerRepository>:<spec.version> resolves to at that moment (HEAD on the manifest); the Job's image is <installerRepository>@<digest>, so what was verified is what runs. The only way not to verify is an explicit, reasoned, recorded opt-out, spec.installerSignature: {mode: skip, reason: ...} (ichor upgrade ichor --skip-signature-check "<reason>"), which the controller records in status.installerImage and an Event and which the CLI and the UI show as a warning. A failed resolution or verification pauses the operation with a reason and creates no Job. The apiserver enforces the digest form for the installer Job through the existing Job-image admission policy. The release workflow verifies the signatures it has just made with the same Go code before it finishes. | Accepted; built; the positive signed path awaits a release tag built by release.yml |
| 0123 | TalosCluster.spec.security gains secureBoot (`mode: Off | Preferred |
| 0124 | a person with the Kubernetes RBAC verb download on talosclusters/talosconfig-reader gets a short-lived talosconfig whose client certificate carries the Talos role os:reader, for talosctl. It is a new TalosCredentialRequest purpose, talosconfig-reader (default lifetime 1 hour, cap 12 hours, longer is refused at admission and again by the controller), signed by the controller for a key the requester keeps. Every issuance is recorded as an Event on the TalosCluster that names the requesting user, written by the admission webhook before the request is admitted (a request whose Event cannot be written is refused). ichor cluster talosconfig NAME and the UI (the cluster page's "..." menu, a download) are the two ways to ask. The shipped roles ichor-operator and ichor-admin carry the verb; ichor-viewer does not. Ichor offers no os:operator talosconfig. | Accepted; built; hardware verification pending |
| 0125 | TalosCluster.spec.security.kubeletServingCertificates (enabled, verifyFromAPIServer, both off by default, for new and existing clusters) makes every kubelet request its serving certificate from the kubernetes.io/kubelet-serving signer (serverTLSBootstrap), and Ichor's controller approves those requests itself, with a narrow rule, instead of asking the user to install a third-party approver. With verifyFromAPIServer the kube-apiserver also verifies the kubelets against the Kubernetes CA (--kubelet-certificate-authority), but only after every machine has been issued a certificate, and it stays on once on. | Accepted; built; hardware verification pending |
| 0126 | the last world egress rule of Ichor's own policies, ichor-ui to its external OIDC issuer on 443, is replaced by an FQDN rule that the controller derives from the issuer the live ichor-ui Deployment is configured with (--oidc-issuer or ICHOR_OIDC_ISSUER) and writes as the CiliumNetworkPolicy ichor-ui-oidc-egress; the static install copy (config/cilium-network-policy, used by ichor init and the installer Job) therefore carries no world egress rule, and the chart renders one only as a fallback (networkPolicy.cilium.dynamicTalosEgress=false or externalCIDRs) and only while ichorUI.oidc.enabled; the controller's /metrics port is no longer open to the whole cluster entity: neither the static copy nor the chart opens it unless the chart's networkPolicy.cilium.metricsNamespaces names a namespace (optionally narrowed by pod labels, metricsPodLabels), and an installer or ichor init install allows a scraper with a policy of the operator's own, which the guide gives. | Accepted; built; hardware verification pending (FQDN rules and metrics policy) |
| 0127 | ~/.ichor/clusters/<name>/secrets.yaml is written as an age file (armored, one scrypt passphrase recipient, filippo.io/age, already a dependency of the etcd backup format) instead of plaintext. ichor init, ichor adopt --recover-secrets and ichor cluster bundle-export seal new files by default; a passphrase comes from --bundle-passphrase-file PATH (any readable path, so /dev/fd/3 and /dev/stdin work), then $ICHOR_BUNDLE_PASSPHRASE, and on a terminal with neither Ichor generates one and prints it once; with neither on a non-terminal the command refuses before it writes anything, and --bundle-plaintext is the explicit, warned opt-out. Every reader (restore, adopt --secrets-bundle, cluster credentials --bundle, bundle-export's read-back, adopt's stored-bundle comparison) recognises an encrypted file by its header and takes the passphrase from the same two sources, then a no-echo prompt. A plaintext file keeps working and warns on every read; ichor secrets encrypt migrates one. The file name and path do not change, the in-cluster Secret is untouched, and the local kubeconfig stays plaintext (decision 6). | Accepted; built; hardware lane B passed 2026-11-05 (restore --recover-cluster from the encrypted file not run) |
| 0128 | Ichor already adopts a running cluster (ADR 0103, 0104) and resets and adds a machine that another cluster configured (ADR 0103 section 2, 0107). A map of those two paths against the cases a user has in front of them (below) shows four gaps, and this ADR closes them without a new object kind and without any new write path to a node. (1) A machine that already runs a configuration signed by an Ichor cluster's own CAs, but has no TalosMachine (it joined the Talos cluster by the user's own tooling after the adoption, or its TalosMachine was deleted), has no way in: discovery skips it, ichor adopt only reads the machines already recorded, and ichor machine reclaim refuses it. ichor adopt --cluster NAME --add-machine ADDRESS (repeatable) now registers such a machine: the same read-only plan, the same three classes of drift, the same held TalosMachine and the same zero-drift release as an adoption, for the listed addresses only, with a trust check that refuses a machine that does not carry the cluster's CAs and names ichor machine reclaim instead. (2) ichor machine reclaim and the UI refusal text now say that a machine of an Ichor-managed cluster is added with --add-machine, not reclaimed, and a stale pending TalosMachine no longer counts as "already an Ichor machine" (hardware finding C7 of ADR 0103). (3) ichor adopt --cilium-values FILE now also supplies the operator's Helm values when an existing Cilium with no Helm release record (the common Talos-guide install, an inline manifest made with helm template) is taken over; until now those values could not be carried and the diff gate could only be accepted. (4) A non-Cilium cluster stays unmanaged-only and Ichor still does not migrate a CNI. The refusal now says why a migration into a managed Cilium is not offered, and the guide states the three honest options (leave the CNI, a new cluster, or a planned outage that ends in a cluster with no CNI, which ADR 0104 adopts as managed Cilium). Cilium's documented live migration is not one of them: it ends on ipam.mode: cluster-pool, which Ichor refuses. | Accepted; built; hardware lane B passed E1, E2, E4 and E5 2026-11-05 (E3 needs a re-run, E6 and E7 not run; see the amendments) |
| 0129 | Ichor's rule is that the UI and the CLI change the world by writing CRs, and that a rule the UI backend enforces must also hold for a CR written with kubectl (ADR 0094 decision on destructive starts, ADR 0108). An audit found four places where it did not. (1) The UI, ichor cni values set and ichor adopt run cilium.ValidateValues on spec.cni.cilium.values; the TalosCluster webhook did not, and the controller only rejects owned keys at render time, so a built-in operator could kubectl patch values the UI refuses (schema violations, unknown keys, a value of a key Ichor owns, an oversized layer) and the controller would hand them to Helm. The TalosCluster webhook now runs the part of ValidateValues that needs no knowledge outside the object, on create when values are set and on update when spec.cni.cilium.values, spec.cni.kubeProxy or spec.cni.tunnel changes. The three guards that need the cluster's machine configuration or the live cluster (bpf.masquerade against forwardKubeDNSToHost, KubePrism in replaced mode, Gateway API CRDs) stay in the UI and the CLI. (2) guardGrant and guardLastAdmin lived only in UserService; a new IchorUser validating webhook applies both to every write of an IchorUser. (3) and (4) are documentation: the credentials hand-off annotation is an attestation, and spec.paused has no effect on an etcd restore that has started. | Accepted |
| 0130 | nowhere in Ichor does a person have to, or get asked to, type a reason. The user decided on 2026-11-05 that every place that asks for a reason goes. What gates a dangerous action does not change: the RBAC verbs (override, start, approve), the typed cluster or machine name, the --yes prompt, and the Events and audit records that say who did what. Only the free text goes: the CLI has no --reason and no --override-reason, --skip-signature-check is a bare boolean flag, the UI dialogs have no reason box, no CRD rule requires a reason, and no controller or backend code reads or prints text a person typed as a reason. Every reason field of a CRD or a Connect message stays (append-only) as a deprecated, unused field. | Accepted |
| 0131 | A user with only the built-in ichor-operator role could write spec.cni.cilium.values (in the UI, with ichor cni values set, or with kubectl) and so choose what the Cilium Helm Job deploys. The Job's ServiceAccount (ichor-cni) holds escalate and bind on ClusterRoles and Roles by name (ADR 0099 decision 2, ADR 0108), so it is as strong as cluster-admin, and the Cilium chart accepts extraContainers, extraInitContainers, extraHostPathMounts, extraVolumes, extraArgs, image overrides and securityContext: a privileged, host-network pod with a host path on every node is a legal value. That is root on every node, and from there the Talos and Kubernetes PKI and the Secrets in ichor-system: operator became admin-equivalent. This ADR (1) makes changing the values a right of its own, the verb update on talosclusters/cilium-values, held by ichor-admin only and checked by the TalosCluster admission webhook as the requesting user, so the UI, the CLI and kubectl obey it alike, and (2) makes the five *.secretsNamespace.name values of the chart Ichor-owned, pinned to cilium-secrets. | Accepted |
| 0132 | The documentation lives only as Markdown in docs/ and is published by a static Astro site in site/ (Preact islands, Tailwind v4, preactcn components, Pagefind search) to GitHub Pages at https://ichor.maesh.dev/; a page is added or changed by editing Markdown alone, and the build fails on a broken internal link. | Accepted |
| 0133 | An ichor TalosOperation carries status.toVersion, the controller copies IchorInstallation.spec.version into it once, when it first advances the upgrade, and never rewrites it; the UI and CLI show it for every phase. | Accepted |
| 0134 | ichor init --merge-kubeconfig (opt-in) merges the cluster-admin kubeconfig into the kubeconfig kubectl would write to (the first existing file of $KUBECONFIG, else its first path, else ~/.kube/config; --kubeconfig wins), as cluster <name>, user and context admin@<name>, renames on a collision instead of overwriting, and makes the new context current. The copy at ~/.ichor/clusters/<name>/kubeconfig is written as before. Without the flag init touches nothing outside ~/.ichor and prints the export KUBECONFIG= line. | Accepted |
| 0135 | "Remove node" is the only way to take a node out of a cluster in the UI, and ichor node remove NODE [--reboot] in the CLI. It runs the existing removal flow (lock, quorum and count checks, storage evacuation, drain, etcd member removal for a control plane), then a graceful-as-today Talos Reset of STATE and EPHEMERAL, which ends with a power off (default) or a reboot into maintenance mode (--reboot, "After wiping" in the UI), then deletes the Kubernetes Node, then clears the finalizer. The wipe is no longer optional. It needs the typed machine name and the override permission, with no reason. The reset action, RemoveMachine and every proto and CRD field stay (ichor node reset was removed on 2026-11-07, see the amendments below); RemoveMachine and a reset operation both run this flow. | Accepted |
| 0136 | a machine that carries the configuration of a cluster Ichor manages but has no approved TalosMachine (an orphan: removed before ADR 0135, so never wiped) is no longer refused by reclaim. ichor machine reclaim ADDRESS --cluster NAME without --talosconfig, and the UI's "Reset and add" with "use this cluster's own credentials", create a MachineReclaim with the new spec.clusterCredentials: true and spec.clusterRef; the controller dials the node with an os:admin credential built from the cluster's own secrets bundle, applies the same assessment (typed UUID, quorum and last-control-plane refusals, graceful reset of STATE and EPHEMERAL, reboot into maintenance mode, the reclaim verb), and, once the machine is back in maintenance mode, deletes its Kubernetes Node as Remove does. A machine that still has an approved TalosMachine is refused with a pointer to Remove node. adopt --add-machine does not derive the hostname and link choice from the node: when it blocks a machine it adds the line "to start over, wipe it with ichor machine reclaim ADDRESS --cluster NAME (or the UI Reset and add)". | Accepted |
| 0137 | an operation the controller parked with Paused reason ClusterTransitioning (the count of approved control planes is even) does not block "Remove node" of a control plane, and the plan says so. While the count is even, planning a new operation (except an etcd restore) is refused with the way out, instead of creating one that is parked at once. A parked operation resumes by itself when the count is odd. | Accepted |