# 0059. IchorInstallation holds the target version; a TalosOperation of kind ichor runs an installer Job that carries its own manifests, snapshots, and rolls back

Status: Accepted; superseded in part by ADR 0090 (no air-gap requirement) and in part by ADR 0101 (decision 5 and the external mode) · Date: 2026-10-26 · Design section: Upgrades (Ichor), API
Decided: Add the cluster-scoped singleton `IchorInstallation` (managementMode, target version, installer repository; installed version reported by the controller), execute `kind: ichor` by creating a one-shot installer Job that runs the target version's own image and embedded manifest (snapshot in ConfigMaps, additive-only CRD check, CRDs then controller then UI with post-checks, rollback from the snapshot), refuse CLI writes more than one minor from the server, and never launch the Job in `external` mode

## Context

The design (Upgrades, Ichor) says: in `ichor` mode the controller only launches the installer Job for the
target version and never upgrades itself; the installer snapshots the current manifests, updates CRDs
(additive, never downgraded), the controller, then the UI, runs post-checks, and rolls back from the snapshot on
failure; in `external` mode it is a PR; the CLI refuses writes more than one minor from the server; the UI
backend sends its version in a header. ADR 0004 deferred `IchorInstallation` to this phase. ADR 0026 left
`kind: ichor` without an executor, ADR 0019 made `ichor init` apply a manifest embedded in the binary, and ADR
0056 wired PR mode into the UI backend.

## Decisions

**1. Shape, and who owns what.** `IchorInstallation` (cluster-scoped, must be named `ichor`, enforced by CEL):
`spec.managementMode` (`ichor` default, `external`), `spec.version` (semver with a leading `v`, required),
`spec.installerRepository` (default `ghcr.io/maeshinshin/ichor-installer`; point it at a mirror if needed).
`status.installedVersion` is written by the controller from its own build version (`internal/version`, linked
with `-ldflags`) and nothing else writes it; conditions `UpToDate` and `ExternalSuggested` (the latter when
the mode is `ichor` but `pkg/gitopstracking.IsTracked` finds the controller's own Deployment tracked).
The `TalosOperation` of kind `ichor` carries **no version**: the target is `IchorInstallation.spec.version`,
read when the Job is created and recorded on the Job in an annotation. The CR says what should be installed and
who may change it (it is in Git, per the design); the operation is the record of one execution. A target
version in both places would be two places to disagree. Consequence: changing `spec.version` while an
operation runs is detected (the annotation differs) and pauses the operation instead of silently finishing the
old target.

**The backbone is not rebuilt.** ADR 0026 calls `AdvanceMachineFunc` once per machine and an Ichor upgrade has
no machines. The executor is therefore idempotent per operation: the Job is named from a hash of the
operation's namespace/name, the first call creates it, every later call (whichever machine, whichever
reconcile, including the new controller after the upgrade restarted this one) looks at the same Job. Machines
complete serially as soon as the Job has. `acquireDisruptiveLock` remains the only `Acquire` caller. Known
limit: a cluster with no approved machine completes the operation vacuously without launching anything; Ichor
manages Talos clusters, so this is the empty case, and the operation's cluster lock is held for the Job's whole
run, which is the point.

**2. Where the target's manifest comes from.** From the target version's own installer image. `ichor installer
run` (a hidden subcommand) applies `installmanifest.Manifest`, the same embedded bytes `ichor init` applies
(ADR 0019), so the image is built with `-tags installembed` (`Dockerfile.installer`, `make
docker-build-installer`) and **fetches nothing at run time**. The
Job runs `<installerRepository>:<version>`. The controller never reads the new manifest; it cannot even tell
what it contains. The cost: one more image per release, and the manifest's own image references (controller,
UI) are pulled as usual.

**3. Snapshot and rollback.** Before any write, the installer saves the live state of every object the
manifest names (and a marker for those that do not exist yet) as gzipped JSON, split into 512 KiB parts, in
ConfigMaps `<run id>-snapshot-N` in the Job's namespace, labelled with the run id. ConfigMaps, not Secrets:
the snapshot holds no Secret by construction (a Secret in the manifest is neither applied nor snapshotted; the
installer's RBAC has no `secrets` rule at all). 512 KiB leaves wide margin under the 1 MiB object limit;
the seven CRDs gzip to a few hundred KiB, so normally it is one part, and an incomplete set of parts is refused
at load rather than trusted. A snapshot that already exists for the run id is reused, never retaken: a re-run
after a crash would otherwise snapshot the half-upgraded cluster over the good one. Rollback runs in reverse
order with `Update` (an exact restore, unlike re-applying, which would leave behind fields the new version
added), deletes objects the upgrade created, and **never deletes a CRD** (that deletes every custom resource of
the kind); a CRD that was new stays, and a CRD that existed is restored to its old schema. Snapshots are kept
after success or failure (evidence, and the manual way back when the rollback itself failed); garbage
collection of old snapshots is not built.

Order and checks: CRDs (each must be `Established`), then every non-UI object (Deployments must finish a
rollout: generation observed, all replicas updated and available), then the UI objects (same). UI objects are
recognised by `ichor-ui` in the name, because kustomize's prefix makes them `ichor-ichor-ui-*`.

**4. CRDs are additive.** Before the snapshot, for each CRD in the manifest that exists, the installer compares
the schemas and refuses if the target would drop a served version, drop a property, change a property's type
(recursively through properties, items and additionalProperties), or change the scope. This judges the schemas, not
a version string, so it also catches a target built from an older tree and a hand-edited live CRD. The refusal
comes before any write and leaves no snapshot. The controller-side `semver` comparison in the executor
(`spec.version` older than the running controller is refused, creating no Job) is a second, earlier line.

**5. `external` mode.** The executor does not launch the Job: it returns an error naming the pull request path,
which pauses the operation (ADR 0026). **Not built: opening that PR.** PR submission (repository connection,
token, the Argo CD Application's source path) is wired inside `internal/uiapi` (ADR 0056), which another unit
was editing; reaching it from the CLI means either a UI service for `IchorInstallation` or an export from
`uiapi`, to be done after that unit lands. Until then `external` mode means "bump the chart version in Git by
hand". The chart's installer RBAC is also off in that mode (below).

**6. CLI refuses writes beyond one minor.** The CLI compares its own `internal/version` with
`IchorInstallation.status.installedVersion` (the controller's report, so it works with or without ichor-ui,
including the `--local` fallback) before a command marked as a write: `machine approve`, `machine reject`,
`patch create/update/delete`. Reads are never refused. It refuses only on a positive finding: no
`IchorInstallation`, no reported version, an unreadable cluster, a `dev` build or different majors mean "cannot
judge" and the command proceeds. Not marked yet: `init`, `adopt`, `restore`, `gitops` (they run before the
server exists or against its absence; revisit when `ichor self-upgrade` lands). `ichor init` now creates the
`IchorInstallation` with its own version (skipped for a `dev` build).

**7. UI response header: not built.** Nothing sends or reads it; the version badge (`ui/src/lib/ichorVersion.ts`)
is what exists. It was to work like this: the backend sends `X-Ichor-Version` on every response from `internal/version`; the SPA compares it with
the version baked into its bundle and prompts a reload when they differ.

## The installer's RBAC, and how far it narrows

A separate ServiceAccount `ichor-installer` (`config/rbac/ichor-installer`, hand-written, compared with the
chart's copy by `dist/chart/chart_test.go`). The controller holds only `jobs: get, create` in its namespace; it
never holds the installer's permissions. The chart creates the identity only with `installer.enabled`
(default false: a chart-installed Ichor is normally Argo CD's, and nothing there should carry it).

Narrowed to exactly the kinds `dist/install.yaml` contains: CRDs `create`/`get` and `patch`/`update` limited by
`resourceNames` to Ichor's own seven; the `ichor-system` namespace `get`/`patch`/`update` by name;
Deployments, Services, ServiceAccounts, Roles and RoleBindings in the namespace only; ConfigMaps `create`/`get`/`list`
(the snapshots); ValidatingWebhookConfigurations and admission policies; no Secrets, no wildcards, no custom
resources, no CRD delete (a test fails the build if a `secrets` or `*` grant appears).

**Not narrowable: `clusterroles`/`clusterrolebindings` with `escalate` and `bind`** (and `roles` with the same in
the namespace). Applying a role that grants more than the applier holds is refused without `escalate`, and a
new version may add roles with names nobody can list in advance, so `resourceNames` cannot do it. This part is
as strong as cluster-admin. What contains it: the identity exists for one Job, only the controller can start
that Job, its image is the one `spec.installerRepository` names, and in `external` mode it does not exist.
Mirroring the installer image into a registry you control is what makes that last point hold. The installer
rewrites its own role during an upgrade (it is part of the manifest); a new version that narrows it takes effect
mid-run, which the rollback uses the old snapshot to undo.

## Built, and not

(The `external` pull request, the version label, snapshot garbage collection and the Job TTL listed as not built
below were built later; see "Amendment (the remaining items, built)".)

1. `IchorInstallation` CRD and reconciler: built. 2. `kind: ichor` executor and the Job: built. 3. The
installer (snapshot, additive check, ordered apply, post-checks, rollback), its image and RBAC: built, tested
against a real API server. 4. CLI skew refusal: built; UI header: **not started** (above). 5. `external`: no Job
launched, built and tested; opening the PR: **not started** (above).

Also not built: `ichor self-upgrade` (creates the operation and, in external mode, the PR) -- today the
operation is written by hand like any TalosOperation; `ichor self-upgrade --local` as the way out of a broken
controller; snapshot garbage collection; `ttlSecondsAfterFinished` on the Job (a finished Job and its snapshot
stay until deleted). Not verified: a real upgrade between two released versions, because there is no second
released version yet and no cluster to run it on.

## Amendment (the remaining items, built)

1. **Installer Job cleanup.** The Job carries `ttlSecondsAfterFinished: 86400` (one day: long enough to read the
   pod log of a failed upgrade, short enough that Jobs do not pile up). The operation does not need the Job once
   it has finished: after a successful upgrade the new controller's version equals `spec.version`, which the
   executor checks before it looks for a Job (decision 1, "Done must be decided before the mode"). An expired
   *failed* Job that is resumed after more than a day starts a fresh attempt under the same run id, which reuses
   the run's snapshot (decision 3), so it is a retry and never a second snapshot of a half-upgraded cluster.
2. **Snapshot garbage collection.** Each snapshot ConfigMap carries a `...-taken` annotation (RFC 3339, nanoseconds;
   `creationTimestamp` has one-second resolution and runs can share a second). After the installer has saved
   (or reused) its snapshot, and before it writes anything, it deletes every snapshot except the newest
   `SnapshotsKept` (3, the current one included) and never the current run's. Failing to delete is logged and does
   not fail the upgrade. The installer's Role gains `delete` on ConfigMaps (still namespace-scoped; the chart copy
   is compared by `dist/chart/chart_test.go`). The decision above that snapshots are kept after success or
   failure still holds, now for the last three.
3. **Version in the UI.** `OperationService.GetIchorInstallation` returns the installed version
   (`status.installedVersion`), `spec.version`, the mode, whether an `ichor` operation is unfinished, and
   `update_available`: `spec.version` is semver-newer than the installed version and no Ichor upgrade is running. A
   cluster without an `IchorInstallation` answers with empty fields, not an error. The app shell shows the
   installed version as a badge in the top bar, plus an "Update available" badge only when the server says so; a
   failed read shows nothing. The `X-Ichor-Version` response header (decision 7, a reload prompt when the bundle
   and backend differ) is still not built (decision 7 says so too); this label (`ui/src/lib/ichorVersion.ts`) is
   separate from it.
4. **`external` mode and Git: a pull request.** What the PR changes is the one thing prmode can edit safely:
   the `IchorInstallation` document in the file the tracking Argo CD Application's plain path source holds,
   field `spec.version` (target `IchorInstallation/ichor`, branch `ichor/upgrade-ichor-<version>`, so asking
   twice finds the first PR). It goes through the same gate as every other write (`planWriteIn`: Argo CD tracks
   it, the cluster's `gitops.repository` is the Application's repository, the source is a plain path). When the
   `CreateOperation`/`PlanOperation` target would change a tracked `IchorInstallation` and a plan exists, the
   backend opens the PR and returns `pull_request_url` with **no operation** (`opens_pull_request` in the plan, so
   the CLI and UI say so before they act); nothing is written to the cluster. When no PR can be opened the old
   refusal stands. In `ichor` mode the next step after merging is to start the upgrade again without a target (the
   installer Job runs). In `external` mode no Job ever runs, so the PR records the target only: the chart or
   manifests Argo CD applies for Ichor (a Helm source, which prmode cannot edit) must be bumped by hand, and the PR
   description says so. That is the conservative reading of "bump the chart version in Git"; editing a Helm
   `targetRevision` or values file is not attempted. An `external` installation that Argo CD does not track is
   refused by the plan (it can neither be written nor proposed). The executor in `external` mode still
   launches no Job and pauses, unchanged.

## Amendment (real hardware)

The executor never created the installer Job: it reads Jobs through the manager's cached client, whose cluster-wide
informer needs `list`/`watch` on `jobs.batch` at cluster scope, while the Role grants `jobs` in `ichor-system` only
(`jobs.batch is forbidden ... at the cluster scope`). The Job cache is now confined to the controller's namespace like
the Secret cache, and the Role grants `get;list;watch;create` on `jobs`. A new envtest spec (ADR 0081, item 6) builds
the manager's cache as the controller's identity bound to the generated RBAC and requires an informer to sync for every
kind read through the cached client, so this class of mismatch fails in CI.

Addendum (ADR 0088): an Ichor upgrade whose renderer emits different output is applied to nodes like any desired-state change (render.Version is part of the input hash); reboot-requiring changes wait for a config rollout.

## Amendment (self-upgrade from an older release)

An upgrade runs the installer Job with the RBAC of the release being replaced, so a rule the new release adds helps only the release after next. On the demo, `ichor upgrade ichor` failed with `cannot patch resource "customresourcedefinitions"` for `ichorusers.ichor.maesh.dev` (missing from the ClusterRole resourceNames) and `cannot delete resource "configmaps"` (snapshot garbage collection), and rolled back.

- The installer applies its own RBAC first. The ClusterRole, ClusterRoleBinding, Role and RoleBinding named `ichor-installer-*` form a step before the CRDs, in the same run. It may already `escalate`/`bind` and patch roles, so it grants itself the new rights, then applies everything else. The snapshot covers these objects too (first entry, so a rollback restores them last and still holds the rights it needs). Kubernetes evaluates RBAC per request and the installer client caches no authorization state, so no restart is needed; applies retry briefly on Forbidden (default 30s) because the authorizer cache can lag the write.
- Snapshot garbage collection now runs after the RBAC step and stays non-fatal. Its `delete` on ConfigMaps is namespace-wide in the release namespace: snapshot names derive from the generated Job name, so `resourceNames` cannot list them.
- Every CRD in `config/crd/bases` must be in the installer ClusterRole `patch`/`update` resourceNames. `dist/chart` tests (`TestInstallerRBAC`) fail when one is missing, in both the kustomize role and the chart template.

Amendment: a rolled-back upgrade leaves `spec.version` at the failed target, so "current" in plans and the CLI is `status.installedVersion` (the running controller's own version), never `spec.version`; `UpToDate=False` states running versus target.

## Amendment (an upgrade keeps what `ichor init` customized)

On the demo, installed with `--ui-url` and `--ui-tls-cert/key`, `ichor upgrade ichor` failed on `Deployment ichor-ichor-ui ... livenessProbe.tcpSocket: Forbidden: may not specify more than 1 handler type` and rolled back. `init` edits the manifest's Deployments and Service after installing them (users-enabled and issuer flags, the internal port and OIDC Secret volumes, the TLS Secret mount and flags, HTTPS probes, the controller's `--user-issuer`). The installer applied the bare manifest by server-side apply under its own field manager, so: container `args` (an atomic list) were replaced, silently dropping every init flag (users and TLS off, and the controller's issuer off); `ports`, `volumes`, `volumeMounts` and Service `ports` (merge lists owned by init's manager) survived by luck; probes merged field by field, leaving init's `httpGet` beside the manifest's `tcpSocket`.

- The customizations are one set of pure functions in `internal/installer` (`CustomizeUI`, `CustomizeController`, `CustomizeService`, over `UISettings`). `ichor init` calls them on the live objects; the installer calls them on the new manifest's objects before it classifies, snapshots or applies anything. They replace whole probes and are idempotent, so there is nothing to drift between the two.
- The settings are not stored anywhere new: the installer re-derives them on every run from the live Deployments (`ReadUISettings`): `--external-url`, `--kube-api-server-url`, `--kube-api-tls-server-name` and the `tls` volume's Secret on ichor-ui, `--oidc-refresh-lifetime` on the controller. Those are what init wrote and what an operator would see. A first install by the installer reads as a plain install.
- Deployments are written by a full `Update` (with the live resourceVersion and annotations carried over, field manager `ichor-installer`) instead of server-side apply, so a field another manager owns can no longer merge into the new spec. The Service and everything else keep server-side apply. The rollback snapshot is untouched: it records the live object before the run and writes it back whole.
- A rolled-back Ichor upgrade leaves a Paused `TalosOperation` with a failure reason and `spec.version` at the failed target. A new `ichor upgrade ichor` is refused while it exists (one unfinished operation per cluster); resuming it only meets the failed installer Job again, which is kept for a day. The refusal now says to run `ichor operation delete NAME` and then `ichor upgrade ichor <version>`; after that, any version at or above the running one is accepted, the failed one included.

## Amendment (the installer snapshots its own RBAC first, then everything else)

- Bug: the snapshot read every object of the new manifest before the installer applied its own RBAC. An older release's `ichor-installer-role` has no rule for a kind that is new in the release (a CiliumNetworkPolicy, for the upgrade from v0.0.1797000000-82b4420), so the snapshot failed Forbidden before anything was applied and every new kind blocked every upgrade from an older release.
- Order now: (1) snapshot stage 0, only the `ichor-installer-*` ClusterRole, ClusterRoleBinding, Role and RoleBinding; (2) apply them; (3) poll a SelfSubjectAccessReview for the first rule of each applied role (verb `get` when the rule has it) until allowed, bounded by `RBACSettle` (default 30s), because RBAC is eventually consistent; (4) snapshot stage 1, everything else (plus the Cilium policies); (5) CRDs, controller, UI, network policies as before. Both stages carry one run id (a stage annotation on the ConfigMaps); a rollback reads them as one list and restores stage 1 first, the RBAC last, so it still holds the rights it needs. A re-run keeps a stage that already exists.
- (Corrected by "Amendment (installer rights are cluster-scoped only)" below: this holds for the installer's own ClusterRole and the Role in the release namespace, not for a new namespaced Role elsewhere.) No bridge is needed for those. Every release since the installer role was introduced (ffb1413) grants the role get/create/patch/update/delete on `clusterroles`, `clusterrolebindings`, `roles` and `rolebindings` plus `escalate` and `bind` on `clusterroles` and `roles`, so the oldest role can rewrite itself without the user's rights. The CLI does not pre-apply anything.
- The snapshot treats NotFound, and a kind whose CRD is not installed (no REST mapping), as "did not exist": nothing to restore, and the rollback skips it. Forbidden or any other error fails the run before it writes, never ignored, because the rollback depends on the snapshot being complete.

## Amendment (installer rights are cluster-scoped only)

- Bug (hardware): an upgrade from an older release failed at the stage-0 snapshot with `roles.rbac.authorization.k8s.io "ichor-installer-kube-system-role" is forbidden ... in the namespace "kube-system"`. Commit 8a7199b moved the Cilium Job to kube-system and gave the installer a namespaced Role there. The old installer role holds no rights in kube-system, so it can neither read nor create that Role, and "RBAC first" cannot bootstrap a namespaced Role in a namespace the old role never reached. The "no bridge is needed" claim above is wrong for any new namespaced RBAC outside the release namespace.
- Rule: installer rights are cluster-scoped only. The installer holds no namespaced Role outside the release namespace. What `installer-kube-system-role` granted (serviceaccounts, roles and rolebindings: create/delete/get/patch/update, and bind/escalate on roles) is folded into `installer-role`'s ClusterRole, which every release's installer can rewrite: since ffb1413 the role holds get/create/patch/update on clusterroles and clusterrolebindings plus bind and escalate on clusterroles. `installer-kube-system-role`, its RoleBinding, their chart copies and the kustomize patches are removed. The controller's `cni-job-role` in kube-system stays; the installer creates it, and the folded rules (with escalate and bind on roles) let it.
- Order: stage 0 is now the installer's ClusterRoles and ClusterRoleBindings only (snapshot, apply, wait until effective); stage 1 is its namespaced Roles and RoleBindings (snapshot, apply, wait); stage 2 is everything else. A future namespaced installer Role elsewhere is read under the already-widened ClusterRole. Rollback restores stage 2, then 1, then 0.
- Cost: the installer's serviceaccount, role and rolebinding rights now hold in every namespace, not only kube-system. The ClusterRole already held escalate and bind on clusterroles, so it was already as strong as cluster-admin (see "The installer's RBAC").
- Amendment (ADR 0113): an Ichor upgrade does not move the cluster's Talos, Kubernetes or Cilium versions. Each Ichor build embeds its own release set, and the cluster follows it through `ichor upgrade cluster`, one step at a time, after the Ichor upgrade has finished.

## Amendment (the operation completes only when the rollout is over)

- Bug (UI demo): `ichor upgrade ichor --follow` reported the operation Completed while the installer Job was still Running and ichor-ui rolled out 10 to 40 seconds later. The new controller resumes the operation as soon as it runs, and the executor returned done as soon as `spec.version` equalled the running version, without looking at the Job.
- Rule: an `ichor` operation is Completed only when (1) the installer Job has succeeded (or is gone: it is removed after a day), (2) the controller Deployment (`control-plane=controller-manager`) and (3) the ichor-ui Deployment (`app.kubernetes.io/component=ui`) have finished rolling out: `observedGeneration` is current and the updated, ready and available replica counts equal the desired count with no old replica left. A failed Job fails the operation with the Job's message, even when the new controller already runs. A Deployment that is not installed is not waited for. The check is the same whichever controller runs it (the old one after the Job succeeds, the new one on resume).
- The Deployments are read through the uncached API reader (`get` and `list` in the release namespace, already granted), not the cached client: a cached read would start a cluster-wide informer (ADR 0081). The CLI `--follow` only polls the operation phase, so it needs no change.

## Amendment (an operation shows what it is waiting for)

- Gap (UI demo): the executor's waiting text ("the installer Job is running", "waiting for the ichor-ui Deployment to finish rolling out (n of m updated replicas are available)") was built on every reconcile and thrown away, so nobody could see what a running operation waited for.
- Rule: `TalosOperation.status.message` (optional, appended) holds the message the controller last passed to `setPhase` while the phase is InProgress. It is replaced when it changes (an unchanged value is a no-op status update) and cleared in every other phase: a pause says why in `failureReason` or the Paused condition, a finished operation in its Events. It applies to every kind that goes through `setPhase`, not only `ichor`.
- Shown by `Operation.message` (proto field 21, `GetOperation` and the list), by `ichor ... --follow` and `operation get --watch` (the progress line ends in `: <message>`, printed whenever the line changes), and by the "Currently" field on the operation detail page. The operations list is a table and does not show it.