# 0122. The installer image runs by digest and its signature is verified before the installer Job Status: Accepted; built; the positive signed path awaits a release tag built by release.yml · Date: 2026-11-04 · Design section: Upgrades (Ichor), Supply chain · Touches: 0059, 0090, 0104, 0105, 0108 Decided: Before the controller creates the installer Job of an Ichor upgrade it pins the installer image to a digest and verifies that digest's keyless cosign signature, offline against an embedded Sigstore trusted root, for the identity `https://github.com/maeshinshin/ichor/.github/workflows/release.yml@refs/tags/` issued by GitHub Actions. The digest is `spec.installerDigest` when set, else what the tag `:` resolves to at that moment (HEAD on the manifest); the Job's image is `@`, so what was verified is what runs. The only way not to verify is an explicit, reasoned, recorded opt-out, `spec.installerSignature: {mode: skip, reason: ...}` (`ichor upgrade ichor --skip-signature-check ""`), which the controller records in `status.installerImage` and an Event and which the CLI and the UI show as a warning. A failed resolution or verification pauses the operation with a reason and creates no Job. The apiserver enforces the digest form for the installer Job through the existing Job-image admission policy. The release workflow verifies the signatures it has just made with the same Go code before it finishes. ## Context ADR 0108 signs the three release images and the chart keylessly with cosign and documents `cosign verify`, but left two things open (its follow-up table): the installer Job's image is `installerRepository:version`, a tag a registry operator or an attacker with write access can repoint after anyone looked, and nothing in Ichor checks a signature. The installer Job's ServiceAccount holds `escalate` and `bind` on ClusterRoles (ADR 0059), so the installer image is the most privileged thing Ichor runs; "as strong as cluster-admin". Audit finding M3 asked for resolution by digest (`spec.installerDigest`) and a signature check before the Job runs. `internal/imagesig` and its `sigstore-go` dependency existed until commit 3c04d972 for the Cilium image-patch path ADR 0115 later removed. The verification code, the embedded trusted root and the way the bundles are read (OCI referrers, Sigstore bundle v0.3, what cosign 3 writes by default; `cosign sign` v3.1.3 is the flake's version) are reused here. ADR 0090 says air-gapped installs are out of scope; mirrors are in scope (`installerRepository`). ## Decisions ### 1. Where the digest comes from Two sources, one rule: the Job never runs a tag. - `spec.installerDigest` (new, `^sha256:[a-f0-9]{64}$`): a person's pin. The controller does not resolve the tag at all, so a pin plus `skip` needs no registry access from the controller. - Otherwise the executor resolves `:` with a manifest HEAD request (`imagesig.ResolveDigest`) when it creates the Job, and uses that digest for both the signature check and the Job. It records `{version, repository, digest, verification, signer|reason, resolvedAt}` in the new `status.installerImage` and puts the digest on the Job (annotation `ichor.maesh.dev/installer-image-digest`). A later reconcile of the same operation reads the Job, never the registry again; a changed `spec.installerDigest` while a Job exists pauses the operation, as a changed `spec.version` does (ADR 0059). Rejected: a release manifest embedded in the CLI or the controller. The controller that runs the upgrade is the old one and cannot hold a digest of a release that did not exist when it was built; the CLI could only name its own version's digests, tying the CLI's version to the target, and the manifest would itself need a trust root. Resolving at upgrade time against the registry the user chose, then verifying a signature that is bound to the digest, needs neither. Rejected: writing the resolved digest into `spec` (the controller never mutates spec); it goes to `status`. The resolve and verify happen once per Job, before it exists: a window between "verified" and "pulled" no longer exists because the pull is by digest (this is the point of M3). ### 2. Who signs, and the identity Keyless cosign with the GitHub Actions OIDC token, which `release.yml` already does for the installer, controller and UI images and the chart (`cosign sign --yes @`, in the `publish` job, the only job with `id-token: write`). The certificate's issuer is `https://token.actions.githubusercontent.com` and its subject alternative name is `https://github.com/maeshinshin/ichor/.github/workflows/release.yml@refs/tags/`. `imagesig.IchorIdentity(spec.version)` is that identity as an anchored, quoted regular expression, so only a signature made by that workflow on the tag equal to `spec.version` verifies; a valid signature of another tag, another workflow, a branch ref or another repository does not. It is the identity `docs/guides/install-with-helm.md` gives to `cosign verify`. The identity is not configurable. A fork that builds and signs its own installer image runs with `skip` (recorded) until a decision on a configurable signer is made; making it configurable needs a second trust decision (who may a repository owner name) that this ADR does not take. A new step in `release.yml`'s `publish` job, after signing, verifies each of the three images and the installer's signature with `ichor`'s own verifier (`internal/tools/verifyimage`, built in the `build` job, which may fetch Go modules, and shipped in the artifact, so `publish` still fetches none). A format or trust-root mismatch between cosign and `sigstore-go` therefore fails the release instead of the first upgrade. ### 3. How verification works offline, with mirrors, and what is trusted - Offline: the bundle carries the Fulcio certificate, the signed certificate timestamp, the Rekor inclusion proof and entry timestamp, so `sigstore-go` verifies it against `internal/imagesig/trusted_root.json` (the Sigstore public-good root, embedded, no TUF fetch) without contacting Sigstore. The only network traffic is the registry: a manifest HEAD, the referrers listing of the digest and the bundle blob. The trusted root changes with an Ichor release; a signature made after Sigstore rotates its keys may not verify on a controller built before it. The recovery is `skip` with a reason, then the next Ichor release carries the new root. (Rejected: a root fetched at run time, which makes the verifier depend on a second online service; a ConfigMap override, which is a way to widen trust that needs its own review.) - Mirrors: the signature is about the digest, not the registry. A mirror that carries the image under the same digest and the image's OCI referrers (for example `oras copy --recursive`; a plain `crane copy` or `docker pull/push` does not copy referrers) verifies unchanged. A mirror that carries the image only fails with "has no cosign signature" and the operator has two honest ways out: copy the referrers, or `skip` with a reason. The controller reads the registry anonymously (the Job's own pull uses the node's credentials, as before); a private mirror that needs a pull secret for the controller's reads is `spec.installerDigest` plus `skip` today. - Egress: the controller's Cilium egress policy (ADR 0100) gets one more dynamic policy, `ichor-controller-installer-image`, maintained by the IchorInstallation reconciler: HTTPS to the host of `spec.installerRepository`, plus `pkg-containers.githubusercontent.com` when that host is `ghcr.io` (GHCR redirects blobs there). It exists whenever Ichor's own controller policy exists, whether or not the update check is on, and is the same shape as the release check's. A registry that redirects blobs to another host needs that host named; until it is, the verification fails with the registry's error and the pause says so (`skip` is the way out). ### 4. The opt-out for unsigned images Development and demo builds (`v0.0.-g`, pushed to a mirror by `make demo-images-push`) are not signed. `spec.installerSignature` is `{mode: required|skip, reason}`; `skip` requires a non-empty reason (CEL on the CRD), is a `version`-subresource write (the same right as changing `spec.version`, `installerRepository` and the digest; the settings webhook records who changed it in an Event), and is never inherited: `CreateOperation` for an Ichor upgrade writes `installerDigest` and `installerSignature` exactly as the request says, so a later upgrade without `--skip-signature-check` writes `required` again. A standing skip cannot outlive the upgrade it was given for through the CLI or the UI backend; a person who edits the CR by hand owns what they leave there. When the controller runs a Job under `skip` it writes `status.installerImage.verification: Skipped` with the reason and the digest, emits a Warning Event on the IchorInstallation (`InstallerSignatureSkipped`) and logs it. Shown, not hidden: - `ichor upgrade ichor --skip-signature-check ""` prints, before it asks for confirmation, a warning that the installer image is not verified and why, and `ichor upgrade ichor --installer-digest sha256:...` pins. The plan the UI backend returns carries the same warning. - The UI shows a destructive badge "Installer image not verified" in the top bar while `spec.installerSignature.mode` is `skip` or the last run was `Skipped`, with the reason and the digest on hover and on the Ichor card of the dashboard. Rejected: a controller flag or chart value that turns verification off for the whole installation (a silent, standing bypass), a verification mode "warn" (an upgrade that proceeds on a failed check teaches people to ignore it), and treating a dev version string as unsigned automatically (a release-looking tag from a hostile mirror would then pass). ### 5. Failure The executor returns a typed error; `advanceIchor` pauses the operation (ADR 0026) with the reason `InstallerImageUnresolved` (the tag did not resolve: registry unreachable, tag missing, egress) or `InstallerImageUnverified` (no signature, a signature that does not verify, wrong identity, wrong digest) and the full message. No Job is created, so the installer ServiceAccount does not run, nothing is snapshotted and nothing is applied. Resuming (ADR 0095's retry annotation, `ichor operation retry`) re-runs resolve and verify. Nothing ever falls back to the tag or to unverified. ### 6. The apiserver enforces the digest form `ichor-controller-job-image` (ADR 0108) now requires, for a container of the `ichor-installer` ServiceAccount's Job, `image == installerRepository + '@' + installerDigest` when `spec.installerDigest` is set, else `installerRepository + '@sha256:<64 chars>'`. A controller that was compromised or buggy and created a tag-form Job is refused by the apiserver. The Cilium Helm Job (`ichor-cni`) keeps `installerRepository:version`: it runs the installer image of the version already installed, under a ServiceAccount that cannot create roles outside its namespace, and its Job exists in Helm-only installs with no IchorInstallation (see Residual). The chart's structural rule for a chart-only install (`admission.jobImageRepository`) accepts both forms. ## Residual risk and what this does not do - The first upgrade onto a release that contains this change is made by the older controller, which creates the Job by tag as before. Protection starts with the upgrade after that. - The installer applies a manifest whose controller and UI images are tags, not digests. A signed installer therefore still pulls `ichor-controller:` and `ichor-ui:` by tag. Closing that needs the installer image to be built after the other two images are pushed so its embedded manifest names their digests; the build job has no registry access today. Follow-up. - The Cilium Helm Job's installer image is tag-resolved (above). - No revocation: a signature stays valid after the key-less certificate expires (that is how keyless works); a compromised release is withdrawn by a new release and the operator's `spec.version`, not by Ichor. - The trusted root is embedded; see 3. - The verification reads signatures only from `installerRepository`; a signature published at a different repository is not looked for. ## Built - `internal/imagesig` (restored, `IchorIdentity`, `ResolveDigest`; the live test for Cilium is gone), `internal/tools/verifyimage`, `release.yml` (verify step, build step for the tool). - `api/v1alpha1/ichorinstallation_types.go`: `spec.installerDigest`, `spec.installerSignature`, `status.installerImage`; CRD and chart CRD regenerated. Fields only added. - `internal/controller`: `ichor_upgrade.go` (resolve, verify, digest Job, Job annotation, record, Event), the pause reasons, the egress policy in the IchorInstallation reconciler. - `config/admission/controller-jobs-policy.yaml` and the chart copy; `internal/webhook/v1alpha1` settings gate (the new fields need the `version` subresource). - `proto/ichor/v1alpha1/operation.proto` (`IchorInstallerOptions` on plan and create; installer-image fields on `GetIchorInstallationResponse`), `internal/uiapi`, `internal/cli/upgrade.go`, `ui/` (badge, alert). - `internal/controller/talosoperation_controller.go`: the two refusal reasons set `status.failureReason` like `StepFailed`, which the CLI, the UI, the retry request (ADR 0095) and the "one unfinished operation" rule key on; a pause with only a condition reason would be resumed by the controller by itself. `controllerPolicyApp` no longer treats a policy the controller wrote itself as Ichor's installed policy. - `ichor version` shows the installer image line; `PlanOperation` warns on a skip; a new Ichor upgrade replaces the installer settings (a pin or skip is written even when the version is already the target, so "retry with a skip" after a refusal works: delete the paused operation, run the upgrade with the flag). - Docs: `docs/guides/install-with-helm.md` ("The installer image on an upgrade"), `test/harness/ui-demo/README.md` (the demo upgrade passes `--installer-digest` and `--skip-signature-check`: the controller cannot resolve a demo tag, which only the nodes' containerd mirror serves), ADR 0108's follow-up table. ## Tests Unit: the identity matches only its tag; a signature verified in a virtual Sigstore passes only for the right identity, issuer and digest; a certificate from another CA fails; a registry with no signature, an unparsable bundle and an attestation referrer are refused; `ResolveDigest` against an in-memory registry. Controller (envtest): the Job carries `repo@digest`, a pinned digest skips resolution, a failed resolution and a failed verification pause with their reasons and create no Job, `skip` records `Skipped`, the reason and a Warning Event, a changed pin pauses. API server (envtest): the Job-image policy accepts the digest form and refuses a tag and a different digest, the CRD refuses `skip` without a reason and a malformed digest. UI backend: the plan warns on skip, create writes and clears the fields. CLI: the flags and the warning. UI: the badge appears for skip. ## Hardware verification (later) See the report that accompanies this ADR; the positive path with a real signature needs a release tag built by `release.yml`. ## Follow-up (2026-11-05): the manifest the installer embeds names the controller and ichor-ui by digest The second residual risk above is closed. A verified installer applies the install manifest embedded in it; that manifest now names the controller and ichor-ui images as `:@sha256:`, so a retagged image is not pulled (the digest wins over the tag when the node pulls; the tag stays for people and for `ichor version`-style reading). Nothing in Ichor parses these two images' tags. - The release builds the installer last. `release.yml` now runs `verify`, `build` (the controller and ichor-ui binaries and the verifier; no token), `publish-core` (pushes those two images, writes `bin/digests.txt`; the first token job), `installer` (no token; downloads the digests, renders the manifest with `make build-installer IMAGE_DIGESTS=bin/digests.txt`, refuses to embed a manifest that does not name them, compiles the installer binary, and packages the chart with the same digests as its default images), and `publish` (pushes the installer and the chart, then signs, verifies and attests all four as before). The job that fetches Go modules and Kustomize still holds no registry token, which is why the render and the installer build are not folded into `publish-core` or `publish`. The cost is a second approval of the protected `release` environment per release (`publish-core` and `publish`). - `make build-installer` takes `IMAGE_DIGESTS=` (what `images.sh` writes to `DIGEST_FILE`; `hack/pin-image.sh` reads it, and fails for an image with no digest instead of falling back to a tag). Without it the images stay tags, so a local `make build-installer`, `make deploy` and `make test` are unchanged. `INSTALL_OUT` redirects the output. `make helm-package IMAGE_DIGESTS=...` stamps the digests into a temporary copy of the chart's default repositories (the templates add no tag to a repository that contains an `@`), so `dist/chart/values.yaml` in the tree is never edited. - `images.sh` gains `build-core`, `push-core`, `build-installer` and `push-installer`; Makefile targets `images-build-core`, `images-push-core`, `images-build-installer`, `images-push-installer`. `make demo-images` now runs all four, so the demo harness exercises the release path and its manifest is digest-pinned (the nodes' containerd mirror serves a pull by digest as it serves one by tag). `demo-images-build` and `demo-images-push`, the two-step flow documented in the getting-started guide, still render a manifest of tags. - The released CLI (`ichor init` applies the manifest embedded in it) is built outside the workflow, by goreleaser or `make release`. The workflow uploads the complete `bin/digests.txt` as the artifact `release-image-digests` (90 days); `ICHOR_IMAGE_DIGESTS= goreleaser release` and `make release IMAGE_DIGESTS=` embed a manifest with the same digests. A snapshot build or a `make release` without the file embeds tags, as before. - The Cilium Helm Job (the third residual risk) now runs the installer image the last Ichor upgrade recorded: when `status.installerImage` is for `spec.version` and `spec.installerRepository` and has a digest, the Job's image is `@` (verified, or recorded as skipped). Without such a record (a fresh `ichor init`, a Helm-only install with no IchorInstallation, a version changed by hand) it is `repository:version` as before; closing that case needs a verified digest the controller has not resolved, which would put the resolve-and-verify step on the Cilium path too, and was not done. The Job-image policy already accepted both forms. Tests: `test/harness/ui-demo` runs `images.sh` against an in-memory registry with a stub `go` (push the two images, check the digest file and the registry's digests, render the manifest with and without digests through `make build-installer`, embed it and refuse a tag-only one, push the installer and append its digest), `hack/pin-image.sh`, and `make helm-package` with digests rendered with `helm template`; `internal/controller` tests the Cilium Job image for a matching record, another version, another repository and no digest. `actionlint` passes on the workflow. Still open: a tag-form manifest in the two-step `demo-images-build` flow and in snapshot CLI builds (neither is a release); the first upgrade onto a release with this change still starts from the older controller's tag-form manifest. Hardware verification (later): push a `v*` tag and check that the installer image's embedded manifest (`ichor installer` Job logs, or `kubectl get deploy -n ichor-system -o jsonpath='{..image}'` after an upgrade) shows `@sha256:` for the controller and ichor-ui; run `make demo-images` against the demo registry and an `ichor init` / `ichor upgrade ichor` with the digest-pinned manifest to see the nodes pull by digest through the mirror; after an Ichor upgrade, the `ichor-cni` Job's pod image is `ichor-installer@sha256:...`. ## Amendment (2026-11-05): the Cilium Helm Job never runs a tag The follow-up above left one case open: without a `status.installerImage` record for the installed version and repository (a fresh `ichor init`, a Helm-only install, a version changed by hand) the Cilium Helm Job ran `repository:version`, a tag that can move, under the `ichor-cni` ServiceAccount, which is close to cluster-admin (ADR 0108). That case is closed, and with it the "residual risk" the Job-image policy carried for the Cilium Job. - `cniJobImage` (`internal/controller/cni_helm.go`) returns `repository@digest` or an error. A recorded digest is used as it is (when it is for `spec.version` and `spec.installerRepository`, and is `spec.installerDigest` when that is set: a record that is not the pin is not trusted). Without one the controller calls `pinInstallerImage`, the function the Ichor upgrade uses: it resolves the tag (or takes the pin), verifies the cosign signature with the same verifier and identity, and honours `spec.installerSignature.mode: skip`, so one setting governs both Jobs. A Helm-only install has no IchorInstallation; its stand-in (the controller's own version, the default repository) is resolved and verified the same way. - This happens in `AdvanceCNIHelmJob`, after the values check and before the Job is created, and only when the Job does not exist yet. On a refusal no Job is started and the error is an `*installerImageError` with the installer path's reasons (`InstallerImageUnresolved`, `InstallerImageUnverified`; a controller without a verifier refuses too). The hand-over reconciler lets go of the cluster lock, sets the `CNIValuesApplied` progress condition to that reason with the message, and requeues; the kube-proxy switch and the chart upgrade return the error, which pauses their operation as a step failure with the same text. `NewCNIKubeProxyAdvancer` and `CNIHandoverReconciler` take the `InstallerImages` the installer path already uses (`cmd/main.go`). - The Job-image policy (`config/admission/controller-jobs-policy.yaml` and the chart copy) now holds both Jobs to `repository@sha256:<64 hex>` (`spec.installerDigest` when set); the Cilium-only `repository:version` rule is gone. The chart's `admission.jobImageRepository` rule (chart-only installs) requires the same digest form. - Not done (ponytail): the digest resolved here is not written to `status.installerImage`, so each new Cilium Job of a Helm-only or fresh install resolves and verifies again (one registry round trip per Job). A skipped check on this path raises no Warning Event of its own; the upgrade's event and `status.installerImage` already show a skip where there was an upgrade. Tests: `TestCNIJobImageIsADigestTheControllerChecked` (record, other version, other repository, pin, skip, unresolvable, unverified, no verifier), `TestAdvanceCNIHelmJobStartsNoJobForAnUnverifiedImage`, the apiserver policy tests in `api/v1alpha1` (the Cilium Job refuses a tag and any digest but the pin). Hardware verification (later): on a cluster with no `status.installerImage` (fresh `ichor init`), enable a Cilium value edit and see the `ichor-cni` Job's pod image be `ichor-installer@sha256:...` and an `InstallerImageUnverified` progress message with an unsigned build and no skip. Amendment 2026-11-05 (ADR 0130): skipping the signature check no longer needs a reason. `spec.installerSignature: {mode: skip}` is valid without one (the CEL rule is removed; `reason` is optional, deprecated and unused), `ichor upgrade ichor --skip-signature-check` is a boolean flag, the Connect `IchorInstallerOptions` carries `skip_signature` (the deprecated `skip_signature_reason` still means skip when an older client sends it), `status.installerImage.reason` is no longer written, and the `InstallerSignatureSkipped` Event, `ichor version` ("signature NOT checked"), the plan warning and the UI badge say the check was skipped without a reason. Everything else (the digest, the opt-out being explicit and never inherited, the version-subresource right) stands. Amendment 2026-11-06 (hardware lane J, P2: a fresh `ichor init` of an unsigned demo build cannot hand Cilium over): since the previous amendment of 2026-11-05 the Cilium Job resolves and verifies the installer's tag when `status.installerImage` has no record for the installed version, and right after `ichor init` there never is one (only an Ichor upgrade writes it). On a cluster whose controller cannot reach the registry (the demo: nodes pull through a containerd mirror, the controller does not) `CNIHandover` went `False/InstallerImageUnresolved` (`Get "https://ghcr.io/v2/": dial tcp ...: i/o timeout`), no Job was created, and the demo needed `spec.installerDigest` and `spec.installerSignature.mode: skip` patched in by hand. The release flow is unaffected, because ghcr.io is reachable there and the signature is real. Decision: `ichor init` takes the two flags of `ichor upgrade ichor`, `--installer-digest sha256:...` and `--skip-signature-check`, and writes them into the IchorInstallation it creates (`spec.installerDigest`, `spec.installerSignature.mode: skip`). The writer is init, a CLI creating the object, not the controller (which never mutates `spec`); the field was already there, so no CRD change. With the pin set `pinInstallerImage` never resolves the tag, and with `skip` it never asks for a signature, so the controller needs no network to run the Cilium Job as `repository@digest`; without the flags nothing changes (the tag is resolved and the signature required). A skipped check is shown at once: init prints a Warning, and `spec.installerSignature` records it on the object (the Cilium path writes no `status.installerImage` record of its own, as the previous amendment says). The digest is not derived by init itself: the installer image embeds the CLI and its manifest, so its own digest cannot be inside them (a digest in the binary would change the digest); it comes from `DIGEST_FILE` of `make demo-images` (the line `ghcr.io/maeshinshin/ichor-installer@sha256:...`), as the upgrade of the demo already takes it. The next `ichor upgrade ichor` replaces both settings with what its own flags say, as before (ADR 0130), so a demo that upgrades keeps passing both flags. The demo harness (`test/harness/ui-demo/README.md`, step 1 and 5) now builds with `DIGEST_FILE` and passes both flags to `init`; there is no other init call in the repository. A released CLI built outside the release workflow with `ICHOR_IMAGE_DIGESTS` could embed the installer's digest too; not done (the registry is reachable there). The `ichor init -f` file has no keys for the two flags yet (ponytail: flags only; a demo is driven by flags). Tests: `TestCreateIchorInstallation_RecordsTheInstallerPin` and `TestCreateIchorInstallation_SkipIsWarnedAbout` and `TestInitInstallerFlags` (red first: no digest or skip written, no warning, a bad digest accepted), `TestCreateIchorInstallation_WithoutAPinLeavesTheInstallerToTheController`; in the controller the case "pinned by init with skip: no record and no registry needed" of `TestCNIJobImageIsADigestTheControllerChecked` (an unreachable registry, an unset status record, the pinned digest and skip: the Job image is `repository@digest`, nothing resolved or verified). Hardware verification (later): `make demo-images ... DIGEST_FILE=bin/demo-digests.txt`, then `ichor init ... --installer-digest $D --skip-signature-check` on an unsigned demo build with no hand patch: `CNIHandover` goes `True` with no `InstallerImageUnresolved`, the `ichor-cni` Job's pod image is `ichor-installer@$D`, and `kubectl get ichorinstallation ichor -o yaml` shows both fields from the start.