# 0099. Argo CD and Kokumi become Ichor's standard stack; Cilium is the only CNI and Ichor's own Helm release

Status: Accepted; superseded in part by ADR 0101 · Date: 2026-10-29 · Design section: Bootstrap, GitOps and Argo CD, CNI · Supersedes in part: 0013, 0019, 0024, 0028, 0048, 0055

Decided: `ichor init` always installs Cilium, Kokumi and Argo CD with no opt-out; Cilium is the only CNI, bootstrapped through Talos inline manifests and then owned by Ichor as a Helm release run by an installer-style Job with user values stored on the TalosCluster; Kokumi and Argo CD are applied once, then managed by Argo CD from artifacts the Ichor controller publishes into Kokumi's in-cluster registry, so no Git repository is needed

## Context

Four things in the repo no longer fit what the user now wants.

1. Argo CD and Kokumi are optional and Ichor installs neither (ADR 0019, 0048). `--argocd`, `--kokumi`, `--repo`, `--gitops-dir` and `gitops export` exist to serve that option.
2. The CNI is a choice (`flannel`, `cilium`, `custom`, ADR 0013), and Cilium is handed to Argo CD after bootstrap (ADR 0024, 0055). Users now need to edit Cilium's Helm values from the Ichor UI, which an Argo CD-owned release in a Git repository cannot offer.
3. Kokumi is built only as a repo-creds Secret (ADR 0048). Its install, its storage and its real data model were never checked against a running release.
4. `docs/design/architecture.md` says Ichor "bootstraps CNI and Argo CD once and hands them over". That sentence stays true for Argo CD and Kokumi. For Cilium it is replaced here.

User decisions this ADR implements (2026-10-29): Argo CD and Kokumi are the standard stack, always installed; Argo CD syncs from Kokumi's in-cluster OCI registry; Cilium is the only CNI and existing flannel/custom clusters may be deleted; Argo CD manages Cilium's neighbours (itself and Kokumi) after a one-time bootstrap, while Cilium stays Ichor-managed and UI-editable; Ichor publishes artifacts into Kokumi so a later "addons" UI can edit component values; tracking detection of Ichor's own CRs by a user's GitOps stays; Cilium-aware NetworkPolicies for the Kokumi and Argo CD namespaces are applied after init.

## Research

All versions below were read on 2026-10-29. The Kokumi release was downloaded and its checksum matched ADR 0048 (`install.yaml` sha256 `345223726800ca3f08f633c76c91a38265bc0c131551b24dc49c69ee0cd7b840`). Its source tarball was read for the docs that ship in `docs/content` (the kokumi.dev pages are older than release 0.17.0: they describe `Product`, `Registry` and `Repository` kinds that 0.17.0 does not contain; the source tree is authoritative).

### Kokumi

- Latest stable: **0.17.0** (2026-10-11), https://github.com/kokumi-dev/kokumi/releases. Apache-2.0, "experimental", API `delivery.kokumi.dev/v1alpha1`. Context7 has no Kokumi entry (the only hit was an unrelated library), so kokumi.dev and the source were used.
- Install: only `kubectl apply -f .../releases/download/0.17.0/install.yaml` (https://kokumi.dev/docs/installation/). There is no Helm chart and no OCI install artifact. The file is 134,389 bytes (14.9 KB gzipped). It holds one Namespace (`kokumi`), 8 CRDs (Approval, Kitchen, Menu, Order, Pantry, Preparation, Recipe, Serving), about 40 RBAC objects, 3 Deployments, 3 Services, 1 PVC, 1 ConfigMap, 1 Secret and 2 ValidatingAdmissionPolicies. Requires Kubernetes >= 1.26 and Argo CD >= 3.3.
- Images (exactly two): `ghcr.io/kokumi-dev/kokumi:0.17.0` (controller manager and API server/UI) and `registry:3.1.2@sha256:ddf75434...` (bundled registry). No scanner is in 0.17.0.
- Workloads and resources set in the manifest: controller manager (requests 10m/64Mi, limit 500m/128Mi), server (same), registry (request 100m/256Mi, limit 256Mi). Sum of requests 120m/384Mi, sum of limits 1.1 CPU/512Mi.
- Registry storage: PVC `kokumi-registry-pvc`, 1Gi, ReadWriteOnce, **no storageClassName**. A Talos cluster has no StorageClass unless the user adds one (Ichor installs no storage), so the registry pod would stay Pending. The registry Service is `type: NodePort` (30050) and the registry accepts anonymous reads and writes over HTTP; the server Secret `kokumi-server-auth` ships `admin`/`admin`. All three need handling (decisions 6 and 7).
- Data model (source `docs/content/docs/architecture.md`, `getting-started.md` at 0.17.0): an **Order** takes a source OCI artifact, renders it, and pushes the result as an immutable **Preparation** to a destination registry (default in-cluster, `oci://kokumi-registry.kokumi.svc.cluster.local:5000/preparation/<name>`). A **Serving** (one per Order, created automatically) points at one Preparation and creates the Argo CD Application. `promotion.mode` is `Automatic` (newest Ready Preparation) or `Manual`. A **Menu** is a template with override policies; a **Pantry** carries registry URL and credentials; a **Recipe** is a render profile.
- Source formats: with no `spec.render`, the source artifact must hold `*.yaml` files at its root, stored as-is ("pre-rendered manifest bundle"). With `spec.render.helm` it must be a Helm chart. In code (`internal/oci/oras_client.go`) the accepted single-layer media types are `application/vnd.cncf.helm.chart.content.v1.tar+gzip` and `application/vnd.cncf.flux.content.v1.tar+gzip` (a gzipped tar of the files).
- Publishing: there is no Kokumi API to publish a source. Ichor pushes an OCI artifact to the registry itself (oras-go or go-containerregistry are both fine; the registry is the CNCF `registry` image, plain HTTP, no auth) and creates an Order whose `spec.source.oci` and `version` name it. Ichor never needs a Product/Menu.
- The Application Kokumi creates (`internal/deployer/argocd.go`): name = Order name, namespace `argocd`, `project: default`, `source {repoURL: oci://<registry>/preparation/<name>, targetRevision: <digest>, path: "."}`, `destination {server: https://kubernetes.default.svc, namespace: <Order namespace>}`, `syncPolicy.automated {prune, selfHeal}` and `ServerSideApply=true`, annotation `delivery.kokumi.dev/allowed-order`. It refuses to update a pre-existing Application without that annotation. Consequence: an Order in namespace `X` named `N` produces Application `N` that deploys into namespace `X`; Ichor creates the Order `argocd` in namespace `argocd` and the Order `kokumi` in namespace `kokumi`.
- Argo CD reads that artifact as an OCI source: one layer, media type `application/vnd.oci.image.layer.v1.tar+gzip` or the Helm chart type by default, `path` selects inside the image (https://argo-cd.readthedocs.io/en/stable/user-guide/oci/). The docs there do not mention `insecureOCIForceHttp`; Kokumi's own Getting Started does, via a repo-creds Secret (`type: oci`, `url: oci://kokumi-registry.kokumi.svc.cluster.local:5000`, `insecureOCIForceHttp: "true"`), which ADR 0048 already builds.

### Argo CD

- Latest stable 3.x: **v3.5.6** (2026-10-20). Also current: v3.4.12, v3.3.18; v3.6.0-rc4 is a prerelease. Kokumi 0.17.0 requires >= 3.3 and its own Getting Started installs v3.5.1; ADR 0019 and 0055 verified shapes against v3.5.x (ADR 0055's e2e ran v3.5.6). Pin **3.5.6**.
- Install forms: `manifests/install.yaml` (1,917,766 bytes, 106 KB gzipped; images `quay.io/argoproj/argocd:v3.5.6`, `redis:8.2.3-alpine`, `dex:v2.45.1`; no resource requests set) or `core-install.yaml` (no API server/UI/dex, and no `default` AppProject, which made ADR 0055's e2e fail with `InvalidSpecError` until one was created; Kokumi's Applications use `project: default`). Or the Helm chart `argo-cd` **10.11.6** (appVersion v3.5.6, https://github.com/argoproj/argo-helm, 237,904 bytes, `kubeVersion >=1.25`, dependency `redis-ha` that is off by default). Pinned manifests give no switches for dex, notifications and the ApplicationSet controller; the chart does.
- Self-management: "Argo CD is able to manage itself since all settings are represented by Kubernetes manifests", and it must use `ServerSideApply=true` (https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/). Kokumi's Applications already set it.
- Not answered by the docs read: a size limit for an OCI source, the version in which OCI sources left beta, and whether `insecureOCIForceHttp` has a documented `argocd-cm` home. All three are checked in the e2e (work unit 14).

### Talos and Cilium (https://docs.siderolabs.com/kubernetes-guides/cni/deploying-cilium, Talos v1.14 docs)

Prescribed and already matched by `pkg/cni/cilium/values.go`: `ipam.mode=kubernetes`; `kubeProxyReplacement=false|true`; `securityContext.capabilities.ciliumAgent` (CHOWN, KILL, NET_ADMIN, NET_RAW, IPC_LOCK, SYS_ADMIN, SYS_RESOURCE, DAC_OVERRIDE, FOWNER, SETGID, SETUID; no SYS_MODULE) and `cleanCiliumState` (NET_ADMIN, SYS_ADMIN, SYS_RESOURCE); `cgroup.autoMount.enabled=false`, `cgroup.hostRoot=/sys/fs/cgroup`; `k8sServiceHost=localhost`, `k8sServicePort=7445` for the kube-proxy-free case; KubePrism enabled on 7445.

Where it matters for this ADR:

- "Talos only creates missing resources from inline manifests -- it never deletes or updates them." To change them, "edit all control plane machine configurations and run `talosctl upgrade-k8s`." So an inline manifest cannot carry a values change at runtime. **Ichor-managed Cilium cannot stay in the inline manifest**; this decides option (b) below. (ADR 0024 recorded, from Talos v1.14.3 source, that `manifest_apply.go` server-side-applies on every Kubernetes upgrade. The guide's wording is "creates only". The two readings agree that nothing is pruned; the difference only affects whether a stale inline entry can roll Cilium back. The hand-over below removes the entry either way.)
- v1.14 and later: use a `KubeFlannelCNIConfig` `$patch: delete` document to drop flannel (Ichor does: `pkg/render/networkinstall.go`), `KubeProxyConfig` `enabled: false` instead of `cluster.proxy.disabled`, and the inline-manifest document instead of the deprecated `cluster.inlineManifests`. `Preset.TalosPatches` still emits the legacy `cluster.proxy.disabled` and render strips it; this ADR cleans that up. The exact kind name of the v1.14 inline-manifest document (`KubeInlineManifestConfig`) is taken from the audit of the same guide; work unit 2 verifies it against the pinned machinery before use.
- The guide recommends inline manifests for production, only on control planes, identical on all of them, and warns that without a CNI the node waits "about 10 minutes" before retrying. Known issue: `bpf.masquerade` with `forwardKubeDNSToHost` can break CoreDNS; `forwardKubeDNSToHost=false` fixes it. Gateway API needs its CRDs installed first, then `gatewayAPI.enabled`, `enableAlpn`, `enableAppProtocol`.
- Differences from today's approach: (1) we use the legacy `cluster.inlineManifests` and `cluster.proxy.disabled` on a v1.14 contract (fixed here); (2) we remove the inline entry after hand-over, which the guide does not discuss and which ADR 0024 verified (kept, retargeted); (3) the guide's hosted/CLI/manifest-apply Helm variants are not used.
- The embedded Cilium 1.18.0 chart ships `values.schema.json` (146,865 bytes). It has **no `additionalProperties: false`**, so a misspelled key passes the schema. Helm already validates against the schema on render, so the existing render path gives type checking for free; an unknown-key check against the chart's own `values.yaml` is added (decision 4).

### Footprint (estimates; nothing was measured on a cluster)

Requests/limits come from the manifests above. Idle working-set figures for Argo CD are typical numbers for a small install and are the main uncertainty; unit 14 measures them.

| Component | Kokumi manifest says | Estimated working set |
|---|---|---|
| Kokumi controller + server | limit 128Mi each | 60-120Mi each; the 128Mi limit is tight while rendering/pulling, so Ichor raises it in the published bundle (unit 4) |
| Kokumi registry | 256Mi | 30-100Mi |
| Argo CD application-controller | none | 200-350Mi |
| Argo CD repo-server | none | 100-250Mi, spikes when it fetches an artifact |
| Argo CD server | none | 80-150Mi |
| Argo CD redis | none | 20-50Mi |
| Total | | about 0.7-1.2 GiB working set, about 1.3-1.7 GiB with sensible limits |

On the 1 control plane (3Gi) + 1 worker (2.5Gi) homelab, Talos and the control-plane components use about 1.2-1.5Gi of the control plane, and Cilium and Ichor add roughly 0.4-0.6Gi across the nodes. With a tainted control plane the stack lands on the 2.5Gi worker: it **fits, with little slack**. A **minimal Argo CD** (Helm chart with `dex.enabled=false`, `notifications.enabled=false`, `applicationSet.enabled=false`, `redis-ha.enabled=false`, no `controller` replicas beyond one) removes dex, notifications and the ApplicationSet controller, about 150-250Mi and 3 pods. It is the default (decision 5). The raw `install.yaml` cannot do that; that is why the chart is embedded instead.

## Decisions

### 1. Always installed, no opt-out

`ichor init` installs Cilium, Kokumi and Argo CD every time. Flags `--cni`, `--argocd`, `--argocd-namespace`, `--kokumi`, `--repo`, `--path`, `--repo-username`, `--repo-password` and `--gitops-dir` go away (`--cni` stays only as a hidden flag that accepts `cilium` for one release so scripts get an explicit message; `--cni-tunnel` stays). The `ichor-system` namespace and Ichor itself are applied as today (ADR 0019). Argo CD, Kokumi and Cilium are Ichor's three bootstrap components.

### 2. Cilium: option (b), a Helm release owned by Ichor

Option (a), keep Cilium in `cluster.inlineManifests` and change values by editing it, is rejected on the Talos guide's own text: Talos creates inline manifests and never updates or deletes them, so a values change would apply only through `talosctl upgrade-k8s`, with no prune. Option (b) is chosen.

- **Bootstrap stays the inline manifest**, rendered from the embedded chart plus the effective values (ADR 0098), delivered as the v1.14 inline-manifest document on a v1.14 contract (gated by `MultidocKubernetesConfigSupported()`, as `networkinstall.go` already gates its flannel choice) and as `cluster.inlineManifests` below it. Only control planes carry it, identical on all, as the guide requires.
- **Adoption.** After the control plane is Ready and Cilium's DaemonSet is Ready, a Helm release `cilium` in `kube-system` is created over the bootstrapped objects with the Helm SDK: `action.Install` with `TakeOwnership: true` (field present in `helm.sh/helm/v3` v3.21.4, `pkg/action/install.go:114`, the version in `go.mod`) and the same chart, release name, namespace and values as the bootstrap render. The result has no object changes; it only records release revision 1 as Secret `sh.helm.release.v1.cilium.v1`. The bootstrap objects are nevertheless rendered with Helm's release-ownership metadata (annotations `meta.helm.sh/release-name: cilium` and `meta.helm.sh/release-namespace: kube-system`, label `app.kubernetes.io/managed-by: Helm`), so that the release is created over objects that already look like its own and `helm`-aware tooling reads them as members. `TakeOwnership` stays: it is what makes the adoption not depend on that metadata (a cluster bootstrapped by an older build has none). This changes the bootstrap bytes once (see unit 7).
- **Operation.** Later changes are `action.Upgrade` with the new effective values: Helm's three-way merge updates changed objects and deletes objects the chart no longer renders (prune), with `Atomic`/rollback on failure so a bad value set does not leave a half-upgraded CNI. `lookup` works (Helm talks to the cluster), so the Hubble certificate reasoning in ADR 0055 (cronJob instead of `helm`) no longer forces a choice; the existing `cronJob` value is kept to avoid changing running clusters, and users may override it.
- **Who runs Helm.** The Helm SDK needs the permissions of everything the chart creates (ClusterRoles, CRDs, DaemonSet, ...), which is cluster-admin in practice (ADR 0059 reached the same conclusion for the installer: `escalate`/`bind` cannot be narrowed). So the controller does not run Helm. It launches a one-shot Job, exactly as the Ichor installer does: image `ghcr.io/maeshinshin/ichor-installer:<version>` (reused; same image, new hidden subcommand `ichor cni helm apply`), its own ServiceAccount `ichor-cni`, created per run, deleted after, with `ttlSecondsAfterFinished`. The Job uses `hostNetwork: true` and `KUBERNETES_SERVICE_HOST=localhost`, `KUBERNETES_SERVICE_PORT=7445` (KubePrism) so it reaches the API server while Cilium restarts; the replaced-mode requirement for KubePrism already makes that port a precondition.
- **Hand-over retargeted.** `CNIHandoverReconciler` (ADR 0024) keeps its shape, condition `CNIHandover` and render-time removal of the inline entry. Its signal changes from "Argo CD reports an Application Synced and Healthy" to "Helm release `cilium` is `deployed` at the effective-values hash and the DaemonSet is ready", read from the release Secret and the DaemonSet. `spec.cni.handover` (`automatic|manual|disabled`) is kept as a deprecated field; `manual` and `disabled` are read as `automatic` (decision 8). The upgrade guard `cniHandoverPending` keeps blocking a Kubernetes upgrade until the entry is gone.
- **Why not a controller-embedded Helm run or a CRD of Argo CD.** Embedding means giving the long-lived controller cluster-admin. An Argo CD Application would hand Cilium to Git or Kokumi, which the user decided against.

### 3. Cilium values: stored on the TalosCluster, merged by Ichor

New field `spec.cni.cilium.values` on `TalosClusterCNI` (an object with `x-kubernetes-preserve-unknown-fields`, append-only, immutable only in its owned subset, below). The effective values are, in order: Ichor defaults (`pkg/cni/cilium/values.go`), then the user's `spec.cni.cilium.values` deep-merged on top, then the **Ichor-owned keys** applied last. The UI shows the effective values read-only and the user values as the editable layer.

Ichor-owned keys (cannot be set by users; a value for one is a validation error naming the key and the reason, not silently ignored): `ipam.mode`, `securityContext.capabilities.ciliumAgent`, `securityContext.capabilities.cleanCiliumState`, `cgroup.autoMount.enabled`, `cgroup.hostRoot` (the Talos guide's requirements), `k8sServiceHost` and `k8sServicePort` (KubePrism), `kubeProxyReplacement` (owned by the kube-proxy switch, `spec.cni.kubeProxy`) and `tunnelProtocol` (owned by `spec.cni.tunnel`, immutable).

### 4. Validation

One function, `cilium.ValidateValues(values, context)`, used by the admission webhook (`internal/webhook/v1alpha1/taloscluster_webhook.go`), the Connect service behind the UI and CLI, and the controller before it launches the Job:

1. Render the embedded chart with the merged values (no cluster, no network). Helm's schema validation against the chart's `values.schema.json` yields type errors with paths. A failed render is an error with Helm's message.
2. Reject any Ichor-owned key.
3. Reject or warn on keys absent from the chart's `values.yaml` tree (the schema does not catch typos). Free-form maps in the chart (`extraArgs`, `annotations`, ...) are exempt.
4. Three semantic guards from the Talos guide: `bpf.masquerade=true` is refused unless the cluster's effective machine config has `machine.features.hostDNS.forwardKubeDNSToHost: false` (CoreDNS breaks otherwise; Ichor does not change the Talos side on its own in this ADR); `gatewayAPI.enabled=true` is refused unless the Gateway API CRDs exist (checked live by the controller, which sets a condition; the webhook cannot check and passes); and in replaced mode a patch that disables KubePrism or moves it off 7445 is refused. The context argument carries what each caller can know; the controller re-checks live before every Job.
5. Size: the merged values must stay under 64 KiB (they are stored on the CR and rendered into the Job).

### 5. Kokumi and Argo CD: applied once, then Argo CD owns them

Bootstrap order, from a fresh `ichor init`:

1. Talos config with the Cilium inline manifest is applied; bootstrap etcd; kubeconfig; Secrets bundle stored (unchanged, ADR 0019 steps 1-4).
2. Cilium becomes Ready (the existing wait). Ichor is applied through the API server (unchanged).
3. `init` creates `IchorInstallation` with a new `spec.stack` (below) and waits. The controller (not the CLI) does 4-8 so there is one implementation for init and for later upgrades.
4. The controller applies the embedded Kokumi manifest (server-side apply, field manager `ichor-bootstrap`) with the transformations in decision 6, then waits for the registry Deployment to be Ready.
5. It renders the Argo CD chart with Helm (release `argocd`, namespace `argocd`, values below), applies it the same way, and waits for the server and repo-server.
6. It creates the Argo CD repo-creds Secret (`type: oci`, the registry URL, `insecureOCIForceHttp: "true"`; no credential in it) and the `default` AppProject if absent. These also ship in the Argo CD bundle so Argo CD keeps them.
7. For each of `kokumi` and `argocd` it publishes a **pre-rendered manifest bundle** to the registry (decision 7) and creates one Order (`delivery.kokumi.dev/v1alpha1`, `promotion.mode: Automatic`, no `spec.render`). Kokumi renders a Preparation from it, its Serving creates Application `kokumi` / `argocd`, and Argo CD adopts the live objects. Because the bundle contains exactly the bytes Ichor applied in 4-5, the first sync shows no diff (the same "adoption produces no diff" rule as ADR 0024). Pre-rendered bundles, not Helm Orders, are used deliberately: the bytes come from Ichor's own Helm SDK and embedded chart, not from the `helm template` inside Kokumi, whose version and default `KubeVersion` Ichor does not control.
8. The policies of decision 10 are applied last, once both namespaces and workloads exist.
9. `IchorInstallation.status.stack` reports per component: published digest, Order ready, Application Synced/Healthy. When both are Healthy, init finishes. This is the stack's hand-over; there is nothing to remove afterwards because nothing was inlined.

Argo CD configuration shipped by Ichor (chart values; user-overridable except where noted): `dex.enabled=false`, `notifications.enabled=false`, `applicationSet.enabled=false`, `redis-ha.enabled=false`, `configs.params."server.insecure"` unset, `configs.cm."application.resourceTrackingMethod"` left at Argo CD's default (Ichor reads it, ADR 0044), `crds.keep=true`. Ichor-owned keys, as for Cilium: the repo-creds Secret, the `default` AppProject, and the `argocd-cm` resource customizations for Ichor's kinds. The Lua health checks of ADR 0046 move into the Argo CD bundle's `argocd-cm`, which resolves that ADR's limitation (Ichor did not own `argocd-cm`; it now does). Argo CD has no external Git repository, so `repositories` is empty.

### 6. Where the manifests come from and their size

Embedded in the `ichor` binary and in the installer image, like Cilium (ADR 0098), each with a `make` fetch target that refuses a file whose SHA-256 differs from the recorded digest:

| Artifact | Form | Size in the binary |
|---|---|---|
| Cilium 1.18.0 | chart `.tgz` (already embedded) | 229 KB |
| Argo CD chart 10.11.6 (app v3.5.6) | chart `.tgz` with `redis-ha` subchart | 238 KB |
| Kokumi 0.17.0 | `install.yaml` | 134 KB raw (store gzipped, 15 KB) |
| Helm's own dependencies | already linked for Cilium | 0 extra |

Pinned artifacts as embedded by `pkg/stack` (unit 4); `make argocd-chart` and `make kokumi-install` fetch them and refuse a file whose SHA-256 differs from the Makefile:

- Argo CD chart: https://github.com/argoproj/argo-helm/releases/download/argo-cd-10.11.6/argo-cd-10.11.6.tgz, sha256 `d8f096d87513b41ff754053a7f9604c943bf29c15436a79226fd51595ea67d05`, 237,904 bytes, in `pkg/stack/argocd/chart/`.
- Kokumi: https://github.com/kokumi-dev/kokumi/releases/download/0.17.0/install.yaml, sha256 `345223726800ca3f08f633c76c91a38265bc0c131551b24dc49c69ee0cd7b840`, 134,389 bytes raw, stored as `pkg/stack/kokumi/install.yaml.gz` (14,611 bytes, `gzip -9 -n`).
- Correction found while implementing: chart 10.11.6 has no `applicationSet.enabled` value, so `pkg/stack/argocd` drops the `argocd-applicationset` templates after rendering (the ApplicationSet CRD stays). `redis-ha.enabled=false` only works once Helm's dependency conditions are processed, which the renderer does. The chart's `redis-secret-init` hook Job is dropped with all hooks, so the renderer emits the `argocd-redis` Secret itself from a caller-supplied password.
- Bundles are pushed with go-containerregistry (already in `go.mod`, previously indirect): one layer `application/vnd.cncf.flux.content.v1.tar+gzip` holding `manifest.yaml`, config type `application/vnd.cncf.flux.config.v1+json`, over plain HTTP.

About 0.4 MB added. The rendered Argo CD bundle is about 1.9 MB raw (the three CRDs dominate), about 0.1 MB as a gzipped tar. That is above what fits comfortably in a machine config, which is why Argo CD and Kokumi are applied through the API server and not through Talos inline manifests. Images are pulled from `ghcr.io`, `quay.io`, `public.ecr.aws` and `docker.io`; air-gapped installs are out of scope (ADR 0090), so nothing is mirrored here.

Transformations applied to the Kokumi manifest before it is applied and published, each a small, tested, deterministic step in `pkg/stack/kokumi`: the registry Service becomes `ClusterIP` (the shipped `NodePort 30050` would expose an anonymous, writable registry on every node); the registry volume becomes an `emptyDir` with a size limit unless the cluster has a default StorageClass (see Risks: artifacts are rebuilt from the Ichor controller on registry loss); the controller and server memory limits are raised to 256Mi; the `kokumi-server-auth` Secret's `admin`/`admin` is replaced by a generated password kept in a Secret `ichor-kokumi-admin` in `ichor-system` and shown once by `ichor init` (a login through Ichor's OIDC issuer, ADR 0094, is future work: Kokumi supports OIDC through its Kitchen resource).

### 7. Publishing into Kokumi, and the data flow for the future "addons" UI

New package `pkg/stack` and a controller section in the `IchorInstallation` reconciler.

- **State.** `IchorInstallation.spec.stack` (new, append-only): `argocd.values` and `kokumi.values` (free-form objects, user layer only), `promotion` (`Automatic` default, or `Manual` so a human promotes in Kokumi's UI). `status.stack.<component>` holds `publishedTag`, `digest`, `orderReady`, `applicationHealth`. This singleton already exists, is cluster-scoped and is the object Ichor upgrades; the addons UI later writes `spec.stack.<component>.values` through a service the same way it writes Cilium's values, and a future third component adds a key, not a new mechanism. Kokumi has no chart, so its `values` initially accept only the keys Ichor maps to patches it controls (memory limits, replica counts); this is stated in the field docs.
- **Render.** Effective values = Ichor defaults, user values, owned keys (decision 3 rules, same validation function with a different chart and owned set). Rendered with the Helm SDK to a single `manifest.yaml`.
- **Publish.** A deterministic tar (sorted entries, zero mtimes, fixed owner) gzipped, pushed as one layer of media type `application/vnd.cncf.flux.content.v1.tar+gzip` to `kokumi-registry.kokumi.svc.cluster.local:5000/ichor/<component>:<chartVersion>-<hash12>`, where the hash covers the rendered bytes, with `org.opencontainers.image.source` and `.version` annotations. The same input always gives the same digest, so a republish after a registry restart restores the same references.
- **Order.** `delivery.kokumi.dev/v1alpha1 Order` `argocd` (namespace `argocd`) and `kokumi` (namespace `kokumi`), `spec.source.oci` = the repository, `version` = the tag, `promotion.mode` from `spec.stack.promotion`, default destination. Changing a value moves `version`, Kokumi renders a Preparation, Automatic promotion moves the Serving, Argo CD syncs. History, approvals and rollback are Kokumi's.
- **Reach.** The controller pushes from inside the cluster; the registry has no authentication, so the policy in decision 10 is what restricts writers. The controller's RBAC gains `orders` in the two namespaces and `get/list` on `servings`/`preparations`; it never writes Argo CD objects.
- **Idempotence and loss.** The reconcile compares the Order's `version` and the registry's manifest digest each time; if the tag is gone (registry rebuilt) it republishes and, if Kokumi does not re-render by itself, bumps an annotation on the Order (unit 6; whether Kokumi re-renders a Preparation whose artifact vanished is an open question).

### 8. Cilium is the only CNI; flannel and custom are removed

User decision: existing flannel/custom clusters may be deleted, so no code path keeps them running.

- **New clusters:** `preset` defaults to `cilium`; the webhook rejects creation with any other value.
- **Stored objects (append-only CRD).** The enum `flannel;cilium;custom`, the `custom` field and its types stay in the schema, marked deprecated in the Go doc comments and in the generated CRD description. Removing an enum value does not trip the installer's additive check (`internal/installer/crdcheck.go` compares served versions, property names, property types, `items` and `additionalProperties`; it does not read `enum`), but removing the `custom` property or its sub-properties would fail with `... would be removed` (line 83), and any removal would also make stored `preset: flannel` objects fail validation on their next update. So **nothing is removed from the schema**, and the enum keeps all three values. The transition rule `self.preset == oldSelf.preset` stays.
- **A stored `preset: flannel|custom` is rejected on update, not ignored.** The webhook refuses any change to such a cluster's `spec` (finalizers, labels, annotations and deletion are allowed) with a message saying the preset was removed in this release and the cluster should be deleted and recreated. Ignoring it would be worse: render would now always strip Talos's flannel document, so the next config apply to a flannel cluster would take its network down. The controllers act the same way: `TalosMachineReconciler` and the renderer refuse to render a cluster whose preset is not `cilium`, set a `CNIUnsupported` condition on the `TalosCluster` and make no change to nodes. Nothing else in the code handles flannel or custom.
- `spec.cni.custom`, `spec.cni.handover=manual|disabled`: stored values are read but have no effect; the webhook rejects setting them to anything but absent / `automatic` on create. `spec.cni.kubeProxy` and `spec.cni.tunnel` stay and work as before.
- Code removed: see "What goes away".

### 9. Kube-proxy replacement now changes the Cilium release itself

`spec.cni.kubeProxy` is already an Ichor-owned input (`kubeProxyReplacement`, `k8sServiceHost`, `k8sServicePort` derive from it). The switch executor (`internal/controller/cni_kubeproxy_switch.go`) currently prints the values the operator must set in Cilium's config and waits for the operator to do it (`ciliumChangeInstructions`). With Ichor owning the release, the CNI stage launches the Helm Job with the new mode, then runs the existing verification (`cilium-config` key and DaemonSet rollout), and only then touches the Talos side (`KubeProxyConfig`, v1.14). The order, the confirmation annotation and the "turn the old side off only after the new side is verified" rule are unchanged. The `ichor-cni` Job is the same one used for a values edit, so a kube-proxy switch and a user edit cannot overlap: the Job takes the existing cluster operation lock.

### 10. NetworkPolicies for the Kokumi and Argo CD namespaces (a separate work unit)

Cilium is up before either namespace exists, so the policies are CiliumNetworkPolicies shipped inside each component's bundle, applied by the controller as the last step of the bootstrap (step 8) and re-applied by Argo CD on every upgrade. The `ichor-system` policies are another agent's work and are not designed here. Each of the `kokumi` and `argocd` namespaces gets a default-deny (ingress and egress) plus explicit allows:

- DNS: egress to `kube-system` `k8s-app=kube-dns` on 53/UDP and 53/TCP, from every pod, with a `dns` rule so the allowed names are visible.
- Argo CD to kube-apiserver: egress `toEntities: [kube-apiserver]` (Cilium's entity, so it works with KubePrism and with the node-local endpoint) from the application-controller, server, repo-server (for the one cluster call it makes).
- Argo CD to the Kokumi registry: egress from the repo-server (and the application-controller, which resolves revisions) to pods `app=registry` in `kokumi` on 5000/TCP; the matching ingress on the registry side.
- Ichor publishing into Kokumi: ingress on the registry (5000) and on `kokumi` to the Ichor controller pod in `ichor-system` (namespace and pod labels), egress from `ichor-system` is the other agent's unit; this unit only provides the receiving rules and names the dependency.
- Kokumi's own needs: controller-manager and server to kube-apiserver; controller-manager to the registry (render and push) and to nothing external, since sources are in-cluster; the server's own port 8080/80 ingress from `host` entity only (for `ichor ui`-style port-forward, which arrives from the node); Argo CD ingress for its server from `host` and, for Ichor's UI reading status, none (it reads CRs through the API server).
- Redis inside `argocd`: ingress only from the application-controller, server and repo-server.
- Intra-namespace health probes: kubelet probes come from the `host` entity and are allowed explicitly.
- Applied "once their namespaces exist": the controller applies them after the workloads' namespaces and Deployments exist, never before, so a default-deny cannot cut off the bootstrap sync.

### 11. Upgrades after hand-over

- **Cilium:** the pinned chart changes with the Ichor release. `ichor upgrade ichor` does not upgrade Cilium on its own; it makes the new chart available (`status.cni.availableChartVersion`, condition `CiliumUpgradeAvailable`). The user starts it (`ichor cni upgrade`, or the UI button) as an operation of the existing kind `cni`, which validates the user's values against the new chart's schema first and fails early if they no longer validate, then runs the Helm Job. Cilium supports upgrading one minor version at a time, so each Ichor release changes the pinned Cilium by at most one minor. Rollback is Helm's.
- **Argo CD and Kokumi:** the new controller (after `ichor upgrade ichor`, ADR 0059) recomputes the bundles from its embedded charts and the stored `spec.stack` values; if the tag differs from the Order's `version`, it publishes and updates the Order. With `promotion: Automatic` Argo CD applies the new Preparation; with `Manual` the new Preparation waits in Kokumi's UI. Argo CD upgrades itself this way (the Application managing Argo CD deploys the new manifests; `ServerSideApply=true` handles the CRD growth), and Kokumi upgrades itself the same way. Since both are Argo CD's after this point, Ichor never runs `kubectl apply` on them again; the first-run apply (decision 5, steps 4-5) happens only when the namespace has no Order of that name.
- A user who edits the Orders or the bundle in Kokumi by hand will see Ichor overwrite the Order's `version` on the next reconcile. The Order is Ichor-owned (label `ichor.maesh.dev/managed`); the values are the supported way to change anything.

### 12. Git tracking detection stays

`pkg/gitopstracking`, ADR 0044 and ADR 0051 are unchanged: the UI and CLI still refuse writes to CRs a user's own Argo CD or Flux tracks, the `spec.gitops` repository connection and PR mode (ADR 0056) stay, and Ichor still reads `argocd-cm` rather than guessing the tracking method. That `argocd-cm` is now normally Ichor's own, but a user may run another Argo CD; the code does not care. `TalosCluster.spec.gitops.mode: strict` keeps meaning "the user's Git is the source". `ichor.maesh.dev` CRs are not placed in Ichor's own Argo CD; Ichor's own Applications track only the Kokumi and Argo CD bundles, so Ichor's own Argo CD never tracks an Ichor CR and never triggers the warning.

## What goes away

Removed (the files are named so each can be deleted by a work unit, with tests):

- `internal/cli/init_gitops.go`, `init_gitops_test.go` (`initArgoCD`, `buildRootApplication`, `buildRepoSecret`, `buildSkeleton`).
- `internal/cli/kokumi.go`, `kokumi_test.go` (the repo-creds helper moves to `pkg/stack`).
- `internal/cli/gitops.go`, `gitops_export.go`, `gitops_export_test.go`, `gitops_handoff.go`, `gitops_handoff_test.go` (the `ichor gitops export|handoff` commands), and `exportCNIApplication`.
- `internal/cli/init.go` flags listed in decision 1 and `initOptions` fields `ArgoCD`, `ArgoCDNamespace`, `Kokumi`, `RepoURL`, `RepoPath`, `RepoUsername`, `RepoPassword`, `GitOpsDir`; `init_plan.go` `CNIPreset` validation (line 268), `init_cni.go` (`flannel` branch, `wantArgoCD`, the Application half of `cniBootstrap`), `init_run.go` (the GitOps steps and the Application wiring), and their tests: `init_cni_test.go`, `init_run_test.go`, `init_leak_test.go`, `helpers_test.go`.
- `pkg/cni/flannel/` (whole directory); `pkg/cni/preset.go`: `ArgoCDApplication`, `HandoverWorkloads` and the Argo-specific doc; `pkg/cni/cilium/cilium.go` `ArgoCDApplication` and its ADR 0055 tests; `Preset.TalosPatches`' legacy `cluster.proxy.disabled`.
- `pkg/gitops` builders and the skeleton (`skeleton.go`, `application.go` `New`) after confirming `pkg/prmode/source.go` only needs the Application read types (it does not use the builders).
- `internal/controller/cnihandover_controller.go`: the Argo CD lookup (`argoApplicationGVK`, `argoApplicationStatus`, `WaitingForArgoCD` reasons), `cniPresetFor`'s flannel branch; `internal/controller/kubernetes_upgrade.go` (`cniHandoverPending` comment and flannel case), `cni_kubeproxy_switch.go` (`ciliumChangeInstructions` and its Argo wording), `etcd_restore.go` (`kube-flannel` in `restoreAgentNames`), `ca_rotation_clients.go` and `config/rbac/role.yaml` (`kube-flannel` resourceName), and the `cni` rule markers that name Argo CD.
- `pkg/render/networkinstall.go`: `talosRunsFlannel`, `cniPresetFlannel`, the `preset == flannel` branch; the flannel document is always removed. `internal/cli/adopt_run.go` lines 222-223: `adopt` of a flannel cluster is refused with the deletion message (adopt keeps working for Cilium clusters).
- Docs: architecture.md ("Bootstrap" step 3, the whole "GitOps and Argo CD" hand-over text, "Kokumi as the delivery layer in front of Argo CD", "CNI" presets text, the Open questions entry on Kokumi) and the repository skeleton block; ADRs 0013, 0019, 0024, 0028, 0048, 0055 gain a "superseded in part by 0099" line.
- Kept: `pkg/gitopstracking`, `pkg/prmode`, `spec.gitops.*`, `internal/uiapi/gitops_*`, the UI GitOps page, `dist/chart` `argocdHealth` (its Lua files now also feed the bundle's `argocd-cm`).
- Kept as deprecated, unused CRD fields (append-only): `spec.cni.preset` values `flannel`/`custom`, `spec.cni.custom` and `TalosClusterCNICustom*` types, `spec.cni.handover` values `manual`/`disabled`, `TalosOperation` fields that mention handoff.

## Work units

Units are small and mostly independent; each names its files. Dependencies are in brackets. Commit incrementally, one Conventional Commit per unit.

1. **CRD fields and deprecations.** `api/v1alpha1/taloscluster_types.go` (add `spec.cni.cilium.values`, change the `preset` default to `cilium`, deprecate `custom`/`flannel`/`handover` values in comments), `api/v1alpha1/ichorinstallation_types.go` (`spec.stack`, `status.stack`), `make manifests generate`, `config/crd/bases/*`, `dist/chart`. Check `internal/installer` additive test still passes against the previous CRDs. [none]
2. **Talos v1.14 render cleanup.** `pkg/cni/cilium/cilium.go` (`TalosPatches` emits `KubeProxyConfig enabled:false`), `pkg/render/networkinstall.go`, `pkg/render/kubeproxy.go`, the inline-manifest document on a v1.14 contract (verify the kind in the pinned machinery), `pkg/fieldmeta/fieldmeta.go`, tests. [none]

   Unit 2 as built: the kind is `KubeInlineManifestConfig` (apiVersion `v1alpha1`, Go type `k8s.KubeInlineManifestConfigV1Alpha1` in `talos/pkg/machinery/config/types/k8s/inline_manifest.go`, v1.14.3; fields `name`, `manifest`). Patches keep carrying the legacy `cluster.inlineManifests` entries (the typed CRD has no document kind for it); `applyPatches` converts them to that document on a `MultidocKubernetesConfigSupported()` contract, and the hand-over removes the entry before conversion, so it works for both forms. `Preset.TalosPatches` no longer emits `cluster.proxy.disabled`; `injectKubeProxy` is the only owner. Hash effect: a v1.14-contract render with a patch inline manifest (every cluster with the CNI entry) changes from `cluster.inlineManifests` to a `KubeInlineManifestConfig` document, so its hash moves once (a rolling config apply, safe class via a new `fieldmeta` rule). Contracts below v1.14 render byte-identically; v1.14 renders without inline manifests are unchanged.
3. **Cilium-only code removal.** Delete `pkg/cni/flannel/`, flannel/custom branches in `internal/cli/init_cni.go`, `init_plan.go`, `internal/controller/*` listed above, `config/rbac/role.yaml`. Webhook: create requires `cilium`, update of a stored flannel/custom `spec` refused, `CNIUnsupported` condition and render refusal. `internal/webhook/v1alpha1/taloscluster_webhook.go`, `pkg/render`, `internal/controller/talosmachine_controller.go`. [1]
4. **`pkg/stack` (render and publish).** New package: embedded Argo CD chart and Kokumi manifest with fetch targets and digests (`Makefile`, `pkg/stack/argocd/chart/`, `pkg/stack/kokumi/install.yaml.gz`), Kokumi transformations, Argo CD default values, deterministic bundle tar, OCI push (oras-go or go-containerregistry; pick by Context7 docs), an in-memory registry fake for tests. [none]
5. **Values engine.** `pkg/values` shared by Cilium, Argo CD and Kokumi: merge order, owned-key rejection, unknown-key check, schema validation, size cap; `pkg/cni/cilium/validate.go` with the three semantic guards. [none]
6. **Stack controller.** `internal/controller/stack.go` inside the `IchorInstallation` reconciler: steps 4-9, Orders (unstructured, no Kokumi Go dependency, like the Argo CD lookups today), status, republish on registry loss, RBAC markers, `config/rbac`, `dist/chart`. [1, 4, 5]
7. **Cilium Helm Job.** Hidden `ichor cni helm apply` in `internal/cli`, Job builder and watcher in `internal/controller/cni_helm.go`, SA `ichor-cni` RBAC in `config/rbac/ichor-installer` style with the chart-comparison test, adoption with `TakeOwnership`, upgrade and rollback, lock acquisition. [1, 5]
   Unit 7 as built: `cilium.Apply` (`pkg/cni/cilium/helm.go`) looks the release up and runs `action.Install` (`TakeOwnership`, wait for the rollout, no rollback: an atomic rollback of an adoption would uninstall a running CNI) when there is none, otherwise `action.Upgrade` (`ResetValues`, wait, `Atomic`). The hidden `ichor cni helm apply --cluster NAME -n NS` reads the TalosCluster, computes `cilium.EffectiveValuesFor` and calls it. The controller side is `internal/controller/cni_helm.go`: `AdvanceCNIHelmJob` creates or reports the Job named by cluster and effective-values hash (`CNIValuesHash`), the hooks for unit 8 (hash to compare the release against) and unit 9 (launch with a new kube-proxy mode; the cluster operation lock is theirs to take). The Job is the installer image on the host network with `KUBERNETES_SERVICE_HOST=localhost`, `KUBERNETES_SERVICE_PORT=7445`, pod label `app.kubernetes.io/name: ichor-cni`, one attempt, 24 h TTL, 25 min deadline. Departures from the text above: (1) the ServiceAccount `ichor-cni` and its RBAC are static, shipped with Ichor (`config/rbac/ichor-cni`, chart `cniHelm.enabled`, default on), not created per run, because the controller could not create the ClusterRoleBinding without holding `bind` and `escalate` itself; (2) Helm's release record Secret is stored in the controller's namespace (`--storage-namespace`, `ichor-system`), not in `kube-system`, so the one Secret permission is a namespaced Role like every other (the chart refuses a cluster-wide Secret rule) and the controller, which already reads Secrets there, can read the record in unit 8; the release itself (name `cilium`, namespace `kube-system`) is as specified; (3) a user value that makes Cilium's chart create Secrets in `kube-system` (`hubble.tls.auto.method=helm`) is not covered by the Job's RBAC, Ichor keeps `cronJob`. Network policy: a `cni` policy for the pod label (egress to the kube-apiserver entity on 6443 and to the host entity on 7445) in `config/cilium-network-policy` and the chart. Hash effect: the bootstrap manifest now carries the ownership metadata and is re-marshalled, so the CNI inline manifest's bytes, and with them the machine config hash of every cluster, change once at this release (a rolling config apply, the manifest content is otherwise the same objects); the goldens in `pkg/cni/cilium/testdata` were regenerated deliberately. Helm's SDK pulls more indirect modules into `go.mod` (SQL storage drivers, oras).

   Unit 7 amendment (found on hardware): the Job and its ServiceAccount run in **`kube-system`**, not `ichor-system`. The Job needs `hostNetwork`, which Talos's default Pod Security admission (enforce `baseline`, exempting only `kube-system`) refuses elsewhere; with the Job in `ichor-system` its pod was never created and the hand-over could not happen. What changed: the `ichor-cni` ServiceAccount and the ClusterRoleBinding's subject are in `kube-system`; the Helm release record's Role/RoleBinding stay in `ichor-system` (the binding's subject is the `kube-system` ServiceAccount); the controller gets a hand-written namespaced Role `cni-job-role` in `kube-system` (`config/rbac/ichor-cni/controller_role.yaml`: `jobs` get/list/watch/create/delete, `pods` get/list, `pods/log` get, `events` get/list) instead of the `ichor-system` markers, and the installer a namespaced Role in `kube-system` for the ServiceAccount, Roles and RoleBindings it applies there; `config/default` puts these back in `kube-system` after kustomize's namespace transformer. The controller's Job informer is scoped to both namespaces. The `cni` network policy is dropped (ADR 0100 amendment). Fail-fast: a Job with no active pod `cniPodGrace` (2 min) after creation is taken to be stuck (its pods cannot be created); `AdvanceCNIHelmJob` deletes it and returns a `CNIHelmJobFailedError` with `NoPod`, the hand-over releases the cluster lock, sets `CNIValuesApplied=False` (`JobFailed`) with the newest `FailedCreate` event message of the Job, and requeues after 30 s to try a fresh Job, instead of holding the lock until the 25-minute deadline.
8. **Hand-over retarget.** `internal/controller/cnihandover_controller.go`, `render` (unchanged removal), `desiredInputsHash` unchanged, `kubernetes_upgrade.go` guard, tests; `spec.cni.handover` read as automatic. [3, 7]

   Unit 8 as built: `CNIHandoverReconciler` drops every Argo CD lookup and ignores `spec.cni.handover`. Once the cilium DaemonSet is ready it runs `AdvanceCNIHelmJob` under the cluster operation lock (identity `ichor-controller/cni-helm`, distinct from `controllerIdentity` so a node rollout holding the lock stops the Job and the reverse; taken through an `acquireDisruptiveLock` method, so it makes the usual shutdown and even-control-plane refusals). When the Job has completed, the newest `sh.helm.release.v1.cilium.v*` record in the controller namespace is `deployed` with a config that hashes to `CNIValuesHash`, and the DaemonSet is fully ready, it sets `CNIHandover=True` and records the hash in the new `status.cni.appliedValuesHash`. A later hash difference starts a new Job and leaves `CNIHandover` True (flipping it would re-render the inline entry); the new condition `CNIValuesApplied` reports `Applying`, `WaitingForLock`, `WaitingForRelease`, `InvalidValues`, and `JobFailed` with the Job's last pod log line (or its reason). A failed Job is not retried until the values change or its 24 h TTL removes it. A finished Job for a hash the values returned to is deleted when the release is deployed at other values, so a revert runs again. `cniHandoverPending` has no `disabled` exemption.
9. **Kube-proxy switch drives the release.** `internal/controller/cni_kubeproxy_switch.go`, `cni_kubeproxy_admin.go`, tests. [7]
10. **`ichor init` rewrite.** `internal/cli/init.go`, `init_run.go`, `init_plan.go`, `init_cni.go`; create `IchorInstallation.spec.stack`; wait on `status.stack`; print the Kokumi admin password once; delete the flags and files above. [3, 6]
11. **`ichor cni values|upgrade` commands and service.** `internal/cli/cni.go` (`values get|set|edit|diff`, `upgrade`), `proto/ichor/v1alpha1` (CniService), `internal/uiapi/cni_service.go` (effective values, validate, diff preview, apply: writes `spec.cni.cilium.values`, honours tracking refusal ADR 0051). [1, 5]
   Unit 11 as built: `CniService` (`GetCiliumValues`, `ValidateCiliumValues`, `DiffCiliumValues`, `SetCiliumValues`, `StartCniUpgrade`) and `ichor cni values get|set|edit|diff`, `ichor cni upgrade`. Writes go through `spec.cni.cilium.values` with a dry-run first; a tracked TalosCluster is refused with the ADR 0051 wording. Gateway API CRD presence and Talos `forwardKubeDNSToHost`/KubePrism are read server-side (one CRD metadata get, and a rendered control-plane config); an unreadable fact stays unknown and its guard passes. `StartCniUpgrade` runs its checks (an upgrade is offered, stored values validate, no other operation is unfinished) and creates the upgrade operation described next.

    The Cilium chart upgrade operation, as built. Shape: a `TalosOperation` of kind `cni` with the new, append-only `spec.cni` (`action: upgrade`, `chartVersion: <target>`), immutable, only allowed on the `cni` kind. Without `spec.cni` a `cni` operation is the kube-proxy switch, as before. `StartCniUpgrade` fills `chartVersion` from `status.cni.availableChartVersion` and creates the operation as the caller (a dry-run create first, so a caller who may not create `talosoperations` starts nothing). Availability: `CNIHandoverReconciler` compares the chart version in the newest `deployed` release record (`chart.metadata.version`) with the embedded `cilium.ChartVersion` on every reconcile after the hand-over and sets `status.cni.availableChartVersion` and `CiliumUpgradeAvailable` (True with the installed and available versions, or False `UpToDate`); it also watches `cni` operations so the end of an upgrade clears them. When the gap is more than one minor, the condition message says the step cannot be applied in one go. Flow: like an Ichor self-upgrade the operation is machine-less (`status.machines` stays empty); the backbone takes the cluster operation lock, and the executor (`cniKubeProxyAdvancer.upgradeChart`) (1) refuses when `spec.cni.chartVersion` is not the chart this controller embeds (Ichor changed after the operation was created), (2) refuses a downgrade or a jump of more than one minor from the release's chart (decision 11), pausing with the message and starting no Job, (3) runs the Cilium Helm Job (`AdvanceCNIHelmJob` with the operation's UID, and its last retry time, mixed into the Job name so a Job the hand-over left for the same values within its 24 h TTL, or a failed one, is not mistaken for this run), which upgrades the release to the embedded chart with the current effective values, (4) waits until the newest release record is `deployed` at the target chart, and (5) waits for the `cilium` DaemonSet to be rolled out, both on the `VerifyingNewSide` clock (30 min), then the operation completes and releases the lock. A failed Job pauses the operation with the Job's reason; Helm's atomic upgrade has already rolled back; `ichor operation resume` runs a new Job. The cluster's `status.cni` is no longer replaced wholesale by the kube-proxy switch (it set only `kubeProxy` and dropped `appliedValuesHash` and `availableChartVersion`); it sets just `kubeProxy`.
12. **UI Cilium settings.** `ui/src/pages/Cilium.tsx` (effective values read-only view, YAML editor for the user layer, validate, diff preview, apply), route in `ui/src/router.tsx` and `app.tsx`, nav entry; reuse the config-editing patterns of ADR 0039; preactcn only. A later `Addons` page uses the same editor against `spec.stack`. [11]
13. **NetworkPolicies for `kokumi` and `argocd`.** `pkg/stack/policies/*.yaml` (CiliumNetworkPolicies per decision 10) included in each bundle, applied last by unit 6, unit tests that parse them and a render test checking default-deny plus each named allow; e2e connectivity check with Cilium. Does not touch `ichor-system` policies. [4, 6]
14. **e2e on the e2e harness.** `test/e2e`: init to Healthy stack, a Cilium values edit applied live, a kube-proxy switch, an Argo CD values edit through Kokumi, a registry restart, measured memory of the stack on the 3Gi + 2.5Gi layout, plus the three unanswered Argo CD points. Hardware batch per the usual rule. [10, 7, 6]
15. **Docs.** `docs/design/architecture.md` rewrite of the sections above, supersession lines in ADRs 0013, 0019, 0024, 0028, 0048, 0055, `docs/adr/README.md` via `make adr-index`. [alongside each unit]
16. **Removal of GitOps export.** Delete `internal/cli/gitops*.go`, `pkg/gitops` builders and the skeleton, UI strings for export, once unit 10 no longer calls them. [10]

Units 1, 2, 4 and 5 can start at once on stub commits; 3 follows 1; 6 and 7 follow 1, 4, 5; 8-13 follow their brackets.

## Consequences

- One path for every cluster: Cilium plus Kokumi plus Argo CD. Nothing to choose, nothing to explain, and the e2e covers one shape.
- Ichor's controller now runs two kinds of Job (installer and Cilium Helm) and writes Kokumi Orders. The controller still never writes Talos nodes outside the machine controller and still never mutates `spec`.
- Cilium is editable from the UI because Ichor, not Git, holds its values. Argo CD never manages Cilium, so a Cilium change cannot be reverted by Argo CD. The cost is a cluster-admin-equivalent Job identity for a few minutes per change (same exposure as the installer).
- The user's own apps get Argo CD with no repository set up; they can add repositories to it. They still must not put Ichor CRs under it (the warning stays).
- Ichor now ships three third-party releases and bumps them: an Ichor release carries Cilium, Argo CD and Kokumi pins, each with a digest in the Makefile.
- Memory on the small homelab is tight (estimated 1.3-1.7 GiB for the stack, a 2.5Gi worker), and the default Kokumi limits of 128Mi are probably too low.
- ADR 0048's "Unknowns" are answered: the Application accepts only what Kokumi generates, so sync waves and `ignoreDifferences` cannot be carried (not needed now that Cilium is not delivered through Kokumi); the Product/Order question is answered by bundle-per-component Orders; air gap is out of scope (ADR 0090); the Argo CD range is >= 3.3 and 3.5.6 is pinned; Kokumi remains experimental `v1alpha1`.

## Risks

- **Kokumi is experimental** (`v1alpha1`, breaking changes in every minor release: AutoDeploy removed in 0.17, Menu made namespaced in 0.16, status fields removed in 0.15 and 0.13). Ichor pins 0.17.0 and must re-read the source for each bump. A broken Kokumi stops value changes for Argo CD and Kokumi but not the running workloads, and does not affect Cilium or the nodes.
- **Registry durability.** The default emptyDir loses the registry on a pod restart or node reboot. Source bundles and the pinned digest are reproducible, but Kokumi's Preparations (rendered artifacts) are not; if Kokumi does not re-render, Argo CD fails to fetch the revision until the Order is bumped. Using a PVC needs a StorageClass Ichor does not provide.
- **Self-management.** Argo CD prunes and self-heals its own manifests: a wrong bundle can remove Argo CD or the Kokumi registry. Mitigations: pre-rendered bundles are identical to the applied bytes, `Automatic` promotion can be set to `Manual`, and Kokumi Applications carry no resources-finalizer.
- **Bootstrap-time diffs.** First sync of an adopted component could show a diff if the bundle and the applied objects differ in defaulted fields; server-side apply makes that benign but the first Healthy takes longer.
- **Anonymous writable registry.** Any pod could push a source bundle that Argo CD later applies cluster-wide; the CiliumNetworkPolicies (decision 10) are the only barrier until Kokumi's `requireSourceVerification` is used with a key held by Ichor (future).
- **The Cilium Job path.** A Cilium upgrade replaces the CNI on live nodes; a Job that fails mid-way relies on Helm's atomic rollback. The Job uses KubePrism; if a cluster has KubePrism disabled the Job uses the in-cluster service address and can lose its API access during the agent restart.
- **Resource pressure** on small clusters (above), including Kokumi's 128Mi limits and Argo CD repo-server spikes.
- **Values footguns.** The schema does not reject unknown keys; the unknown-key check is best effort and some Cilium values only work with others.

## Open questions

- Does Kokumi 0.17.0 re-render a Preparation whose artifact has vanished from the registry, or must the Order be changed? (Unit 6 and the e2e.)
- Does Argo CD accept the Preparation artifact's layer media type (Kokumi pushes `application/vnd.kokumi.preparation.v1`) with its default `ARGOCD_REPO_SERVER_OCI_LAYER_MEDIA_TYPES`? Kokumi's own Getting Started works with Argo CD 3.5.1; our version is not tested.
- How large a bundle can Argo CD take from an OCI source (the docs name no limit; the Argo CD bundle is about 1.9 MB raw)?
- Is `hostNetwork` plus `localhost:7445` the right way for the Cilium Job to keep its API access, or should the Job tolerate a short API outage and retry?
- Should the registry use a PVC when a default StorageClass exists, and which size? 1Gi is Kokumi's default; Preparations are never garbage collected.
- Should Ichor render the Talos side of `bpf.masquerade` (`forwardKubeDNSToHost: false`) itself instead of refusing the value?
- Whether to wire Kokumi's login to Ichor's OIDC issuer (ADR 0094) in this change or later; until then the generated admin password is the credential.
- Does `ichor adopt` of an existing Cilium cluster need an adoption path for an Argo CD-managed Cilium Application (a cluster that went through ADR 0024)? It is out of scope here; those clusters are re-created like flannel ones unless the user asks for it.
- The exact Talos v1.14 inline-manifest document kind and its fields must be confirmed against the pinned machinery (unit 2).

## Amendment (2026-10-29): how decision 10 is realised (unit 13)

- The policies are `pkg/stack/policies/{kokumi,argocd}.yaml`, embedded and appended to each bundle by `kokumi.Render` / `argocd.Render`, so the published bundle keeps them and Argo CD does not prune them. `Options.OmitPolicies` leaves them out for the first apply (steps 4-5) and `stack.Policies(component)` returns them alone for step 8.
- The Argo CD chart renders Kubernetes NetworkPolicies by default (`global.networkPolicy.create`), and Cilium unions those with CiliumNetworkPolicies; the `argocd-server` one allows all ingress. `DefaultValues` switches them off.
- The apiserver egress uses port 6443, as ADR 0100 does. Neither Kokumi 0.17.0 nor Argo CD ships a webhook, so no apiserver ingress exists; a test fails if one appears, because it would also need `remote-node` (ADR 0100 amendment).
- Ichor's side of the publish flow is an egress rule to `kokumi` `app=registry` :5000 in `config/cilium-network-policy/policies.yaml` and the chart copy. A test requires both directions.
- Users reach `argocd-server` and `kokumi-server` (both ClusterIP) by `kubectl port-forward`, which the policies admit from the `host` entity on 8080. An ingress or gateway needs a policy of the operator's own.

## Amendment (as built: an unprivileged controller, and no Secret in a bundle)

Two decisions changed unit 6 after it landed.

1. **The controller does not apply Kokumi or Argo CD; a one-shot Job does (pattern of ADR 0059).** Unit 6 gave the controller `escalate`/`bind` on (cluster)roles and create/patch on CRDs, workloads, RBAC and admission policies so it could server-side apply the two manifests itself, which made it cluster-admin in effect. Decision 5, steps 4 and 5 now run as a Job `ichor-stack-<component>` in the controller's namespace, using the installer image and the hidden command `ichor stack apply kokumi|argocd` (`internal/stackapply`). It runs as its own ServiceAccount `ichor-stack` (`config/rbac/ichor-stack`, chart `rbac/ichor-stack.yaml`, always present, compared by `TestStackRBAC`; a test in `internal/stackapply` fails when a bundle holds a kind the role does not grant). The Job renders with `OmitPolicies`, applies Namespace, then the Secrets, then the rest, waits for the component's Deployments, and applies the component's CiliumNetworkPolicies last (unit 13). Policies are therefore applied per component, not once after both. The pod carries the label `app.kubernetes.io/name: ichor-stack`; `config/cilium-network-policy` and the chart allow it egress to `kube-apiserver:6443` only. It listens on nothing, so no ingress rule exists for it (the `[kube-apiserver, remote-node]` ingress rule of the ADR 0100 amendment stays on the controller's webhook port). The controller creates the Job (`jobs: get, create` in its namespace, already granted), treats a completed Job as step done, a failed Job as an error (delete the Job to retry), and does everything else: Secrets in its namespace, publishing, Orders, status. The Job is skipped once the component's Order exists. The generated values reach the Job through `secretKeyRef` env vars, so the Job needs no rights on the controller's namespace; user values for Argo CD go as `--values=<json>`. The controller's ClusterRole lost namespaces, CRDs, RBAC, admission policies, workloads, ConfigMaps/Services/ServiceAccounts and Secrets create/patch; it keeps `orders: get, create, patch` and `applications: get`. The Job's role is as strong as cluster-admin (it must create roles it does not itself hold) for the minutes it runs; that is the cost of the first apply and is why the identity is separate.

2. **No Secret is ever published.** Kokumi's registry is anonymous, so a Secret in a bundle is readable by any pod that can reach it. `stack.SplitSecrets` removes every `kind: Secret` from the rendered manifest; the controller hashes and pushes only what remains (a test unpacks the pushed layer and asserts it holds no Secret and neither generated value). The Secrets (`kokumi-server-auth`, `argocd-redis`, `argocd-secret`, `ichor-kokumi-registry`) are created by the Job directly, once, with `Create` and `AlreadyExists` ignored, so a rerun never resets them. They are not part of any bundle, so no Argo CD Application tracks them and an Application with prune enabled cannot prune what it does not track; as a second line they carry `argocd.argoproj.io/sync-options: Prune=false` and `compare-options: IgnoreExtraneous`. Kokumi's generated Application could not be read here (no Kokumi source in the tree; its install.yaml holds no Application), so the guarantee rests on Argo CD's tracking rule, not on the Application's settings: confirm on hardware that the Secrets survive the first sync.

## Amendment (2026-10-29): the unmanaged preset exists for adoption only

ADR 0103 adds `spec.cni.preset: unmanaged` (with `spec.cni.unmanaged.talosCNI`) so that `ichor adopt --cni-unmanaged` can bring in a cluster whose CNI Ichor leaves alone. Decision 8 is unchanged: Cilium is the only CNI Ichor installs or operates, `flannel` and `custom` stay refused, and `ichor init` has no flag for `unmanaged`. The webhook refuses `unmanaged` on create unless the object carries the adoption annotation `ichor.maesh.dev/adopted`. A foreign Cilium can also be taken over as this ADR's release through the same `TakeOwnership` path (ADR 0103, 1.2).

Amendment 2026-11-05 (ADR 0129): decision 4 said the webhook runs `ValidateValues`; it did not. The TalosCluster webhook now runs its static part (schema, unknown and owned keys, size cap) when the values, `kubeProxy` or `tunnel` change. The three guards that need the machine configuration or the live cluster are still the Connect service and the CLI only, and the controller does not yet re-check before a Job as this decision says; that re-check is open.

Amendment 2026-11-05 (ADR 0129, second amendment): the open re-check is closed. Before it creates a Cilium Helm Job (hand-over, values change, kube-proxy switch, chart upgrade) the controller runs `ValidateValues` with the same facts the UI reads (`internal/cnifacts`); on failure it starts no Job, sets `CNIValuesApplied=False` with reason `InvalidValues` and requeues.

Amendment 2026-11-05 (hardware lane H): the Job's ClusterRole `ichor-cni-role` (decision 2) had no rule for the objects the Cilium chart renders with `gatewayAPI.enabled=true`, so the upgrade failed on its first `get` of the GatewayClass `cilium` (`cannot get resource "gatewayclasses"`). It now holds `create`/`delete`/`get`/`list`/`patch`/`update` on `gatewayclasses` (`gateway.networking.k8s.io`) and on `validatingadmissionpolicies` and `validatingadmissionpolicybindings` (`admissionregistration.k8s.io`), and `bind`/`escalate` on the two Roles the chart then adds in `cilium-secrets` (`cilium-gateway-secrets`, `cilium-operator-gateway-secrets`; ADR 0108). The cilium-operator ClusterRole's extra Gateway API rules are covered by the existing `escalate` on that name. `TestCNIEscalationNamesOnlyCiliumsRoles` now renders the chart with the Gateway API on as well, and `TestCNIRoleCoversTheGatewayAPIObjects` fails when the role (in `config/rbac/ichor-cni` or the chart copy) lacks Helm's verbs on those kinds. Other optional chart features render kinds the role still lacks (`ingressController.enabled`: IngressClass, EndpointSlice and two Roles in `cilium-secrets`; `hubble.ui.enabled`: ClusterRole `hubble-ui`; `clustermesh.useAPIServer`: ClusterRole `clustermesh-apiserver`; `bgpControlPlane.enabled`: Role `cilium-bgp-control-plane-secrets`; `envoyConfig.enabled`: Role `cilium-envoy-config-secrets`); they are not granted until a user needs them.

Amendment 2026-11-05 (optional Cilium features): the five features the previous amendment left out are now granted, none of them is refused by `ValidateValues` (the owned keys are `ipam.mode`, the capability sets, `cgroup.*`, `k8sService*`, `kubeProxyReplacement` and `tunnelProtocol`). `ichor-cni-role` now holds `create`/`delete`/`get`/`list`/`patch`/`update` on `ingressclasses` (`networking.k8s.io`) and `endpointslices` (`discovery.k8s.io`) for `ingressController.enabled`, and `bind`/`escalate`, by name and never a wildcard, on the ClusterRoles `hubble-ui` (`hubble.ui.enabled`, which the chart only renders with `hubble.relay.enabled`) and `clustermesh-apiserver` (`clustermesh.useAPIServer`), and on the Roles `cilium-ingress-secrets` and `cilium-operator-ingress-secrets` (`ingressController.enabled`, in `cilium-secrets`), `cilium-bgp-control-plane-secrets` (`bgpControlPlane.enabled`, in `kube-system`), `cilium-envoy-config-secrets` (`envoyConfig.enabled`, in `cilium-secrets`) and `clustermesh-apiserver-generate-certs` (`clustermesh.useAPIServer` with the cronJob certificate method, in `kube-system`). The Deployments, Services, ServiceAccounts, ConfigMaps, Jobs, CronJobs and RoleBindings those features add were already covered. `clustermesh.useAPIServer` works only with `clustermesh.apiserver.tls.auto.method=cronJob`: the chart default, `helm`, renders Secrets (`cilium-ca`, `clustermesh-apiserver-*-cert`) in `kube-system`, which the Job cannot manage and must not be given (the same reason Ichor keeps `hubble.tls.auto.method=cronJob`). `TestCNIEscalationNamesOnlyCiliumsRoles` now renders every optional feature, and `TestCNIRoleCoversTheOptionalFeatureObjects` fails per feature when the role (in `config/rbac/ichor-cni` or the chart copy) lacks Helm's verbs on a kind the chart renders, or when the chart renders a kind the test does not map.

Amendment 2026-11-05 (clustermesh certificates): `ValidateValues` now refuses `clustermesh.useAPIServer=true` unless `clustermesh.apiserver.tls.auto.method=cronJob` (or `clustermesh.apiserver.tls.auto.enabled=false`, certificates supplied by the user), so the default `helm` method fails at the webhook and in the controller's pre-Job check with a message, not in the Job on a Secret it may not create.

Amendment 2026-11-05 (the Job cannot change its own confinement): `ichor-cni-role` held `create`/`delete`/`get`/`list`/`patch`/`update` on every `validatingadmissionpolicies`, `validatingadmissionpolicybindings`, `gatewayclasses`, `ingressclasses` and `endpointslices`, so the Cilium Job could update or delete the `ichor-controller-jobs` policies that confine it (ADR 0108). `get`, `update`, `patch` and `delete` on those five resources are now granted by `resourceNames`, exactly the names the pinned Cilium chart renders with every optional feature on: `gateway.cilium.io` (the policy and its binding), the GatewayClass `cilium`, the IngressClass `cilium` and the EndpointSlice `cilium-ingress` (a fixed name; a ClusterRole rule, not a namespaced Role, since the name is the limit). `create` and `list` cannot be limited by name in RBAC and stay, so the Job can still create a policy of another name; it can no longer touch `ichor-controller-jobs`, `ichor-controller-job-image` or their bindings. `TestCNIRoleChangesOnlyTheObjectsCiliumRenders` renders the chart with each optional feature, requires the names to equal what it renders (a chart bump that renames one fails there), requires the name-limited verbs to carry `resourceNames`, and refuses the four Ichor policy names, in `config/rbac/ichor-cni` and in the chart copy. `TestCNIRoleCoversTheGatewayAPIObjects` and `TestCNIRoleCoversTheOptionalFeatureObjects` still ask for Helm's verbs on each kind. Hardware verification (later): enable `gatewayAPI` and `ingressController`, then disable them, and see the Helm Job create, update and delete those objects without `forbidden`.

Amendment 2026-11-05 (ADR 0131): the user layer of the Cilium values can no longer be written by `ichor-operator`. Changing `spec.cni.cilium.values` needs `update` on `talosclusters/cilium-values`, which only `ichor-admin` holds, because the chart behind them can render privileged host-path pods that the Job's near-cluster-admin ServiceAccount creates (decision 2). The five `*.secretsNamespace.name` values (`ingressController`, `gatewayAPI`, `envoyConfig`, `bgpControlPlane`, `tls`) are Ichor-owned keys pinned to `cilium-secrets`, so the Roles named in the `bind`/`escalate` lists land only in Cilium's own namespace; BGP's chart default of `kube-system` moves there too, which changes where the `cilium-bgp-control-plane-secrets` Role of the previous amendment is created.

Amendment 2026-11-06 (hardware run I): `ValidateValues` gained two refusals and a text fix. (1) `gatewayAPI.enabled=true` is refused unless the kube-proxy mode is `replaced`: the Cilium 1.20 Gateway API prerequisites say "Cilium must be configured with the kube-proxy replacement, using `kubeProxyReplacement=true`" (docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api), and on hardware the operator logged "Gateway API support requires kube-proxy-replacement enabled" and never accepted the GatewayClass. The mode is known to the webhook, the UI and the controller alike (the controller passes the mode Cilium runs now, so a values edit made together with the switch waits for it). (2) After the schema, owned-key and guard checks pass and the user set something, the chart is rendered once with the effective values, so the chart's own `validate.yaml` (for example `cluster.id` without `cluster.name`) refuses at admission instead of failing the Helm Job again and again. Measured on the workstation (Intel N100): the check costs about 80 ms and 36 MB of allocation per call; the schema validation that already ran costs about 30 ms and 14 MB, nearly all of it Helm compiling the 1 MB JSON schema on every call (not cached, a possible later saving). Rendering is offline, so what the Job decides from the live cluster stays the Job's: `monitoring.coreos.com/v1` is declared present (the chart refuses a ServiceMonitor without that CRD) and values that make the chart look up a ConfigMap (`k8sServiceHostRef`) are not rendered. (3) The clustermesh refusal no longer ends mid-sentence ("...which Ichor's Helm Job may not manage"). Tests: `TestValidateValues` (gateway API while kube-proxy runs, cluster.id without cluster.name, a ServiceMonitor passing), the webhook table in `taloscluster_cilium_values_test.go`.

Amendment 2026-11-06 (hardware lane I, finding B2: the Job holds what Cilium's feature Roles grant; `escalate` by name does not apply to a create): on a real cluster the Cilium Helm Job could not create the objects the chart renders for the optional features: the Roles `cilium-gateway-secrets`, `cilium-operator-gateway-secrets`, `cilium-ingress-secrets`, `cilium-operator-ingress-secrets` and `cilium-envoy-config-secrets` (namespace `cilium-secrets`), `cilium-bgp-control-plane-secrets` and `clustermesh-apiserver-generate-certs` (`kube-system`), and the ClusterRoles `hubble-ui` and `clustermesh-apiserver`. The apiserver answered `attempting to grant RBAC permissions not currently held`, although `kubectl auth can-i escalate roles/<name>` said yes and the names were in the `escalate`/`bind` rules (ADR 0108). Cause, from the Kubernetes 1.36 source (`pkg/registry/rbac/escalation_check.go`): `RoleEscalationAuthorized`, called by the Role and ClusterRole storage on both create and update, asks the authorizer for `escalate` with `Name: requestInfo.Name`; on a create the request path has no name (it is in the body), so a rule with `resourceNames` never matches, and only an update (name in the path) is authorized that way. `BindingAuthorized` takes the name from `roleRef`, so `bind` by name does work on a create. The 2026-11-05 amendments and ADR 0108 assumed the name was honoured on create; the earlier hubble-generate-certs failure (ADR 0103 amendment 3) was the same cause.

Decision: the Job's ServiceAccount holds the rules those Roles and ClusterRoles grant, so the check passes without `escalate` (a held rule also authorizes the bindings, which need `bind` otherwise). Nothing is wildcarded and each grant is as narrow as the chart's own rule:

- ClusterRole `ichor-cni-role` (config `cni-role`) gains, read-only, exactly what `hubble-ui` and `clustermesh-apiserver` grant: `""` `componentstatuses`, `namespaces`, `nodes`, `services` `get`/`list`/`watch` (the new part is `componentstatuses`, `nodes` and `watch` on `namespaces` and `services`; the rest the Job already held); `apiextensions.k8s.io` `customresourcedefinitions` `get`/`list`/`watch`; `cilium.io` `ciliumendpoints`, `ciliumendpointslices`, `ciliumidentities`, `ciliumnodes` `get`/`list`/`watch`; `discovery.k8s.io` `endpointslices` `get`/`watch` (unnamed; `list` was held; the writes stay limited to `cilium-ingress`).
- Role `ichor-cni-features-role` (config `cni-features-role`, `config/rbac/ichor-cni/features_role.yaml`) with its RoleBinding, in `kube-system`: `secrets` `get`/`list`/`watch`, and `update` on `clustermesh-apiserver-server-cert`, `clustermesh-apiserver-admin-cert`, `clustermesh-apiserver-remote-cert`, `clustermesh-apiserver-local-cert`. With `cni-hubble-role` (create, `cilium-ca`) this covers `cilium-bgp-control-plane-secrets` and `clustermesh-apiserver-generate-certs`.
- Role `ichor-cni-secrets-role` (config `cni-secrets-role`) with its RoleBinding, in `cilium-secrets`: `secrets` `create`/`delete`/`get`/`list`/`patch`/`update`/`watch`, the union of the five Roles Cilium creates there. That namespace holds Gateway, Ingress and Envoy TLS Secrets and nothing of Ichor's.
- The `bind` and `escalate` names are cut to the base objects the bootstrap or the takeover already created and Helm only updates, which the Job does not hold (an update carries the name): ClusterRoles `cilium`, `cilium-operator`; Roles `cilium-config-agent`, `cilium-operator-ztunnel`. Dropped: `hubble-ui`, `clustermesh-apiserver` and the nine Roles above (held now), and `hubble-generate-certs`, `cilium-tlsinterception-secrets` and `cilium-operator-tlsinterception-secrets` (held already or now: the first by `cni-hubble-role`, the others by `cni-secrets-role`).

The widening, honestly: the Job gains read access to nodes, CRDs and the Cilium custom resources cluster-wide, `watch` on every Secret in `kube-system` (it already held `get` and `list` there), update of four named clustermesh Secrets, and every Secret verb on every Secret in `cilium-secrets`. This is on top of an identity that is already close to cluster-admin (create/update of Roles and ClusterRoles, `bind` and `escalate` on `cilium` and `cilium-operator`, and creation of Deployments and ServiceAccounts anywhere), so it adds little, and these features are admin-only to enable (ADR 0131). It is a real, bounded grant, not a cosmetic one, and an operator who never enables the features still carries it.

`cilium-secrets` exists only where Cilium runs, and a RoleBinding into a missing namespace fails an install. The installer therefore skips an object in `cilium-secrets` when the apply answers `not found` (`internal/installer`, test `TestRunSkipsObjectsInTheCiliumSecretsNamespaceWhereItDoesNotExist`; a missing namespace anywhere else still fails the run), and the chart has `cniHelm.ciliumSecretsRoles` (`auto` by default: render the pair when `helm install` or `helm upgrade` finds the namespace through `lookup`, which `helm template` and Argo CD cannot, so they leave it out and gateway, ingress and envoyConfig then fail with the same `forbidden`; `true` always, `false` never). The installer's own `bind`/`escalate` names list `ichor-cni-features-role` and `ichor-cni-secrets-role`, as `TestInstallerEscalationNamesEveryRoleOfTheInstallManifest` requires.

Enforcement: `TestCNIJobHoldsWhatTheCiliumRolesItCreatesGrant` (`dist/chart/cni_coverage_test.go`) replaces `TestCNIEscalationNamesOnlyCiliumsRoles` and `TestCNIRoleHoldsEveryRuleTheCiliumChartGrants`. It renders the Cilium chart with no optional feature (both kube-proxy modes, every tunnel) and with each of the six, and for every Role and ClusterRole it checks, in `config/rbac` and in the chart, that every rule is covered by the Job's own permissions with the apiserver's `covers` semantics (group, resource, verb and resourceName, a rule without names covering named ones, a ClusterRole judged against the cluster-wide bindings and a Role against those plus the RoleBindings of its namespace). An uncovered object must be a base object whose name is in the `bind` and `escalate` lists; an uncovered feature-only object fails, and so does a name in the lists that is covered anyway or not rendered as a base object. Red before the change: all nine feature objects above failed in both the config and the chart copy, and the lists held ten names the Job should hold. `TestCNIRoleChangesOnlyTheObjectsCiliumRenders` now allows the one unnamed `get` on `endpointslices`.

Known limit, not fixed here: the installer Job creates the Roles and ClusterRoles of a new release under its `bind`/`escalate` names, and the same rule means a Role or ClusterRole that is new in a release and grants rules the installer does not hold (these two, `ichor-cni-hubble-role`) cannot be created by it on an upgrade from a release that lacked it; a fresh `ichor init` is not affected. Hardware verification (later): on a Cilium cluster enable each of `gatewayAPI`, `ingressController` (NodePort), `hubble.relay`+`hubble.ui`, `bgpControlPlane`, `envoyConfig` and `clustermesh.useAPIServer` (cronJob method, `cluster.id`/`cluster.name`, `clustermesh.config.enabled`) as an admin, without the temporary ClusterRole of lane I, see the Helm Job finish `Applied` each time, and disable them again; and install on a flannel cluster to see the `cilium-secrets` pair skipped.

Amendment 2026-11-06 (correction: the installer is not affected by the create limit): the previous amendment's "known limit" said the installer Job could not create a Role or ClusterRole that is new in a release and grants more than it holds. That is wrong for the installer, and the paragraph is superseded by this one. The installer applies with server-side apply, which is a PATCH on the object's own URL: the apiserver handles it as an update with the name in the path (`requestInfo.Name` set, including when the apply creates the object), so `escalate` limited by `resourceNames` authorizes it. Only a plain POST has no name, which is what Helm sends, so the limit hits the Cilium Job and not the installer. `TestRunCreatesANewRoleThatGrantsMoreThanTheInstallerHolds` (`internal/installer/rbacfirst_test.go`) pins it under an enforced RBAC apiserver: an installer whose `bind`/`escalate` are by name only is refused a plain create of a Role with rules it does not hold (`attempting to grant RBAC permissions not currently held`), yet a run creates a new Role and a new ClusterRole of that kind through its apply, after it has widened its own role by name first (the existing stage 0 and 1 order, `TestRunUpgradesUnderAnOldInstallerRoleWithoutTheNewKindsRule`). The test passed before any installer change, so none was made: holding every rule of every Role in the install manifest, which was considered, would have given the installer Secret and node access it never needs. A future change of the installer to a plain create for Roles or ClusterRoles fails that test.

Amendment 2026-11-06 (hardware lane J, P1: `ichor init` lost a race against Cilium's bootstrap): `ichor init` failed on a fresh cluster, twice out of two, with `init: applying Ichor's own Role "ichor-cni-secrets-role": namespaces "cilium-secrets" not found`. The skip of the previous amendment is in `internal/installer` only; init applies the install manifest through its own path (`applyIchorManifest`), and at that moment the namespace does not exist yet: it is rendered by the Cilium chart into the bootstrap manifest that Talos applies from `cluster.inlineManifests`, which on hardware happened eight minutes after init's apply. A rerun is not possible (the node already took its configuration), so every fresh init was affected, and so was an `ichor adopt` of a cluster on another CNI.

Decision: `applyIchorManifest` takes whether Cilium is the cluster's CNI (init: always, `--cni cilium` is the only preset; adopt: the detected CNI). With Cilium, before it applies an object in `cilium-secrets` it creates that Namespace (name only, as a server-side apply by the installer's field manager, so it owns nothing but the name; an existing one is left as it is). Without Cilium it skips an object in `cilium-secrets` when the apply answers `not found`, and prints the same "skipping ... Cilium does not run here" line as the installer; a missing namespace anywhere else still fails. The Role and RoleBinding therefore exist from the first minute, and the Cilium features (Gateway API, Ingress, Envoy config, BGP) never meet a missing Role.

Why creating, not waiting or deferring: waiting for Cilium's bootstrap is a race with no bound (eight minutes here), and init has to finish before Cilium can be Ready; the controller cannot create the pair later, because it holds no Secret rights and no `bind`/`escalate` and must not be given them (the installer Job and the Cilium Job are the identities that hold them, ADR 0108); leaving the pair out until the next `ichor upgrade` makes every feature fail with `forbidden` until then. Who owns the namespace: the Cilium chart (`tls.secretsNamespace.create`, true by default, and ADR 0131 pins the five `*.secretsNamespace.name` values to `cilium-secrets`). Both orders work: if Talos applies the bootstrap manifest to the namespace init created, it gets the chart's labels and the Helm ownership annotations as any object updated in place; and the Helm install of the hand-over runs with `TakeOwnership` (`pkg/cni/cilium/helm.go`), so it adopts the namespace whether or not it carries them. The chart renders no more than name and labels for it, so there is nothing for the two to fight over. The installer's own apply stays idempotent: a cluster that still lacks the namespace when an upgrade runs skips the pair, and the next run that finds the namespace applies it (test `TestRunAppliesTheCiliumSecretsRoleOnceTheNamespaceExists`, which also keeps the envtest namespace-free for the skip test). The Helm chart is unchanged: `cniHelm.ciliumSecretsRoles: auto` renders the pair when `lookup` finds the namespace (a cluster whose Cilium already runs), `true` always, `false` never; a flannel or other cluster gets neither the namespace nor the pair from either path.

Tests: `TestApplyIchorManifest_CreatesCiliumSecretsForCilium` (red: the exact error above; a fake API server that answers `not found` for an apply into a missing namespace), `TestApplyIchorManifest_SkipsCiliumSecretsWithoutCilium` (red, same error), `TestApplyIchorManifest_MissingNamespaceElsewhereStillFails`.

Hardware verification (later): a fresh `ichor init --cni cilium` on a node in maintenance mode completes the "Install Ichor" step with no `cilium-secrets` error; once Cilium's bootstrap has run, `kubectl -n cilium-secrets get role ichor-cni-secrets-role,cilium-tlsinterception-secrets` and the RoleBinding exist, the namespace carries Cilium's labels (Talos did update, not refuse, the pre-existing object), `helm`-side hand-over reports `Adopted`, and enabling `gatewayAPI` as an admin (ADR 0131) applies in about 20 seconds.