0099. Argo CD and Kokumi become Ichor's standard stack; Cilium is the only CNI and Ichor's own Helm release

Status: Accepted; superseded in part by ADR 0101 · Date: 2026-10-29 · Design section: Bootstrap, GitOps and Argo CD, CNI · Supersedes in part: 0013, 0019, 0024, 0028, 0048, 0055

Decided: ichor init always installs Cilium, Kokumi and Argo CD with no opt-out; Cilium is the only CNI, bootstrapped through Talos inline manifests and then owned by Ichor as a Helm release run by an installer-style Job with user values stored on the TalosCluster; Kokumi and Argo CD are applied once, then managed by Argo CD from artifacts the Ichor controller publishes into Kokumi's in-cluster registry, so no Git repository is needed

Context

Four things in the repo no longer fit what the user now wants.

1. Argo CD and Kokumi are optional and Ichor installs neither (ADR 0019, 0048). --argocd, --kokumi, --repo, --gitops-dir and gitops export exist to serve that option.
2. The CNI is a choice (flannel, cilium, custom, ADR 0013), and Cilium is handed to Argo CD after bootstrap (ADR 0024, 0055). Users now need to edit Cilium's Helm values from the Ichor UI, which an Argo CD-owned release in a Git repository cannot offer.
3. Kokumi is built only as a repo-creds Secret (ADR 0048). Its install, its storage and its real data model were never checked against a running release.
4. docs/design/architecture.md says Ichor "bootstraps CNI and Argo CD once and hands them over". That sentence stays true for Argo CD and Kokumi. For Cilium it is replaced here.

User decisions this ADR implements (2026-10-29): Argo CD and Kokumi are the standard stack, always installed; Argo CD syncs from Kokumi's in-cluster OCI registry; Cilium is the only CNI and existing flannel/custom clusters may be deleted; Argo CD manages Cilium's neighbours (itself and Kokumi) after a one-time bootstrap, while Cilium stays Ichor-managed and UI-editable; Ichor publishes artifacts into Kokumi so a later "addons" UI can edit component values; tracking detection of Ichor's own CRs by a user's GitOps stays; Cilium-aware NetworkPolicies for the Kokumi and Argo CD namespaces are applied after init.

Research

All versions below were read on 2026-10-29. The Kokumi release was downloaded and its checksum matched ADR 0048 (install.yaml sha256 345223726800ca3f08f633c76c91a38265bc0c131551b24dc49c69ee0cd7b840). Its source tarball was read for the docs that ship in docs/content (the kokumi.dev pages are older than release 0.17.0: they describe Product, Registry and Repository kinds that 0.17.0 does not contain; the source tree is authoritative).

Kokumi

• Latest stable: 0.17.0 (2026-10-11), https://github.com/kokumi-dev/kokumi/releases. Apache-2.0, "experimental", API delivery.kokumi.dev/v1alpha1. Context7 has no Kokumi entry (the only hit was an unrelated library), so kokumi.dev and the source were used.
• Install: only kubectl apply -f .../releases/download/0.17.0/install.yaml (https://kokumi.dev/docs/installation/). There is no Helm chart and no OCI install artifact. The file is 134,389 bytes (14.9 KB gzipped). It holds one Namespace (kokumi), 8 CRDs (Approval, Kitchen, Menu, Order, Pantry, Preparation, Recipe, Serving), about 40 RBAC objects, 3 Deployments, 3 Services, 1 PVC, 1 ConfigMap, 1 Secret and 2 ValidatingAdmissionPolicies. Requires Kubernetes >= 1.26 and Argo CD >= 3.3.
• Images (exactly two): ghcr.io/kokumi-dev/kokumi:0.17.0 (controller manager and API server/UI) and registry:3.1.2@sha256:ddf75434... (bundled registry). No scanner is in 0.17.0.
• Workloads and resources set in the manifest: controller manager (requests 10m/64Mi, limit 500m/128Mi), server (same), registry (request 100m/256Mi, limit 256Mi). Sum of requests 120m/384Mi, sum of limits 1.1 CPU/512Mi.
• Registry storage: PVC kokumi-registry-pvc, 1Gi, ReadWriteOnce, no storageClassName. A Talos cluster has no StorageClass unless the user adds one (Ichor installs no storage), so the registry pod would stay Pending. The registry Service is type: NodePort (30050) and the registry accepts anonymous reads and writes over HTTP; the server Secret kokumi-server-auth ships admin/admin. All three need handling (decisions 6 and 7).
• Data model (source docs/content/docs/architecture.md, getting-started.md at 0.17.0): an Order takes a source OCI artifact, renders it, and pushes the result as an immutable Preparation to a destination registry (default in-cluster, oci://kokumi-registry.kokumi.svc.cluster.local:5000/preparation/<name>). A Serving (one per Order, created automatically) points at one Preparation and creates the Argo CD Application. promotion.mode is Automatic (newest Ready Preparation) or Manual. A Menu is a template with override policies; a Pantry carries registry URL and credentials; a Recipe is a render profile.
• Source formats: with no spec.render, the source artifact must hold *.yaml files at its root, stored as-is ("pre-rendered manifest bundle"). With spec.render.helm it must be a Helm chart. In code (internal/oci/oras_client.go) the accepted single-layer media types are application/vnd.cncf.helm.chart.content.v1.tar+gzip and application/vnd.cncf.flux.content.v1.tar+gzip (a gzipped tar of the files).
• Publishing: there is no Kokumi API to publish a source. Ichor pushes an OCI artifact to the registry itself (oras-go or go-containerregistry are both fine; the registry is the CNCF registry image, plain HTTP, no auth) and creates an Order whose spec.source.oci and version name it. Ichor never needs a Product/Menu.
• The Application Kokumi creates (internal/deployer/argocd.go): name = Order name, namespace argocd, project: default, source {repoURL: oci://<registry>/preparation/<name>, targetRevision: <digest>, path: "."}, destination {server: https://kubernetes.default.svc, namespace: <Order namespace>}, syncPolicy.automated {prune, selfHeal} and ServerSideApply=true, annotation delivery.kokumi.dev/allowed-order. It refuses to update a pre-existing Application without that annotation. Consequence: an Order in namespace X named N produces Application N that deploys into namespace X; Ichor creates the Order argocd in namespace argocd and the Order kokumi in namespace kokumi.
• Argo CD reads that artifact as an OCI source: one layer, media type application/vnd.oci.image.layer.v1.tar+gzip or the Helm chart type by default, path selects inside the image (https://argo-cd.readthedocs.io/en/stable/user-guide/oci/). The docs there do not mention insecureOCIForceHttp; Kokumi's own Getting Started does, via a repo-creds Secret (type: oci, url: oci://kokumi-registry.kokumi.svc.cluster.local:5000, insecureOCIForceHttp: "true"), which ADR 0048 already builds.

Argo CD

• Latest stable 3.x: v3.5.6 (2026-10-20). Also current: v3.4.12, v3.3.18; v3.6.0-rc4 is a prerelease. Kokumi 0.17.0 requires >= 3.3 and its own Getting Started installs v3.5.1; ADR 0019 and 0055 verified shapes against v3.5.x (ADR 0055's e2e ran v3.5.6). Pin 3.5.6.
• Install forms: manifests/install.yaml (1,917,766 bytes, 106 KB gzipped; images quay.io/argoproj/argocd:v3.5.6, redis:8.2.3-alpine, dex:v2.45.1; no resource requests set) or core-install.yaml (no API server/UI/dex, and no default AppProject, which made ADR 0055's e2e fail with InvalidSpecError until one was created; Kokumi's Applications use project: default). Or the Helm chart argo-cd 10.11.6 (appVersion v3.5.6, https://github.com/argoproj/argo-helm, 237,904 bytes, kubeVersion >=1.25, dependency redis-ha that is off by default). Pinned manifests give no switches for dex, notifications and the ApplicationSet controller; the chart does.
• Self-management: "Argo CD is able to manage itself since all settings are represented by Kubernetes manifests", and it must use ServerSideApply=true (https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/). Kokumi's Applications already set it.
• Not answered by the docs read: a size limit for an OCI source, the version in which OCI sources left beta, and whether insecureOCIForceHttp has a documented argocd-cm home. All three are checked in the e2e (work unit 14).

Talos and Cilium (https://docs.siderolabs.com/kubernetes-guides/cni/deploying-cilium, Talos v1.14 docs)

Prescribed and already matched by pkg/cni/cilium/values.go: ipam.mode=kubernetes; kubeProxyReplacement=false|true; securityContext.capabilities.ciliumAgent (CHOWN, KILL, NET_ADMIN, NET_RAW, IPC_LOCK, SYS_ADMIN, SYS_RESOURCE, DAC_OVERRIDE, FOWNER, SETGID, SETUID; no SYS_MODULE) and cleanCiliumState (NET_ADMIN, SYS_ADMIN, SYS_RESOURCE); cgroup.autoMount.enabled=false, cgroup.hostRoot=/sys/fs/cgroup; k8sServiceHost=localhost, k8sServicePort=7445 for the kube-proxy-free case; KubePrism enabled on 7445.

Where it matters for this ADR:

• "Talos only creates missing resources from inline manifests -- it never deletes or updates them." To change them, "edit all control plane machine configurations and run talosctl upgrade-k8s." So an inline manifest cannot carry a values change at runtime. Ichor-managed Cilium cannot stay in the inline manifest; this decides option (b) below. (ADR 0024 recorded, from Talos v1.14.3 source, that manifest_apply.go server-side-applies on every Kubernetes upgrade. The guide's wording is "creates only". The two readings agree that nothing is pruned; the difference only affects whether a stale inline entry can roll Cilium back. The hand-over below removes the entry either way.)
• v1.14 and later: use a KubeFlannelCNIConfig $patch: delete document to drop flannel (Ichor does: pkg/render/networkinstall.go), KubeProxyConfig enabled: false instead of cluster.proxy.disabled, and the inline-manifest document instead of the deprecated cluster.inlineManifests. Preset.TalosPatches still emits the legacy cluster.proxy.disabled and render strips it; this ADR cleans that up. The exact kind name of the v1.14 inline-manifest document (KubeInlineManifestConfig) is taken from the audit of the same guide; work unit 2 verifies it against the pinned machinery before use.
• The guide recommends inline manifests for production, only on control planes, identical on all of them, and warns that without a CNI the node waits "about 10 minutes" before retrying. Known issue: bpf.masquerade with forwardKubeDNSToHost can break CoreDNS; forwardKubeDNSToHost=false fixes it. Gateway API needs its CRDs installed first, then gatewayAPI.enabled, enableAlpn, enableAppProtocol.
• Differences from today's approach: (1) we use the legacy cluster.inlineManifests and cluster.proxy.disabled on a v1.14 contract (fixed here); (2) we remove the inline entry after hand-over, which the guide does not discuss and which ADR 0024 verified (kept, retargeted); (3) the guide's hosted/CLI/manifest-apply Helm variants are not used.
• The embedded Cilium 1.18.0 chart ships values.schema.json (146,865 bytes). It has no additionalProperties: false, so a misspelled key passes the schema. Helm already validates against the schema on render, so the existing render path gives type checking for free; an unknown-key check against the chart's own values.yaml is added (decision 4).

Footprint (estimates; nothing was measured on a cluster)

Requests/limits come from the manifests above. Idle working-set figures for Argo CD are typical numbers for a small install and are the main uncertainty; unit 14 measures them.

On the 1 control plane (3Gi) + 1 worker (2.5Gi) homelab, Talos and the control-plane components use about 1.2-1.5Gi of the control plane, and Cilium and Ichor add roughly 0.4-0.6Gi across the nodes. With a tainted control plane the stack lands on the 2.5Gi worker: it fits, with little slack. A minimal Argo CD (Helm chart with dex.enabled=false, notifications.enabled=false, applicationSet.enabled=false, redis-ha.enabled=false, no controller replicas beyond one) removes dex, notifications and the ApplicationSet controller, about 150-250Mi and 3 pods. It is the default (decision 5). The raw install.yaml cannot do that; that is why the chart is embedded instead.

Decisions

1. Always installed, no opt-out

ichor init installs Cilium, Kokumi and Argo CD every time. Flags --cni, --argocd, --argocd-namespace, --kokumi, --repo, --path, --repo-username, --repo-password and --gitops-dir go away (--cni stays only as a hidden flag that accepts cilium for one release so scripts get an explicit message; --cni-tunnel stays). The ichor-system namespace and Ichor itself are applied as today (ADR 0019). Argo CD, Kokumi and Cilium are Ichor's three bootstrap components.

2. Cilium: option (b), a Helm release owned by Ichor

Option (a), keep Cilium in cluster.inlineManifests and change values by editing it, is rejected on the Talos guide's own text: Talos creates inline manifests and never updates or deletes them, so a values change would apply only through talosctl upgrade-k8s, with no prune. Option (b) is chosen.

• Bootstrap stays the inline manifest, rendered from the embedded chart plus the effective values (ADR 0098), delivered as the v1.14 inline-manifest document on a v1.14 contract (gated by MultidocKubernetesConfigSupported(), as networkinstall.go already gates its flannel choice) and as cluster.inlineManifests below it. Only control planes carry it, identical on all, as the guide requires.
• Adoption. After the control plane is Ready and Cilium's DaemonSet is Ready, a Helm release cilium in kube-system is created over the bootstrapped objects with the Helm SDK: action.Install with TakeOwnership: true (field present in helm.sh/helm/v3 v3.21.4, pkg/action/install.go:114, the version in go.mod) and the same chart, release name, namespace and values as the bootstrap render. The result has no object changes; it only records release revision 1 as Secret sh.helm.release.v1.cilium.v1. The bootstrap objects are nevertheless rendered with Helm's release-ownership metadata (annotations meta.helm.sh/release-name: cilium and meta.helm.sh/release-namespace: kube-system, label app.kubernetes.io/managed-by: Helm), so that the release is created over objects that already look like its own and helm-aware tooling reads them as members. TakeOwnership stays: it is what makes the adoption not depend on that metadata (a cluster bootstrapped by an older build has none). This changes the bootstrap bytes once (see unit 7).
• Operation. Later changes are action.Upgrade with the new effective values: Helm's three-way merge updates changed objects and deletes objects the chart no longer renders (prune), with Atomic/rollback on failure so a bad value set does not leave a half-upgraded CNI. lookup works (Helm talks to the cluster), so the Hubble certificate reasoning in ADR 0055 (cronJob instead of helm) no longer forces a choice; the existing cronJob value is kept to avoid changing running clusters, and users may override it.
• Who runs Helm. The Helm SDK needs the permissions of everything the chart creates (ClusterRoles, CRDs, DaemonSet, ...), which is cluster-admin in practice (ADR 0059 reached the same conclusion for the installer: escalate/bind cannot be narrowed). So the controller does not run Helm. It launches a one-shot Job, exactly as the Ichor installer does: image ghcr.io/maeshinshin/ichor-installer:<version> (reused; same image, new hidden subcommand ichor cni helm apply), its own ServiceAccount ichor-cni, created per run, deleted after, with ttlSecondsAfterFinished. The Job uses hostNetwork: true and KUBERNETES_SERVICE_HOST=localhost, KUBERNETES_SERVICE_PORT=7445 (KubePrism) so it reaches the API server while Cilium restarts; the replaced-mode requirement for KubePrism already makes that port a precondition.
• Hand-over retargeted. CNIHandoverReconciler (ADR 0024) keeps its shape, condition CNIHandover and render-time removal of the inline entry. Its signal changes from "Argo CD reports an Application Synced and Healthy" to "Helm release cilium is deployed at the effective-values hash and the DaemonSet is ready", read from the release Secret and the DaemonSet. spec.cni.handover (automatic|manual|disabled) is kept as a deprecated field; manual and disabled are read as automatic (decision 8). The upgrade guard cniHandoverPending keeps blocking a Kubernetes upgrade until the entry is gone.
• Why not a controller-embedded Helm run or a CRD of Argo CD. Embedding means giving the long-lived controller cluster-admin. An Argo CD Application would hand Cilium to Git or Kokumi, which the user decided against.

3. Cilium values: stored on the TalosCluster, merged by Ichor

New field spec.cni.cilium.values on TalosClusterCNI (an object with x-kubernetes-preserve-unknown-fields, append-only, immutable only in its owned subset, below). The effective values are, in order: Ichor defaults (pkg/cni/cilium/values.go), then the user's spec.cni.cilium.values deep-merged on top, then the Ichor-owned keys applied last. The UI shows the effective values read-only and the user values as the editable layer.

Ichor-owned keys (cannot be set by users; a value for one is a validation error naming the key and the reason, not silently ignored): ipam.mode, securityContext.capabilities.ciliumAgent, securityContext.capabilities.cleanCiliumState, cgroup.autoMount.enabled, cgroup.hostRoot (the Talos guide's requirements), k8sServiceHost and k8sServicePort (KubePrism), kubeProxyReplacement (owned by the kube-proxy switch, spec.cni.kubeProxy) and tunnelProtocol (owned by spec.cni.tunnel, immutable).

4. Validation

One function, cilium.ValidateValues(values, context), used by the admission webhook (internal/webhook/v1alpha1/taloscluster_webhook.go), the Connect service behind the UI and CLI, and the controller before it launches the Job:

1. Render the embedded chart with the merged values (no cluster, no network). Helm's schema validation against the chart's values.schema.json yields type errors with paths. A failed render is an error with Helm's message.
2. Reject any Ichor-owned key.
3. Reject or warn on keys absent from the chart's values.yaml tree (the schema does not catch typos). Free-form maps in the chart (extraArgs, annotations, ...) are exempt.
4. Three semantic guards from the Talos guide: bpf.masquerade=true is refused unless the cluster's effective machine config has machine.features.hostDNS.forwardKubeDNSToHost: false (CoreDNS breaks otherwise; Ichor does not change the Talos side on its own in this ADR); gatewayAPI.enabled=true is refused unless the Gateway API CRDs exist (checked live by the controller, which sets a condition; the webhook cannot check and passes); and in replaced mode a patch that disables KubePrism or moves it off 7445 is refused. The context argument carries what each caller can know; the controller re-checks live before every Job.
5. Size: the merged values must stay under 64 KiB (they are stored on the CR and rendered into the Job).

5. Kokumi and Argo CD: applied once, then Argo CD owns them

Bootstrap order, from a fresh ichor init:

1. Talos config with the Cilium inline manifest is applied; bootstrap etcd; kubeconfig; Secrets bundle stored (unchanged, ADR 0019 steps 1-4).
2. Cilium becomes Ready (the existing wait). Ichor is applied through the API server (unchanged).
3. init creates IchorInstallation with a new spec.stack (below) and waits. The controller (not the CLI) does 4-8 so there is one implementation for init and for later upgrades.
4. The controller applies the embedded Kokumi manifest (server-side apply, field manager ichor-bootstrap) with the transformations in decision 6, then waits for the registry Deployment to be Ready.
5. It renders the Argo CD chart with Helm (release argocd, namespace argocd, values below), applies it the same way, and waits for the server and repo-server.
6. It creates the Argo CD repo-creds Secret (type: oci, the registry URL, insecureOCIForceHttp: "true"; no credential in it) and the default AppProject if absent. These also ship in the Argo CD bundle so Argo CD keeps them.
7. For each of kokumi and argocd it publishes a pre-rendered manifest bundle to the registry (decision 7) and creates one Order (delivery.kokumi.dev/v1alpha1, promotion.mode: Automatic, no spec.render). Kokumi renders a Preparation from it, its Serving creates Application kokumi / argocd, and Argo CD adopts the live objects. Because the bundle contains exactly the bytes Ichor applied in 4-5, the first sync shows no diff (the same "adoption produces no diff" rule as ADR 0024). Pre-rendered bundles, not Helm Orders, are used deliberately: the bytes come from Ichor's own Helm SDK and embedded chart, not from the helm template inside Kokumi, whose version and default KubeVersion Ichor does not control.
8. The policies of decision 10 are applied last, once both namespaces and workloads exist.
9. IchorInstallation.status.stack reports per component: published digest, Order ready, Application Synced/Healthy. When both are Healthy, init finishes. This is the stack's hand-over; there is nothing to remove afterwards because nothing was inlined.

Argo CD configuration shipped by Ichor (chart values; user-overridable except where noted): dex.enabled=false, notifications.enabled=false, applicationSet.enabled=false, redis-ha.enabled=false, configs.params."server.insecure" unset, configs.cm."application.resourceTrackingMethod" left at Argo CD's default (Ichor reads it, ADR 0044), crds.keep=true. Ichor-owned keys, as for Cilium: the repo-creds Secret, the default AppProject, and the argocd-cm resource customizations for Ichor's kinds. The Lua health checks of ADR 0046 move into the Argo CD bundle's argocd-cm, which resolves that ADR's limitation (Ichor did not own argocd-cm; it now does). Argo CD has no external Git repository, so repositories is empty.

6. Where the manifests come from and their size

Embedded in the ichor binary and in the installer image, like Cilium (ADR 0098), each with a make fetch target that refuses a file whose SHA-256 differs from the recorded digest:

Pinned artifacts as embedded by pkg/stack (unit 4); make argocd-chart and make kokumi-install fetch them and refuse a file whose SHA-256 differs from the Makefile:

• Argo CD chart: https://github.com/argoproj/argo-helm/releases/download/argo-cd-10.11.6/argo-cd-10.11.6.tgz, sha256 d8f096d87513b41ff754053a7f9604c943bf29c15436a79226fd51595ea67d05, 237,904 bytes, in pkg/stack/argocd/chart/.
• Kokumi: https://github.com/kokumi-dev/kokumi/releases/download/0.17.0/install.yaml, sha256 345223726800ca3f08f633c76c91a38265bc0c131551b24dc49c69ee0cd7b840, 134,389 bytes raw, stored as pkg/stack/kokumi/install.yaml.gz (14,611 bytes, gzip -9 -n).
• Correction found while implementing: chart 10.11.6 has no applicationSet.enabled value, so pkg/stack/argocd drops the argocd-applicationset templates after rendering (the ApplicationSet CRD stays). redis-ha.enabled=false only works once Helm's dependency conditions are processed, which the renderer does. The chart's redis-secret-init hook Job is dropped with all hooks, so the renderer emits the argocd-redis Secret itself from a caller-supplied password.
• Bundles are pushed with go-containerregistry (already in go.mod, previously indirect): one layer application/vnd.cncf.flux.content.v1.tar+gzip holding manifest.yaml, config type application/vnd.cncf.flux.config.v1+json, over plain HTTP.

About 0.4 MB added. The rendered Argo CD bundle is about 1.9 MB raw (the three CRDs dominate), about 0.1 MB as a gzipped tar. That is above what fits comfortably in a machine config, which is why Argo CD and Kokumi are applied through the API server and not through Talos inline manifests. Images are pulled from ghcr.io, quay.io, public.ecr.aws and docker.io; air-gapped installs are out of scope (ADR 0090), so nothing is mirrored here.

Transformations applied to the Kokumi manifest before it is applied and published, each a small, tested, deterministic step in pkg/stack/kokumi: the registry Service becomes ClusterIP (the shipped NodePort 30050 would expose an anonymous, writable registry on every node); the registry volume becomes an emptyDir with a size limit unless the cluster has a default StorageClass (see Risks: artifacts are rebuilt from the Ichor controller on registry loss); the controller and server memory limits are raised to 256Mi; the kokumi-server-auth Secret's admin/admin is replaced by a generated password kept in a Secret ichor-kokumi-admin in ichor-system and shown once by ichor init (a login through Ichor's OIDC issuer, ADR 0094, is future work: Kokumi supports OIDC through its Kitchen resource).

7. Publishing into Kokumi, and the data flow for the future "addons" UI

New package pkg/stack and a controller section in the IchorInstallation reconciler.

• State. IchorInstallation.spec.stack (new, append-only): argocd.values and kokumi.values (free-form objects, user layer only), promotion (Automatic default, or Manual so a human promotes in Kokumi's UI). status.stack.<component> holds publishedTag, digest, orderReady, applicationHealth. This singleton already exists, is cluster-scoped and is the object Ichor upgrades; the addons UI later writes spec.stack.<component>.values through a service the same way it writes Cilium's values, and a future third component adds a key, not a new mechanism. Kokumi has no chart, so its values initially accept only the keys Ichor maps to patches it controls (memory limits, replica counts); this is stated in the field docs.
• Render. Effective values = Ichor defaults, user values, owned keys (decision 3 rules, same validation function with a different chart and owned set). Rendered with the Helm SDK to a single manifest.yaml.
• Publish. A deterministic tar (sorted entries, zero mtimes, fixed owner) gzipped, pushed as one layer of media type application/vnd.cncf.flux.content.v1.tar+gzip to kokumi-registry.kokumi.svc.cluster.local:5000/ichor/<component>:<chartVersion>-<hash12>, where the hash covers the rendered bytes, with org.opencontainers.image.source and .version annotations. The same input always gives the same digest, so a republish after a registry restart restores the same references.
• Order. delivery.kokumi.dev/v1alpha1 Order argocd (namespace argocd) and kokumi (namespace kokumi), spec.source.oci = the repository, version = the tag, promotion.mode from spec.stack.promotion, default destination. Changing a value moves version, Kokumi renders a Preparation, Automatic promotion moves the Serving, Argo CD syncs. History, approvals and rollback are Kokumi's.
• Reach. The controller pushes from inside the cluster; the registry has no authentication, so the policy in decision 10 is what restricts writers. The controller's RBAC gains orders in the two namespaces and get/list on servings/preparations; it never writes Argo CD objects.
• Idempotence and loss. The reconcile compares the Order's version and the registry's manifest digest each time; if the tag is gone (registry rebuilt) it republishes and, if Kokumi does not re-render by itself, bumps an annotation on the Order (unit 6; whether Kokumi re-renders a Preparation whose artifact vanished is an open question).

8. Cilium is the only CNI; flannel and custom are removed

User decision: existing flannel/custom clusters may be deleted, so no code path keeps them running.

• New clusters: preset defaults to cilium; the webhook rejects creation with any other value.
• Stored objects (append-only CRD). The enum flannel;cilium;custom, the custom field and its types stay in the schema, marked deprecated in the Go doc comments and in the generated CRD description. Removing an enum value does not trip the installer's additive check (internal/installer/crdcheck.go compares served versions, property names, property types, items and additionalProperties; it does not read enum), but removing the custom property or its sub-properties would fail with ... would be removed (line 83), and any removal would also make stored preset: flannel objects fail validation on their next update. So nothing is removed from the schema, and the enum keeps all three values. The transition rule self.preset == oldSelf.preset stays.
• A stored preset: flannel|custom is rejected on update, not ignored. The webhook refuses any change to such a cluster's spec (finalizers, labels, annotations and deletion are allowed) with a message saying the preset was removed in this release and the cluster should be deleted and recreated. Ignoring it would be worse: render would now always strip Talos's flannel document, so the next config apply to a flannel cluster would take its network down. The controllers act the same way: TalosMachineReconciler and the renderer refuse to render a cluster whose preset is not cilium, set a CNIUnsupported condition on the TalosCluster and make no change to nodes. Nothing else in the code handles flannel or custom.
• spec.cni.custom, spec.cni.handover=manual|disabled: stored values are read but have no effect; the webhook rejects setting them to anything but absent / automatic on create. spec.cni.kubeProxy and spec.cni.tunnel stay and work as before.
• Code removed: see "What goes away".

9. Kube-proxy replacement now changes the Cilium release itself

spec.cni.kubeProxy is already an Ichor-owned input (kubeProxyReplacement, k8sServiceHost, k8sServicePort derive from it). The switch executor (internal/controller/cni_kubeproxy_switch.go) currently prints the values the operator must set in Cilium's config and waits for the operator to do it (ciliumChangeInstructions). With Ichor owning the release, the CNI stage launches the Helm Job with the new mode, then runs the existing verification (cilium-config key and DaemonSet rollout), and only then touches the Talos side (KubeProxyConfig, v1.14). The order, the confirmation annotation and the "turn the old side off only after the new side is verified" rule are unchanged. The ichor-cni Job is the same one used for a values edit, so a kube-proxy switch and a user edit cannot overlap: the Job takes the existing cluster operation lock.

10. NetworkPolicies for the Kokumi and Argo CD namespaces (a separate work unit)

Cilium is up before either namespace exists, so the policies are CiliumNetworkPolicies shipped inside each component's bundle, applied by the controller as the last step of the bootstrap (step 8) and re-applied by Argo CD on every upgrade. The ichor-system policies are another agent's work and are not designed here. Each of the kokumi and argocd namespaces gets a default-deny (ingress and egress) plus explicit allows:

• DNS: egress to kube-system k8s-app=kube-dns on 53/UDP and 53/TCP, from every pod, with a dns rule so the allowed names are visible.
• Argo CD to kube-apiserver: egress toEntities: [kube-apiserver] (Cilium's entity, so it works with KubePrism and with the node-local endpoint) from the application-controller, server, repo-server (for the one cluster call it makes).
• Argo CD to the Kokumi registry: egress from the repo-server (and the application-controller, which resolves revisions) to pods app=registry in kokumi on 5000/TCP; the matching ingress on the registry side.
• Ichor publishing into Kokumi: ingress on the registry (5000) and on kokumi to the Ichor controller pod in ichor-system (namespace and pod labels), egress from ichor-system is the other agent's unit; this unit only provides the receiving rules and names the dependency.
• Kokumi's own needs: controller-manager and server to kube-apiserver; controller-manager to the registry (render and push) and to nothing external, since sources are in-cluster; the server's own port 8080/80 ingress from host entity only (for ichor ui-style port-forward, which arrives from the node); Argo CD ingress for its server from host and, for Ichor's UI reading status, none (it reads CRs through the API server).
• Redis inside argocd: ingress only from the application-controller, server and repo-server.
• Intra-namespace health probes: kubelet probes come from the host entity and are allowed explicitly.
• Applied "once their namespaces exist": the controller applies them after the workloads' namespaces and Deployments exist, never before, so a default-deny cannot cut off the bootstrap sync.

11. Upgrades after hand-over

• Cilium: the pinned chart changes with the Ichor release. ichor upgrade ichor does not upgrade Cilium on its own; it makes the new chart available (status.cni.availableChartVersion, condition CiliumUpgradeAvailable). The user starts it (ichor cni upgrade, or the UI button) as an operation of the existing kind cni, which validates the user's values against the new chart's schema first and fails early if they no longer validate, then runs the Helm Job. Cilium supports upgrading one minor version at a time, so each Ichor release changes the pinned Cilium by at most one minor. Rollback is Helm's.
• Argo CD and Kokumi: the new controller (after ichor upgrade ichor, ADR 0059) recomputes the bundles from its embedded charts and the stored spec.stack values; if the tag differs from the Order's version, it publishes and updates the Order. With promotion: Automatic Argo CD applies the new Preparation; with Manual the new Preparation waits in Kokumi's UI. Argo CD upgrades itself this way (the Application managing Argo CD deploys the new manifests; ServerSideApply=true handles the CRD growth), and Kokumi upgrades itself the same way. Since both are Argo CD's after this point, Ichor never runs kubectl apply on them again; the first-run apply (decision 5, steps 4-5) happens only when the namespace has no Order of that name.
• A user who edits the Orders or the bundle in Kokumi by hand will see Ichor overwrite the Order's version on the next reconcile. The Order is Ichor-owned (label ichor.maesh.dev/managed); the values are the supported way to change anything.

12. Git tracking detection stays

pkg/gitopstracking, ADR 0044 and ADR 0051 are unchanged: the UI and CLI still refuse writes to CRs a user's own Argo CD or Flux tracks, the spec.gitops repository connection and PR mode (ADR 0056) stay, and Ichor still reads argocd-cm rather than guessing the tracking method. That argocd-cm is now normally Ichor's own, but a user may run another Argo CD; the code does not care. TalosCluster.spec.gitops.mode: strict keeps meaning "the user's Git is the source". ichor.maesh.dev CRs are not placed in Ichor's own Argo CD; Ichor's own Applications track only the Kokumi and Argo CD bundles, so Ichor's own Argo CD never tracks an Ichor CR and never triggers the warning.

What goes away

Removed (the files are named so each can be deleted by a work unit, with tests):

• internal/cli/init_gitops.go, init_gitops_test.go (initArgoCD, buildRootApplication, buildRepoSecret, buildSkeleton).
• internal/cli/kokumi.go, kokumi_test.go (the repo-creds helper moves to pkg/stack).
• internal/cli/gitops.go, gitops_export.go, gitops_export_test.go, gitops_handoff.go, gitops_handoff_test.go (the ichor gitops export|handoff commands), and exportCNIApplication.
• internal/cli/init.go flags listed in decision 1 and initOptions fields ArgoCD, ArgoCDNamespace, Kokumi, RepoURL, RepoPath, RepoUsername, RepoPassword, GitOpsDir; init_plan.go CNIPreset validation (line 268), init_cni.go (flannel branch, wantArgoCD, the Application half of cniBootstrap), init_run.go (the GitOps steps and the Application wiring), and their tests: init_cni_test.go, init_run_test.go, init_leak_test.go, helpers_test.go.
• pkg/cni/flannel/ (whole directory); pkg/cni/preset.go: ArgoCDApplication, HandoverWorkloads and the Argo-specific doc; pkg/cni/cilium/cilium.go ArgoCDApplication and its ADR 0055 tests; Preset.TalosPatches' legacy cluster.proxy.disabled.
• pkg/gitops builders and the skeleton (skeleton.go, application.go New) after confirming pkg/prmode/source.go only needs the Application read types (it does not use the builders).
• internal/controller/cnihandover_controller.go: the Argo CD lookup (argoApplicationGVK, argoApplicationStatus, WaitingForArgoCD reasons), cniPresetFor's flannel branch; internal/controller/kubernetes_upgrade.go (cniHandoverPending comment and flannel case), cni_kubeproxy_switch.go (ciliumChangeInstructions and its Argo wording), etcd_restore.go (kube-flannel in restoreAgentNames), ca_rotation_clients.go and config/rbac/role.yaml (kube-flannel resourceName), and the cni rule markers that name Argo CD.
• pkg/render/networkinstall.go: talosRunsFlannel, cniPresetFlannel, the preset == flannel branch; the flannel document is always removed. internal/cli/adopt_run.go lines 222-223: adopt of a flannel cluster is refused with the deletion message (adopt keeps working for Cilium clusters).
• Docs: architecture.md ("Bootstrap" step 3, the whole "GitOps and Argo CD" hand-over text, "Kokumi as the delivery layer in front of Argo CD", "CNI" presets text, the Open questions entry on Kokumi) and the repository skeleton block; ADRs 0013, 0019, 0024, 0028, 0048, 0055 gain a "superseded in part by 0099" line.
• Kept: pkg/gitopstracking, pkg/prmode, spec.gitops.*, internal/uiapi/gitops_*, the UI GitOps page, dist/chart argocdHealth (its Lua files now also feed the bundle's argocd-cm).
• Kept as deprecated, unused CRD fields (append-only): spec.cni.preset values flannel/custom, spec.cni.custom and TalosClusterCNICustom* types, spec.cni.handover values manual/disabled, TalosOperation fields that mention handoff.

Work units

Units are small and mostly independent; each names its files. Dependencies are in brackets. Commit incrementally, one Conventional Commit per unit.

1. CRD fields and deprecations. api/v1alpha1/taloscluster_types.go (add spec.cni.cilium.values, change the preset default to cilium, deprecate custom/flannel/handover values in comments), api/v1alpha1/ichorinstallation_types.go (spec.stack, status.stack), make manifests generate, config/crd/bases/*, dist/chart. Check internal/installer additive test still passes against the previous CRDs. [none]

2. Talos v1.14 render cleanup. pkg/cni/cilium/cilium.go (TalosPatches emits KubeProxyConfig enabled:false), pkg/render/networkinstall.go, pkg/render/kubeproxy.go, the inline-manifest document on a v1.14 contract (verify the kind in the pinned machinery), pkg/fieldmeta/fieldmeta.go, tests. [none]

   Unit 2 as built: the kind is KubeInlineManifestConfig (apiVersion v1alpha1, Go type k8s.KubeInlineManifestConfigV1Alpha1 in talos/pkg/machinery/config/types/k8s/inline_manifest.go, v1.14.3; fields name, manifest). Patches keep carrying the legacy cluster.inlineManifests entries (the typed CRD has no document kind for it); applyPatches converts them to that document on a MultidocKubernetesConfigSupported() contract, and the hand-over removes the entry before conversion, so it works for both forms. Preset.TalosPatches no longer emits cluster.proxy.disabled; injectKubeProxy is the only owner. Hash effect: a v1.14-contract render with a patch inline manifest (every cluster with the CNI entry) changes from cluster.inlineManifests to a KubeInlineManifestConfig document, so its hash moves once (a rolling config apply, safe class via a new fieldmeta rule). Contracts below v1.14 render byte-identically; v1.14 renders without inline manifests are unchanged.

3. Cilium-only code removal. Delete pkg/cni/flannel/, flannel/custom branches in internal/cli/init_cni.go, init_plan.go, internal/controller/* listed above, config/rbac/role.yaml. Webhook: create requires cilium, update of a stored flannel/custom spec refused, CNIUnsupported condition and render refusal. internal/webhook/v1alpha1/taloscluster_webhook.go, pkg/render, internal/controller/talosmachine_controller.go. [1]

4. pkg/stack (render and publish). New package: embedded Argo CD chart and Kokumi manifest with fetch targets and digests (Makefile, pkg/stack/argocd/chart/, pkg/stack/kokumi/install.yaml.gz), Kokumi transformations, Argo CD default values, deterministic bundle tar, OCI push (oras-go or go-containerregistry; pick by Context7 docs), an in-memory registry fake for tests. [none]

5. Values engine. pkg/values shared by Cilium, Argo CD and Kokumi: merge order, owned-key rejection, unknown-key check, schema validation, size cap; pkg/cni/cilium/validate.go with the three semantic guards. [none]

6. Stack controller. internal/controller/stack.go inside the IchorInstallation reconciler: steps 4-9, Orders (unstructured, no Kokumi Go dependency, like the Argo CD lookups today), status, republish on registry loss, RBAC markers, config/rbac, dist/chart. [1, 4, 5]

7. Cilium Helm Job. Hidden ichor cni helm apply in internal/cli, Job builder and watcher in internal/controller/cni_helm.go, SA ichor-cni RBAC in config/rbac/ichor-installer style with the chart-comparison test, adoption with TakeOwnership, upgrade and rollback, lock acquisition. [1, 5] Unit 7 as built: cilium.Apply (pkg/cni/cilium/helm.go) looks the release up and runs action.Install (TakeOwnership, wait for the rollout, no rollback: an atomic rollback of an adoption would uninstall a running CNI) when there is none, otherwise action.Upgrade (ResetValues, wait, Atomic). The hidden ichor cni helm apply --cluster NAME -n NS reads the TalosCluster, computes cilium.EffectiveValuesFor and calls it. The controller side is internal/controller/cni_helm.go: AdvanceCNIHelmJob creates or reports the Job named by cluster and effective-values hash (CNIValuesHash), the hooks for unit 8 (hash to compare the release against) and unit 9 (launch with a new kube-proxy mode; the cluster operation lock is theirs to take). The Job is the installer image on the host network with KUBERNETES_SERVICE_HOST=localhost, KUBERNETES_SERVICE_PORT=7445, pod label app.kubernetes.io/name: ichor-cni, one attempt, 24 h TTL, 25 min deadline. Departures from the text above: (1) the ServiceAccount ichor-cni and its RBAC are static, shipped with Ichor (config/rbac/ichor-cni, chart cniHelm.enabled, default on), not created per run, because the controller could not create the ClusterRoleBinding without holding bind and escalate itself; (2) Helm's release record Secret is stored in the controller's namespace (--storage-namespace, ichor-system), not in kube-system, so the one Secret permission is a namespaced Role like every other (the chart refuses a cluster-wide Secret rule) and the controller, which already reads Secrets there, can read the record in unit 8; the release itself (name cilium, namespace kube-system) is as specified; (3) a user value that makes Cilium's chart create Secrets in kube-system (hubble.tls.auto.method=helm) is not covered by the Job's RBAC, Ichor keeps cronJob. Network policy: a cni policy for the pod label (egress to the kube-apiserver entity on 6443 and to the host entity on 7445) in config/cilium-network-policy and the chart. Hash effect: the bootstrap manifest now carries the ownership metadata and is re-marshalled, so the CNI inline manifest's bytes, and with them the machine config hash of every cluster, change once at this release (a rolling config apply, the manifest content is otherwise the same objects); the goldens in pkg/cni/cilium/testdata were regenerated deliberately. Helm's SDK pulls more indirect modules into go.mod (SQL storage drivers, oras).

   Unit 7 amendment (found on hardware): the Job and its ServiceAccount run in kube-system, not ichor-system. The Job needs hostNetwork, which Talos's default Pod Security admission (enforce baseline, exempting only kube-system) refuses elsewhere; with the Job in ichor-system its pod was never created and the hand-over could not happen. What changed: the ichor-cni ServiceAccount and the ClusterRoleBinding's subject are in kube-system; the Helm release record's Role/RoleBinding stay in ichor-system (the binding's subject is the kube-system ServiceAccount); the controller gets a hand-written namespaced Role cni-job-role in kube-system (config/rbac/ichor-cni/controller_role.yaml: jobs get/list/watch/create/delete, pods get/list, pods/log get, events get/list) instead of the ichor-system markers, and the installer a namespaced Role in kube-system for the ServiceAccount, Roles and RoleBindings it applies there; config/default puts these back in kube-system after kustomize's namespace transformer. The controller's Job informer is scoped to both namespaces. The cni network policy is dropped (ADR 0100 amendment). Fail-fast: a Job with no active pod cniPodGrace (2 min) after creation is taken to be stuck (its pods cannot be created); AdvanceCNIHelmJob deletes it and returns a CNIHelmJobFailedError with NoPod, the hand-over releases the cluster lock, sets CNIValuesApplied=False (JobFailed) with the newest FailedCreate event message of the Job, and requeues after 30 s to try a fresh Job, instead of holding the lock until the 25-minute deadline.

8. Hand-over retarget. internal/controller/cnihandover_controller.go, render (unchanged removal), desiredInputsHash unchanged, kubernetes_upgrade.go guard, tests; spec.cni.handover read as automatic. [3, 7]

   Unit 8 as built: CNIHandoverReconciler drops every Argo CD lookup and ignores spec.cni.handover. Once the cilium DaemonSet is ready it runs AdvanceCNIHelmJob under the cluster operation lock (identity ichor-controller/cni-helm, distinct from controllerIdentity so a node rollout holding the lock stops the Job and the reverse; taken through an acquireDisruptiveLock method, so it makes the usual shutdown and even-control-plane refusals). When the Job has completed, the newest sh.helm.release.v1.cilium.v* record in the controller namespace is deployed with a config that hashes to CNIValuesHash, and the DaemonSet is fully ready, it sets CNIHandover=True and records the hash in the new status.cni.appliedValuesHash. A later hash difference starts a new Job and leaves CNIHandover True (flipping it would re-render the inline entry); the new condition CNIValuesApplied reports Applying, WaitingForLock, WaitingForRelease, InvalidValues, and JobFailed with the Job's last pod log line (or its reason). A failed Job is not retried until the values change or its 24 h TTL removes it. A finished Job for a hash the values returned to is deleted when the release is deployed at other values, so a revert runs again. cniHandoverPending has no disabled exemption.

9. Kube-proxy switch drives the release. internal/controller/cni_kubeproxy_switch.go, cni_kubeproxy_admin.go, tests. [7]

10. ichor init rewrite. internal/cli/init.go, init_run.go, init_plan.go, init_cni.go; create IchorInstallation.spec.stack; wait on status.stack; print the Kokumi admin password once; delete the flags and files above. [3, 6]

11. ichor cni values|upgrade commands and service. internal/cli/cni.go (values get|set|edit|diff, upgrade), proto/ichor/v1alpha1 (CniService), internal/uiapi/cni_service.go (effective values, validate, diff preview, apply: writes spec.cni.cilium.values, honours tracking refusal ADR 0051). [1, 5] Unit 11 as built: CniService (GetCiliumValues, ValidateCiliumValues, DiffCiliumValues, SetCiliumValues, StartCniUpgrade) and ichor cni values get|set|edit|diff, ichor cni upgrade. Writes go through spec.cni.cilium.values with a dry-run first; a tracked TalosCluster is refused with the ADR 0051 wording. Gateway API CRD presence and Talos forwardKubeDNSToHost/KubePrism are read server-side (one CRD metadata get, and a rendered control-plane config); an unreadable fact stays unknown and its guard passes. StartCniUpgrade runs its checks (an upgrade is offered, stored values validate, no other operation is unfinished) and creates the upgrade operation described next.

    The Cilium chart upgrade operation, as built. Shape: a TalosOperation of kind cni with the new, append-only spec.cni (action: upgrade, chartVersion: <target>), immutable, only allowed on the cni kind. Without spec.cni a cni operation is the kube-proxy switch, as before. StartCniUpgrade fills chartVersion from status.cni.availableChartVersion and creates the operation as the caller (a dry-run create first, so a caller who may not create talosoperations starts nothing). Availability: CNIHandoverReconciler compares the chart version in the newest deployed release record (chart.metadata.version) with the embedded cilium.ChartVersion on every reconcile after the hand-over and sets status.cni.availableChartVersion and CiliumUpgradeAvailable (True with the installed and available versions, or False UpToDate); it also watches cni operations so the end of an upgrade clears them. When the gap is more than one minor, the condition message says the step cannot be applied in one go. Flow: like an Ichor self-upgrade the operation is machine-less (status.machines stays empty); the backbone takes the cluster operation lock, and the executor (cniKubeProxyAdvancer.upgradeChart) (1) refuses when spec.cni.chartVersion is not the chart this controller embeds (Ichor changed after the operation was created), (2) refuses a downgrade or a jump of more than one minor from the release's chart (decision 11), pausing with the message and starting no Job, (3) runs the Cilium Helm Job (AdvanceCNIHelmJob with the operation's UID, and its last retry time, mixed into the Job name so a Job the hand-over left for the same values within its 24 h TTL, or a failed one, is not mistaken for this run), which upgrades the release to the embedded chart with the current effective values, (4) waits until the newest release record is deployed at the target chart, and (5) waits for the cilium DaemonSet to be rolled out, both on the VerifyingNewSide clock (30 min), then the operation completes and releases the lock. A failed Job pauses the operation with the Job's reason; Helm's atomic upgrade has already rolled back; ichor operation resume runs a new Job. The cluster's status.cni is no longer replaced wholesale by the kube-proxy switch (it set only kubeProxy and dropped appliedValuesHash and availableChartVersion); it sets just kubeProxy.

12. UI Cilium settings. ui/src/pages/Cilium.tsx (effective values read-only view, YAML editor for the user layer, validate, diff preview, apply), route in ui/src/router.tsx and app.tsx, nav entry; reuse the config-editing patterns of ADR 0039; preactcn only. A later Addons page uses the same editor against spec.stack. [11]

13. NetworkPolicies for kokumi and argocd. pkg/stack/policies/*.yaml (CiliumNetworkPolicies per decision 10) included in each bundle, applied last by unit 6, unit tests that parse them and a render test checking default-deny plus each named allow; e2e connectivity check with Cilium. Does not touch ichor-system policies. [4, 6]

14. e2e on the e2e harness. test/e2e: init to Healthy stack, a Cilium values edit applied live, a kube-proxy switch, an Argo CD values edit through Kokumi, a registry restart, measured memory of the stack on the 3Gi + 2.5Gi layout, plus the three unanswered Argo CD points. Hardware batch per the usual rule. [10, 7, 6]

15. Docs. docs/design/architecture.md rewrite of the sections above, supersession lines in ADRs 0013, 0019, 0024, 0028, 0048, 0055, docs/adr/README.md via make adr-index. [alongside each unit]

16. Removal of GitOps export. Delete internal/cli/gitops*.go, pkg/gitops builders and the skeleton, UI strings for export, once unit 10 no longer calls them. [10]

Units 1, 2, 4 and 5 can start at once on stub commits; 3 follows 1; 6 and 7 follow 1, 4, 5; 8-13 follow their brackets.

Consequences

• One path for every cluster: Cilium plus Kokumi plus Argo CD. Nothing to choose, nothing to explain, and the e2e covers one shape.
• Ichor's controller now runs two kinds of Job (installer and Cilium Helm) and writes Kokumi Orders. The controller still never writes Talos nodes outside the machine controller and still never mutates spec.
• Cilium is editable from the UI because Ichor, not Git, holds its values. Argo CD never manages Cilium, so a Cilium change cannot be reverted by Argo CD. The cost is a cluster-admin-equivalent Job identity for a few minutes per change (same exposure as the installer).
• The user's own apps get Argo CD with no repository set up; they can add repositories to it. They still must not put Ichor CRs under it (the warning stays).
• Ichor now ships three third-party releases and bumps them: an Ichor release carries Cilium, Argo CD and Kokumi pins, each with a digest in the Makefile.
• Memory on the small homelab is tight (estimated 1.3-1.7 GiB for the stack, a 2.5Gi worker), and the default Kokumi limits of 128Mi are probably too low.
• ADR 0048's "Unknowns" are answered: the Application accepts only what Kokumi generates, so sync waves and ignoreDifferences cannot be carried (not needed now that Cilium is not delivered through Kokumi); the Product/Order question is answered by bundle-per-component Orders; air gap is out of scope (ADR 0090); the Argo CD range is >= 3.3 and 3.5.6 is pinned; Kokumi remains experimental v1alpha1.

Risks

• Kokumi is experimental (v1alpha1, breaking changes in every minor release: AutoDeploy removed in 0.17, Menu made namespaced in 0.16, status fields removed in 0.15 and 0.13). Ichor pins 0.17.0 and must re-read the source for each bump. A broken Kokumi stops value changes for Argo CD and Kokumi but not the running workloads, and does not affect Cilium or the nodes.
• Registry durability. The default emptyDir loses the registry on a pod restart or node reboot. Source bundles and the pinned digest are reproducible, but Kokumi's Preparations (rendered artifacts) are not; if Kokumi does not re-render, Argo CD fails to fetch the revision until the Order is bumped. Using a PVC needs a StorageClass Ichor does not provide.
• Self-management. Argo CD prunes and self-heals its own manifests: a wrong bundle can remove Argo CD or the Kokumi registry. Mitigations: pre-rendered bundles are identical to the applied bytes, Automatic promotion can be set to Manual, and Kokumi Applications carry no resources-finalizer.
• Bootstrap-time diffs. First sync of an adopted component could show a diff if the bundle and the applied objects differ in defaulted fields; server-side apply makes that benign but the first Healthy takes longer.
• Anonymous writable registry. Any pod could push a source bundle that Argo CD later applies cluster-wide; the CiliumNetworkPolicies (decision 10) are the only barrier until Kokumi's requireSourceVerification is used with a key held by Ichor (future).
• The Cilium Job path. A Cilium upgrade replaces the CNI on live nodes; a Job that fails mid-way relies on Helm's atomic rollback. The Job uses KubePrism; if a cluster has KubePrism disabled the Job uses the in-cluster service address and can lose its API access during the agent restart.
• Resource pressure on small clusters (above), including Kokumi's 128Mi limits and Argo CD repo-server spikes.
• Values footguns. The schema does not reject unknown keys; the unknown-key check is best effort and some Cilium values only work with others.

Open questions

• Does Kokumi 0.17.0 re-render a Preparation whose artifact has vanished from the registry, or must the Order be changed? (Unit 6 and the e2e.)
• Does Argo CD accept the Preparation artifact's layer media type (Kokumi pushes application/vnd.kokumi.preparation.v1) with its default ARGOCD_REPO_SERVER_OCI_LAYER_MEDIA_TYPES? Kokumi's own Getting Started works with Argo CD 3.5.1; our version is not tested.
• How large a bundle can Argo CD take from an OCI source (the docs name no limit; the Argo CD bundle is about 1.9 MB raw)?
• Is hostNetwork plus localhost:7445 the right way for the Cilium Job to keep its API access, or should the Job tolerate a short API outage and retry?
• Should the registry use a PVC when a default StorageClass exists, and which size? 1Gi is Kokumi's default; Preparations are never garbage collected.
• Should Ichor render the Talos side of bpf.masquerade (forwardKubeDNSToHost: false) itself instead of refusing the value?
• Whether to wire Kokumi's login to Ichor's OIDC issuer (ADR 0094) in this change or later; until then the generated admin password is the credential.
• Does ichor adopt of an existing Cilium cluster need an adoption path for an Argo CD-managed Cilium Application (a cluster that went through ADR 0024)? It is out of scope here; those clusters are re-created like flannel ones unless the user asks for it.
• The exact Talos v1.14 inline-manifest document kind and its fields must be confirmed against the pinned machinery (unit 2).

Amendment (2026-10-29): how decision 10 is realised (unit 13)

• The policies are pkg/stack/policies/{kokumi,argocd}.yaml, embedded and appended to each bundle by kokumi.Render / argocd.Render, so the published bundle keeps them and Argo CD does not prune them. Options.OmitPolicies leaves them out for the first apply (steps 4-5) and stack.Policies(component) returns them alone for step 8.
• The Argo CD chart renders Kubernetes NetworkPolicies by default (global.networkPolicy.create), and Cilium unions those with CiliumNetworkPolicies; the argocd-server one allows all ingress. DefaultValues switches them off.
• The apiserver egress uses port 6443, as ADR 0100 does. Neither Kokumi 0.17.0 nor Argo CD ships a webhook, so no apiserver ingress exists; a test fails if one appears, because it would also need remote-node (ADR 0100 amendment).
• Ichor's side of the publish flow is an egress rule to kokumi app=registry :5000 in config/cilium-network-policy/policies.yaml and the chart copy. A test requires both directions.
• Users reach argocd-server and kokumi-server (both ClusterIP) by kubectl port-forward, which the policies admit from the host entity on 8080. An ingress or gateway needs a policy of the operator's own.

Amendment (as built: an unprivileged controller, and no Secret in a bundle)

Two decisions changed unit 6 after it landed.

1. The controller does not apply Kokumi or Argo CD; a one-shot Job does (pattern of ADR 0059). Unit 6 gave the controller escalate/bind on (cluster)roles and create/patch on CRDs, workloads, RBAC and admission policies so it could server-side apply the two manifests itself, which made it cluster-admin in effect. Decision 5, steps 4 and 5 now run as a Job ichor-stack-<component> in the controller's namespace, using the installer image and the hidden command ichor stack apply kokumi|argocd (internal/stackapply). It runs as its own ServiceAccount ichor-stack (config/rbac/ichor-stack, chart rbac/ichor-stack.yaml, always present, compared by TestStackRBAC; a test in internal/stackapply fails when a bundle holds a kind the role does not grant). The Job renders with OmitPolicies, applies Namespace, then the Secrets, then the rest, waits for the component's Deployments, and applies the component's CiliumNetworkPolicies last (unit 13). Policies are therefore applied per component, not once after both. The pod carries the label app.kubernetes.io/name: ichor-stack; config/cilium-network-policy and the chart allow it egress to kube-apiserver:6443 only. It listens on nothing, so no ingress rule exists for it (the [kube-apiserver, remote-node] ingress rule of the ADR 0100 amendment stays on the controller's webhook port). The controller creates the Job (jobs: get, create in its namespace, already granted), treats a completed Job as step done, a failed Job as an error (delete the Job to retry), and does everything else: Secrets in its namespace, publishing, Orders, status. The Job is skipped once the component's Order exists. The generated values reach the Job through secretKeyRef env vars, so the Job needs no rights on the controller's namespace; user values for Argo CD go as --values=<json>. The controller's ClusterRole lost namespaces, CRDs, RBAC, admission policies, workloads, ConfigMaps/Services/ServiceAccounts and Secrets create/patch; it keeps orders: get, create, patch and applications: get. The Job's role is as strong as cluster-admin (it must create roles it does not itself hold) for the minutes it runs; that is the cost of the first apply and is why the identity is separate.

2. No Secret is ever published. Kokumi's registry is anonymous, so a Secret in a bundle is readable by any pod that can reach it. stack.SplitSecrets removes every kind: Secret from the rendered manifest; the controller hashes and pushes only what remains (a test unpacks the pushed layer and asserts it holds no Secret and neither generated value). The Secrets (kokumi-server-auth, argocd-redis, argocd-secret, ichor-kokumi-registry) are created by the Job directly, once, with Create and AlreadyExists ignored, so a rerun never resets them. They are not part of any bundle, so no Argo CD Application tracks them and an Application with prune enabled cannot prune what it does not track; as a second line they carry argocd.argoproj.io/sync-options: Prune=false and compare-options: IgnoreExtraneous. Kokumi's generated Application could not be read here (no Kokumi source in the tree; its install.yaml holds no Application), so the guarantee rests on Argo CD's tracking rule, not on the Application's settings: confirm on hardware that the Secrets survive the first sync.

Amendment (2026-10-29): the unmanaged preset exists for adoption only

ADR 0103 adds spec.cni.preset: unmanaged (with spec.cni.unmanaged.talosCNI) so that ichor adopt --cni-unmanaged can bring in a cluster whose CNI Ichor leaves alone. Decision 8 is unchanged: Cilium is the only CNI Ichor installs or operates, flannel and custom stay refused, and ichor init has no flag for unmanaged. The webhook refuses unmanaged on create unless the object carries the adoption annotation ichor.maesh.dev/adopted. A foreign Cilium can also be taken over as this ADR's release through the same TakeOwnership path (ADR 0103, 1.2).

Amendment 2026-11-05 (ADR 0129): decision 4 said the webhook runs ValidateValues; it did not. The TalosCluster webhook now runs its static part (schema, unknown and owned keys, size cap) when the values, kubeProxy or tunnel change. The three guards that need the machine configuration or the live cluster are still the Connect service and the CLI only, and the controller does not yet re-check before a Job as this decision says; that re-check is open.

Amendment 2026-11-05 (ADR 0129, second amendment): the open re-check is closed. Before it creates a Cilium Helm Job (hand-over, values change, kube-proxy switch, chart upgrade) the controller runs ValidateValues with the same facts the UI reads (internal/cnifacts); on failure it starts no Job, sets CNIValuesApplied=False with reason InvalidValues and requeues.

Amendment 2026-11-05 (hardware lane H): the Job's ClusterRole ichor-cni-role (decision 2) had no rule for the objects the Cilium chart renders with gatewayAPI.enabled=true, so the upgrade failed on its first get of the GatewayClass cilium (cannot get resource "gatewayclasses"). It now holds create/delete/get/list/patch/update on gatewayclasses (gateway.networking.k8s.io) and on validatingadmissionpolicies and validatingadmissionpolicybindings (admissionregistration.k8s.io), and bind/escalate on the two Roles the chart then adds in cilium-secrets (cilium-gateway-secrets, cilium-operator-gateway-secrets; ADR 0108). The cilium-operator ClusterRole's extra Gateway API rules are covered by the existing escalate on that name. TestCNIEscalationNamesOnlyCiliumsRoles now renders the chart with the Gateway API on as well, and TestCNIRoleCoversTheGatewayAPIObjects fails when the role (in config/rbac/ichor-cni or the chart copy) lacks Helm's verbs on those kinds. Other optional chart features render kinds the role still lacks (ingressController.enabled: IngressClass, EndpointSlice and two Roles in cilium-secrets; hubble.ui.enabled: ClusterRole hubble-ui; clustermesh.useAPIServer: ClusterRole clustermesh-apiserver; bgpControlPlane.enabled: Role cilium-bgp-control-plane-secrets; envoyConfig.enabled: Role cilium-envoy-config-secrets); they are not granted until a user needs them.

Amendment 2026-11-05 (optional Cilium features): the five features the previous amendment left out are now granted, none of them is refused by ValidateValues (the owned keys are ipam.mode, the capability sets, cgroup.*, k8sService*, kubeProxyReplacement and tunnelProtocol). ichor-cni-role now holds create/delete/get/list/patch/update on ingressclasses (networking.k8s.io) and endpointslices (discovery.k8s.io) for ingressController.enabled, and bind/escalate, by name and never a wildcard, on the ClusterRoles hubble-ui (hubble.ui.enabled, which the chart only renders with hubble.relay.enabled) and clustermesh-apiserver (clustermesh.useAPIServer), and on the Roles cilium-ingress-secrets and cilium-operator-ingress-secrets (ingressController.enabled, in cilium-secrets), cilium-bgp-control-plane-secrets (bgpControlPlane.enabled, in kube-system), cilium-envoy-config-secrets (envoyConfig.enabled, in cilium-secrets) and clustermesh-apiserver-generate-certs (clustermesh.useAPIServer with the cronJob certificate method, in kube-system). The Deployments, Services, ServiceAccounts, ConfigMaps, Jobs, CronJobs and RoleBindings those features add were already covered. clustermesh.useAPIServer works only with clustermesh.apiserver.tls.auto.method=cronJob: the chart default, helm, renders Secrets (cilium-ca, clustermesh-apiserver-*-cert) in kube-system, which the Job cannot manage and must not be given (the same reason Ichor keeps hubble.tls.auto.method=cronJob). TestCNIEscalationNamesOnlyCiliumsRoles now renders every optional feature, and TestCNIRoleCoversTheOptionalFeatureObjects fails per feature when the role (in config/rbac/ichor-cni or the chart copy) lacks Helm's verbs on a kind the chart renders, or when the chart renders a kind the test does not map.

Amendment 2026-11-05 (clustermesh certificates): ValidateValues now refuses clustermesh.useAPIServer=true unless clustermesh.apiserver.tls.auto.method=cronJob (or clustermesh.apiserver.tls.auto.enabled=false, certificates supplied by the user), so the default helm method fails at the webhook and in the controller's pre-Job check with a message, not in the Job on a Secret it may not create.

Amendment 2026-11-05 (the Job cannot change its own confinement): ichor-cni-role held create/delete/get/list/patch/update on every validatingadmissionpolicies, validatingadmissionpolicybindings, gatewayclasses, ingressclasses and endpointslices, so the Cilium Job could update or delete the ichor-controller-jobs policies that confine it (ADR 0108). get, update, patch and delete on those five resources are now granted by resourceNames, exactly the names the pinned Cilium chart renders with every optional feature on: gateway.cilium.io (the policy and its binding), the GatewayClass cilium, the IngressClass cilium and the EndpointSlice cilium-ingress (a fixed name; a ClusterRole rule, not a namespaced Role, since the name is the limit). create and list cannot be limited by name in RBAC and stay, so the Job can still create a policy of another name; it can no longer touch ichor-controller-jobs, ichor-controller-job-image or their bindings. TestCNIRoleChangesOnlyTheObjectsCiliumRenders renders the chart with each optional feature, requires the names to equal what it renders (a chart bump that renames one fails there), requires the name-limited verbs to carry resourceNames, and refuses the four Ichor policy names, in config/rbac/ichor-cni and in the chart copy. TestCNIRoleCoversTheGatewayAPIObjects and TestCNIRoleCoversTheOptionalFeatureObjects still ask for Helm's verbs on each kind. Hardware verification (later): enable gatewayAPI and ingressController, then disable them, and see the Helm Job create, update and delete those objects without forbidden.

Amendment 2026-11-05 (ADR 0131): the user layer of the Cilium values can no longer be written by ichor-operator. Changing spec.cni.cilium.values needs update on talosclusters/cilium-values, which only ichor-admin holds, because the chart behind them can render privileged host-path pods that the Job's near-cluster-admin ServiceAccount creates (decision 2). The five *.secretsNamespace.name values (ingressController, gatewayAPI, envoyConfig, bgpControlPlane, tls) are Ichor-owned keys pinned to cilium-secrets, so the Roles named in the bind/escalate lists land only in Cilium's own namespace; BGP's chart default of kube-system moves there too, which changes where the cilium-bgp-control-plane-secrets Role of the previous amendment is created.

Amendment 2026-11-06 (hardware run I): ValidateValues gained two refusals and a text fix. (1) gatewayAPI.enabled=true is refused unless the kube-proxy mode is replaced: the Cilium 1.20 Gateway API prerequisites say "Cilium must be configured with the kube-proxy replacement, using kubeProxyReplacement=true" (docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api), and on hardware the operator logged "Gateway API support requires kube-proxy-replacement enabled" and never accepted the GatewayClass. The mode is known to the webhook, the UI and the controller alike (the controller passes the mode Cilium runs now, so a values edit made together with the switch waits for it). (2) After the schema, owned-key and guard checks pass and the user set something, the chart is rendered once with the effective values, so the chart's own validate.yaml (for example cluster.id without cluster.name) refuses at admission instead of failing the Helm Job again and again. Measured on the workstation (Intel N100): the check costs about 80 ms and 36 MB of allocation per call; the schema validation that already ran costs about 30 ms and 14 MB, nearly all of it Helm compiling the 1 MB JSON schema on every call (not cached, a possible later saving). Rendering is offline, so what the Job decides from the live cluster stays the Job's: monitoring.coreos.com/v1 is declared present (the chart refuses a ServiceMonitor without that CRD) and values that make the chart look up a ConfigMap (k8sServiceHostRef) are not rendered. (3) The clustermesh refusal no longer ends mid-sentence ("...which Ichor's Helm Job may not manage"). Tests: TestValidateValues (gateway API while kube-proxy runs, cluster.id without cluster.name, a ServiceMonitor passing), the webhook table in taloscluster_cilium_values_test.go.

Amendment 2026-11-06 (hardware lane I, finding B2: the Job holds what Cilium's feature Roles grant; escalate by name does not apply to a create): on a real cluster the Cilium Helm Job could not create the objects the chart renders for the optional features: the Roles cilium-gateway-secrets, cilium-operator-gateway-secrets, cilium-ingress-secrets, cilium-operator-ingress-secrets and cilium-envoy-config-secrets (namespace cilium-secrets), cilium-bgp-control-plane-secrets and clustermesh-apiserver-generate-certs (kube-system), and the ClusterRoles hubble-ui and clustermesh-apiserver. The apiserver answered attempting to grant RBAC permissions not currently held, although kubectl auth can-i escalate roles/<name> said yes and the names were in the escalate/bind rules (ADR 0108). Cause, from the Kubernetes 1.36 source (pkg/registry/rbac/escalation_check.go): RoleEscalationAuthorized, called by the Role and ClusterRole storage on both create and update, asks the authorizer for escalate with Name: requestInfo.Name; on a create the request path has no name (it is in the body), so a rule with resourceNames never matches, and only an update (name in the path) is authorized that way. BindingAuthorized takes the name from roleRef, so bind by name does work on a create. The 2026-11-05 amendments and ADR 0108 assumed the name was honoured on create; the earlier hubble-generate-certs failure (ADR 0103 amendment 3) was the same cause.

Decision: the Job's ServiceAccount holds the rules those Roles and ClusterRoles grant, so the check passes without escalate (a held rule also authorizes the bindings, which need bind otherwise). Nothing is wildcarded and each grant is as narrow as the chart's own rule:

• ClusterRole ichor-cni-role (config cni-role) gains, read-only, exactly what hubble-ui and clustermesh-apiserver grant: "" componentstatuses, namespaces, nodes, services get/list/watch (the new part is componentstatuses, nodes and watch on namespaces and services; the rest the Job already held); apiextensions.k8s.io customresourcedefinitions get/list/watch; cilium.io ciliumendpoints, ciliumendpointslices, ciliumidentities, ciliumnodes get/list/watch; discovery.k8s.io endpointslices get/watch (unnamed; list was held; the writes stay limited to cilium-ingress).
• Role ichor-cni-features-role (config cni-features-role, config/rbac/ichor-cni/features_role.yaml) with its RoleBinding, in kube-system: secrets get/list/watch, and update on clustermesh-apiserver-server-cert, clustermesh-apiserver-admin-cert, clustermesh-apiserver-remote-cert, clustermesh-apiserver-local-cert. With cni-hubble-role (create, cilium-ca) this covers cilium-bgp-control-plane-secrets and clustermesh-apiserver-generate-certs.
• Role ichor-cni-secrets-role (config cni-secrets-role) with its RoleBinding, in cilium-secrets: secrets create/delete/get/list/patch/update/watch, the union of the five Roles Cilium creates there. That namespace holds Gateway, Ingress and Envoy TLS Secrets and nothing of Ichor's.
• The bind and escalate names are cut to the base objects the bootstrap or the takeover already created and Helm only updates, which the Job does not hold (an update carries the name): ClusterRoles cilium, cilium-operator; Roles cilium-config-agent, cilium-operator-ztunnel. Dropped: hubble-ui, clustermesh-apiserver and the nine Roles above (held now), and hubble-generate-certs, cilium-tlsinterception-secrets and cilium-operator-tlsinterception-secrets (held already or now: the first by cni-hubble-role, the others by cni-secrets-role).

The widening, honestly: the Job gains read access to nodes, CRDs and the Cilium custom resources cluster-wide, watch on every Secret in kube-system (it already held get and list there), update of four named clustermesh Secrets, and every Secret verb on every Secret in cilium-secrets. This is on top of an identity that is already close to cluster-admin (create/update of Roles and ClusterRoles, bind and escalate on cilium and cilium-operator, and creation of Deployments and ServiceAccounts anywhere), so it adds little, and these features are admin-only to enable (ADR 0131). It is a real, bounded grant, not a cosmetic one, and an operator who never enables the features still carries it.

cilium-secrets exists only where Cilium runs, and a RoleBinding into a missing namespace fails an install. The installer therefore skips an object in cilium-secrets when the apply answers not found (internal/installer, test TestRunSkipsObjectsInTheCiliumSecretsNamespaceWhereItDoesNotExist; a missing namespace anywhere else still fails the run), and the chart has cniHelm.ciliumSecretsRoles (auto by default: render the pair when helm install or helm upgrade finds the namespace through lookup, which helm template and Argo CD cannot, so they leave it out and gateway, ingress and envoyConfig then fail with the same forbidden; true always, false never). The installer's own bind/escalate names list ichor-cni-features-role and ichor-cni-secrets-role, as TestInstallerEscalationNamesEveryRoleOfTheInstallManifest requires.

Enforcement: TestCNIJobHoldsWhatTheCiliumRolesItCreatesGrant (dist/chart/cni_coverage_test.go) replaces TestCNIEscalationNamesOnlyCiliumsRoles and TestCNIRoleHoldsEveryRuleTheCiliumChartGrants. It renders the Cilium chart with no optional feature (both kube-proxy modes, every tunnel) and with each of the six, and for every Role and ClusterRole it checks, in config/rbac and in the chart, that every rule is covered by the Job's own permissions with the apiserver's covers semantics (group, resource, verb and resourceName, a rule without names covering named ones, a ClusterRole judged against the cluster-wide bindings and a Role against those plus the RoleBindings of its namespace). An uncovered object must be a base object whose name is in the bind and escalate lists; an uncovered feature-only object fails, and so does a name in the lists that is covered anyway or not rendered as a base object. Red before the change: all nine feature objects above failed in both the config and the chart copy, and the lists held ten names the Job should hold. TestCNIRoleChangesOnlyTheObjectsCiliumRenders now allows the one unnamed get on endpointslices.

Known limit, not fixed here: the installer Job creates the Roles and ClusterRoles of a new release under its bind/escalate names, and the same rule means a Role or ClusterRole that is new in a release and grants rules the installer does not hold (these two, ichor-cni-hubble-role) cannot be created by it on an upgrade from a release that lacked it; a fresh ichor init is not affected. Hardware verification (later): on a Cilium cluster enable each of gatewayAPI, ingressController (NodePort), hubble.relay+hubble.ui, bgpControlPlane, envoyConfig and clustermesh.useAPIServer (cronJob method, cluster.id/cluster.name, clustermesh.config.enabled) as an admin, without the temporary ClusterRole of lane I, see the Helm Job finish Applied each time, and disable them again; and install on a flannel cluster to see the cilium-secrets pair skipped.

Amendment 2026-11-06 (correction: the installer is not affected by the create limit): the previous amendment's "known limit" said the installer Job could not create a Role or ClusterRole that is new in a release and grants more than it holds. That is wrong for the installer, and the paragraph is superseded by this one. The installer applies with server-side apply, which is a PATCH on the object's own URL: the apiserver handles it as an update with the name in the path (requestInfo.Name set, including when the apply creates the object), so escalate limited by resourceNames authorizes it. Only a plain POST has no name, which is what Helm sends, so the limit hits the Cilium Job and not the installer. TestRunCreatesANewRoleThatGrantsMoreThanTheInstallerHolds (internal/installer/rbacfirst_test.go) pins it under an enforced RBAC apiserver: an installer whose bind/escalate are by name only is refused a plain create of a Role with rules it does not hold (attempting to grant RBAC permissions not currently held), yet a run creates a new Role and a new ClusterRole of that kind through its apply, after it has widened its own role by name first (the existing stage 0 and 1 order, TestRunUpgradesUnderAnOldInstallerRoleWithoutTheNewKindsRule). The test passed before any installer change, so none was made: holding every rule of every Role in the install manifest, which was considered, would have given the installer Secret and node access it never needs. A future change of the installer to a plain create for Roles or ClusterRoles fails that test.

Amendment 2026-11-06 (hardware lane J, P1: ichor init lost a race against Cilium's bootstrap): ichor init failed on a fresh cluster, twice out of two, with init: applying Ichor's own Role "ichor-cni-secrets-role": namespaces "cilium-secrets" not found. The skip of the previous amendment is in internal/installer only; init applies the install manifest through its own path (applyIchorManifest), and at that moment the namespace does not exist yet: it is rendered by the Cilium chart into the bootstrap manifest that Talos applies from cluster.inlineManifests, which on hardware happened eight minutes after init's apply. A rerun is not possible (the node already took its configuration), so every fresh init was affected, and so was an ichor adopt of a cluster on another CNI.

Decision: applyIchorManifest takes whether Cilium is the cluster's CNI (init: always, --cni cilium is the only preset; adopt: the detected CNI). With Cilium, before it applies an object in cilium-secrets it creates that Namespace (name only, as a server-side apply by the installer's field manager, so it owns nothing but the name; an existing one is left as it is). Without Cilium it skips an object in cilium-secrets when the apply answers not found, and prints the same "skipping ... Cilium does not run here" line as the installer; a missing namespace anywhere else still fails. The Role and RoleBinding therefore exist from the first minute, and the Cilium features (Gateway API, Ingress, Envoy config, BGP) never meet a missing Role.

Why creating, not waiting or deferring: waiting for Cilium's bootstrap is a race with no bound (eight minutes here), and init has to finish before Cilium can be Ready; the controller cannot create the pair later, because it holds no Secret rights and no bind/escalate and must not be given them (the installer Job and the Cilium Job are the identities that hold them, ADR 0108); leaving the pair out until the next ichor upgrade makes every feature fail with forbidden until then. Who owns the namespace: the Cilium chart (tls.secretsNamespace.create, true by default, and ADR 0131 pins the five *.secretsNamespace.name values to cilium-secrets). Both orders work: if Talos applies the bootstrap manifest to the namespace init created, it gets the chart's labels and the Helm ownership annotations as any object updated in place; and the Helm install of the hand-over runs with TakeOwnership (pkg/cni/cilium/helm.go), so it adopts the namespace whether or not it carries them. The chart renders no more than name and labels for it, so there is nothing for the two to fight over. The installer's own apply stays idempotent: a cluster that still lacks the namespace when an upgrade runs skips the pair, and the next run that finds the namespace applies it (test TestRunAppliesTheCiliumSecretsRoleOnceTheNamespaceExists, which also keeps the envtest namespace-free for the skip test). The Helm chart is unchanged: cniHelm.ciliumSecretsRoles: auto renders the pair when lookup finds the namespace (a cluster whose Cilium already runs), true always, false never; a flannel or other cluster gets neither the namespace nor the pair from either path.

Tests: TestApplyIchorManifest_CreatesCiliumSecretsForCilium (red: the exact error above; a fake API server that answers not found for an apply into a missing namespace), TestApplyIchorManifest_SkipsCiliumSecretsWithoutCilium (red, same error), TestApplyIchorManifest_MissingNamespaceElsewhereStillFails.

Hardware verification (later): a fresh ichor init --cni cilium on a node in maintenance mode completes the "Install Ichor" step with no cilium-secrets error; once Cilium's bootstrap has run, kubectl -n cilium-secrets get role ichor-cni-secrets-role,cilium-tlsinterception-secrets and the RoleBinding exist, the namespace carries Cilium's labels (Talos did update, not refuse, the pre-existing object), helm-side hand-over reports Adopted, and enabling gatewayAPI as an admin (ADR 0131) applies in about 20 seconds.