# 0024. CNI hand-over: no registry, render-time removal, a separate controller Status: Accepted (amended 2026-10-26); superseded in part by ADR 0099 · Date: 2026-10-25 · Design section: GitOps and Argo CD Decided: No CNI hand-over registry (one real member); removal happens at render time off a status condition; a separate controller; the generated Application syncs automatically (prune, selfHeal), amended 2026-10-26 ## Context ADR 0019 built `ichor init` through step 5 and deliberately excluded step 6 (hand-over): "it is a continuous reconciliation loop against Argo CD's own status (watch, poll, retry) ... It belongs to the controller, as a later unit's work." This is that unit. The design document's "GitOps and Argo CD" section originally described hand-over as "one mechanism with a component registry, not three special cases," implying three components (CNI, Ichor, Argo CD) each go through inject-into-`cluster.inlineManifests` → adopted by Argo CD → removed. Building `ichor init` (ADR 0019) showed that isn't what actually happens: - **Ichor** is applied directly through the API server (`applyIchorManifest`), never inlined. It never needs hand-over. - **Argo CD** is never installed by Ichor at all. - **The CNI** is the only one that genuinely has to be inlined (a node isn't Ready and nothing can be scheduled before a CNI exists), and the only one Argo CD later adopts. So the registry would have exactly one member. ## Decision: no registry Ponytail's own rule applies directly here: don't build an abstraction over a single implementation. `pkg/cni.Preset` already declares everything a CNI hand-over needs (`BootstrapManifest`, `ArgoCDApplication`, `HandoverWorkloads`) as part of the one interface every CNI implements. A "component registry" would be a second interface wrapping the first for no caller that exists. The hand-over reconciler calls `preset.ArgoCDApplication` directly to get the Name/Namespace it needs to look up; there is nothing to register. ## Decision: removal happens at render time, not by rewriting a `TalosConfigPatch` Why removal has to actually happen, not just "Argo CD adopting it": Talos's `ManifestApplyController` (checked against Talos v1.14.3's source) only ever server-side-applies `cluster.inlineManifests` and re-applies them on every Kubernetes upgrade -- it has no delete path. If the CNI's inlined entry were left in place, an upgrade would silently roll the CNI back to whatever Ichor originally inlined, overwriting anything Argo CD has since changed. The same controller refuses to apply over an object carrying another controller's inventory annotation, which sounds like protection -- but Argo CD's own tracking marker (configured by `argocd-cm`'s `application.resourceTrackingMethod`, read rather than assumed, per the design doc) is not that annotation, so adoption alone does not stop the re-apply. Given that, removing the entry has to be a real, visible state change, not an implicit side effect of Argo CD's status. It is implemented in `pkg/render.BuildConfig`: once `TalosCluster.status` carries the `CNIHandover` condition as `True`, every patch's `ClusterConfig.ClusterInlineManifests` has the entry named `cni.InlineManifestName` filtered out before merging (`dropHandedOverInlineManifest`/`withoutInlineManifest`, `pkg/render/render.go`). This follows the design's existing precedent (control-plane taint folding already happens in render, not as a stored patch) and principle 5 (the controller renders from ingredients; it does not rewrite them): no `TalosConfigPatch` object is ever mutated -- a patch carrying the entry is deep-copied only for the filtered copy that gets merged, and every other patch passes through as the original pointer (verified in `pkg/render/render_test.go`'s `TestCNIHandoverDropsInlineManifest`, including a pointer-identity check). ## Decision: automatic / manual / disabled, as a CR field only `TalosClusterCNI.Handover` (`api/v1alpha1/taloscluster_types.go`) is one of `automatic` (default), `manual`, or `disabled`, matching the design's "Hand-over can be automatic, manual (`ichor gitops handoff cni`), or disabled when Argo CD isn't used." This unit only builds the field and the reconciler's branching on it: `disabled` and `manual` both set the `CNIHandover` condition `False` (with a reason naming which) and never requeue. The `ichor gitops handoff cni` CLI command that would flip a manual cluster to adopted is explicitly another agent's work and is not built here. ## Decision: a separate controller, not a bump to `TalosClusterReconciler` `CNIHandoverReconciler` (`internal/controller/cnihandover_controller.go`) is its own controller watching `TalosCluster`, not folded into `TalosClusterReconciler`. `TalosClusterReconciler`'s own doc comment already warns `DiscoverAll` can block a single `TalosCluster`'s reconcile for up to ~68s at `spec.discovery`'s caps. Raising that reconciler's `MaxConcurrentReconciles` would not have helped: controller-runtime's workqueue deduplicates and serializes by object key *within one controller*, so a single `TalosCluster` only ever has one in-flight reconcile through one controller's queue regardless of concurrency settings -- a slow discovery pass would still block a hand-over check sharing the same queue. Two controllers watching the same Kind each get an independent work queue, so a slow `DiscoverAll` reconcile in one never delays the other. ## Decision: Argo CD absent is "not here yet," never a hang or a hard error `automatic` mode reads the CNI's Argo CD `Application` as `unstructured.Unstructured` (the standard controller-runtime pattern for a foreign CRD this project neither owns nor vendors Go types for) at the exact Name/Namespace `preset.ArgoCDApplication` itself returns -- which is also how "adoption produces no diff" is satisfied: the same template that was injected is the only thing ever looked up, so there's no second, independently-guessed shape to drift from (`pkg/cni/cilium`'s `TestArgoCDApplication` already covers the template's own content; this reconciler's test (`internal/controller/cnihandover_controller_test.go`) covers that the lookup uses that template's Name/Namespace). Both `apierrors.IsNotFound` (Argo CD exists but hasn't adopted the Application yet) and `apimeta.IsNoMatchError` (Argo CD's CRDs aren't installed at all -- the realistic "this cluster doesn't run Argo CD" case) are treated identically: the `CNIHandover` condition is set `False` with reason `WaitingForArgoCD`, and the reconcile returns a bounded `RequeueAfter` (`handoverRequeueInterval`, 30s), never a blocking wait or an error. A cluster that genuinely never runs Argo CD should set `spec.cni.handover: disabled` to stop this polling; left at the `automatic` default it polls forever and says so plainly, which satisfies "does not break" (nothing blocks, nothing retries in a tight loop) without guessing that the operator meant `disabled`. > Update (ADR 0055): the handed-over Application now also carries `ignoreDifferences` on `CiliumIdentity`; release name, namespace and values are unchanged and a test compares them with the bootstrap values. ## Consequences - `pkg/cni.InlineManifestName` is a new exported constant duplicating `internal/cli`'s own unexported `cniKey`: both `pkg/render` and `internal/controller` need the exact string Ichor inlined the manifest under, and `internal/cli` is off-limits to this unit, so there is no shared home below it. `pkg/cni` is safe for both to import (its own subpackages `cilium`/`flannel` import it, not the reverse). - `CNIHandoverReconciler.cniPresetFor` duplicates `internal/cli`'s own preset switch (`"cilium"`/`"flannel"`, `"custom"` deliberately absent, same as ADR 0013) for the same reason: a controller cannot import the CLI package. - Not built here: the `ichor gitops handoff cni` CLI command; any change to `internal/cli`, `internal/uiapi`, `pkg/upgrade`, or the e2e suite. > Update (ADR 0055 addendum, 2026-10-26): the generated Cilium Application now carries `syncPolicy.automated {prune: true, selfHeal: true}` and `PruneLast=true`. Without it Argo CD never synced the Application, so it never became Synced + Healthy and the hand-over never completed (found on real hardware). Ichor still never changes the sync policy of an existing Application; it only chooses one when it creates its own. > Update (2026-10-28): `cluster.inlineManifests`, `cluster.extraManifests` and `cluster.extraManifestHeaders` are `safe` in `pkg/fieldmeta`. Found on real hardware: after the hand-over Ichor stops rendering the CNI entry, and with the default (reboot-required) class the single control plane sat `Ready=False/RebootRequired` for that pending removal (the machine controller never reboots), a later CNI operation paused on "unrelated pending configuration change at cluster.inlineManifests", and only `ichor config apply` cleared it, by rebooting the node (~3 min API outage). Talos v1.14.3 needs no reboot: `controllers/k8s/control_plane.go:510-552` rebuilds `ExtraManifestsConfig` live from the active MachineConfig on control planes only (`controlplaneMapFunc`, `:44-66`); `extra_manifest.go:131-141` destroys the no-longer-listed `k8s.Manifest` resource; `manifest_apply.go:230-393` only server-side-applies (skips inventoried objects, `:335-338`) and has no delete path, so removing an entry deletes nothing in the cluster. The machine controller therefore applies the removal live right after the hand-over and nothing reboot-required is left behind. ## Amendment (2026-10-28): the hand-over status is a render input, so its removal is a change, not drift The machine controller classes a node/render mismatch as drift when `desiredInputsHash` (the generations of the cluster, machine, patches, storage, plus the registry credentials) equals the hash recorded at the last convergence. The hand-over writes `status.conditions[CNIHandover]`, which moves no generation, so the hash stayed put while the render dropped the CNI's `cluster.inlineManifests` entry. The removal read as drift and, under the default `warnOnly`, was never applied (`NodeDrifted`). `desiredInputsHash` now also folds in whether `CNIHandover` is True and `status.cni.kubeProxy` (ADR 0066), the two status values the render reads. The removal is then an intended change and is applied live under any `driftPolicy`, since `cluster.inlineManifests` is safe and needs no reboot. Hashes of clusters that have neither are unchanged.