0055. Cilium stays Synced under Argo CD with ignoreDifferences on CiliumIdentity and ServerSideApply

Status: Accepted (amended 2026-10-26); superseded in part by ADR 0099 · Date: 2026-10-26 · Design section: CNI, GitOps and Argo CD

Decided: The generated Cilium Application carries ignoreDifferences: [{group: cilium.io, kind: CiliumIdentity}] and ServerSideApply=true, without RespectIgnoreDifferences=true; Hubble certificates stay on hubble.tls.auto.method=cronJob; Kokumi cannot carry either setting today, so that case is an open question; the generated Cilium and root Applications use automated sync with prune and selfHeal (addendum 2026-10-26)

Context

The design's last open question was which Cilium and Calico values avoid a permanent OutOfSync under Argo CD. The user chose ignoreDifferences and named a reference: their own homelab Argo CD, read-only on 2026-10-26.

What was observed there (the user's cluster, not a cluster Ichor handed over): Application cilium, Cilium chart 1.20.0, status Synced with selfHeal: true, syncOptions: [ServerSideApply=true], and ignoreDifferences with the single entry {group: cilium.io, kind: CiliumIdentity}. argocd-cm held no Cilium-specific customization. Its values were ipam.mode: kubernetes, kubeProxyReplacement: true, securityContext.capabilities, cgroup, k8sServiceHost: localhost, k8sServicePort: 7445, plus homelab-only choices (Gateway API, L2 announcements, Hubble UI service type and metrics) and no hubble.tls.auto.method.

Decision

Argo CD on its own (Ichor writes the Application, pkg/cni/cilium): the Application gets the one ignoreDifferences entry and keeps ServerSideApply=true and CreateNamespace=true. Release name (cilium), namespace (kube-system) and values are unchanged from the bootstrap manifest, so ADR 0024's no-diff hand-over still holds; a test now compares the two byte for byte.

RespectIgnoreDifferences=true is not added. Argo CD's sync-options documentation (argo-cd.readthedocs.io, stable, read 2026-10-26; the page names no version) says ignoreDifferences is by default used only to compute sync status, and that the option makes a sync pre-patch the desired state so ignored fields are not applied over live ones; it is only effective when the resource already exists. CiliumIdentity objects are created by the Cilium operator and appear nowhere in the chart (no template under install/kubernetes/cilium/templates at tag v1.20.0 mentions the kind), so there is no desired manifest to pre-patch. The reference has no such option and is Synced. gitops.New therefore changed: it adds the option only when an entry names fields (jsonPointers, jqPathExpressions or managedFieldsManagers), which is the case ADR 0015 introduced the pairing for. A kind-only entry gets no option. Still one constructor.

Hubble certificates stay cronJob. Read in the Cilium v1.20.0 chart: hubble.tls.auto.method defaults to helm. The tls-helm templates call genCA/genSignedCert on every render, try to reuse existing material through lookup (which helm template, what Argo CD runs, cannot do because it has no cluster access), and label each Secret cilium.io/helm-template-non-idempotent: "true". So under Argo CD a re-render yields new certificates. The reference runs the default and is Synced; the likely reason is that Argo CD's repo-server caches rendered manifests per revision, so a re-render only happens on a new revision or hard refresh. That reason was not tested. cronJob renders no random material (the Job and CronJob generate certificates in the cluster), so the diff cannot arise and the behaviour does not depend on cache timing. It is also the same value in the bootstrap manifest and the Application. Kept.

Values compared with the reference. All five Talos essentials are already present, with identical content: ipam.mode: kubernetes; kubeProxyReplacement; securityContext.capabilities (ciliumAgent, cleanCiliumState); cgroup.autoMount.enabled: false and hostRoot: /sys/fs/cgroup; k8sServiceHost: localhost and k8sServicePort: 7445 (these two only when kube-proxy is replaced, as in Talos's guide). Nothing was added. Not added: gatewayAPI, l2announcements, bgpControlPlane, Hubble relay/UI/metrics, which are homelab choices.

Chart version. The reference runs 1.20.0; Ichor pins 1.18.0 (ChartVersion). The observation is of a newer chart than the one Ichor installs. The pin is not bumped here: ADR 0013 ties the bump to a review of the values against the new chart. Whether the entry behaves the same on 1.18.0 is checked in the e2e run.

Caveat. Neither the Argo CD docs for ignoreDifferences (diffing guide, read 2026-10-26) nor the observation explain mechanically how a group/kind-only entry affects the result; every documented example names a field, and the page does not say what a kind-only entry does. The decision rests on the observed Synced state, not on a derived mechanism. Ichor's own confirmation is the e2e on a cluster Ichor handed over; until then this is a copy of what worked elsewhere.

Argo CD alone versus Argo CD with Kokumi

Source for the Kokumi column: kokumi-dev/kokumi at commit 4126144 (2026-10-21): internal/argocd/application.go builds the Application as a literal with syncPolicy.automated {prune, selfHeal: true} and syncOptions: ["ServerSideApply=true"] and no ignoreDifferences; no api/v1alpha1 type carries Application fields; applyApplication in internal/controller/serving_controller.go compares the whole spec and updates the Application when it differs, so an edit to the Application is reverted. kokumi.dev's overview page says nothing on the subject. This is a reading of the source of an experimental v1alpha1 project, not a statement of its contract. The CiliumIdentity problem exists with Kokumi too: Argo CD compares the artifact's rendered manifests with live regardless of who made the Application.

What is not decided: whether the argocd-cm route works. Its documented form (resource.customizations.ignoreDifferences.<group>_<kind>) takes jsonPointers, jqPathExpressions or managedFieldsManagers, and a kind-only entry is not documented there, so the equivalent of the Application entry is unknown. Ichor does not write argocd-cm (ADR 0046).

Calico

There is no reference. Calico is phase 11. Its values are decided when it is built, the same way: observe a working Argo CD-managed Calico.

Consequences

• gitops.New no longer adds RespectIgnoreDifferences=true for kind-only entries (ADR 0015 amended).
• The design's Open questions shrinks to Calico and the Kokumi case.
• Chart 1.18.0 versus the 1.20.0 observation is an explicit gap for e2e.

E2E observation (2026-10-26)

Run on a throwaway 3-control-plane Talos v1.14.3 cluster (ichor-e2e harness, ICHOR_E2E_CNI=none), Kubernetes v1.37.1. Cilium chart 1.18.0 was installed from helm template with the values pkg/cni/cilium produces, applied with server-side apply (the stand-in for the inline manifest: today BootstrapManifest returns only the values document, see ADR 0013). Argo CD v3.5.6 core-install.yaml was installed on the inner cluster only. The Application is the one ArgoCDApplication returns, with one added value (tunnelProtocol: geneve, below) in both the install and the Application.

• Before the first sync: OutOfSync, 25 resources, and the only difference in every one is the argocd.argoproj.io/tracking-id annotation (from argocd app diff --core; no other line differed). That includes the Hubble cronJob resources, so the chosen method produces no render difference. The hand-over is "no diff" in content, but not Synced until Argo CD writes its tracking annotation, which needs one sync.
• The Ichor Application has no syncPolicy.automated. Argo CD never syncs it by itself. One manual sync (13:23 JST) succeeded and the Application was Synced/Healthy; the DaemonSet did not change generation and no Cilium pod restarted. It stayed Synced for the 15 minutes watched (13:23 to 13:38; the last reconcile was 13:38:12) with no further operation.
• With automated: {prune: true, selfHeal: true} (as in the homelab) added to the same Application: Synced/Healthy for another 10 minutes (13:40 to 13:50), no new sync operation, no restarts.
• Isolation: with ignoreDifferences removed (checked: spec.ignoreDifferences empty), with selfHeal on: still Synced/Healthy for 5 minutes (13:46 to 13:50) with 9 CiliumIdentity objects present. So on this cluster the kind-only entry was not needed, and nothing was shown about what it does. Consistent with the Context above: the chart renders no CiliumIdentity, and they are cluster-scoped objects Argo CD does not track. The entry is kept (homelab reference). Not tested: a Cilium upgrade, or features that create more operator-owned objects.
• Not exercised: the CNIHandover controller (cluster set to handover: disabled) and a Talos-inlined bootstrap manifest.

Things the run turned up:

• Argo CD core-install.yaml has no default AppProject; the Application reported InvalidSpecError until one was created.
• VXLAN (UDP 8472) between the nodes is dropped on talos-cluster01: Cilium's default tunnel gave 1/3 reachable, pod to service and pod DNS timed out, and Argo CD's repo-server could not resolve anything. Switching to tunnelProtocol: geneve fixed it at once. Cause on the outer network not investigated. Any Cilium e2e on this harness needs it.
• Chart 1.18.0 ran on Kubernetes v1.37.1 here without trouble in this short run.

Addendum (2026-10-26): generated Applications sync automatically

Decided by the user: with Argo CD in use, automated sync is fine; without Argo CD this does not apply. The e2e run above showed that an Application without syncPolicy.automated stays OutOfSync until someone syncs it, so the CNI hand-over (ADR 0024) never completes.

Distinction. "Ichor never touches an Application's sync policy" means Ichor never changes the sync policy of an Application that already exists. An Application Ichor generates (only on the Argo CD / hand-over path) chooses its sync policy once, at creation, in pkg/gitops.New (ADR 0015).

Every generated Application (there are no others; gitops export reuses these two builders, flannel's ArgoCDApplication returns nil):

Prune, per Application (Argo CD v3.5 docs, sync-options and auto_sync):

• CNI: prune: true. Prune removes only resources Argo CD tracks that the pinned chart no longer renders, i.e. after a deliberate chart or value change. prune: false would not stop that change; it would leave the app OutOfSync ("the app will be out of sync if Argo CD expects a resource to be pruned"), which blocks the Synced + Healthy signal the hand-over waits for. The risk that a dropped resource takes the network down is reduced by PruneLast=true (pruning runs as a last implicit wave, after the new resources are healthy). A user can protect any single resource with the argocd.argoproj.io/sync-options: Prune=false annotation, which overrides the policy.
• root: prune: true. It deletes a child Application only when its manifest is removed from Git, which is the intent. Ichor sets no resources-finalizer on children, so deleting a child Application does not cascade-delete the workloads it deployed.

Without --argocd (or with handover: disabled) no Application is generated and nothing changes; TestBuildCNIBootstrap_Cilium and _Flannel check it.