# 0066. The kube-proxy switch is a TalosOperation of kind cni; Ichor changes the Talos side and removes kube-proxy, and the CNI's owner changes the CNI Status: Accepted · Date: 2026-10-26 · Design section: CNI Decided: Add `TalosOperation` kind `cni` whose per-machine step applies the Talos-side kube-proxy mode (rendered by `pkg/render` from `status.cni.kubeProxy`, falling back to `spec.cni.kubeProxy`); replacing waits for Cilium to serve Services before touching any node and deletes the kube-proxy objects last, enabling brings kube-proxy up and verifies it before asking Cilium to step aside; Ichor never writes the CNI's configuration (hand-over or not) and leaves node-level kube-proxy rules to the next reboot ## Context The design (CNI) requires the `spec.cni.kubeProxy` switch to run new side up, verify, old side down, and says Ichor must delete kube-proxy itself. ADR 0013 added the admission guard and noted that nothing executed the sequence. ADR 0026 is the backbone (lock, status-held state machine, pause-on-failure), ADR 0057 the Kubernetes executor, ADR 0024 the hand-over. ## Decision 1: what Talos does (Talos v1.14.3 source) - `internal/app/machined/pkg/controllers/k8s/control_plane.go` (around line 447) copies `K8sProxyConfig().Enabled()` into `BootstrapManifestsConfig.ProxyEnabled`; `manifest.go` (line 238) renders the `10-kube-proxy` manifest only `if cfg.ProxyEnabled`: the ConfigMap `kube-proxy-` (labels `k8s-app=kube-proxy`, `tier=node`, only when a proxy config is set), the DaemonSet, ServiceAccount and ClusterRoleBinding, all named `kube-proxy` (`k8stemplates/kube-proxy.go`). It is the same manifest path the Kubernetes upgrade syncs (ADR 0057). - `manifest_apply.go` has no delete (or prune) call at all. Enabling therefore creates the DaemonSet; disabling only stops Talos managing it. Confirmed: the design text and `grep` for Delete/Destroy/Prune in that file. - At the `v1.14` contract the setting is the `KubeProxyConfig` document's `enabled` field (default mode `nftables`), and the legacy `cluster.proxy` block cannot coexist with that document (validation error "can't be used with KubeProxyConfig document"). `pkg/typedconfig` has no `KubeProxyConfig` kind, and the Cilium preset's `TalosPatches` emits the legacy block, so a replaced cluster at the v1.14 contract failed to render (reproduced in `pkg/render/kubeproxy_test.go`). Nothing in Talos clears the rules kube-proxy programmed. ## Decision 2: what represents the switch A `TalosOperation` of kind `cni` (enum value added). It needs the cluster lock because the Service data path can drop during a switch, and a restart must resume from status: both are the backbone's. The per-machine step is "apply the rendered configuration without a reboot", exactly the shape of ADR 0057, so the control planes go one at a time and then the workers. `pkg/render` now owns the Talos side. `status.cni.kubeProxy` (new, written only by the executor) wins over `spec.cni.kubeProxy`; replaced renders disabled, and enabled renders an explicit flag only once a switch set the status, so a cluster that never switched keeps its configuration hash. From the moment Ichor owns the value (the cluster replaces kube-proxy, or a switch set the status) `cluster.proxy.disabled` from patches is dropped (as the hand-over drops the inline manifest); an enabled cluster that never switched renders exactly as before, at the contract-appropriate form: the `KubeProxyConfig` document from v1.14, the legacy block before. `fieldmeta` classifies the change as reboot-required, so the machine controller refuses it on every node and only the operation applies it, one node at a time. ## Decision 3: the order Replacing (`enabled` to `replaced`): 1. Cilium's live `kube-proxy-replacement` is true and its DaemonSet is fully rolled out. Until then no node is touched and no status changes. 2. `status.cni.kubeProxy` is set to replaced; each machine is switched (`enabled: false`), control planes first. 3. After every machine of the operation is done: delete the DaemonSet, ServiceAccount, ClusterRoleBinding and the `kube-proxy`-named ConfigMaps (through the admin kubeconfig, like ADR 0057), then confirm the DaemonSet is gone. A control plane excluded by the hold annotation fails the step instead (it would re-apply what is deleted). Enabling (`replaced` to `enabled`): 1. `status.cni.kubeProxy` is set to enabled and the control planes are switched; Talos creates the DaemonSet. 2. When all machines are done, the kube-proxy DaemonSet must exist and be fully ready. 3. Only then does the operation wait for Cilium to be switched off (`kube-proxy-replacement` not true, agents rolled out). Cilium stays the owner of that change. ## Decision 4: Ichor never writes the CNI's configuration Before the hand-over Cilium's values are an entry in a user-owned `TalosConfigPatch`, after it they are in Git (ADR 0024); the controller never mutates `spec`. So Ichor does not edit either. The operation sets `AwaitingCNIChange` on itself with the exact values and where they go (the inline-manifest entry before the hand-over, the Application's Helm values in Git after), and waits up to 24 hours (a pull request may be in review). The UI and CLI open that change (PR mode, ADR 0056); that is their work, not this executor's. ## Decision 5: how "the new side works" is judged Replacing: `cilium-config`'s `kube-proxy-replacement` through `Preset.CheckConsistency`, plus the `cilium` DaemonSet with `observedGeneration` current and every pod updated and ready. Enabling: the kube-proxy DaemonSet exists with every pod updated and ready. Each wait is `VerifyingNewSide` (30 minutes); past its clock the step fails and the operation pauses (ADR 0026), never retried automatically. A preset whose live state Ichor cannot read (`custom`) is refused before anything changes. ponytail: no probe of an actual Service connection from a pod, and no per-agent `cilium status`; the agents rolling out on the changed ConfigMap is the proxy for it. A synthetic probe needs a workload Ichor does not run. ## Decision 6: node rules Left alone. kube-proxy's rules (nftables by default) remain until the node reboots, which clears them on Talos (ephemeral); Cilium's documented manual cleanup is `iptables-save | grep -v KUBE | iptables-restore` on each node, which Talos has no shell for. `kube-proxy --cleanup` would need a privileged per-node Job. Deferred (`ponytail:` in `RemoveKubeProxy`): the stale rules are bypassed by Cilium's socket load balancing. Amended: after the removal the executor connects to the `kubernetes` Service's cluster IP from the controller (`KUBERNETES_SERVICE_HOST`) and reports done only once that answers; until then it waits under the verification clock (`VerifyingNewSide`, 30 minutes) with a message that says the old rules stay on each node until it reboots. ## Consequences - Existing replaced clusters at the v1.14 contract render again; the legacy block in their patches is ignored. - Applying `KubeProxyConfig.enabled` without a reboot is assumed from the controller reading the config live (as ADR 0057 does for images). Confirmed on hardware for the config apply (no reboot), but not for the DaemonSet appearing: see the amendment at the end. - The operation is created by hand (or by the UI/CLI); the admission guard (ADR 0013) still needs the annotation on the spec change. ## E2E observation (2026-10-26, through `ichor cni kube-proxy replaced`) Same cluster as ADR 0057's observation (Cilium 1.18.0 with kube-proxy, `handover: disabled`, legacy `cluster.proxy` form because the contract was v1.13, so the v1.14 `KubeProxyConfig` document path was not run). - Confirmation text: "Switch kube-proxy on cluster ... from enabled to replaced. Configuration is applied without a reboot; the kubelet and control plane components restart on each machine. 4 machine(s) ...". The annotation was not left on the TalosCluster afterwards. - The operation sat in `AwaitingCNIChange` ("live kube-proxy-replacement is false ... set kubeProxyReplacement: true, with k8sServiceHost: localhost and the KubePrism port ...") and touched no node until Cilium was changed. I applied the chart's `kubeProxyReplacement: true` values by server-side apply (the stand-in for editing the inline entry, as there was none here). Within about 15 s the operation reported "the Cilium agents are rolling out" and then started on the first control plane. - A cluster with a control plane still on the old Kubernetes version failed the step with "unrelated pending configuration change at cluster.apiServer.image ... converge it first", as designed (my cluster was half upgraded; see ADR 0057). - After converging: all four machines switched (Talos `cluster.proxy.disabled: true` on every node, 4 s for the three control planes already applied), `cilium-config` `kube-proxy-replacement: true`. - **Bug: the operation ends `Paused` on the worker**, the last machine, with `removing kube-proxy: fetching an admin kubeconfig from the Talos API: rpc error: code = Unimplemented desc = kubeconfig is only available on control plane nodes`. The post-switch deletion of the DaemonSet and its objects is run through whichever machine finishes last; a cluster with workers always ends on a worker. It needs to use a control plane for the admin kubeconfig. I removed the DaemonSet and ServiceAccount by hand and then checked: from a pod, the cluster DNS ClusterIP (`10.249.0.10`) answered and a TCP/TLS connection to the `kubernetes` Service ClusterIP was established. - Not observed: `replaced` to `enabled`; the "values in Git" wait after a hand-over. - Fixed: the removal goes through a control plane's Talos client when the last machine is a worker, and deletes the DaemonSet, ServiceAccount, ClusterRoleBinding and every `kube-proxy` or `kube-proxy-` ConfigMap by name. ## Amendment: one-shot, Git-friendly confirmation On an Argo CD-tracked cluster the Git change that switches `spec.cni.kubeProxy` was rejected by the admission policy (ADR 0013) because the confirmation annotation was missing, so the tester committed the annotation into Git where it stayed as a standing consent. The annotation value now names the target mode (`replaced` or `enabled`) and the policy admits a switch only when the value equals the new `spec.cni.kubeProxy`. A leftover annotation therefore cannot admit a different later switch, and it is safe to commit for the one switch. The server sets it to the target for the one write it makes and removes it again (it used to write an actor name). For a tracked cluster `ichor cni kube-proxy ` prints the exact YAML to commit (spec field plus annotation) and says to run the same command again, which finds the spec already changed and starts the operation; the old text said "start the operation without a target", which was wrong. ## Amendment (2026-10-28): Talos never recreates kube-proxy after Ichor removed it; Ichor restores it when enabling Observed on hardware, `replaced` to `enabled`: the configuration applied without a reboot (`cluster.proxy.disabled: false`), but the kube-proxy DaemonSet was never created and the operation sat in `VerifyingNewSide` ("the kube-proxy DaemonSet has not been created yet") for over 19 minutes. The `talos-bootstrap-manifests-inventory` ConfigMap in kube-system still listed the old kube-proxy objects. Cause, in Talos v1.14.3 `internal/app/machined/pkg/controllers/k8s/manifest_apply.go` (`ManifestApplyController.apply`): for every object of the rendered manifests it does `if inv.Contains(objMeta) { continue }` before it looks at the cluster, and it never removes an entry from the inventory for an object that disappeared from the cluster (only the v1.13 namespaced-cluster-wide migration removes entries). The controller also only runs when its inputs change (the Kubernetes secrets, the `k8s.Manifest` resources, the etcd service), and it only creates, never deletes. So an object Talos applied once, which Ichor deleted when it replaced kube-proxy, is skipped forever: the apply is not tied to a node being the bootstrap node, nor to a kubelet or apid event; the inventory alone decides. (`ManifestController` does render `10-kube-proxy` again once `cfg.ProxyEnabled` is true; it is the apply that skips it.) Decision: when enabling, and only while the kube-proxy DaemonSet is missing, the cni executor restores the objects itself (`KubeProxyAdmin.RestoreKubeProxy`). It reads the `k8s.Manifest` resources Talos rendered on a control plane (`TalosClient.BootstrapManifests`, the `talosctl get manifests` equivalent), keeps only the kube-proxy objects (the DaemonSet, ServiceAccount and the `kube-proxy` or `kube-proxy-` ConfigMaps in kube-system, the ClusterRoleBinding `kube-proxy`) and server-side-applies them under Talos's own field manager, through the admin kubeconfig a control plane issues (a worker's step goes through a control plane's client, as for the removal). It is idempotent, bounded by the existing 30-minute `VerifyingNewSide` clock, and runs again on each requeue only while the DaemonSet is absent; if Talos has not rendered the manifest yet it applies nothing and the wait continues. This restores Talos's own component from Talos's own rendering; the inventory is left as it is (editing Talos's bookkeeping would not wake its controller, and the entries are correct again once the objects exist). Rejected: editing the inventory ConfigMap, because Talos would still need an input event to re-run the apply. ## Amendment (2026-10-28): safe unrelated pending paths ride along with the switch The executor refused any pending path outside `KubeProxyConfig` / `cluster.proxy`, which paused a switch with no way out when a safe leftover (the CNI's `cluster.inlineManifests` removal, ADR 0024 amendment) was pending. It now classifies each unrelated path with `fieldmeta.Assess` and applies those that are safe and need no reboot together with the switch. Unrelated reboot-requiring, install-time-only, API-server-restarting or version-unsupported paths are still refused with the "converge it first" error. ## Amendment (2026-10-28): a spec-first change reports KubeProxySwitchPending, not RebootRequired On a Git-tracked cluster the user commits `spec.cni.kubeProxy: replaced` (and the consent annotation) before running `ichor cni kube-proxy replaced`. Until the switch sets `status.cni.kubeProxy` the render follows spec, so the control plane differed from its node at `KubeProxyConfig.enabled` and showed `Ready=False/RebootRequired` with "start a config rollout". That points at the wrong action: `ichor config apply` would reboot the node onto the new mode and skip the switch's checks. The reverse direction never showed it because status already overrides the render. Chosen: report it differently, keep the render. The alternative, holding the render at the current mode, needs a "current mode" when `status.cni.kubeProxy` is unset, and nothing records one: a fresh cluster created `replaced` and a running cluster edited spec-first look the same from the TalosCluster, and the fresh one must render from spec. Guessing from the machine's applied hash would make the render depend on node history. - The signal that a switch is pending, in either direction, is the consent annotation (`ichor.maesh.dev/confirm-kube-proxy-switch`) naming the spec's mode while `status.cni.kubeProxy` is unset or differs. Spec alone cannot tell a switch from a patch-made `cluster.proxy.disabled` change, and a first version that keyed on "spec says replaced, no status" missed the reverse (a cluster created `replaced`, edited spec-first to `enabled`). The admission rule already requires the annotation for any switch, so Git-first commits carry it. - While that holds, a TalosMachine whose only reboot-class paths are `KubeProxyConfig*` / `cluster.proxy*` (and no API server restart on a control plane) reports `Ready=False/KubeProxySwitchPending`: "run `ichor cni kube-proxy replaced` (or the UI's kube-proxy switch) to apply it without a reboot". It does not set `status.rebootPaths`, so `ichor config apply` and the rollout planner (which select on those paths) do not pick the machine up. - A machine that also has other reboot-class paths keeps `RebootRequired` and is selectable, so the rollout step (`configRolloutAdvancer.begin`) refuses any machine with a kube-proxy path pending in this state ("run `ichor cni kube-proxy ` first"). A rollout therefore never reboots a node onto a kube-proxy mode, bypassing the switch's Cilium check and ordering. Once the switch sets the status the guard no longer applies. - Without the annotation (a patch edit of `cluster.proxy.disabled`, an annotation for the other mode, or a finished switch whose status equals the spec) nothing changes: the change stays `RebootRequired` as before. ## Amendment (2026-10-29): only the operation changes Cilium's kube-proxy mode; the consent annotation is removed after the operation exists Observed on hardware (replaced, through the Cilium Helm Job): writing `spec.cni.kubeProxy=replaced` alone made Cilium run `kube-proxy-replacement: true` while kube-proxy still ran, with no operation and no guard. Cause: the hand-over reconciler (ADR 0099 unit 8) hashed the effective values with `spec.cni.kubeProxy`, which derive `kubeProxyReplacement` and `k8sService*`, saw a new hash and ran the `ichor-cni` Job at once. The operation then found nothing to wait for and completed in about a second. - The hand-over reconciler, the upgrade operation and every other Cilium Helm Job outside a switch now take the mode Cilium runs now (`currentKubeProxyMode`): `cilium-config`'s `kube-proxy-replacement`; when there is no `cilium-config` yet, `status.cni.kubeProxy`, then the spec (nothing was switched, the bootstrap rendered it). They never read the spec's mode for a changed value. `status.cni.kubeProxy` alone is not the signal, because enabling sets it first (kube-proxy must come back before Cilium steps aside), so the reconciler could otherwise run the Cilium Job ahead of kube-proxy; `cilium-config` is only ever changed by the switch's own Job. - Only the switch executor applies the target mode: `requireCilium` passes it to `AdvanceCNIHelmJob`, which names the Job by the hash of the target values and gives it `--kube-proxy=` (`ichor cni helm apply` reads the mode from the flag, not the spec). Both Cilium-first (replaced) and kube-proxy-first (enabled) ordering therefore run under the operation's lock and verification, and the `VerifyingNewSide` wait shows while the Job runs. - No fight: once the switch has changed `cilium-config` and `status.cni.kubeProxy`, the reconciler's hash is the target mode's, equal to the switch Job's, so it adopts that Job and records `appliedValuesHash`; it never runs a Job that reverts it. While the operation holds the cluster lock the reconciler waits anyway. - The consent annotation is removed by the server after the operation is created, not before: the spec write (with the annotation) is retried on a conflict with a fresh read, then the operation is created, then the annotation is removed (also retried). A conflict can no longer leave a changed mode with no operation. The removal happens for every `cni` start, including one whose spec was already changed (a retry, or a Git-first change that carried the annotation), which is the case where it used to be left behind.