0057. The Kubernetes upgrade applies the rendered configuration node by node; talosctl's upgrade-k8s is not reused whole

Status: Accepted · Date: 2026-10-26 · Design section: Upgrades Decided: Execute kind: Kubernetes as a per-node AdvanceMachineFunc that applies the freshly rendered machine configuration without a reboot (only the component image fields may differ), pre-pulls the images, verifies the node from the Kubernetes API, and syncs the bootstrap manifests once the last control plane runs the target; it refuses a skipped minor, an unsupported Talos pairing and an unfinished CNI hand-over before touching a node, and only warns about removed APIs in use

Context

The design's "Upgrades / Kubernetes" lists: one minor at a time, warn about removed APIs, pre-pull images, switch the control-plane components node by node through machine config, update kube-proxy (skipped when replaced), roll the kubelets, sync the bootstrap manifests, flag outstanding hand-overs first, and says reusing talosctl upgrade-k8s is the first thing to evaluate. ADR 0026 supplies the lock, the status-held state machine, pause on failure, maxUnavailable, hold annotations and ordering; ADR 0043 is the Talos-kind executor this follows.

Evaluating upgrade-k8s (Talos v1.14.3 source)

• cmd/talosctl/cmd/talos/upgrade-k8s.go builds a kubernetes.UpgradeOptions and calls pkg/cluster/kubernetes.Upgrade (talos_managed.go). That package is public, but Upgrade is one blocking call over every node: it takes an UpgradeProvider (a talosctl cluster client with a kubeconfig), loops upgradeStaticPod over all control planes per component, then kube-proxy, then kubelets, then PerformManifestsSync, and keeps its progress in goroutine and watch state. It cannot resume after a crash from TalosOperation status, and it cannot be split into the backbone's per-machine steps.
• It changes configuration with patchNodeConfig (patch.go): read the node's active config, patch an image field, ApplyConfiguration in NO_REBOOT mode. Ichor's controller renders each machine's whole configuration from the CRs and treats a node that differs as drift (design step 5), so a patch made behind its back would be contradicted by the next reconcile. Two findings from there carry over: every Kubernetes component image change applies with no reboot (the same call, NO_REBOOT, for apiserver, controller-manager, scheduler, kube-proxy and kubelet), and the pre-pull uses the ImagePull RPC in the CRI namespace for the static pods and the system namespace for the kubelet (prePullImages).
• The removed-API pre-check (go-kubernetes/kubernetes/upgrade/checks.go, a different module, ADR 0022) keeps a per-minor table of removed flags, feature gates and API resources that someone has to extend for every Kubernetes release. Most of what it checks (flags and gates on the static pods) cannot be set through Ichor's typed configuration in the first place.
• go-kubernetes's ssa.Manager and manifests are reusable as a library, but adding the module moves k8s.io/api, apimachinery, client-go, cli-runtime, component-base, kubectl from v0.36.0 to v0.36.2 (tried with go get and reverted) for every package in the repository, to save about forty lines.

So: not reused as a whole, and not as a dependency. What is reused is what it proves: the apply mode, the pre-pull RPC and namespaces, the order (control planes, then workers) and the bootstrap-manifest sync as the last cluster-wide step.

Decision

The per-node switch is "apply the rendered configuration". pkg/render already derives every component image from spec.kubernetesVersion; for a control plane the diff against the node is exactly KubeAPIServerConfig.image, KubeControllerManagerConfig.image, KubeSchedulerConfig.image, KubeProxyConfig.image and KubeletConfig.image (a worker: KubeletConfig.image), measured with ChangedPaths on a render at two versions. Nothing hand-writes a component switch.

The existing apply path cannot do it by itself, and that is useful. pkg/fieldmeta has no rule for these paths (it only knows the legacy cluster.apiServer.* prefixes, and cluster.proxy and the kubelet are deliberately absent), so TalosMachineReconciler classifies a version change as reboot-required and refuses it on every node (RebootRequired). That keeps the roll gated by the operation: no node changes until the executor's turn, one control plane at a time. The executor applies with NO_REBOOT itself, accepting a changed path only if it is one of the image paths above or fieldmeta.Assess says it is reboot-free and not install-time-only; any other pending change fails the step ("unrelated pending configuration change") instead of riding along. fieldmeta is not edited: a rule there would let the machine controller apply the roll on every node at once.

Per call, everything is re-derived (the ADR 0043 discipline). If the node's configuration hash differs from the render, the step has not been applied: pre-checks, dry run, pre-pull of the changed non-proxy images, apply, and the KubernetesUpgrade condition (a resume marker and the verification clock, not a source of truth). If they are equal it only verifies: the Node's kubelet version and Ready, and for a control plane its three static pods (mirror pods in kube-system, found by k8s-app label and node) on the target image tag and Ready. Not yet true is "still settling"; past 20 minutes from the marker the step errors and ADR 0026 pauses the operation.

Pre-checks (before any node is changed; failures pause with a message):

1. upgrade.KubernetesSkew(node's kubelet version, target): first caller. Judged per node against what that node actually runs, so a skipped minor, a downgrade, or a node left behind is refused.
2. upgrade.KubernetesPairing(target, the node's running Talos version).
3. The node is Ready.
4. CNI hand-over: block, as an error. The bootstrap-manifest sync re-applies the cluster's inline manifests; while Argo CD is taking the CNI over and Ichor still renders the inline entry (the CNIHandover condition is not True, ADR 0024), that rolls the CNI back, which can take the network down: not something to warn about and continue. Blocked: preset with a hand-over (cilium, via the same cniPresetFor/ArgoCDApplication decision the hand-over controller makes), handover automatic or manual, condition not True. Allowed: no CNI hand-over (flannel, custom), condition True, and handover: disabled (nothing ever takes over, so the inline manifest is the source of truth and re-applying it is the intended state). It is an error rather than a wait so a blocked operation does not hold the cluster lock indefinitely; the operator finishes the hand-over (or sets disabled on a cluster without Argo CD) and creates the operation again.
5. Removed APIs: warn, not gate. The API server's own apiserver_requested_deprecated_apis metric reports, per requested API version, the release that removes it (pkg/upgrade.RemovedAPIsInUse), so there is no table to maintain. APIs removed by the target minor or earlier that something requested are recorded in the RemovedAPIsInUse condition on the TalosOperation; a failed check is recorded as Unknown. Limits: it sees only APIs requested since that API server started, and one API server answers the scrape.

Bootstrap manifests and kube-proxy run once, in the control plane whose verification finds every kube-apiserver pod on the target and Ready (observed state, no control-plane list, so countplanes_guard_test.go is untouched): KubernetesAdmin.SyncBootstrapManifests server-side-applies the node's k8s.Manifest resources under Talos's own field manager (talos) through an admin kubeconfig from the Talos API. The controller's own ServiceAccount therefore gets no write access to ClusterRoles, DaemonSets or CRDs; it gains get on DaemonSets, and get;list on Pods and Nodes it already had. When spec.cni.kubeProxy is replaced nothing about kube-proxy is checked (no DaemonSet is shipped); otherwise the step waits for the kube-proxy DaemonSet to carry the target and finish rolling out. This is before the workers, not after the kubelets as in upgrade-k8s; kube-proxy may be newer than a kubelet but not than the API server. ponytail: no pruning and no inventory, unlike ssa.Manager; an object a later Talos stops shipping stays until removed.

A rendering bug found on the way (fixed in pkg/render): spec.kubernetesVersion is vX.Y.Z (the CRD pattern) but generate.NewInput expects it without the v and adds its own, so every component image rendered as ...:vv1.36.0. The v is now trimmed before the call, with a regression test.

Consequences

• TalosClient gains PullImage, Kubeconfig and BootstrapManifests; KubernetesAdmin is the seam for the two cluster-wide actions, with a fake in the tests. The real admin client (kubeconfig, SSA, /metrics) is covered by the repository's tests only through the pure parser; it has not been run against a cluster.
• upgrade.KubernetesSkew has its first caller; KubernetesMinorOf and RemovedAPIsInUse are new.
• pkg/fieldmeta still lacks reboot rules for the multi-document Kubernetes component kinds. Any machine whose rendered config changes one of those kinds outside an operation is reported RebootRequired; left as is on purpose (above).
• Amended: every kubelet roll (the Kubelets stage, ADR 0087) now begins storage evacuation, cordons and drains the node with the config rollout's helpers, rolls the kubelet, waits for Ready, uncordons and ends the evacuation; a failure pauses and ends the evacuation too. The controller's own node goes last in the Kubelets stage (holdOwnNodeLast, as for config rollouts) and is drained and evacuated like the rest; its eviction restarts the controller, and the step resumes from the node's state.
• Deferred: the Ichor kind is still unwired.

E2E observation (2026-10-26, through ichor upgrade kubernetes)

e2e harness, 3 control planes + 1 worker, Talos v1.14.3, Cilium 1.18.0 (handover: disabled), controller run in the cluster. No newer patch than v1.37.1 exists on registry.k8s.io, so the cluster booted at v1.36.4 and the target was v1.36.5. The cluster's configs were rendered by Ichor itself (all four machines Converged, i.e. zero drift) at versionContract v1.13: at v1.14 the typed patches cannot express pod/service subnets, the install disk or "no flannel" (those are separate documents, rejected when combined with the legacy Config fields).

• The confirmation text appeared as designed: "Upgrade Kubernetes on cluster ... from v1.36.4 to v1.36.5 ... 4 machine(s) (3 control plane, 1 worker): control planes one at a time, then workers (at most 1 at once), in this order: ...". Answering n aborted with nothing written.

• The vv fix holds: after the first control plane, kube-apiserver, kube-controller-manager and kube-scheduler ran registry.k8s.io/...:v1.36.5, the kubelet reported v1.36.5, the three pods were Ready.

• Bug (not fixed here; internal/controller belongs to another change): the step fails with rpc error: code = DeadlineExceeded from ApplyNoReboot. begin runs the version query, the removed-API scrape, the dry run and the apply all under the one 15 s etcdDialTimeout context (dialCtx). Two runs, the first failed on the first control plane (nothing changed on the node), the second succeeded on it and failed on the second. Each leaves the operation Paused; a failed machine is not retried (ADR 0026), delete and start again. The apply of a control-plane image change blocks while Talos restarts the static pods, the same shape as the upgradeCallTimeout bug in ADR 0043. Needs its own bound for the apply. Stopped after the second failure; the cluster was finished with talosctl upgrade-k8s, which is outside the product.

• Not observed: the bootstrap-manifest sync and the worker step (the operation never got past the second control plane).

• Fixed: each Talos call in begin (and the kube-proxy switch and CA rotation applies) now has its own deadline, the apply bounded by upgradeCallTimeout.

• Addendum: the executors write the machine condition through a re-read with retry on conflict (setMachineCondition), so the machine reconciler writing the same status during the long calls no longer pauses the first machine. Finding, not fixed: this executor applies the control-plane images and the kubelet of one node together, where upgrade-k8s upgrades every control plane first and the kubelets afterwards, so on a minor upgrade the first node's kubelet is newer than the apiservers it reaches through the endpoint (see the report on endpoint type node).

• Addendum (ADR 0087): fixed. The executor now runs in four stages (apiservers, then controller-manager and scheduler, then kube-proxy and the bootstrap-manifest sync, then kubelets), so the manifest sync no longer waits for "the last control plane" and no kubelet is ever newer than an API server.

• Amendment (ADR 0113): the target of a Kubernetes upgrade is the release set's Kubernetes, and other targets are refused before anything is written. The rollout runs this step after the Talos step, and after any configuration step the executor needs for an unrelated pending change. The executor's pairing and skew checks are unchanged.