0087. The Kubernetes upgrade runs in three stages: control-plane components, then kube-proxy, then kubelets

Status: Accepted · Date: 2026-10-27 · Design section: Upgrades

Decided: the kubernetes executor visits the machines in four stages, recorded in status.kubernetes.stage: (1) APIServers — each control plane, one at a time, gets kube-apiserver at the target with controller-manager, scheduler, kubelet and kube-proxy pinned to the version its node runs; (2) ControlPlaneComponents — each control plane gets controller-manager and scheduler at the target; (3) KubeProxy — each control plane gets kube-proxy at the target, then the bootstrap manifests are synced and the DaemonSet's rollout awaited; (4) Kubelets — control planes, then workers within maxUnavailable, get the full rendered configuration, which is the cluster's desired state; pkg/render takes the pins as a system-layer input (SystemInputs.Staged) that only the executor sets, nothing is written to spec, status.machines is cleared at each stage change, an operation with no recorded stage starts at APIServers, and the plan names the stages and warns (does not refuse) about a minor upgrade of several control planes behind an endpoint of type node

Amended (hardware, v1.36.5 to v1.37.1): four stages, not three. The first stage below, ControlPlanes, is now two: APIServers (kube-apiserver on every control plane, one at a time) then ControlPlaneComponents (controller-manager and scheduler, one control plane at a time). See "Amendment: apiservers first" at the end; where this text says three stages and ControlPlanes, read that.

Context

ADR 0057 applied a node's control-plane images and its kubelet in one apply, one node at a time. On a minor upgrade the first node's kubelet and kube-proxy were the new minor while the other control planes' API servers were still the old one, which breaks the Kubernetes version-skew policy (a kubelet may not be newer than the API server it talks to). On hardware, v1.36.5 to v1.37.1 behind a single-node endpoint left the upgraded node's components unready and the executor paused after 20 minutes. talosctl upgrade-k8s (Talos v1.14.3, pkg/cluster/kubernetes/talos_managed.go, Upgrade) orders it differently: the three static-pod components on every control plane, then kube-proxy, then the kubelets, then the manifest sync.

Decision

Stages. The backbone (ADR 0026) is unchanged in what it guarantees (lock, pause on failure, maxUnavailable, hold annotation, order label, per-machine status). For this kind kubernetesBatch picks the machines for the current stage and, when a stage has none left, moves to the next one and clears status.machines in the same status write, so a restarted controller resumes from the stage and the machines recorded in it.

So at no step is any kubelet newer than any API server, and the node's final configuration is what TalosMachineReconciler renders, so it sees no drift afterwards. A patch upgrade takes the same path (one code path; the intermediate states are harmless). A held control plane is excluded from every stage.

The intermediate render. render.SystemInputs.Staged (KubeletVersion, KubeProxyVersion) is applied below the patch layers with machinery's own stdpatches.WithKubeletImage and WithKubeProxyImage, so the contract-dependent form (the KubeletConfig and KubeProxyConfig documents from v1.14, the legacy fields before) is machinery's choice, and a patch that sets an image itself still wins. A worker gets no kube-proxy pin (it has no such document). The pinned version is not stored: until the Kubelets stage the kubelet of a node has not been touched, so the executor reads it from the node's Node.status.nodeInfo.kubeletVersion on every call and gets the same answer.

Everything is still re-derived. In each stage a call compares the node's configuration hash with the render for that stage: different means pre-checks, dry run, pre-pull of the changed images, apply; equal means only verify. The images that change are the stage's own, so the kubelet image is pulled in the Kubelets stage only. The skew, Talos pairing, Ready and CNI hand-over checks run at the start of every stage's apply.

kube-proxy and the manifest sync move to the middle stage. ADR 0057 synced from the last control plane. Now the sync runs in the KubeProxy stage, from each control plane once it has the target kube-proxy image (server-side apply is idempotent, so repeating it is harmless); the manifests are generated by the node being asked, which is why each control plane is first given the image. upgrade-k8s syncs after the kubelets; Ichor does it before them because a kube-proxy newer than a kubelet is inside the skew policy and the rollout is verified before any kubelet moves. The mode is read with render.KubeProxyMode, so a switch recorded in status.cni.kubeProxy is honoured.

Plan. PlanOperation returns the three stages in stages and the CLI prints them ("no kubelet is ever newer than an API server"). For a minor upgrade of a cluster with more than one approved control plane whose spec.endpoint.type is node, the plan carries a warning: the endpoint is a single apiserver, so the whole cluster loses the API while that node's apiserver is replaced; use a virtual IP or a load balancer. It is not a refusal. Only a move known to stay within one minor is silent; an unknown target (the spec already says it) warns.

Consequences

• Each control plane receives up to three applies instead of one, and each is an apply without a reboot; the wall-clock time of an upgrade grows by the second and third control-plane pass. Workers still get one.
• TalosOperation.status.kubernetes.stage is a new status field. The UI shows the plan's stages with its existing "passes" wording (written for the CA rotation) and does not show the current stage; a follow-up.
• A single control plane behind any endpoint is staged too: its apiserver restarts first, the kubelet later, with the API unreachable only while the apiserver restarts. That is the case that failed on hardware; this change has not been run there.
• With an endpoint of type node and several control planes, the API stays unreachable for clients using that node's address while its apiserver restarts, in every stage that restarts it; staging does not change that, which is why the plan warns.
• Not verified against a real cluster. Tests use fakes and envtest: they assert the apply order and the rendered versions of each step, that no kubelet is newer than any apiserver after any apply, resume by a fresh reconciler on every pass, and pause on a failed apply in each stage.

Amendment: apiservers first

What happened. On hardware (3 control planes and 1 worker, Talos v1.14.3, v1.36.5 to v1.37.1) the ControlPlanes stage moved the first control plane's apiserver, controller-manager and scheduler together. The v1.37.1 apiserver and controller-manager became Ready; the v1.37.1 scheduler never did (readiness 500 for 20 minutes): failed to list *v1.DeviceTaintRule: the server could not find the requested resource. The v1.37 scheduler needs resource.k8s.io/v1 DeviceTaintRule, which only v1.37 apiservers serve, and the other two apiservers were still v1.36.5. The executor waited for all three pods and paused after 20 minutes.

Why the scheduler saw old apiservers. Talos writes the controller-manager and scheduler kubeconfigs with the secrets controller's LocalEndpoint (internal/app/machined/pkg/controllers/secrets/root.go:118-128, kubernetes.go:136,157): with KubePrism enabled, which is the default, that is https://127.0.0.1:7445, a local load balancer over every apiserver in the cluster; without it, https://localhost:<apiserver port>, the node's own apiserver. So a new scheduler or controller-manager may talk to any apiserver, not only the one next to it, and cannot be assumed to find the APIs it needs until every apiserver is new.

What talosctl upgrade-k8s does (Talos v1.14.3, pkg/cluster/kubernetes/talos_managed.go): Upgrade (line 150) loops over the services kube-apiserver, kube-controller-manager, kube-scheduler in that order, and for each service upgradeStaticPod (line 221) visits every control plane node in turn and waits for that node's pod before the next. So all apiservers move first, then all controller-managers, then all schedulers; then kube-proxy on every control plane (line 156), the kubelets (line 160), and the manifest sync (line 168). Ichor already follows that for kube-proxy and the kubelets; it now follows it for the apiserver.

Decision. The first stage is split:

So at no step is a controller-manager, scheduler or kubelet newer than any apiserver. Controller-manager and scheduler are done together per control plane (upgrade-k8s does all controller-managers, then all schedulers); the stricter rule that matters, no component newer than any apiserver, holds either way, and this keeps the number of applies per control plane at four.

Render. render.StagedKubernetes gains ControllerManagerVersion and SchedulerVersion, applied with machinery's WithKubeControllerManagerImage and WithKubeSchedulerImage below the patch layers like the other pins (none on a worker). The pin is the node's Node.status.nodeInfo.kubeletVersion, read on every call: a node's controller-manager and scheduler run the version its kubelet does until their stage, and the kubelet is untouched until the last one, so the answer is the same on every call and nothing is stored.

Resume. status.kubernetes.stage is one of the four stages; every call compares the node's configuration hash with the render for the current stage, so a restarted controller either applies or only verifies.

Plan and UI. The plan lists four stages; the UI shows "Stage N of 4".

Not verified on hardware. Tests (fakes and envtest) assert the apply order APIServers cp1 cp2 cp3, ControlPlaneComponents cp1 cp2 cp3, KubeProxy, Kubelets, and after every apply that no controller-manager or scheduler is newer than any apiserver and no kubelet is newer than any apiserver.

No compatibility with pre-release stage layouts: Ichor had no release, so the ControlPlanes stage and the unstaged-operation rule are gone; an operation with no recorded stage starts at APIServers.