0009. Reboot necessity is predicted, because Talos will not answer it

Status: Accepted · Date: 2026-10-25 · Design section: 5.6 (configuration reconcile)

Decided: Keep predicting reboot necessity from pkg/fieldmeta's field-path table; Talos itself won't answer the question

Context

pkg/fieldmeta predicts, from a small table keyed by field path, whether a configuration change needs a node reboot. It defaults to reboot-required and carries eight hand-written classifications. Its own author flagged the obvious weakness: those eight are Talos domain knowledge, so a machinery bump cannot tell us when Talos quietly changes the behaviour underneath them.

Reading machinery v1.14.3's API suggested the prediction might be unnecessary. ApplyConfigurationRequest.Mode has AUTO, with REBOOT deprecated in favour of it; DryRun is a bool on the same request; and the response carries Mode ("States which mode was actually chosen"), ModeDetails and Warnings. That reads exactly like Talos offering to answer the question itself, per node and per version, which would be strictly better than predicting it.

It was verified before anything was deleted, and the answer is no.

AUTO is a hardcoded alias for NO_REBOOT. From Talos v1.14.3, internal/app/machined/internal/server/v1alpha1/v1alpha1_server.go:

case machine.ApplyConfigurationRequest_AUTO:
    in.Mode = machine.ApplyConfigurationRequest_NO_REBOOT
    fallthrough

Nothing in that RPC ever sets Mode = REBOOT, and the switch never looks at the diff. ModeDetails is a static string. Confirmed empirically against a real node: three dry runs — one changing only a DNS nameserver, one changing only the installer image tag, one applying a whole worker configuration — returned byte-identical Mode and ModeDetails.

The real decision lives elsewhere. Each subsystem's own controller reacts to the already-written configuration asynchronously and asks for a reboot through a RebootRequest resource, which a separate controller consumes. There is no point in the API where Talos summarises that into "this change needs a reboot".

Decision

Keep pkg/fieldmeta, and keep predicting. Its table is the only mechanism that can answer the question where Ichor needs it answered.

Two places need it, and neither can dry-run:

• A validating webhook on TalosConfigPatch, at admission time. There may be no node to ask: the cluster may not be bootstrapped, the target machine may not exist, and admission has to answer synchronously anyway.
• The reconciler, before it applies. Design 5.6 step 4 requires taking the cluster lock, running the node-operation hook and draining before a reboot-causing apply. Learning the answer from Talos after applying is too late by construction.

So the design's wording — the mode is derived from the diff — stands, and now has evidence behind it rather than being the only idea anyone had.

Two findings from the same investigation are worth acting on separately:

• Warnings on the apply response is genuinely useful, just not for this. It comes from ValidateAtRuntime and surfaces deprecated-field migration hints, missing cross-document configuration (etcd encryption on a control plane, a cluster ID or secret missing when discovery is on) and KubeNodeConfig label mismatches. The reconciler should surface it in status rather than dropping it.
• Dry run is genuinely side-effect free. Verified: a node's MachineStatus version and timestamp were unchanged after a batch of dry runs, not even rewritten with the same value. So it is safe to use for the impact preview's validation half, even though it cannot answer the reboot half.

Consequences

• The eight hand-written entries stay, with the risk their author named: a Talos behaviour change will not announce itself. The mitigation is the default — anything not in the table is reboot-required — so the failure mode of a stale table is an unnecessary reboot, not a node left running a configuration it does not have.
• Whenever the pinned Talos version moves, those eight are worth re-reading against the release notes. That is a cheap, bounded review, and better than the alternative of trusting an API that does not answer.
• The reconciler should pass DryRun first for validation and warnings, then apply for real — two calls, with the mode decided by the table in between.

A note for the e2e harness, found along the way

A real apply of a full worker configuration to a node in maintenance mode starts Talos's UnattendedInstallController, which installs to disk and then asks for a reboot through RebootRequest. On the e2e harness both test nodes went unreachable at that point — TCP resets rather than timeouts, so the kernel was up with nothing listening — and did not return within the test window. Cluster pods were confirmed to have egress to factory.talos.dev, so it is not obviously a network block, and it was not root-caused.

Resolved. It was not the install: Talos's certificate controller blocks apid on a reachable control-plane endpoint for worker nodes only, so an isolated worker never starts apid at all. A control-plane config comes up normally. The details, and the four other harness requirements the same investigation turned up, are in ADR 0007.

Addendum: multi-document and registry paths

The table originally knew only the legacy Config paths and two standalone kinds, so every other path fell to reboot-required. Ichor's own render hit that: adding TalosCluster.spec.registries was refused with RebootRequired (seen in ADR 0061's test). The v1.14.3 source was read to separate what Talos applies live from what it does not. A rule was added only where the consumer was found; the default is unchanged.

Ichor's default render at the v1.14 contract (control plane and worker) is the legacy Config document (machine.*, cluster.*) plus the standalone kinds DiscoveryServiceConfig, DiscoveryIdentityConfig, ResolverConfig, VolumeConfig (EPHEMERAL), FilesystemTrimConfig, SecurityProfileConfig, HostnameConfig, KubeClusterConfig, KubeNodeConfig, KubeletConfig, KubeNetworkConfig, KubeAPIServerCAConfig, KubePrismConfig, and on control planes also KubeAggregatorCAConfig, KubeAdmissionControlConfig, KubeAuditPolicyConfig, KubeAuthenticationConfig, KubeAuthorizerConfig, KubeEtcdEncryptionConfig, KubeServiceAccountConfig, KubeAPIServerConfig, KubeControllerManagerConfig, KubeSchedulerConfig, KubeProxyConfig, KubeFlannelCNIConfig, KubeCoreDNSConfig. Patches add SysctlConfig, KernelModuleConfig, VolumeConfig, UserVolumeConfig, RawVolumeConfig and legacy Config fragments. Registries are injected into the legacy machine.registries, not into a RegistryMirrorConfig document, so the path that mattered was machine.registries.mirrors.<host>.endpoints.

Mode handling, for reference: ApplyConfiguration in internal/app/machined/internal/server/v1alpha1/v1alpha1_server.go writes the config in NO_REBOOT and never inspects the diff; whether anything needs a reboot is decided by the controllers that consume the resource. Paths below are relative to the siderolabs/talos module at v1.14.3 (pkg/machinery/... is the machinery module).

(controllers/ is internal/app/machined/pkg/controllers/.)

The Kubernetes component images are deliberately not no-reboot. KubeAPIServerConfig.image, KubeControllerManagerConfig.image, KubeSchedulerConfig.image, KubeProxyConfig.image and KubeletConfig.image would apply live in Talos, but a safe rule makes the machine controller apply all nodes the moment spec.kubernetesVersion changes, which breaks the one-node-at-a-time order ADR 0057's executor provides (it applies the image fields itself with an explicit no-reboot apply). The explicit .image entries in pkg/fieldmeta carve them out of their document's rule, and a test pins all five. The legacy cluster.apiServer.image, cluster.controllerManager.image and cluster.scheduler.image are still covered by the older cluster.* safe rules, so on a pre-1.14 contract the same hazard exists; it predates this addendum and is left for its own decision.

Read but not verified, so left at the default: KubeProxyConfig (the open question about the kube-proxy DaemonSet stands), KubeletConfig fields other than the image and machine.kubelet.* (kubelet restart behaviour not traced), KubeNodeConfig.nodeIP, ResolverConfig.hostDNS, HostnameConfig, DiscoveryServiceConfig/DiscoveryIdentityConfig, SecurityProfileConfig, KubeClusterConfig, KubeNetworkConfig, the CA, admission, audit, authentication, authorizer, etcd-encryption, service-account, Prism, Flannel and CoreDNS kinds, VolumeConfig/FilesystemTrimConfig beyond provisioning. Ichor's inputs never vary most of these after the first apply.

Addendum: an unchanged hostname needs no reboot

fieldmeta stays path-only, so HostnameConfig.* and machine.network.hostname are reboot-required. assessChange (internal/controller/hostname_noop.go) drops those paths when spec.hostname equals the node's current Nodename, so the release that began rendering spec.hostname does not demand a reboot of every existing node; a real rename stays reboot-required and its message says the Kubernetes node keeps its old name until the reboot.

Addendum: registry auth and TLS apply without a reboot

The first addendum kept registry auth at reboot-required because machinery's doc comment on RegistryConfig.RegistryAuth (pkg/machinery/config/types/v1alpha1/v1alpha1_types.go:2058) says "changes to the registry auth will not be picked up by the CRI containerd plugin without a reboot". The v1.14.3 machined source shows that comment is stale. Auth is written into the CRI config file (GenerateCRIConfig, containerd.go:24-50), which containerd reads only at start; but ServiceController.reconcile (internal/app/machined/pkg/controllers/cri/service.go:84-112) restarts the CRI service whenever the SpecVersion of that file (or the base runtime spec) changes, and CRIConfigController (controllers/files/cri_config.go:139-180) re-renders it on every registries change. Running pods keep running while containerd restarts. TLS and mirrors become hosts.toml files and need no restart at all. So machine.registries.config, RegistryAuthConfig, RegistryTLSConfig and RegistryMirrorConfig are now safe (the legacy host key contains dots, so the rule is the whole subtree). Unverified: Talos versions older than v1.14.3 were not read; Ichor targets v1.14.