0125. Kubelet serving certificates are requested by the kubelets and approved by the controller

Status: Accepted; built; hardware verification pending · Date: 2026-11-04 · Design section: Security · Touches: 0101, 0108

Decided: TalosCluster.spec.security.kubeletServingCertificates (enabled, verifyFromAPIServer, both off by default, for new and existing clusters) makes every kubelet request its serving certificate from the kubernetes.io/kubelet-serving signer (serverTLSBootstrap), and Ichor's controller approves those requests itself, with a narrow rule, instead of asking the user to install a third-party approver. With verifyFromAPIServer the kube-apiserver also verifies the kubelets against the Kubernetes CA (--kubelet-certificate-authority), but only after every machine has been issued a certificate, and it stays on once on.

Context

ADR 0108 left "kubelet serving-certificate bootstrap" as "not modelled". Today a Talos kubelet serves a self-signed certificate, so the kube-apiserver cannot verify it when it opens logs, exec, attach, port-forward and node-proxy connections, and metrics-server needs --kubelet-insecure-tls. Talos's documentation enables CA-signed serving certificates with a KubeletConfig (config.serverTLSBootstrap: true; before v1.14 machine.kubelet.extraConfig.serverTLSBootstrap) and then tells the user to install a community controller (kubelet-serving-cert-approver) that approves the requests, because kube-controller-manager signs kubelet-serving requests but never approves them (Kubernetes, "Certificate Signing Requests", signers). ADR 0101 forbids Ichor installing third-party operators. The controller already is the only writer to the machines and already holds the one thing such an approver needs to be strict: the list of machines it manages and an authenticated Talos API to each.

Decision

• Rendering (no raw YAML). enabled renders the standalone KubeletConfig document with config.serverTLSBootstrap: true on every node when the version contract is v1.14 or later (Talos v1.14 KubeletConfig, "config" is the kubelet's own configuration file; serverTLSBootstrap is not among kubelet.ProtectedConfigurationFields), and machine.kubelet.extraConfig.serverTLSBootstrap: true on an older contract. The deprecated extraArgs: rotate-server-certificates form is not used (Talos's documentation calls it deprecated by the kubelet). Talos stops and starts the kubelet service when its configuration changes (KubeletServiceController), so fieldmeta classes KubeletConfig.config and machine.kubelet.extraConfig as no-reboot; the image field keeps its reboot-required class (the Kubernetes upgrade owns it). A kubelet whose request is not yet approved keeps serving the certificate it had, so enabling the option cannot take a node down. refreshKubeletCerts in Talos removes the kubelet PKI directory when the client certificate no longer matches an accepted CA, so a Kubernetes CA rotation makes the kubelets request fresh serving certificates too.
• The approver. KubeletServingReconciler (one controller, keyed by TalosCluster, woken by CertificateSigningRequest events and a timer) lists the CSRs of the signer kubernetes.io/kubelet-serving that are neither approved nor denied, for clusters with the option enabled, and decides each one:
  • It finds the machine by the node name in the common name (TalosMachine.status.nodeName or spec.hostname). No such machine in an enabled cluster: the CSR is left pending (it may be another cluster's node or a user's; the user can approve it by hand), never denied.
  • The node's identity is read from the node, not from the CSR or from the Kubernetes Node object (a compromised kubelet can write its own Node's addresses): over the authenticated Talos API, the NodeIP resource kubelet (the addresses Talos gives the kubelet as --node-ip), the Nodename and the HostnameStatus. If the node cannot be read, the CSR stays pending and is retried; the rule fails closed.
  • Denied, with a reason on the condition and an Event, when the request is malformed or asks for more than the node's identity: the PEM does not parse or its signature is invalid; spec.username is not system:node:<name> with <name> the common name, or system:nodes is not among spec.groups; the subject is not exactly CN=system:node:<name>, O=system:nodes; a SAN other than DNS or IP (email, URI), no SAN, a DNS name that is not the node name or hostname, or an IP that is not one of the node's addresses; usages other than digital signature and server auth (with or without key encipherment, the two sets the signer permits); an extension other than SAN in the request.
  • Approved otherwise: the condition Approved with reason IchorKubeletServing and a message naming the node. Nothing else is ever approved or denied: the controller does not touch another signer, and a request it did not decide stays as it is.
  • After kube-controller-manager signs it (status.certificate), the controller parses the certificate and records TalosMachine.status.kubeletServingCertificate (request, issuedAt, notAfter). It never deletes a CSR (kube-controller-manager's cleaner does).
• RBAC (least privilege). A ClusterRole with get;list;watch on certificatesigningrequests, update on certificatesigningrequests/approval, and approve on signers with resourceNames: [kubernetes.io/kubelet-serving]. Not sign, not create, not other signers. The rules are added to the controller's own ClusterRole (config/rbac/role.yaml, generated from the reconciler's markers, and the chart's manager-role), and TestControllerRBACApprovesOnlyKubeletServing in dist/chart/chart_test.go fails if the controller ever holds any other right on certificates.k8s.io (another signer, sign, create).
• Condition. KubeletServingCertificates on the TalosCluster: True when every machine has an unexpired issued certificate recorded; False, naming the machines still waiting, otherwise; removed when the option is off. It is neutral in the UI and CLI (its name does not read as a health word), so a rollout in progress does not paint the cluster unhealthy; both already list every condition.
• verifyFromAPIServer and ordering. Setting --kubelet-certificate-authority before a kubelet has a CA-signed certificate would break logs and exec for that node. The flag is therefore rendered (as KubeAPIServerConfig.extraArgs, or cluster.apiServer.extraArgs on an older contract; the path is /system/secrets/kubernetes/kube-apiserver/ca.crt, the same CA file Talos gives kube-controller-manager as --cluster-signing-cert-file, so it signs and verifies with one CA) only while the condition KubeletServingVerified is True. The controller sets that condition when verifyFromAPIServer is set and KubeletServingCertificates is True, and removes it when either option is switched off. It is a latch: a machine that joins later does not take the flag away (that would restart every apiserver each time a node is added); its first requests are approved within seconds of the kubelet starting. Enabling verification restarts the kube-apiserver on each control plane (fieldmeta apiServerRestart, as the other apiserver options).
• Default. Off, for new and existing clusters, like every option of ADR 0108. Ichor already approves nothing it was not asked to; switching it on is one field. docs/guides/hardening.md says what it costs.
• Metrics-server and other clients. A user's metrics-server (not installed by Ichor, ADR 0101) verifies the kubelet against the cluster CA that every pod receives, which in a Talos cluster is the Kubernetes CA, and the certificates carry the node's IP, so --kubelet-insecure-tls is no longer needed. Clients that pin the kubelet's self-signed certificate break; there are none in Ichor.

Rejected

• Installing kubelet-serving-cert-approver (ADR 0101), or documenting it as a step: it would approve any CSR whose names match the Node object's reported addresses, and the Node's addresses come from the kubelet itself.
• An approver that trusts Node.status.addresses. A compromised node could first set another node's address on its own Node (NodeRestriction allows it) and then be issued a certificate for it, which the apiserver would accept for that address when verifyFromAPIServer is on. The Talos API is the independent source.
• Denying every request the rule cannot place (an unknown node name). The CSR might belong to a node Ichor does not manage, or arrive before the machine's status is written; pending is reversible, denied is not.
• rotate-server-certificates in extraArgs: deprecated by the kubelet; the configuration-file field is the supported spelling and renders as a typed document.
• Rendering --kubelet-certificate-authority together with enabled. Without the ordering above, a node whose request is not yet approved would break logs and exec until it is.
• A condition that follows every machine (no latch). Adding a node would remove the flag and restart every kube-apiserver.

Risks

• Until the machine controller has applied the kubelet change to every node, those kubelets still serve self-signed certificates; the condition says so.
• A Kubernetes CA rotation changes the CA the apiserver verifies against; Talos clears the kubelet PKI on the node when the kubelet client certificate stops matching, so new serving certificates are requested and approved. To be confirmed on hardware (below).
• The controller needs the Talos API of the node to approve; a node whose API is unreachable (firewall, down) keeps its CSR pending until it is back.

Test

Unit and envtest: rendering at both contract families and the unchanged bytes when the option is unset; the approver against real CSRs on envtest (a good request approved; wrong requester, wrong group, wrong subject, a foreign SAN, an email/URI SAN, bad usages, an extra extension and a bad signature denied with a reason; an unknown node and an unreadable node left pending; another signer untouched); the condition and the latch; the RBAC invariant in dist/chart/chart_test.go. Hardware verification is listed in docs/guides/hardening.md and in the batch report.