# 0125. Kubelet serving certificates are requested by the kubelets and approved by the controller Status: Accepted; built; hardware verification pending · Date: 2026-11-04 · Design section: Security · Touches: 0101, 0108 Decided: `TalosCluster.spec.security.kubeletServingCertificates` (`enabled`, `verifyFromAPIServer`, both off by default, for new and existing clusters) makes every kubelet request its serving certificate from the `kubernetes.io/kubelet-serving` signer (`serverTLSBootstrap`), and Ichor's controller approves those requests itself, with a narrow rule, instead of asking the user to install a third-party approver. With `verifyFromAPIServer` the kube-apiserver also verifies the kubelets against the Kubernetes CA (`--kubelet-certificate-authority`), but only after every machine has been issued a certificate, and it stays on once on. ## Context ADR 0108 left "kubelet serving-certificate bootstrap" as "not modelled". Today a Talos kubelet serves a self-signed certificate, so the kube-apiserver cannot verify it when it opens logs, exec, attach, port-forward and node-proxy connections, and metrics-server needs `--kubelet-insecure-tls`. Talos's documentation enables CA-signed serving certificates with a `KubeletConfig` (`config.serverTLSBootstrap: true`; before v1.14 `machine.kubelet.extraConfig.serverTLSBootstrap`) and then tells the user to install a community controller (`kubelet-serving-cert-approver`) that approves the requests, because kube-controller-manager signs `kubelet-serving` requests but never approves them (Kubernetes, "Certificate Signing Requests", signers). ADR 0101 forbids Ichor installing third-party operators. The controller already is the only writer to the machines and already holds the one thing such an approver needs to be strict: the list of machines it manages and an authenticated Talos API to each. ## Decision - **Rendering (no raw YAML).** `enabled` renders the standalone `KubeletConfig` document with `config.serverTLSBootstrap: true` on every node when the version contract is v1.14 or later (Talos v1.14 `KubeletConfig`, "config" is the kubelet's own configuration file; `serverTLSBootstrap` is not among `kubelet.ProtectedConfigurationFields`), and `machine.kubelet.extraConfig.serverTLSBootstrap: true` on an older contract. The deprecated `extraArgs: rotate-server-certificates` form is not used (Talos's documentation calls it deprecated by the kubelet). Talos stops and starts the kubelet service when its configuration changes (`KubeletServiceController`), so fieldmeta classes `KubeletConfig.config` and `machine.kubelet.extraConfig` as no-reboot; the image field keeps its reboot-required class (the Kubernetes upgrade owns it). A kubelet whose request is not yet approved keeps serving the certificate it had, so enabling the option cannot take a node down. `refreshKubeletCerts` in Talos removes the kubelet PKI directory when the client certificate no longer matches an accepted CA, so a Kubernetes CA rotation makes the kubelets request fresh serving certificates too. - **The approver.** `KubeletServingReconciler` (one controller, keyed by TalosCluster, woken by CertificateSigningRequest events and a timer) lists the CSRs of the signer `kubernetes.io/kubelet-serving` that are neither approved nor denied, for clusters with the option enabled, and decides each one: - It finds the machine by the node name in the common name (`TalosMachine.status.nodeName` or `spec.hostname`). No such machine in an enabled cluster: the CSR is left pending (it may be another cluster's node or a user's; the user can approve it by hand), never denied. - The node's identity is read from the node, not from the CSR or from the Kubernetes Node object (a compromised kubelet can write its own Node's addresses): over the authenticated Talos API, the `NodeIP` resource `kubelet` (the addresses Talos gives the kubelet as `--node-ip`), the `Nodename` and the `HostnameStatus`. If the node cannot be read, the CSR stays pending and is retried; the rule fails closed. - Denied, with a reason on the condition and an Event, when the request is malformed or asks for more than the node's identity: the PEM does not parse or its signature is invalid; `spec.username` is not `system:node:` with `` the common name, or `system:nodes` is not among `spec.groups`; the subject is not exactly `CN=system:node:, O=system:nodes`; a SAN other than DNS or IP (email, URI), no SAN, a DNS name that is not the node name or hostname, or an IP that is not one of the node's addresses; usages other than `digital signature` and `server auth` (with or without `key encipherment`, the two sets the signer permits); an extension other than SAN in the request. - Approved otherwise: the condition `Approved` with reason `IchorKubeletServing` and a message naming the node. Nothing else is ever approved or denied: the controller does not touch another signer, and a request it did not decide stays as it is. - After kube-controller-manager signs it (`status.certificate`), the controller parses the certificate and records `TalosMachine.status.kubeletServingCertificate` (request, issuedAt, notAfter). It never deletes a CSR (kube-controller-manager's cleaner does). - **RBAC (least privilege).** A ClusterRole with `get;list;watch` on `certificatesigningrequests`, `update` on `certificatesigningrequests/approval`, and `approve` on `signers` with `resourceNames: [kubernetes.io/kubelet-serving]`. Not `sign`, not `create`, not other signers. The rules are added to the controller's own ClusterRole (config/rbac/role.yaml, generated from the reconciler's markers, and the chart's `manager-role`), and `TestControllerRBACApprovesOnlyKubeletServing` in `dist/chart/chart_test.go` fails if the controller ever holds any other right on `certificates.k8s.io` (another signer, `sign`, `create`). - **Condition.** `KubeletServingCertificates` on the TalosCluster: True when every machine has an unexpired issued certificate recorded; False, naming the machines still waiting, otherwise; removed when the option is off. It is neutral in the UI and CLI (its name does not read as a health word), so a rollout in progress does not paint the cluster unhealthy; both already list every condition. - **`verifyFromAPIServer` and ordering.** Setting `--kubelet-certificate-authority` before a kubelet has a CA-signed certificate would break logs and exec for that node. The flag is therefore rendered (as `KubeAPIServerConfig.extraArgs`, or `cluster.apiServer.extraArgs` on an older contract; the path is `/system/secrets/kubernetes/kube-apiserver/ca.crt`, the same CA file Talos gives kube-controller-manager as `--cluster-signing-cert-file`, so it signs and verifies with one CA) only while the condition `KubeletServingVerified` is True. The controller sets that condition when `verifyFromAPIServer` is set and `KubeletServingCertificates` is True, and removes it when either option is switched off. It is a latch: a machine that joins later does not take the flag away (that would restart every apiserver each time a node is added); its first requests are approved within seconds of the kubelet starting. Enabling verification restarts the kube-apiserver on each control plane (fieldmeta `apiServerRestart`, as the other apiserver options). - **Default.** Off, for new and existing clusters, like every option of ADR 0108. Ichor already approves nothing it was not asked to; switching it on is one field. `docs/guides/hardening.md` says what it costs. - **Metrics-server and other clients.** A user's metrics-server (not installed by Ichor, ADR 0101) verifies the kubelet against the cluster CA that every pod receives, which in a Talos cluster is the Kubernetes CA, and the certificates carry the node's IP, so `--kubelet-insecure-tls` is no longer needed. Clients that pin the kubelet's self-signed certificate break; there are none in Ichor. ## Rejected - Installing `kubelet-serving-cert-approver` (ADR 0101), or documenting it as a step: it would approve any CSR whose names match the Node object's reported addresses, and the Node's addresses come from the kubelet itself. - An approver that trusts `Node.status.addresses`. A compromised node could first set another node's address on its own Node (NodeRestriction allows it) and then be issued a certificate for it, which the apiserver would accept for that address when `verifyFromAPIServer` is on. The Talos API is the independent source. - Denying every request the rule cannot place (an unknown node name). The CSR might belong to a node Ichor does not manage, or arrive before the machine's status is written; pending is reversible, denied is not. - `rotate-server-certificates` in `extraArgs`: deprecated by the kubelet; the configuration-file field is the supported spelling and renders as a typed document. - Rendering `--kubelet-certificate-authority` together with `enabled`. Without the ordering above, a node whose request is not yet approved would break logs and exec until it is. - A condition that follows every machine (no latch). Adding a node would remove the flag and restart every kube-apiserver. ## Risks - Until the machine controller has applied the kubelet change to every node, those kubelets still serve self-signed certificates; the condition says so. - A Kubernetes CA rotation changes the CA the apiserver verifies against; Talos clears the kubelet PKI on the node when the kubelet client certificate stops matching, so new serving certificates are requested and approved. To be confirmed on hardware (below). - The controller needs the Talos API of the node to approve; a node whose API is unreachable (firewall, down) keeps its CSR pending until it is back. ## Test Unit and envtest: rendering at both contract families and the unchanged bytes when the option is unset; the approver against real CSRs on envtest (a good request approved; wrong requester, wrong group, wrong subject, a foreign SAN, an email/URI SAN, bad usages, an extra extension and a bad signature denied with a reason; an unknown node and an unreadable node left pending; another signer untouched); the condition and the latch; the RBAC invariant in `dist/chart/chart_test.go`. Hardware verification is listed in `docs/guides/hardening.md` and in the batch report.