# 0106. HA findings: a patch is dry-rendered at admission, a restore refuses an in-cluster backup store, an unreachable node says so, and a dead control plane can be removed Status: Accepted; hardware-verified (H2 and H7 re-verified 2026-10-30) · Date: 2026-10-30 · Design section: Configuration, Backup and restore, Nodes · Touches: 0003, 0065 Decided: a `TalosConfigPatch` is dry-rendered by a validating webhook (create and update) against every approved machine of its cluster that it selects, with the controller's own `render.Render`, and refused with the render error when the render would fail and did not already fail without the patch; the generated `Config` mirror makes `machine`, `cluster`, `machine.type` and `machine.certSANs` optional (an additive loosening) so a sparse `Config` document no longer repeats them and no longer reads as "needs a reboot"; `ichor restore --live` and the `TalosEtcdBackup` condition `StoreInCluster` refuse or warn when the backup endpoint is inside the cluster being restored; a `TalosMachine` whose node cannot be read reports `Ready=False` `NodeUnreachable` with the endpoint and, when the Kubernetes Node shows another InternalIP, the likely new address (spec is never changed); `ichor node remove` of a control plane below the declared count is allowed when its etcd member is unhealthy or unreachable. ## Context A hardware run on an Ichor-made three control plane cluster found four gaps (P2, P3, P4, H7). Each section below is one decision; later sections are added by the commit that implements them. ## Decisions ### 1. A patch that cannot render is refused at admission (P2) A `Config` patch that set `cluster.apiServer.*` was accepted, then every machine went `RenderFailed`: "kube-apiserver config is already set in v1alpha1 config (.cluster.apiServer)". Under a multi-document contract the base config already carries a `KubeAPIServerConfig` document and Talos refuses the legacy block next to it. The controller hides the cause in the condition message, so the author saw only "rendering machine configuration failed". The webhook `vtalosconfigpatch-v1alpha1.kb.io` (`internal/webhook/v1alpha1/talosconfigpatch_webhook.go`) reads the cluster, its secrets bundle and system inputs the way the controller does, renders each approved machine the patch selects with and without the patch, and refuses when only the render with the patch fails, with the render error in the message. It does not check when the secrets bundle is not readable yet, or the cluster is missing (nothing to render against), nor a machine that already fails without the patch (the patch did not break it, and the fix for the other cause must not be blocked). Delete is not checked. The cost is two renders per selected machine per write; renders take milliseconds. CRD validation forced `machine.type` and `machine.certSANs` on a partial `Config` document because the mirror generator (ADR 0003) marks every machinery field without `omitempty` as required, and machinery tags those because a full machine config always has them. A patch is a sparse overlay, and the strategic merge ignores empty values, so `internal/tools/mirrorgen` now has a small table, `sparseFields`, that emits `Config.machine`, `Config.cluster`, `MachineConfig.type` and `MachineConfig.certSANs` as optional. This only relaxes a requirement; no field is removed or renamed. Other required fields (for example `controlPlane.endpoint` inside a `controlPlane` block) stay required, because they are required once their parent is present. The "applying it needs a reboot" note came from the same cause: the zero-valued `machine.type`, `machine.certSANs` and the document `version` were counted as set paths, and `pkg/fieldmeta` treats an unknown path as reboot-required. `patchPaths` now drops the `version` leaf and empty or null leaves. The `cluster.apiServer.*` paths themselves are an API server restart, not a reboot (`pkg/fieldmeta`), and the preview now says so by reporting no reboot. ### 2. A restore refuses a backup store inside the cluster (P3) `ichor restore --live` with the S3 store in the same cluster (a minio Service) deadlocked: the restore marker is written to the bucket after the Kubernetes API, and so the in-cluster S3, is down. `etcdbackup.InClusterReason` (`internal/etcdbackup/incluster.go`) says a destination is in the cluster when the endpoint host is `*.svc` or `*.svc.cluster.local`, or an IP inside the pod or service ranges of `spec.network` (Talos's defaults, `10.244.0.0/16` and `10.96.0.0/12`, for a range the cluster did not declare). The etcd-restore plan sets `Blocked` to "... the backup store must live outside the cluster being restored ..."; `CreateOperation` refuses a blocked plan and `--override` is not consulted, because no reason can make the marker writable. The same check sets the warning condition `StoreInCluster` (True) on the `TalosEtcdBackup`, removed when the endpoint moves out; `Ready` is unaffected, backups still work. Not covered: a short name such as `minio.minio` or a host that resolves to a cluster address (only the literal forms are recognised), and a `TalosOperation` written directly with `kubectl`, which the controller does not re-check. ### 3. An unreachable node says so (P4) A node whose IP changed kept `Ready=True` `Converged` with the stale endpoint. The machine controller already re-probes a converged machine every `reachabilityRecheck` (two minutes) and a failed read already sets `Ready=False` `CurrentConfigUnavailable`, but that reason names neither the endpoint nor the error, and it never changes however long the node stays away. The first failed read now sets a `NodeReachable=False` condition (its `lastTransitionTime` is when the failures began; the first successful read removes it, so no new status field is needed). Within `nodeUnreachableGrace` (60 seconds, long enough for a reboot) the reason stays `CurrentConfigUnavailable`; after it, `Ready=False` reason `NodeUnreachable` with "the node at has not answered for : ". When the Kubernetes Node of the machine (by `spec.hostname` or `status.nodeName`, else by `systemUUID` equal to the machine UUID) reports an InternalIP other than the dialled host, the message adds "spec.endpoint may need updating to ". `spec` is never edited; a machine with a hostname is still followed automatically through `status.observedEndpoint` (ADR 0086), which this does not change. ### 4. A dead control plane can be removed below the declared count (H7) `ichor node remove ` was refused because removing it takes the cluster below `spec.controlPlane.count`, and "approve a replacement first" cannot finish while the dead etcd member is still listed (the new control plane's etcd stays in Preparing). Three places refuse, and two now make an exception for a control plane whose member is dead; the quorum and last-control-plane rules never bend (a two control plane cluster still cannot lose one, dead or not): - The removal in the controller (`removal.go`, authoritative) reads etcd through a healthy sibling (`deadEtcdMember`, `removal_dead.go`): the sibling must list the member, and the departing machine must be unreachable, its etcd status must fail, or it must see no leader. A member that answers with a leader is healthy and the refusal stands, now saying so. Names that cannot be told, a sibling that cannot be dialed, or a member etcd does not list leave the refusal in place (nothing proves it dead); the one exception is a removal that already proceeded for a dead member (the `Removal` condition reason `DeadMemberRemoval` is the marker), so a round that resumes after the member left etcd goes on. When it proceeds it emits a Warning Event `DeadMemberRemoved` saying the count is below the declared one and a replacement should be approved, and it skips the storage evacuation and the drain, which wait for a kubelet that is not there and would never finish. - The plan (`internal/uiapi/node_action.go`, `ichor node remove`, UI) cannot read etcd, so it allows the removal when the machine's `Ready` is False with reason `NodeUnreachable` (section 3), with a warning in the plan; the controller still decides. `node reset` keeps the count check, because a reset needs the node. - The admission webhook (`ValidateDelete`) only ever checked quorum, not the declared count, and is unchanged. ## Hardware results (2026-10-30) HA lane on an Ichor-made three control plane cluster (e2e harness). - H1 bring-up, H3 one-control-plane-at-a-time rollout, H4 Kubernetes v1.37.1 to v1.37.2 and H5 CA rotation on three control planes passed. - H2 control plane loss: the API stayed available (0 hard failures). A halted node could not rejoin on the e2e harness because its pod IP changed. A single-replica controller or UI takes about 6 to 7 minutes to reschedule. - H6 `restore --live` passes with an out-of-cluster S3; an in-cluster S3 is now refused (section 2). - H7 dead control plane replacement is now allowed (11ac8f3e, section 4); re-verified below. ### Re-verification of H2 and H7 (2026-10-30, build `v0.0.1793300581-daa68524`) Fresh Ichor-made cluster: `ichor init --preset compact-ha --cni cilium --cni-tunnel geneve --endpoint-type loadBalancer --endpoint-address ` (an outer LoadBalancer Service selecting the harness group label, port 6443), Talos v1.14.3, Kubernetes v1.37.1, three control planes plus a fourth (`ICHOR_UI_DEMO_HA=extra`). The nodes boot the disk first (harness change `0e218b96`). The Cilium LB also balanced to nodes still in maintenance mode, so the endpoint answered only intermittently until all three control planes were up (bring-up only); the later steps used a control plane's address directly once cp4 joined the group. - **H7 passed.** A healthy control plane (cp2) below the declared count: `ichor node remove ichor-e2e-hb-cp2 --yes` was refused: "node remove: control planes 3 -> 2, below the declared count of 3; either approve a replacement control plane first (then remove this one), or lower spec.controlPlane.count first (to 1, 3 or 5) if you mean to shrink the cluster". Then cp3 was halted for good (19:35:46); `NodeUnreachable` after about 3 minutes. `ichor node remove ichor-e2e-hb-cp3 --yes --confirm ichor-e2e-hb-cp3` (19:39:01) printed the plan with "does not answer, so it is removed although that leaves fewer control planes than the declared 3 ... Approve a replacement afterwards". The controller logged and emitted the Warning Event `DeadMemberRemoved` at 19:39:22 ("removing control plane ... below the declared count of 3: its etcd member is unreachable (rpc error: code = DeadlineExceeded ...); approve a replacement to get back to the declared count"), skipped the drain and evacuation, removed the etcd member and deleted the TalosMachine; two members were left by 19:39:48 (47 s after the command; no wait for a drain). The Kubernetes Node `ichor-e2e-hb-cp3` stays `NotReady,SchedulingDisabled`, as the plan says. - cp4 (`ICHOR_UI_DEMO_HA=extra`): its address added to `spec.discovery.addresses` at 19:41:15, the pending machine approved as a control plane at 19:41:20; three Ready nodes (cp, cp2, cp4) 370 s after the approval; `etcd members` showed cp, cp2 and cp4 (three members, none a learner), all answering with one leader, so the replacement joined etcd. - Observation (not fixed): the cluster keeps probing every `spec.discovery.addresses` entry after the machines are approved and emits a `AlreadyConfigured` Warning Event on the TalosCluster for each address every few minutes, including the stale pod IP of a node that has left (10.244.1.59). It is noise here; on a network where addresses are reused it would probe an unrelated host. ## Later additions - **Endpoint identity check** (a45bafe3): a machine whose endpoint answers as another node is reported as `EndpointIdentityMismatch`. - **NodeUnreachable** (77b7b6ac, section 3): verified on the demo; the hint was correct and `Ready` returned after `spec.endpoint` was updated.