0054. Approving control planes is capped at count+1, and schedulable false needs a worker

Status: Accepted · Date: 2026-10-26 · Design section: Validation, Nodes Decided: The TalosMachine webhook refuses an approval (update to approved, or create already approved) that would leave more than spec.controlPlane.count+1 approved control planes, counted by deleteguard.CountApprovedControlPlanes; the TalosCluster webhook refuses schedulable turning "false" with no approved worker, and the TalosMachine webhook refuses an approved worker leaving (delete, un-approve, re-role, move) when that would leave none; both webhooks now match CREATE/UPDATE as needed, not DELETE only; the race admission cannot close is closed by a controller-side hold of the newest surplus approvals (ControlPlaneOverCap)

Context

The design's Validation section lists "no approving more control planes than configured" and "no schedulable: false with zero workers". Neither was enforced: ValidateCreate/ValidateUpdate returned nil, and the webhook configurations (generated from the +kubebuilder:webhook markers, copied into the chart) matched DELETE only, so those methods were never even called. Only the lower bound (ADR 0010, 0032) existed.

Decision

Upper bound. A control plane is replaced 3 -> 4 -> 3, adding before removing (ADR 0032). A plain approved <= count would make that impossible. The rule is: approving is refused when the control planes already approved exceed spec.controlPlane.count; so the approved total may reach count+1, never more. count is odd (1, 3, 5), so count+1 is exactly the one even, Transitioning state the design allows (transitioningControlPlanes), and it pairs with CheckControlPlaneCount: that check refuses removal below count, this one refuses approval above count+1. Together the live total stays in [count, count+1] outside a deliberate change of spec.controlPlane.count. Lowering count below the live total is not blocked (that is how a shrink starts); raising it widens the cap. Cases: 1 -> 2 ok, 2 -> 3 refused on count 1; 3 -> 4 ok, 4 -> 5 refused on count 3.

The check runs on the transition into "approved control plane of cluster X" only (update, or a create that is already approved, since a GitOps apply can approve by creating), so ordinary edits cost nothing. A machine whose TalosCluster does not exist or is deleting is let through: there is nothing to compare to, and the controller does not act without the cluster.

Schedulable. Two operations can reach "unschedulable, no workers", and each is checked where it happens: TalosCluster update, only on the transition into "false" (a cluster is created false before any worker is discovered, per the standard and large presets, and an unrelated edit to a cluster that has since lost its workers must not be blocked); and removing an approved worker from a cluster whose schedulable is "false", which is ValidateDelete (after CheckDelete, as for control planes) and the ValidateUpdate cases that make an approved worker stop being one. A cluster being deleted skips the check so teardown can cascade. Only approved workers count; a pending one has not joined.

Counting. All counting lives in pkg/deleteguard (CountApprovedControlPlanes, new CountApprovedWorkers, one shared countApproved), read through mgr.GetAPIReader(). No new TalosMachineList appears in the webhook package. The pure rules are CheckControlPlaneApproval and CheckLastWorker.

Webhook scope. The markers now say create;update;delete for TalosMachine and update;delete for TalosCluster, and config/webhook/manifests.yaml, the chart template and the e2e harness follow. The chart test asserts these exact operation lists instead of "DELETE only". This widens the fail-closed blast radius of ADR 0006: with the webhook down, spec and metadata writes to those CRs are refused, not only deletes. Status writes use the /status subresource and do not match. That is the same direction of failure ADR 0006 chose.

The race admission cannot close

Admission is not serialized with the commit it gates. Two concurrent approvals at the live count count both see count and both pass, leaving count+2; two concurrent last-worker removals both see two workers. This is the same TOCTOU as ADR 0010's delete guard, and a lock does not close it for the same reason: the lock would have to span the apiserver commit. The webhook stays as the synchronous, early refusal that is right in the single-writer case (a person at a terminal, one UI click) and costs one List.

The real fix is ordering at the actor that changes the world, which for the cap is the controller. It is built (see the next section); the worker rule needs no controller twin: a transient "no workers" is a scheduling problem that the next approval resolves, not a safety loss like etcd membership.

The controller-side hold

TalosMachineReconciler.Reconcile recounts approved control planes before it loads the bundle, renders or applies anything. If the total exceeds spec.controlPlane.count+1, the machines beyond the first count+1 are held: Ready=False, reason ControlPlaneOverCap, nothing rendered, no dry run, no apply, and a 15 s requeue so the hold lifts by itself once a sibling is deleted, un-approved, or count is raised.

• Which is newer. Approved control planes are sorted by spec.approvalDecision.at, ties broken by name (ascending), and the tail past count+1 is held. A machine with no recorded decision time (a GitOps apply that created it already approved) falls back to its creationTimestamp. The sort and the cut live in deleteguard.ControlPlaneOverCap, which shares one listing (approvedMachines) with CountApprovedControlPlanes; so the controller gained no TalosMachineList declaration, the guard allow-list is unchanged, and the decision is made in the one package that already owns the count. Read through APIReader.
• Already an etcd member. The hold only stops the controller from touching the node. It never removes the member, resets the node, or changes spec: the controller does nothing destructive without a user action. A held machine that had already joined stays a member, its config stays as last applied, and Ready says why. The operator resolves it by deleting a surplus control plane (the ordinary guarded removal path, which the hold does not block) or raising count. A held machine that is the only one the user wants removed can be removed the same way; no special case.
• No lock. The recount does not take the cluster lock. acquireDisruptiveLock refuses while the cluster is Transitioning, and count+1 is the Transitioning state, so taking it here would block exactly the legitimate replacement; and a hold is not a disruptive operation (it changes nothing on any node). The lock would also not close the race: two approvals commit independently of any controller lock, and the recount sees both once they are committed. Idempotence is what makes it safe: every reconcile recomputes the same ordered cut from live state, so two controllers or two passes agree on the same held set.
• Residual window. A node approved just before the surplus approval is visible can already have been configured; that is the case above (it is older, so it is not the one held). The hold only prevents the newest surplus from joining, which is the point.

Consequences

• ValidateUpdate/ValidateCreate on TalosMachine and ValidateUpdate on TalosCluster are now live. A new APIReader field on the cluster validator.
• Approving a control plane at exactly count always succeeds, even when it was not meant as a replacement; the cluster then reads Transitioning until a member is removed or count is raised.
• The webhook reads TalosClusters; the controller's existing RBAC already allows it.

Alternatives considered

approved <= count. Makes 3 -> 4 -> 3 impossible; rejected.

Count only while Transitioning is true on the cluster status. Status can be a reconcile stale and the condition is derived from the very count being checked; the live count is the source.

A ValidatingAdmissionPolicy. CEL cannot list sibling objects.

Addendum: one filter for "the machines that count"

The control-plane count, the approval cap and several controllers each narrowed a TalosMachineList by hand (cluster, approval, role), and they disagreed about a machine that is being deleted: the count included it, findEtcdSurvivor skipped it. This is a policy matter, not an etcd one: members leave etcd one at a time, so a removal in progress cannot break quorum. What it broke was the declared count. With 4 control planes of which one is stuck mid-removal (finalizer still held), counting it let a second removal pass 4 -> 3 although only 3 were staying, so the effective count fell to 2 (test: "does not count another control plane that is already being deleted toward the declared count").

deleteguard.ApprovedMachines(ctx, reader, ns, cluster, MachineFilter{Role, IncludeDeleting, KeepDeleting}) is now the only function that lists TalosMachines and narrows them to approved ones of a cluster. By default a machine with a DeletionTimestamp is not counted. KeepDeleting names the machine whose own removal is being decided, which the quorum and count rules count "including the one being removed". IncludeDeleting is for a caller that must reach every node (the whole-cluster shutdown). CountApprovedControlPlanes, CountApprovedWorkers, ControlPlaneOverCap, findEtcdSurvivor, snapshotCandidates, candidateMachines, othersDone, shutdownPlan and the TalosStorage readiness check use it; awaitPowerOn reads each recorded machine by name. Only machinesOfCluster (it must enqueue unapproved and deleting machines too) still lists by hand, and is the one entry left in countplanesAllowedFuncs (ADR 0010's guard).

Side effects of the default: a deleting machine is no longer picked as a snapshot source, a rolling-operation candidate, or waited for by the kube-proxy switch, and does not make a TalosStorage look unconverged.