0067. Whole-cluster shutdown is a TalosOperation kind the controller runs, with a derived mark that refuses other disruptive work

Status: Accepted · Date: 2026-10-26 · Design section: Upgrades Decided: a TalosOperation of kind shutdown stops a cluster: workers first (drained), then every control plane and the controller's own node in one forced burst; while any shutdown operation has not Completed, acquireDisruptiveLock (both of them) refuses everything else; the operation completes by itself once every machine answers on a new boot; no CLI command, no etcd snapshot taken by the shutdown

Context

Nothing stopped a whole cluster in a planned way. Bringing it back up already works without help: the machine controller re-checks an unreachable node every 30 seconds (ADR 0060) and does not mistake a booting node for drift. What is missing is stopping, in an order that does no harm, and keeping Ichor itself from "repairing" the cluster while it is being stopped or started.

Facts checked in Talos v1.14.3 (internal/app/machined/...):

Server.Shutdown (v1alpha1_server.go) starts SequenceShutdown in a goroutine and returns at once: the RPC acknowledges the request, not the stop.
Sequencer.Shutdown (runtime/v1alpha1/v1alpha1_sequencer.go) runs the drain phase (CordonAndDrainNode: wait for the node to be cordoned, then evict through the kubelet's own kubeconfig) only when !in.GetForce(); then stops pods and services. The ShutdownRequest.force comment (api/machine/machine.proto) is "without first cordoning and draining". The drain needs the Kubernetes API.
Shutdown does not remove the etcd member. LeaveEtcd appears only in the Reset sequence (graceful, not worker). etcd is stopped as a reverse dependency of cri (StopServicesEphemeral). So the member list keeps every control plane, quorum is a majority of the registered members, and powering the machines on restores it with no data lost. The danger is only in the order: the API is gone once quorum is.

Decision 1: where it runs -- a TalosOperation kind, the controller runs it; no CLI exception

A new TalosOperationSpec.kind value, shutdown. The user (or UI, or kubectl) writes the CR; the controller is the only writer to the nodes. This keeps the design's principle whole; init, adopt and full-loss recovery are CLI exceptions only because the cluster, or the controller, does not exist yet, and neither is true here.

The self-managed case decides the shape. A controller that runs inside the cluster it stops dies as soon as its node goes down, so it cannot be the one to do something after the control planes stop. The answer is not to let the CLI do the last stage but to make the last stage not need the controller afterwards: all the requests of the final stage are sent in one burst (Decision 2), and the controller's own node is asked last in it (NODE_NAME, from the downward API, matched with TalosMachine.spec.hostname; empty or unmatched when the controller runs elsewhere). The controller needs to be alive only until it has sent the requests, which are acknowledged in milliseconds and take a node seconds to act on. Everything the controller needs afterwards is in the CR.

TalosOperationReconciler dispatches this kind to its own state machine (shutdown.go) instead of AdvanceMachineFunc, which ADR 0026 left as the one place that branches by kind. It is not a rollout (workers before control planes, a mark to hold, a wait for a human), and forcing it through Advance would have needed the opposite tier order and a different "done". It still takes the cluster lock, honours spec.paused, and keeps its state only in status.

A CLI command was not added. No operation kind has one (upgrades are written as CRs by hand or by the installer); a cluster shutdown verb needs an ichor-ui RPC and its proto, which is a separate piece of work. The CR is

apiVersion: ichor.maesh.dev/v1alpha1
kind: TalosOperation
metadata: {generateName: shutdown-, namespace: <ns>}
spec: {clusterRef: {name: <cluster>}, kind: shutdown}

Decision 2: order

Workers, except the controller's own node, in parallel, Shutdown(force=false). Talos cordons and drains each before stopping it (above), and the API is still up for that. Each is recorded in status.machines with its boot ID before the request is sent. The operation waits until every one has stopped answering (or answers on another boot).
Everything else in one burst, Shutdown(force=true): every control plane, then the controller's own node last. Forced, because the drain needs an API that is about to go and the workers are already down; with control planes schedulable (a small cluster) their pods are killed by Talos stopping them, which is the point of a shutdown. No leader-last order: a sequence with quorum lost half way needs a controller alive after the API is gone, and a shutdown stops etcd cleanly on every member anyway (nothing to transfer). Quorum is lost in this stage and only in it.

A machine that does not answer when first looked at is recorded and not asked (it is already down). Upgrade holds are ignored: they exclude a machine from a rollout, not from stopping the cluster. Only approved machines are stopped.

Recording before sending makes the request at most once per machine per operation: a controller killed between the two leaves a record and never sends twice, which matters because the controller can come back up (the cluster has been powered on) while the operation still exists. A request that fails synchronously removes its record, and those not yet sent, so the next reconcile retries. If the controller dies in the milliseconds between recording and the request, that machine stays up with its boot ID unchanged; delete the operation and apply a new one.

Decision 3: the mark, and the lock

The mark is derived, not stored: a TalosOperation of kind shutdown on the cluster whose phase is not Completed. shutdownBlocker lists operations through the API reader and acquireDisruptiveLock -- both, the machine controller's (reboot-needing change, removal, including the removal that completes a 3-4-3 replacement) and the operation controller's (every upgrade kind) -- calls it before anything else and returns a shuttingDownError. The callers report ClusterShuttingDown (a machine's Ready or Removal condition; an upgrade operation goes Paused and retries every 30 seconds, so it resumes by itself when the mark goes). TestAcquireDisruptiveLockIsTheOnlyAcquireCallSite still holds.

Derived because nothing can leave it stale and deleting the operation is the escape hatch for a shutdown that cannot finish (a machine that never comes back). A shutdown is blocked only by another shutdown that has already recorded a machine, so two created together do not block each other forever. Drift correction that needs no reboot is not gated by the lock today and is left alone: it reads the node, and a node that cannot be read is not drifted.

Decision 4: starting up

Ichor cannot power machines on. The operation sits in phase ShutDown (new), holding no lock, and every 30 seconds (when a controller is running) probes the machines it asked to stop. A machine is back when it answers with a boot ID other than the recorded one, or, for one recorded as already down, when it answers at all. When all are back the phase becomes Completed, the condition Shutdown goes False, and the mark is gone: upgrades paused by it resume, removals proceed, the machine controller works as before. Nobody has to remove the mark; the operator only powers on. A machine deleted in the meantime is not waited for. A controller in the stopped cluster resumes this on its own boot.

Decision 5: etcd backup

No snapshot is taken by the shutdown. A shutdown loses no etcd data, TalosEtcdBackup has no run-now trigger to call, and taking a snapshot before the workers are stopped delays a stop the operator asked for. If a fresh snapshot is wanted, wait for the next scheduled one before applying the operation. The "stale" condition may show while the cluster is off; the next successful snapshot after power-on clears it, so it was left alone.

Consequences

TalosClient gained Shutdown(ctx, force); TalosOperationMachineStatus gained bootID; phase ShutDown and kind shutdown were added to the CRD; the controller Deployment gets NODE_NAME.
Not covered: a node reported unreachable by a network fault is treated as stopped and not asked; a worker whose drain is stuck behind a PodDisruptionBudget keeps stage 1 waiting (Talos's own drain timeout applies; delete the operation to give up).
Not tested here, deliberately: real machines (no real node or kubectl is touched by the tests).
ichor cluster shutdown and the UI button landed in ADR 0068.

E2E observation (2026-10-26, through `ichor cluster shutdown`)

Confirmation text: "SHUT DOWN cluster ... Every machine is POWERED OFF and stays off until you start it again ... workers first, drained and stopped together, then the control planes at once: 1. worker 2. cp, cp, cp".
Started 19:09:20. The worker's node ended at 19:09:37 (about 17 s), the Kubernetes API was gone at about 19:10:11, control planes ha-0 and ha-2 stopped by 19:10:50 and ha-1 by 19:11:24. So the worker went first and the control planes after it. The controller's own node (ha-0) was not the last to stop; the order in which Talos finishes stopping is not the order the requests were sent in, and the operation's status could not be read afterwards (etcd was off), so what the order of the requests was was not observed.
Starting an upgrade while the shutdown was InProgress was refused by ichor upgrade kubernetes 1.37.1: failed_precondition: operation "shutdown-qdr77" (shutdown) on this cluster is not finished (phase "InProgress") (the operation-in-flight check of ADR 0068, not the ClusterShuttingDown pause of acquireDisruptiveLock).
Power-on did not complete, and this harness cannot test it as it is. Starting the nodes again created new launcher pods with different pod IPs (10.244.1.44/.96/.90/.197 became 10.244.1.250, 10.244.0.54, 10.244.1.144, 10.244.2.150). Every address in the cluster (etcd peer URLs, the cluster endpoint, the TalosMachine endpoints, the certificate SANs) is the old pod IP, so a three-member etcd cannot form quorum after a power cycle on this network. In addition no Talos port (50000, 6443, 10250) answered on the new addresses for more than 20 minutes after the start, and I had no console to see why. "The operation becomes Completed by itself" was therefore not observed. A test needs node addresses that survive a stop and start (a secondary interface with static addresses, which the outer cluster does not have, or a single-node cluster).

Amendment (drain assumption re-checked, Talos v1.14.3)

Shutdown(force=false) draining is confirmed in the source: Sequencer.Shutdown (internal/app/machined/pkg/runtime/v1alpha1/v1alpha1_sequencer.go:363-381) appends CordonAndDrainNode when the request is not forced and node registration is not skipped, and the shutting-down stage cordons the node (controllers/k8s/node_cordoned_spec.go:73-76). Nothing was changed here. The assumption that Reboot drains the same way, which ADR 0080 had also made, was wrong and is fixed there: Ichor drains before its own reboots.

Addendum (ADR 0086): the wait for a machine to return dials DialEndpoint(), so a node that came back at a new address is seen once the address is recorded in status.
Amendment: a worker counts as stopped only after it stayed unreachable for 90 seconds (a brief network loss does not count; the clock is in memory, so a restarted controller starts it again), and when stage 1 has waited 15 minutes the operation pauses with a reason that names the PodDisruptionBudgets allowing no disruption (cluster-wide, not narrowed to the waiting nodes).