0053. ichor restore --recover-cluster: full-loss etcd recovery from the bundle and a snapshot

Status: Accepted · Date: 2026-10-26 · Design section: Backup, recovery, certificates

Decided: ichor restore --recover-cluster runs standalone (no Kubernetes, no Ichor): it reads the local secrets bundle, takes S3 credentials from environment variables, picks the newest snapshot that decrypts and passes the etcd checksum (or the one named by --snapshot, which is refused if broken), asks for confirmation, then uploads it with EtcdRecover to one freshly installed control plane and calls Bootstrap{RecoverEtcd: true}; restoring into a live cluster is a separate controller-run operation (ADR 0089)

Context

Goal 4: restore from the secrets bundle and a snapshot alone. ADR 0047 fixed the stored format and left restore to this unit. After a total loss the controller is gone too, so the command cannot lean on any CR, Secret or the Connect API (ADR 0049); it is one of the CLI operations that predate the cluster, like init and adopt.

Decisions

1. Talos side. The sequence is talosctl bootstrap --recover-from (cmd/talosctl/cmd/talos/bootstrap.go, Talos 1.14.3): Client.EtcdRecover streams the snapshot, then Client.Bootstrap with RecoverEtcd: true. In the server (v1alpha1_server.go), EtcdRecover writes /var/lib/etcd.snapshot atomically and needs a control plane (CheckControlplane); Bootstrap needs IsBootstrapAllowed (the cri service healthy), a machine config with an etcd CA, time in sync, and an empty /var/lib/etcd (AlreadyExists otherwise); the etcd service then runs recoverFromSnapshot (services/etcd.go), which restores with manager.Status and the etcd hash check (we do not pass RecoverSkipHashCheck: our snapshots always carry the trailer). So the target is one control plane that is installed and configured (not in maintenance mode) with a machine config made from the same bundle, with empty etcd data; the CLI dials it with the bundle's admin identity (the dialAdmin recipe) and exactly that one node. Other control planes join by themselves afterwards. Applying the machine config to the replacement node is not done here (the CRs that render it lived in the lost etcd); the operator installs the node first, for example with talosctl apply-config --insecure from the config they kept. Deferred: the design's "controller re-reconciles every node the way adopt does and waits for confirmation before applying anything" is the restored controller's job and is not built.
2. Bundle. Same file ichor init writes (internal/cli/bundlefile.go): ~/.ichor/clusters/<cluster>/secrets.yaml, or --secrets-bundle for a copy kept elsewhere (the machine doing the recovery is often not the one that ran init). Decoding errors carry the path only, never content. The derived passphrase and the bundle are never printed; restore_test.go asserts none of the bundle's scalars, the passphrase or the S3 keys appear in stdout, stderr or any error, including when an S3 or Talos error string contains the credentials (every such error passes through a scrubber holding them).
3. S3 credentials. Non-secret addressing is flags (--s3-endpoint, --s3-bucket, --s3-region, --s3-prefix, --s3-ca-file). Secrets are only environment variables, ICHOR_S3_ACCESS_KEY_ID, ICHOR_S3_SECRET_ACCESS_KEY, ICHOR_S3_SESSION_TOKEN, with the standard AWS_* ones as fallback; no flag exists for them, so they never appear in ps. (Environment is readable by the same user via /proc, which is the same exposure as the bundle file; a credentials-file option is deferred.)
4. Which snapshot. Default newest. Each candidate, newest first, is downloaded, decrypted and run through etcdbackup.Verifier (size and sha256 trailer) before anything is sent or even confirmed; one that fails is skipped, reported on stderr, and never sent. Falling back to an older snapshot loses data, so the confirmation names the snapshot actually chosen and its time. --snapshot <name> is exclusive: broken means an error, never a substitute. A bundle that is not the cluster's fails age's authentication on every object, and the final error says nothing was sent. The plaintext is kept in memory, not on disk (it holds every Secret); ponytail: a multi-GB etcd would need a protected spool file instead.
5. Confirmation. The shared confirm and the global --yes; the prompt states the node, the snapshot and its time, that state reverts, and that it cannot be undone. Declining contacts nothing.

Locking

The cluster lock (acquireDisruptiveLock, clusterlock.Acquire) is a Kubernetes object, and in full loss there is no Kubernetes. The command is the only actor by construction, and it is refused by Talos unless etcd data is empty. acquire_guard_test.go is unaffected: this code never calls Acquire. The live-cluster restore, which does have a cluster, must take the lock.

Not done

• Restore into a live cluster (UI and CLI) is built, in ADR 0089: ichor restore --live and the UI create an etcd-restore operation; the controller takes the lock, wipes etcd on every control plane, recovers the snapshot on one with the calls above, lets the others rejoin and reconciles every machine again; it reuses fetchVerified (now etcdbackup.FetchVerified). --recover-cluster is still the way back when that operation cannot finish, for example when the controller ran on the only control plane that was wiped.

• A restore round trip against a real Talos (e2e) is the design's regular proof and is not part of this unit.

• Tests use an in-memory etcdbackup.Store plus a fake Talos client; the real S3 client is covered by internal/etcdbackup's TLS fake S3, which lives in a _test.go file and is not importable.

• Amendment 2026-10-27: loadBundleFile now decodes through secretsbundle.Decode, which fills Bundle.Clock; before, --recover-cluster panicked in dialRecover on a nil Clock.