# 0047. TalosEtcdBackup: verified, bundle-keyed age snapshots in S3 Status: Accepted · Date: 2026-10-26 · Design section: Backup, recovery, certificates Decided: `TalosEtcdBackup` snapshots one healthy non-leader control plane over Talos's `EtcdSnapshot` stream, checks the etcd sha256 trailer, encrypts with age using a key derived from the secrets bundle, and stores `/etcd-.snapshot.age` in S3 via minio-go; restore reads exactly that ## Context Goal 4 of the design is "restore from the secrets bundle and a snapshot alone". Nothing existed: no type, no controller, no storage format. This unit builds the backup half. Restore (live and `--recover-cluster`) is the next unit, so the stored format below is written for it and lives in `internal/etcdbackup`, which both sides import so they cannot drift. ## Decisions 1. **How a snapshot is taken.** Talos's `MachineService.EtcdSnapshot` streams the bbolt database from the node's local etcd member (`internal/app/machined/internal/server/v1alpha1/v1alpha1_server.go`, `Server.EtcdSnapshot`, Talos 1.14.3: it opens `etcd.NewLocalClient` and calls `Snapshot`). It is control-plane only (`CheckControlplane`) and any one member suffices, since a snapshot is one consistent MVCC read. The controller asks every approved control plane for `EtcdStatus` and tries healthy followers first (no errors, not a learner, `member_id != leader`), then healthy leaders, stopping at the first that yields a verified snapshot. A single-control-plane cluster snapshots its leader. Machinery's `Client.EtcdSnapshot` and `Client.EtcdStatus` are used; `talosctl etcd snapshot` is the reference. 2. **Verification.** etcd's snapshot is the bbolt file followed by the sha256 of it. `talosctl` checks only `size % 512 == 32`; `etcdbackup.Verifier` also checks the hash while streaming. It does not open the database (that needs etcd's packages); it proves the stream was complete and intact. A snapshot that fails is never uploaded: the encrypted stream is staged to a spool file (an `emptyDir`; the root filesystem is read-only), uploaded only after verification, and the next control plane is tried. 3. **S3 client: `minio-go/v7`.** `go.mod` had no S3-capable dependency (no aws SDK, no minio, no gocloud, checked with `go mod graph`), so any choice adds one. `aws-sdk-go-v2` pulls in dozens of modules; minio-go adds nine small ones, speaks SigV4 against AWS, MinIO, Ceph and others, and supports a custom CA. 4. **Encryption: age with a scrypt recipient, passphrase derived from the bundle.** Snapshots contain Secrets, so they are always encrypted; there is nothing to configure. The key is `hex(HKDF-SHA256(secret = bundle.Cluster.Secret, salt = bundle.Cluster.ID, info = "ichor.maesh.dev/etcd-backup/v1"))`. It is recomputable from the bundle and stored nowhere, so (a) the bucket alone reveals nothing, and (b) "bundle + snapshot" really is sufficient, with no second key to lose. A key stored with the backups would make encryption pointless; a key held separately would add a thing recovery depends on. The cost is that the bundle is the root of trust for backups, which it already is for the cluster. age gives streaming authenticated encryption without us writing crypto; the file also decrypts with the plain `age` CLI given the passphrase. Changing `hkdfInfo` or the derivation orphans every existing backup, so it is part of the format. `bundle.Cluster.Secret` is not rotated by Talos CA rotation. 5. **Credentials.** `spec.destination.s3.credentialsSecretRef` names a Secret with `accessKeyID`, `secretAccessKey`, optional `sessionToken` and `ca.crt`. It is read from the **controller's own namespace** (`POD_NAMESPACE`), never the backup's (ADR 0011), the same rule as `loadBundle` and `render.ResolveRegistryAuth`; a test puts the backup elsewhere with a decoy Secret to prove it. Errors name the Secret and key, not values. Every error string is passed through a scrubber holding the credentials, the derived passphrase and the bundle's secret scalars before it reaches status or a log, because S3 and Talos error text is not ours. 6. **Git.** The spec holds only references, so it is safe to commit and the design lists `TalosEtcdBackup` as Git-tracked. Status holds object key, time, size and machine name only. **Follow-up:** ADR 0028's `ichor gitops export` allow-list (`internal/cli`) does not include this kind, so an exported repository misses it; adding it was out of this unit's scope. 7. **No cluster lock.** A snapshot is a read of one member; it does not stop, drain or restart anything. The lock exists for disruptive operations (`acquireDisruptiveLock` is the only `Acquire` caller and the guard test keeps it so). A node mid-upgrade fails its snapshot and the next is tried. The restore unit, which does stop etcd, must take the lock. ## Stored format (what restore reads) - Key: `/etcd-.snapshot.age`, UTC. Sorting the names sorts by time. Objects under the prefix not matching the pattern are never listed, restored or pruned. Use one prefix per cluster. - Content: an age file (scrypt recipient) whose plaintext is **exactly what `EtcdSnapshot` streamed**: the bbolt file plus the 32-byte sha256 trailer, i.e. the input of `talosctl bootstrap --recover-from` / `Client.EtcdRecover`. Restore decrypts, may re-run `etcdbackup.Verifier`, and feeds it to Talos unchanged. - To restore: read the bundle, `etcdbackup.Passphrase(bundle)`, `Store.List` (newest first), `Store.Get`, `Decrypt`. Nothing else is needed. - Retention: after each successful upload, keep the newest `retention.keepLast` (default 24) and delete the rest. Pruning errors are reported but do not fail the backup. A suspended backup prunes nothing. ## Schedule, status, metrics `spec.schedule` is a five-field cron in UTC (robfig/cron/v3, no dependencies), default hourly. Due-ness is computed from `status.lastAttemptTime`, so a restarted controller resumes; after a failure it retries every 5 minutes rather than waiting for the next tick. `Ready` is True after success, False after a failure or when the newest success is older than 2.5 schedule intervals (`Stale`). Metrics: `ichor_etcd_backup_last_success_timestamp_seconds` and `ichor_etcd_backup_failures_total`, labelled by namespace and name; alert rules over them belong with the chart's monitoring option. ## Deferred - `ponytail:` only S3; no GCS/Azure/local. Path-style vs virtual-host is minio-go's auto-detection; no override field. - `ponytail:` no deep verification (opening the bbolt file); a periodic restore test in e2e is the real proof, per the design. - Snapshots are staged on local disk (one snapshot of space in the `emptyDir`); a streaming multipart upload would avoid that but cannot un-upload a bad snapshot. - The chart's `manager-role` carries these rules; it already lacked `talosoperations` rules before this change (not touched here). - Amendment 2026-10-27: snapshots are sealed with scrypt work factor 2^10 (1 MiB) instead of age's default 2^18 (~270 MiB of heap per derivation, which OOM-killed the controller on real hardware); the key is a 256-bit HKDF output so the cost bought nothing. Decrypt caps at the same 2^10. There is no compatibility with pre-release backups: none sealed with 2^18 exist (the only ones were in torn-down test buckets), so the legacy decrypt path is dropped. The manager memory limit is 256Mi (request 128Mi): normal use is tens of MiB plus ~100 MiB of caches, and no legacy derivation needs covering. For the S3 test harness, `docker.io/bitnamilegacy/minio:2025.7.23-debian-12-r3` was pullable on 2026-10-27; `quay.io/minio/minio` and `docker.io/minio/minio` return 401 anonymously. - Amendment 2026-10-28: a backup can be taken on demand. `ichor etcd backup now NAME`, the Backups page's "Back up now" button and the `OperationService.RequestEtcdBackup` RPC set the annotation `ichor.maesh.dev/backup-now=` on the `TalosEtcdBackup` as the caller (same RBAC as the other operation writes: `update` on `talosetcdbackups`), so neither UI nor CLI touches Talos. The controller, when that time is newer than the new `status.lastManualBackup`, records it there and takes one snapshot in the same reconcile, outside the schedule, through the same path as a scheduled one (same key format, verification, encryption, pruning); the schedule then counts from that snapshot. A request is acted on once, whether the snapshot succeeded or not: a failure shows in `Ready` and is retried on the 5-minute failure interval like any failed attempt. A missing, unparsable or not-newer annotation is ignored, and a suspended backup ignores it. The pattern is ADR 0095's retry annotation. The RPC returns `FailedPrecondition` for a suspended backup and for a request still waiting, and writes a time never earlier than `lastManualBackup` plus one second. `EtcdBackup` in the API now also carries the destination (endpoint, region, bucket, prefix; never the credentials Secret), `keepLast`, the last success time, the last error (the Ready condition's message while it is Failed), `suspended`, `lastManualBackup` and whether a request is pending, and the Backups page shows them. The controller now also reconciles on annotation changes, not only on spec generation changes.