# 0084. Admin-gated support bundle, etcd snapshot and packet capture downloads Status: Accepted · Date: 2026-10-27 · Design section: UI backend, Trust, Secrets Decided: three read-only downloads from a node's Talos API are served by the UI backend (`DiagnosticsService`, server-streaming byte chunks over the existing Connect transport): `DownloadSupportBundle` (one node, or every approved node of a cluster), `DownloadEtcdSnapshot` (a control plane node) and `CapturePackets` (one interface, 30 s by default and 5 min at most, 100 MiB at most, optional validated BPF filter); each needs the Kubernetes RBAC verb `download` on the subresource `talosmachines/support-bundle`, `talosmachines/etcd-snapshot` or `talosmachines/packet-capture`, checked with a `SelfSubjectAccessReview` as the caller, and granted to nobody by default, including the `ichor-ui` ServiceAccount; the chart grants it to named groups or, if the operator insists, to the ServiceAccount; every download records a Warning Event `DiagnosticsDownload` on the TalosMachine before any byte is read and is refused if that Event cannot be recorded; the backend mints a Talos client certificate per call with the least role the call needs (`os:reader`, `os:etcd:backup`, `os:operator`); nothing is stored; the UI adds an Actions menu on the node page, the CLI adds `ichor node support-bundle`, `ichor node pcap` and `ichor etcd snapshot` ## Context ADR 0082 listed three actions the Talos SDK enables that only read from a node: a support bundle (item 3), an etcd snapshot (item 5) and a packet capture (item 9). It proposed them as `TalosOperation` kinds, because the rule is that the controller is the only writer to Talos nodes. These three write nothing to a node, and an operation would have to put the result somewhere: a Secret or a CR would hold secrets at rest (the snapshot holds every Kubernetes Secret), which the design forbids. So they are served on demand by the UI backend, to the caller, and kept nowhere. Writes still go only through CRs and the controller. The calls were checked in Talos v1.14.3 (`pkg/machinery` and the node's RBAC table in `internal/app/machined/pkg/system/services/machined.go`). ## Decision **API.** `DiagnosticsService` in `proto/ichor/v1alpha1/diagnostics.proto`: - `GetDiagnosticsPermissions(machine)` returns three booleans, so the UI shows only what the user may start. The downloads ask again. - `DownloadSupportBundle(machine | cluster)`, `DownloadEtcdSnapshot(machine)` and `CapturePackets(machine, interface, bpf_filter, duration_seconds, max_bytes)` each return a stream of `DownloadChunk{filename, content_type, data}`; `filename` and `content_type` are on the first chunk. A Connect stream rather than a separate HTTP download endpoint, because the browser cannot attach the OIDC bearer token to a plain link, and the stream goes through the same interceptors (authentication, metrics) as every other call. **Authorisation.** The verb is `download`, the resource `talosmachines`, the subresource one of `support-bundle`, `etcd-snapshot`, `packet-capture`, the object name the machine's. A subresource rather than three pseudo-resources, because RBAC treats `talosmachines/etcd-snapshot` as a resource of its own: a role with `*` on `talosmachines` (the `talosmachine_*_role.yaml` helpers) does not include it, and the three can be granted separately. The pattern is ADR 0064's `override`: a `SelfSubjectAccessReview` made as the user, so Kubernetes decides. Differences from override: - With OIDC on, the review is made as the impersonated user. - With OIDC off, the review is made as the backend's own ServiceAccount. It holds nothing by default, so a default install refuses every download. It is not refused by a special rule; it is simply not granted, and a chart value can grant it (below). - In the CLI's in-process fallback the review is made as whoever the kubeconfig is. - A cluster support bundle checks every machine in it; a denial on any machine refuses the whole download. **Chart.** `ichor-diagnostics-role` (a ClusterRole with the three subresources and the verb) is a helper role like `ichor-override-role` (`rbac.helpers.enabled`), and is also rendered when a grant is asked for. `ichorUI.diagnostics.groups` (a list of group names, with the OIDC groups prefix applied) binds it to those groups; `ichorUI.diagnostics.serviceAccount: true` binds it to `ichor-ui`'s ServiceAccount, which lets anyone who can reach ichor-ui download secrets and is documented as such. Both are empty by default. `ichor-ui-role` is unchanged; `rbac_enforced_test.go` runs `GetDiagnosticsPermissions` and the authorisation check as the generated role and requires every download to be denied, and `dist/chart/chart_test.go` pins the defaults. **Audit.** Before any byte is read, after the authorisation and request validation succeed, the backend creates a Warning Event with reason `DiagnosticsDownload` on the TalosMachine: who (the impersonated user, or the actor name), which download, when, and for a capture the interface, duration and whether a filter is set. It never carries content. If the Event cannot be created the download is refused (`Internal`), so an unaudited download of an etcd snapshot cannot happen. The Event is created with the backend's own client, as ADR 0065 does; the `ichor-ui` role already has `events` `create`. A refused request (bad input, no permission) leaves no Event. **Talos identity.** ADR 0018 said the UI backend holds a read-only Talos identity. These calls need more on the node: `EtcdSnapshot` is open to `os:admin`, `os:operator` and `os:etcd:backup`, and `PacketCapture` to `os:admin` and `os:operator` only. The backend therefore mints a client certificate per download from the secrets bundle with the least role that call accepts: `os:reader` for the bundle, `os:etcd:backup` for the snapshot (a role made for exactly this), `os:operator` for the capture. The certificate lives for the call. The backend had the secrets bundle already (it reads it for every Talos connection), so this adds no new secret at rest, but it does mean a compromised backend could mint those certificates for any call, not only these three. The code path has no method that changes a node (`DownloadTalosClient` has three methods), and `architecture.md`'s Trust section says so. Kubernetes RBAC on the verb is the control that decides who gets them. **Caps, enforced on the server whatever the caller asks.** - Packet capture: `duration_seconds` 1 to 300, default 30; `max_bytes` 1 to 100 MiB, default 100 MiB; the capture stops at the first limit reached and the file is complete up to that point (a classic pcap, so a cut mid-packet at the byte limit is tolerated by readers). The interface name is checked against a short pattern. The BPF filter is accepted in `tcpdump -dd` format, the form Talos takes, and validated: every line must parse, at most 4096 instructions, and the program must pass `golang.org/x/net/bpf`'s checks (jumps inside the program, ending in a return). Compiling a tcpdump expression such as `tcp port 80` needs libpcap, which the pure-Go static binaries do not link; that is left out, and the UI and CLI help show how to produce the `-dd` text. - Support bundle: 256 MiB for the archive, 10 minutes, each log file 8 MiB at most (the tail), 5000 lines per service, 2000 processes, 2000 resources per type. - Etcd snapshot: 8 GiB (etcd's own recommended maximum) and 30 minutes. - At most two downloads run at once in one backend (`ResourceExhausted` beyond that), and a backend that starts draining cancels them. - When a limit stops a download after bytes were sent, the stream ends with an error (`ResourceExhausted` or `DeadlineExceeded`), so the client knows the file is incomplete; the CLI writes to a temporary file and renames it only on success, and the browser saves nothing on an error. **What the support bundle contains.** Per node, under `/`: `version.json`, `dmesg.log` (kernel ring buffer; the `Command line:` line has credential-shaped arguments masked by ADR 0073's rule), `services.yaml` (id, state, health) with `logs/.log` (last 5000 lines of every Talos service), `processes.json` (pid, ppid, state, threads, CPU, memory, the executable's base name; never command lines, arguments or cgroup labels, as ADR 0082), `mounts.json`, `memory.json`, `containers.json` (ids, pods, names, images with credentials removed, status, pid; no environment), and `resources//.yaml`, every COSI resource type that Talos does not mark sensitive, except the whole `secrets` and `config` namespaces and the `KernelCmdlines` and `MachineConfigs` types. `collection-errors.txt` lists what could not be read (an error text is cut at 300 bytes). A `README.txt` at the root says what is and is not in the file. `talosctl support` also collects the logs and manifests of kube-system pods through a kubeconfig fetched with `os:admin`; that is not done, because the Kubernetes API is reachable by other means and the kubeconfig is a credential. The archive is a zip, not encrypted (`talosctl support` encrypts to the Sidero Labs recipients by default, which an operator of Ichor does not want). **What is still sensitive.** Service logs and the resource dumps are free text. Hostnames, IP addresses, MAC addresses, serials, extension configuration and anything a program wrote to a log (a token in a URL, say) remain. The etcd snapshot contains every Kubernetes Secret, unencrypted unless etcd encryption at rest is enabled in the cluster, and the packet capture records whatever is on the wire. Each file is therefore shown behind an "this file may contain secrets" confirmation in the UI, and the CLI writes it with mode 0600. **Delivery.** UI: the stream is read into memory and saved with an object URL, refused above 1 GiB (the CLI has no such limit; the dialog and the error say so). The bearer token travels in a header and the request is a Connect POST with a JSON or protobuf content type, so a cross-site page cannot start a download (no ambient cookie, and a simple form cannot send that content type); with OIDC off there is no identity, and the ServiceAccount holds no grant by default. CLI: `ichor node support-bundle NODE | --cluster NAME`, `ichor etcd snapshot --node NODE`, `ichor node pcap NODE --interface eth0 --duration 30s [--filter TEXT|@FILE] [--max-size BYTES]`; each takes `-o FILE` (overriding the global `-o yaml|json` for these commands), writes mode 0600 and prints a reminder that the file may hold secrets. They go through ichor-ui like the other commands. If ichor-ui requires OIDC, the CLI falls back to the in-process path (`internal/cli`'s `localClients`) with the user's kubeconfig, which then also needs to read the secrets bundle and reach the node's Talos port from the user's machine; if ichor-ui is on its own ServiceAccount and refuses, the error says to run again with `--local`. **UI.** A node page "Actions" dropdown (preactcn `dropdown-menu`) with "Download support bundle", "Download etcd snapshot" (control planes only) and "Capture packets…" (a dialog with the interface chosen from the node's own links, a duration and a filter). Each ends in a preactcn `alert-dialog` that says the file may contain secrets and that the download is recorded. The menu is not rendered at all when `GetDiagnosticsPermissions` says the user may do none of them, and an action the user may not do is omitted. Results show as `sonner` toasts. The toast helper and Toaster mounted in `app.tsx` are this change's own: no shared one was on main when it was written. ## Consequences - A default install can download nothing. An operator who wants it binds `ichor-diagnostics-role` to a group (OIDC on) or sets `ichorUI.diagnostics.serviceAccount` (OIDC off, and then any client of ichor-ui can). - The UI backend's Talos trust is wider than ADR 0018's: per call, it can act as `os:operator` for a packet capture. `os:operator` can also reboot a node, so the one call site that uses it must stay the packet capture. - The etcd snapshot goes through ichor-ui's memory-free `io.Copy` to the client; the backend holds at most a 64 KiB chunk at a time except for the support bundle (one file at a time, 8 MiB). - Not verified on hardware: nothing here has run against a real Talos node. The Talos side is the vendored client (`client.EtcdSnapshot`, `PacketCapture`, `Dmesg`, `Logs`, COSI `List`) and the roles are read from the node's RBAC table; the fake client in the tests stands in for the node. A Talos version that changes that table makes a download fail visibly with the node's refusal. - Whether `os:operator` or `os:admin` is needed for `PacketCapture` on the Talos versions Ichor supports besides 1.14 is not checked. Amendment: `GetDiagnosticsPermissions` is replaced by `GetMyPermissions(machine?, cluster?)`, one call that answers these three flags and ADR 0093's credentials flag with concurrent `SelfSubjectAccessReview`s; the pages fetch it once and pass the flags down.