0084. Admin-gated support bundle, etcd snapshot and packet capture downloads

Status: Accepted · Date: 2026-10-27 · Design section: UI backend, Trust, Secrets Decided: three read-only downloads from a node's Talos API are served by the UI backend (DiagnosticsService, server-streaming byte chunks over the existing Connect transport): DownloadSupportBundle (one node, or every approved node of a cluster), DownloadEtcdSnapshot (a control plane node) and CapturePackets (one interface, 30 s by default and 5 min at most, 100 MiB at most, optional validated BPF filter); each needs the Kubernetes RBAC verb download on the subresource talosmachines/support-bundle, talosmachines/etcd-snapshot or talosmachines/packet-capture, checked with a SelfSubjectAccessReview as the caller, and granted to nobody by default, including the ichor-ui ServiceAccount; the chart grants it to named groups or, if the operator insists, to the ServiceAccount; every download records a Warning Event DiagnosticsDownload on the TalosMachine before any byte is read and is refused if that Event cannot be recorded; the backend mints a Talos client certificate per call with the least role the call needs (os:reader, os:etcd:backup, os:operator); nothing is stored; the UI adds an Actions menu on the node page, the CLI adds ichor node support-bundle, ichor node pcap and ichor etcd snapshot

Context

ADR 0082 listed three actions the Talos SDK enables that only read from a node: a support bundle (item 3), an etcd snapshot (item 5) and a packet capture (item 9). It proposed them as TalosOperation kinds, because the rule is that the controller is the only writer to Talos nodes. These three write nothing to a node, and an operation would have to put the result somewhere: a Secret or a CR would hold secrets at rest (the snapshot holds every Kubernetes Secret), which the design forbids. So they are served on demand by the UI backend, to the caller, and kept nowhere. Writes still go only through CRs and the controller.

The calls were checked in Talos v1.14.3 (pkg/machinery and the node's RBAC table in internal/app/machined/pkg/system/services/machined.go).

Decision

API. DiagnosticsService in proto/ichor/v1alpha1/diagnostics.proto:

• GetDiagnosticsPermissions(machine) returns three booleans, so the UI shows only what the user may start. The downloads ask again.
• DownloadSupportBundle(machine | cluster), DownloadEtcdSnapshot(machine) and CapturePackets(machine, interface, bpf_filter, duration_seconds, max_bytes) each return a stream of DownloadChunk{filename, content_type, data}; filename and content_type are on the first chunk. A Connect stream rather than a separate HTTP download endpoint, because the browser cannot attach the OIDC bearer token to a plain link, and the stream goes through the same interceptors (authentication, metrics) as every other call.

Authorisation. The verb is download, the resource talosmachines, the subresource one of support-bundle, etcd-snapshot, packet-capture, the object name the machine's. A subresource rather than three pseudo-resources, because RBAC treats talosmachines/etcd-snapshot as a resource of its own: a role with * on talosmachines (the talosmachine_*_role.yaml helpers) does not include it, and the three can be granted separately. The pattern is ADR 0064's override: a SelfSubjectAccessReview made as the user, so Kubernetes decides. Differences from override:

• With OIDC on, the review is made as the impersonated user.
• With OIDC off, the review is made as the backend's own ServiceAccount. It holds nothing by default, so a default install refuses every download. It is not refused by a special rule; it is simply not granted, and a chart value can grant it (below).
• In the CLI's in-process fallback the review is made as whoever the kubeconfig is.
• A cluster support bundle checks every machine in it; a denial on any machine refuses the whole download.

Chart. ichor-diagnostics-role (a ClusterRole with the three subresources and the verb) is a helper role like ichor-override-role (rbac.helpers.enabled), and is also rendered when a grant is asked for. ichorUI.diagnostics.groups (a list of group names, with the OIDC groups prefix applied) binds it to those groups; ichorUI.diagnostics.serviceAccount: true binds it to ichor-ui's ServiceAccount, which lets anyone who can reach ichor-ui download secrets and is documented as such. Both are empty by default. ichor-ui-role is unchanged; rbac_enforced_test.go runs GetDiagnosticsPermissions and the authorisation check as the generated role and requires every download to be denied, and dist/chart/chart_test.go pins the defaults.

Audit. Before any byte is read, after the authorisation and request validation succeed, the backend creates a Warning Event with reason DiagnosticsDownload on the TalosMachine: who (the impersonated user, or the actor name), which download, when, and for a capture the interface, duration and whether a filter is set. It never carries content. If the Event cannot be created the download is refused (Internal), so an unaudited download of an etcd snapshot cannot happen. The Event is created with the backend's own client, as ADR 0065 does; the ichor-ui role already has events create. A refused request (bad input, no permission) leaves no Event.

Talos identity. ADR 0018 said the UI backend holds a read-only Talos identity. These calls need more on the node: EtcdSnapshot is open to os:admin, os:operator and os:etcd:backup, and PacketCapture to os:admin and os:operator only. The backend therefore mints a client certificate per download from the secrets bundle with the least role that call accepts: os:reader for the bundle, os:etcd:backup for the snapshot (a role made for exactly this), os:operator for the capture. The certificate lives for the call. The backend had the secrets bundle already (it reads it for every Talos connection), so this adds no new secret at rest, but it does mean a compromised backend could mint those certificates for any call, not only these three. The code path has no method that changes a node (DownloadTalosClient has three methods), and architecture.md's Trust section says so. Kubernetes RBAC on the verb is the control that decides who gets them.

Caps, enforced on the server whatever the caller asks.

• Packet capture: duration_seconds 1 to 300, default 30; max_bytes 1 to 100 MiB, default 100 MiB; the capture stops at the first limit reached and the file is complete up to that point (a classic pcap, so a cut mid-packet at the byte limit is tolerated by readers). The interface name is checked against a short pattern. The BPF filter is accepted in tcpdump -dd format, the form Talos takes, and validated: every line must parse, at most 4096 instructions, and the program must pass golang.org/x/net/bpf's checks (jumps inside the program, ending in a return). Compiling a tcpdump expression such as tcp port 80 needs libpcap, which the pure-Go static binaries do not link; that is left out, and the UI and CLI help show how to produce the -dd text.
• Support bundle: 256 MiB for the archive, 10 minutes, each log file 8 MiB at most (the tail), 5000 lines per service, 2000 processes, 2000 resources per type.
• Etcd snapshot: 8 GiB (etcd's own recommended maximum) and 30 minutes.
• At most two downloads run at once in one backend (ResourceExhausted beyond that), and a backend that starts draining cancels them.
• When a limit stops a download after bytes were sent, the stream ends with an error (ResourceExhausted or DeadlineExceeded), so the client knows the file is incomplete; the CLI writes to a temporary file and renames it only on success, and the browser saves nothing on an error.

What the support bundle contains. Per node, under <machine name>/: version.json, dmesg.log (kernel ring buffer; the Command line: line has credential-shaped arguments masked by ADR 0073's rule), services.yaml (id, state, health) with logs/<service>.log (last 5000 lines of every Talos service), processes.json (pid, ppid, state, threads, CPU, memory, the executable's base name; never command lines, arguments or cgroup labels, as ADR 0082), mounts.json, memory.json, containers.json (ids, pods, names, images with credentials removed, status, pid; no environment), and resources/<namespace>/<type>.yaml, every COSI resource type that Talos does not mark sensitive, except the whole secrets and config namespaces and the KernelCmdlines and MachineConfigs types. collection-errors.txt lists what could not be read (an error text is cut at 300 bytes). A README.txt at the root says what is and is not in the file. talosctl support also collects the logs and manifests of kube-system pods through a kubeconfig fetched with os:admin; that is not done, because the Kubernetes API is reachable by other means and the kubeconfig is a credential. The archive is a zip, not encrypted (talosctl support encrypts to the Sidero Labs recipients by default, which an operator of Ichor does not want).

What is still sensitive. Service logs and the resource dumps are free text. Hostnames, IP addresses, MAC addresses, serials, extension configuration and anything a program wrote to a log (a token in a URL, say) remain. The etcd snapshot contains every Kubernetes Secret, unencrypted unless etcd encryption at rest is enabled in the cluster, and the packet capture records whatever is on the wire. Each file is therefore shown behind an "this file may contain secrets" confirmation in the UI, and the CLI writes it with mode 0600.

Delivery. UI: the stream is read into memory and saved with an object URL, refused above 1 GiB (the CLI has no such limit; the dialog and the error say so). The bearer token travels in a header and the request is a Connect POST with a JSON or protobuf content type, so a cross-site page cannot start a download (no ambient cookie, and a simple form cannot send that content type); with OIDC off there is no identity, and the ServiceAccount holds no grant by default. CLI: ichor node support-bundle NODE | --cluster NAME, ichor etcd snapshot --node NODE, ichor node pcap NODE --interface eth0 --duration 30s [--filter TEXT|@FILE] [--max-size BYTES]; each takes -o FILE (overriding the global -o yaml|json for these commands), writes mode 0600 and prints a reminder that the file may hold secrets. They go through ichor-ui like the other commands. If ichor-ui requires OIDC, the CLI falls back to the in-process path (internal/cli's localClients) with the user's kubeconfig, which then also needs to read the secrets bundle and reach the node's Talos port from the user's machine; if ichor-ui is on its own ServiceAccount and refuses, the error says to run again with --local.

UI. A node page "Actions" dropdown (preactcn dropdown-menu) with "Download support bundle", "Download etcd snapshot" (control planes only) and "Capture packets…" (a dialog with the interface chosen from the node's own links, a duration and a filter). Each ends in a preactcn alert-dialog that says the file may contain secrets and that the download is recorded. The menu is not rendered at all when GetDiagnosticsPermissions says the user may do none of them, and an action the user may not do is omitted. Results show as sonner toasts. The toast helper and Toaster mounted in app.tsx are this change's own: no shared one was on main when it was written.

Consequences

• A default install can download nothing. An operator who wants it binds ichor-diagnostics-role to a group (OIDC on) or sets ichorUI.diagnostics.serviceAccount (OIDC off, and then any client of ichor-ui can).
• The UI backend's Talos trust is wider than ADR 0018's: per call, it can act as os:operator for a packet capture. os:operator can also reboot a node, so the one call site that uses it must stay the packet capture.
• The etcd snapshot goes through ichor-ui's memory-free io.Copy to the client; the backend holds at most a 64 KiB chunk at a time except for the support bundle (one file at a time, 8 MiB).
• Not verified on hardware: nothing here has run against a real Talos node. The Talos side is the vendored client (client.EtcdSnapshot, PacketCapture, Dmesg, Logs, COSI List) and the roles are read from the node's RBAC table; the fake client in the tests stands in for the node. A Talos version that changes that table makes a download fail visibly with the node's refusal.
• Whether os:operator or os:admin is needed for PacketCapture on the Talos versions Ichor supports besides 1.14 is not checked.

Amendment: GetDiagnosticsPermissions is replaced by GetMyPermissions(machine?, cluster?), one call that answers these three flags and ADR 0093's credentials flag with concurrent SelfSubjectAccessReviews; the pages fetch it once and pass the flags down.