# 0108. Security hardening

Status: Accepted; built, hardware-verified (see the Hardware results sections); the five items it deferred as follow-ups are all built since (ADR 0120 to 0127, see the amendments after "Follow-ups") · Date: 2026-11-02 · Design section: Trust, Deployment, Nodes · Touches: 0011, 0041, 0058, 0063, 0089, 0103, 0105, 0107

Decided: Ichor is hardened area by area after five read-only audits (2026-11-02): ichor-ui and its auth, the controller's privileges and admission, workloads and the supply chain, the security of the clusters Ichor creates, and secrets and credentials. Fixes to Ichor itself are made directly. For the clusters Ichor creates, every new hardening is a typed option that is off by default, for new and existing clusters alike; disk encryption stays opt-in (the user's decision, 2026-11-02). Each area below records its decisions and what was built.

## Context

The user asked for security to be hardened "to the extreme" (2026-11-02). Five audits read the code and manifests and listed findings with file:line evidence, an exploit scenario and a fix. Asked how far new clusters' defaults should go, the user chose that all cluster-side hardening is opt-in, and that disk encryption stays opt-in.

## Admission and confused deputy

The controller does things on a CR author's behalf with rights the author does not have: it pushes a rendered machine configuration (the cluster's CA keys) to a machine, resolves a credential Secret, sends snapshots to an S3 endpoint, and creates Jobs whose ServiceAccounts can create roles. The audit (rbac-admission F1 to F9, secrets S5) found places where the author chose the target or the Secret without the matching right.

### Decisions

1. **A machine's endpoint is an address the cluster declares.** The `TalosMachine` webhook judges `spec.endpoint` on create and whenever the endpoint or `clusterRef` changes (an update that leaves both alone is not judged, so a machine stays editable after its discovery entry goes away). Never allowed, for anyone: an unspecified, loopback, link-local or multicast IP, or one inside the pod or service ranges the cluster declares in `spec.network` (`pkg/endpointguard`). Talos's undeclared defaults (`10.244.0.0/16`, `10.96.0.0/12`) are deliberately not assumed: the machines of an outer cluster live in those ranges (the e2e harness). Allowed: an address in the cluster's `spec.discovery.addresses` or `cidrs`; any other address needs the new right `update` on `talosmachines/endpoint` (a hostname cannot be classified here, so it needs the right too).
2. **Approving needs a right.** Moving `spec.approvalState` to `approved`, including at creation, needs `approve` on `talosmachines/approval`, asked by SubjectAccessReview as the requesting user like `talosclusters/authentication`. `ichor-operator` and `ichor-admin` hold it (they could approve before, through `update`); `ichor-admin` also holds `talosmachines/endpoint`. The `talosmachine-editor-role` helper holds neither: an editor can no longer approve. `talosmachine-admin-role` holds both.
3. **The controller re-checks before it dials.** `provision()` runs `endpointguard.ForbiddenResolved` on the dial endpoint (including `status.observedEndpoint`, and what a hostname resolves to) before the insecure maintenance probe, and sets `Ready=False` `EndpointRefused` without dialling. It does not require discovery membership: `ichor init` and `ichor adopt` write machines at addresses they were handed, and the controller cannot ask who wrote the CR.
4. **`MachineReclaim.spec.address`** gets the same forbidden classes in the webhook (the cluster's ranges are not known there; the reclaim verb and the foreign talosconfig remain the other gates).
5. **Where credentials go is part of the Secret reference.** A change to `spec.registries[].endpoints` of a registry that keeps its `authSecretRef`, and a change to `spec.destination.s3` endpoint, bucket, prefix, region or the new `insecure`, needs `update` on `<resource>/secret-references`, like the Secret name. A plain `http://` S3 endpoint is refused unless `spec.destination.s3.insecure: true` (new optional field; judged when the endpoint or the flag changes, so a stored `http://` backup keeps working; the controller does not re-check).
6. **Jobs.** Two ValidatingAdmissionPolicies bound to the controller's ServiceAccount (`config/admission/controller-jobs-policy.yaml`, mirrored in the chart): the first allows only the installer Job (`ichor-installer`, in the Ichor namespace, `installer run`) and the Cilium Job (`ichor-cni`, in kube-system, `cni helm apply`), one container, no command override, only `KUBERNETES_SERVICE_HOST`, `KUBERNETES_SERVICE_PORT` and `HOME` as environment, dropped capabilities, no privilege escalation, `emptyDir` volumes only, `hostNetwork` for the Cilium Job only, an `activeDeadlineSeconds` of at most an hour (the installer Job now sets 30 minutes; the Cilium Job already set 25). The second ties the image to `IchorInstallation.spec.installerRepository:version` and does nothing while no IchorInstallation exists (a `parameterNotFoundAction: Allow` skips the binding). The chart adds `admission.jobImageRepository` (default `ghcr.io/maeshinshin/ichor-installer`) for an install without an IchorInstallation; set it to a mirror or empty it. Neither Job uses a hostPath today, so none is allowed.
7. **Escalation verbs are named.** The installer's `bind` and `escalate` list every ClusterRole and Role of the install manifest (the installer applies its own role first, so a release that adds a role extends the list before using it); the Cilium Job's list the roles the pinned chart creates. `dist/chart` tests compare both lists with `dist/install.yaml` and with the chart as rendered under Ichor's values. Residual: the installer keeps `escalate` on its own role by design (ADR 0059 stage 0), so a compromised installer can still rewrite its role; the Job policy and the `ichorinstallations/version` right are what keep that process from being started with another image.
8. **System namespaces.** No `TalosCluster`, `TalosMachine`, `TalosOperation` or `TalosEtcdBackup` may be created in `kube-system`, `kube-public` or `kube-node-lease`: the cluster lock is a Lease named after the cluster, written with the controller's cluster-wide Lease rights. The leader-election role loses its unused ConfigMap rule. Moving the lock to a Role in the Ichor namespace (audit F5c) is not done.
9. **Finalizers.** The override finalizer check also sees `UPDATE` of `TalosOperation` (the `etcd-restore` and `ca-rotation` finalizers). Every Ichor finalizer is now covered (`machine-removal`, `etcd-restore`, `ca-rotation`).
10. **Smaller.** The controller's cluster-wide `pods delete` is gone; the one direct delete (stale agent pods after an etcd restore) is a Role in kube-system (`config/rbac/pod_restart_role.yaml`). The `*-admin-role` helpers list their verbs instead of `*`, so `override` is never implied. `IchorInstallation` deletion needs the `allow-delete` annotation like a cluster. The kube-proxy switch policy reads its fields behind `has()`. Every webhook has `timeoutSeconds: 5`.

### As built

- `pkg/endpointguard` (`Forbidden`, `ForbiddenResolved`, `InDiscovery`); webhook checks in `internal/webhook/v1alpha1` (`talosmachine_webhook.go`, `machinereclaim_webhook.go`, `override_webhook.go`, `system_namespace.go`); constants `MachineApproveVerb`, `MachineApprovalSubresource`, `MachineEndpointVerb`, `MachineEndpointSubresource` in `api/v1alpha1/talosmachine_gates.go`.
- New CRD field `TalosEtcdBackup.spec.destination.s3.insecure` (optional, append-only).
- Tests: envtest webhook suites (`machine_gate_test.go`, `secret_references_gate_test.go`, `system_namespace_test.go`, `override_operation_finalizer_test.go`), the Job policy against a real apiserver in `api/v1alpha1/controller_jobs_policy_test.go` and against the real Job builders in `internal/controller/controller_jobs_policy_test.go`, and the chart tests in `dist/chart/hardening_test.go` and `user_roles_test.go`.

### Flows that change

- **ADR 0011 (discovery).** Unchanged for discovery-created machines: the endpoint is a declared address. Approving now needs `approve` on `talosmachines/approval`.
- **ADR 0103 (adopt, add by IP, reclaim).** Add by IP and reclaim (the UI API, and `ichor machine reclaim --cluster`) add the address to `spec.discovery.addresses` before anything else, and the controller creates the pending machine from it, so the endpoint is declared. `ichor init` and `ichor adopt` write approved machines at addresses outside discovery, so the user needs `approve` and `update` on `talosmachines/endpoint` (`ichor-admin` has both; cluster-admin has everything). Reclaim addresses in loopback, link-local, unspecified or multicast ranges are refused.
- **ADR 0086 (observed endpoint).** Unchanged: `status.observedEndpoint` is written by the controller, and the controller refuses to dial a forbidden one.
- **ADR 0106 (S3 store).** An in-cluster or external store over `http://` now needs `insecure: true`; `https://` is unchanged.
- **ADR 0059 (installer).** An installer Job that is still running after 30 minutes is killed.

### Not verified on hardware

Nothing here ran on a cluster other than envtest. To prove: an Ichor upgrade through the installer Job (named bind/escalate lists, Job policy, pod-restart Role in kube-system), a Cilium chart upgrade through the Cilium Job, an etcd restore (agent pod deletion), `ichor init` and `ichor adopt` as `ichor-admin`, and a Helm-only install with the default `admission.jobImageRepository`.

## ichor-ui and its auth

Decided and built (audit findings F2 to F18 of the ichor-ui audit; F1, the impersonation scope, is under "Credentials and the ichor-ui identity").

- **Request sizes (F2).** `connect.WithReadMaxBytes` on every handler: 256 KiB, and 1 MiB for the services that carry pasted documents (patches, Cilium values, reclaim's talosconfig). The limit applies to the decompressed message and is checked before the interceptors, so an unauthenticated caller cannot make the pod allocate. The mux caps any body at 2 MiB, the issuer's endpoints at 64 KiB, the internal listener at 4 KiB. Every RPC takes one request message (the streaming ones are server streams), so the limits do not touch streaming.
- **Timeouts (F5).** Main listener: `ReadHeaderTimeout` 10 s, `ReadTimeout` 60 s, `IdleTimeout` 120 s, `MaxHeaderBytes` 64 KiB, and no `WriteTimeout` because `StreamNodeLogs` and the downloads run for minutes; a request body is read within `ReadTimeout`, and a stream is bounded by its token (below). Internal listener (kube-apiserver only): 5 s, 15 s, 15 s write, 60 s idle, 16 KiB headers.
- **Security headers (F4).** One middleware on the whole listener, issuer pages included: CSP (`default-src 'none'`, `script-src 'self'`, `connect-src 'self'`, `img-src 'self' data:`, `font-src 'self'`, `base-uri 'none'`, `form-action 'self'`, `frame-ancestors 'none'`), HSTS one year without preload, `X-Frame-Options: DENY`, `nosniff`, `Referrer-Policy: no-referrer`, a minimal `Permissions-Policy`, `Cross-Origin-Opener-Policy: same-origin`. The theme snippet moved from `index.html` to `ui/public/theme-init.js`, so scripts need no `'unsafe-inline'`. **`style-src` keeps `'unsafe-inline'`**: two stock pieces create a `<style>` element at runtime (the Radix select viewport and the sonner shim), and editing preactcn components was ruled out; inline CSS cannot run script, so the residual risk is CSS injection only. With an external OIDC provider the browser itself fetches the provider's discovery document and token endpoint, whose origins are not known to ichor-ui, so `connect-src` widens to `'self' https:` in that case only; scripts stay `'self'`.
- **Token revocation (F6).** `authenticateLocal` compares the token's `iat` with the credential's `notBefore` (same cached read as `mustChange`, 10 s), so RevokeSessions and a password change end tokens already issued, not only refresh grants. `notBefore` has whole-second resolution: a token issued in the same second as the revoke survives it. The UI signs out after its own password change. The kube-apiserver cannot do this check; a token already given to `kubectl` lives out its lifetime.
- **Refresh rotation (F7).** A refresh grant returns a new refresh token and retires the old. State is one entry per live family (`families` in the user's credentials entry, so all replicas agree and RevokeSessions or a password change drops them): the id of the current token and the family's expiry, at most 16 per user. Writes are optimistic-lock patches of the Secret, retried on conflict. Presenting a retired token deletes the family (the owner signs in again) and logs it. Rotation keeps the session's start and expiry, so the absolute lifetime is not extended. A token from before rotation is accepted once and starts a family. A race between two processes refreshing the same token is serialised by the CLI's file lock, so it does not trip reuse detection.
- **Login limits (F8, F9, F10, F11).** Failures are counted per (source, user), so a stranger cannot lock `admin` out; per source with a looser limit (30 free failures, at most 1 min); and per user across sources, where 30 failures in an hour add a 2 s delay to every sign-in of that user and never refuse one. Tables hold 50 000 keys and evict the entries with the fewest failures first, so flooding fresh names cannot push out a slowed key. The source is the TCP peer, or with `--trusted-proxy-cidrs` the rightmost `X-Forwarded-For` address outside the trusted list. Counters stay per replica (a Lease per attempt was too expensive): N replicas give N times the free guesses. `/oidc/authorize`, `/login` and `/token` have a per-source token bucket (20, 10 and 20 per second, bursts 100, 50 and 100). A ticket buys one successful login (a typo may retry). Logins hash on their own argon2 slot with a queue of 8 and answer 503 beyond it; password operations of a signed-in user have another slot, so one cannot starve the other (the memory is the same two slots as before).
- **Role grants (F12).** CreateUser and UpdateUser refuse to grant a role the caller does not hold, unless the caller is admin. This is in code, not a ValidatingAdmissionPolicy; an external (OIDC) identity has no Ichor roles and is still gated by RBAC alone.
- **Streams (F13).** A stream gets a deadline at its token's expiry and ends with Unauthenticated ("your sign-in expired"). A disable inside the token lifetime is not seen by a running stream.
- **Errors (F14).** Internal, unknown, data-loss and unavailable errors reach the browser as the code plus a reference; the full error is logged with that reference. Errors the caller caused are unchanged.
- **/metrics (F15).** It stays on the main port but needs a Kubernetes token that may `get` the non-resource URL `/metrics` (the TokenReview and SubjectAccessReview of the controller's secure endpoint). The ichor-ui role gains `create` on `tokenreviews` and `subjectaccessreviews`; the ServiceMonitors send the pod's token; the scraping Prometheus needs the `metrics-reader` ClusterRole.
- **TLS (F16).** TLS 1.2 or newer, with only ECDHE and AES-GCM or ChaCha20 suites on 1.2 (1.3 suites are fixed by Go), curves X25519 and P-256, on both listeners. The `certreload` mutex is unchanged.
- **Dependencies and keys (F17, F18).** `oras-go` v2.6.2 (govulncheck: no vulnerability reached); `oidctrust` refuses an RSA signing key under 2048 bits.
- **No sign-in (F3).** With neither OIDC nor Ichor users, ichor-ui serves only the procedures in `readOnlyProcedures` (a new RPC is closed until listed; downloads are not listed). `--allow-unauthenticated` (`ICHOR_ALLOW_UNAUTHENTICATED`, chart `ichorUI.allowUnauthenticated`) restores the open mode explicitly; `/auth/config` says `anonymous: readonly|open` and the UI banner shows which. Compatibility: the installer reads a live ichor-ui with no users, no OIDC and no flag as "ran open" and sets the flag with a `NOTICE`, so an upgrade changes nothing; `ichor init` and `ichor adopt` start ichor-ui with the flag, `ichor init` without `--ui-url` prints a warning at the end, and `--ui-url` drops the flag. A Helm upgrade of such an install turns read-only until `ichorUI.allowUnauthenticated=true` is set (the chart prints a note). Making users the `ichor init` default was left out of scope.

Not done here: shared login counters across replicas (documented per replica), a ValidatingAdmissionPolicy for role grants, re-checking a disabled user inside a running stream before its token ends, narrowing the Role's `secrets: get` to names (F19, with the credentials work).

## Credentials and the ichor-ui identity

Findings: S1 (ichor-ui holds every CA private key through `get secrets`), F1/F7 (ichor-ui may impersonate any user and group), F19, S2, S3, S4, S6.

### 1. ichor-ui holds no CA key (S1)

Decided: the controller is the only reader of the secrets bundle. ichor-ui asks it for a credential by creating a `TalosCredentialRequest` (new kind, namespaced next to the cluster, short-lived), and the controller answers in the request's `status`.

- **Channel.** A CR, not a controller-local endpoint: it keeps "the UI changes the world by writing CRs", it is authorised and audited by the Kubernetes API server (the audit log shows who created the request, as the impersonated user, and which purpose), and it needs no new listener, certificate or NetworkPolicy. An endpoint would also have had to trust identity claims made by the caller, which is what the API server already checks.
- **No private key crosses the channel.** The requester generates the key pair in memory (Ed25519) and sends a PKCS#10 CSR in `spec.csr`. The controller checks the CSR signature (proof of possession), discards its subject and SANs, builds the subject from the purpose, signs with the CA, and writes the certificate and the CA(s) to trust into `status`. A reader of the request sees only public data. Nothing is written to a Secret, so ichor-ui needs no `get secrets` for credentials. The controller deletes a request 2 minutes after it was created.
- **Purposes** (`spec.purpose`) and the Kubernetes check the validating webhook makes with a SubjectAccessReview as the requester (new file `internal/webhook/v1alpha1/credentialrequest_webhook.go`; the other webhook files are untouched):

  | purpose | certificate | default TTL, cap | SAR as the requester |
  | --- | --- | --- | --- |
  | `public` | none; returns the cluster ID and the CA certificates | n/a | `issue` on `talosclusters/public` (ichor-ui's ServiceAccount) |
  | `node-reader` | Talos `os:reader`, signed by the current OS CA | 10 min, 15 min | `issue` on `talosclusters/node-reader` (ichor-ui's ServiceAccount) |
  | `support-bundle` | Talos `os:reader` | 10 min, 15 min | `download` on `talosmachines/support-bundle` (the signed-in user) |
  | `etcd-snapshot` | Talos `os:etcd:backup` | 10 min, 15 min | `download` on `talosmachines/etcd-snapshot` |
  | `packet-capture` | Talos `os:operator` | 10 min, 15 min | `download` on `talosmachines/packet-capture` |
  | `credentials-talos` | Talos `os:admin` or `os:reader`, signed by the rotation's new OS CA during a rotation, trust = old + new | 365 d, 365 d | `download` on `talosclusters/credentials` |
  | `credentials-kubernetes` | `admin` in `system:masters`, same rotation rule | 365 d, 365 d | `download` on `talosclusters/credentials` |

  The verbs on the download rows are the ones ADR 0084 and 0093 already use, so installed bindings keep working. The two `issue` rows are new and granted only to ichor-ui's ServiceAccount; they are for the UI's own reads (node pages, logs, effective configuration), which today already run as the ServiceAccount after the caller proved `get` on the machine. A request that fails the SAR is denied at admission and never reaches the controller; the controller re-checks purpose, TTL cap and CSR itself.
- **Who creates it.** The three download purposes and `credentials-*` are created as the signed-in user (impersonation), so the SAR and the audit log name that user. That needs plain `create` on `taloscredentialrequests`: the shipped roles `ichor-viewer`, `ichor-operator` and `ichor-admin` carry it and the webhook is the gate. `public` and `node-reader` are created as ichor-ui's ServiceAccount. ichor-ui caches the `node-reader` credential (one per cluster, in memory, renewed after half its TTL) and the `public` data (1 minute).
- **ichor-ui's Role** (namespace of the controller) loses `get` and `create` on all Secrets. What remains is by `resourceNames`: `ichor-user-credentials` (get, patch, update; Ichor users) and the issuer's own Secrets that it reads by name. The one `create` that stays is the MachineReclaim credential Secret, narrowed by the existing ValidatingAdmissionPolicy. The bundle Secret, S3 and registry Secrets are no longer readable by it. Where ichor-ui needs a rendering (effective configuration, CNI facts) it renders with a placeholder bundle whose keys are random and whose public parts (cluster ID, CA certificates) come from `public`; every secret field is masked in the output anyway.
- **Residual risk, stated.** A compromised ichor-ui can no longer exfiltrate a CA key, mint certificates offline, outlive a CA rotation, or forge Talos identities beyond a 15 minute window. It can still ask the controller for what the identity it acts as may ask for. With users off that is the ServiceAccount (no download verbs by default); with users on, it can only impersonate Ichor users (section 2) and each purpose is checked against that user's RBAC.
- CRD change: a new kind only; no existing field is touched.

### 1a. What was built, and where it differs from the first design

- `TalosCredentialRequest` (`api/v1alpha1/taloscredentialrequest_types.go`; scaffolded with kubebuilder), its reconciler (`internal/controller/taloscredentialrequest_controller.go`), the signing code (`internal/credentials/issue.go`: `NewCSR`, `IssueTalos`, `IssueKubernetes`, `PurposeLifetime`, `TalosRoles`) and the webhook (`internal/webhook/v1alpha1/credentialrequest_webhook.go`, a new file; registered in `cmd/main.go` and the chart's webhook list).
- ichor-ui: `internal/uiapi/credissuer.go` (create the request, poll its status with the ServiceAccount, cache the reader identity and the public facts). The Talos client factories take a `*TalosIdentity` (CA, certificate, key) instead of a bundle; `loadBundle` is gone. `GetMyKubeconfig` takes the Kubernetes CA from `public`.
- **Registry credentials.** Rendering the effective configuration read the registry `authSecretRef` Secrets by name, which would have kept `get secrets` on the whole namespace. The backend now renders through `maskedSecrets`, which answers every Secret read with a placeholder: the view shows them masked anyway. The cost: a missing or malformed registry Secret is no longer noticed in that view (the controller still reports it on the machine).
- **Acknowledging a CA rotation's credentials** checks that the new Kubernetes CA is accepted with a certificate of purpose `credentials-kubernetes` that lives ten minutes, asked as the caller. That needs the `download` verb on `talosclusters/credentials`, which acknowledging did not need before (it needed only to update the operation). The alternative, a purpose the backend could ask for as itself, would have let a compromised backend obtain a cluster-admin certificate at will, which is what this change removes. The UI shows the download and the continue buttons together to the people who hold the verb; the CLI path (`ichor cluster credentials --acknowledge`) is unchanged.
- **Roles.** The `ichor-credentials-role` and `ichor-diagnostics-role` ClusterRoles also carry `create` on `taloscredentialrequests`; the webhook is the gate, so plain creation is harmless. ichor-ui's ClusterRole gets `create` and `get` on the requests and the two `issue` verbs. Its namespaced Role keeps only `create` on Secrets (MachineReclaim credentials) and, with Ichor users, the named `ichor-user-credentials` rules. The installer's ClusterRole lists the new CRD.
- **Upgrade notes.** An install with `ichorUI.oidc.enabled` and empty `impersonate.groups` or `impersonate.users` no longer renders: list the groups and users, or set `ichorUI.oidc.impersonate.anyUser=true` for users. The controller image and the chart must be upgraded together (the backend needs the controller to answer its requests; a new backend with an old controller waits 20 seconds and fails each read).
- Not done: a UI choice of the `credentials-talos` lifetime shorter than a day (a rotation hand-off must outlive the rotation); the controller's and the CLI's own dial certificates keep a year (section 3).

### 2. Impersonation (F1, F7)

Decided: ichor-ui's ServiceAccount may impersonate only named groups and only named users, written by the chart or the controller; there is no "any name" mode unless the operator says so in a value.

- **Groups.** `resourceNames` is always emitted: `ichor:viewer`, `ichor:operator`, `ichor:admin` plus `ichorUI.oidc.impersonate.groups`. With OIDC on, the chart fails to render when `impersonate.groups` is empty (before, empty meant any group). A `system:` group can therefore never be impersonated, structurally.
- **Users.** Kubernetes cannot match a name by prefix, and a user such as `system:kube-controller-manager` is as dangerous as a group. With Ichor users (ADR 0094) the controller keeps the list: it maintains the ClusterRole `ichor-ui-impersonate` with one `resourceNames` entry `ichor:<name>` per IchorUser for users and one `ichor:<role>` per role any of them has for groups (custom roles are allowed, so the three built-in groups alone would not do). An empty list of rules grants nothing; a rule is never written without `resourceNames`. It reconciles on every IchorUser change and every two minutes, and does not watch ClusterRoles (that would need list and watch on all of them). It holds `escalate` and `update` on that one ClusterRole by `resourceNames`, nothing more, so it can grant an impersonation right it does not hold. The binding is installed by the chart. With OIDC on, `ichorUI.oidc.impersonate.users` must be listed, or the operator sets `ichorUI.oidc.impersonate.anyUser: true` to accept the residual risk knowingly; an empty list without the flag fails rendering. A new Ichor user can sign in a few seconds after creation (the controller's reconcile).
- **Rejected.** A Kubernetes structured authorisation config with CEL: its CEL only filters which requests go to an authorisation webhook, it cannot deny by itself, and it needs a webhook service in the managed cluster. A ValidatingAdmissionPolicy cannot see the impersonator. Forwarding a minted token instead of impersonating would move the problem to the signing key ichor-ui already holds for the issuer.
- **Residual risk, stated.** With OIDC and `anyUser: true`, a compromised ichor-ui can act as any user by name (not as a `system:` group), including users bound to powerful roles. Without the flag (the default after this change) it can act only as the listed users, which are the users Ichor is already trusted to represent.

### 3. Lifetimes (S4)

Decided: in-process dial certificates live 10 minutes (ichor-ui; minted per call by the controller on request, section 1). The controller's own dial certificate and the CLI's stay as they are: those processes hold the CA key anyway, so a shorter certificate gains nothing. Files handed to operators keep the default of one year (the user's decision: downloads are Ichor-side and a shorter default would break existing workflows) and gain a choice: `--ttl` on `ichor cluster credentials` (default and cap 8760h) with `--read-only` for an `os:reader` talosconfig, and the same two choices on the UI credentials download (1 year, 30 days or 1 day; read-only).

### 4. etcd backup key (S2)

Decided: backups are sealed with a key derived (HKDF-SHA256, info `ichor.maesh.dev/etcd-backup/v2`, salt cluster ID) from the secretbox encryption secret of the bundle, which only control-plane machine configs carry and which Ichor never rotates (so a CA rotation keeps old backups readable, ADR 0063). The v1 key (from `Cluster.Secret`, present in every worker config) is derived too and tried second when reading, so existing backups restore; every new backup is v2. Rotating the secretbox key is therefore not part of the CA rotation (section 6). Built: `etcdbackup.DeriveKey`, `Key{Current, Legacy}`, `Decrypt` tries both.

### 5. Maintenance pin (S3)

Decided: the first apply goes through an unauthenticated TLS connection, so Ichor pins what it can. The SPKI fingerprint of the maintenance certificate seen when the machine is discovered (re-recorded while the machine is not approved) is stored in the new `status.maintenanceFingerprint`, and the controller refuses to send the initial configuration, which carries the CA keys, to a node that presents another one. A machine that never went through discovery (added by IP, reclaimed) is pinned at its first provisioning probe. Within one `ichor init` run the identity read and the apply share one pin. Talos issues a new maintenance certificate when a node reboots or changes address; an operator who has checked the new fingerprint (named in the machine's Ready condition) accepts it with the annotation `ichor.maesh.dev/trust-maintenance-fingerprint`. This narrows the window to the time between discovery and apply; it cannot authenticate the first sight, so the provisioning network still must be separate (design, Trust). Built: `internal/controller/maintenancepin.go`.

### 6. CA rotation coverage (S6)

Decided: documented as a follow-up. Ichor's rotation covers the Talos API CA and the Kubernetes API CA. The etcd CA, the aggregator CA, the service-account signing key and the secretbox key are not rotated. Kubernetes needs a dual-key period for the last two (the old key must keep verifying or decrypting until every token and Secret is re-written), which is a multi-stage operation with a re-encryption pass, not an extension of the current two-CA stages; the etcd CA rotation needs every member restarted with both CAs trusted in a defined order; and the backup key (section 4) derives from the secretbox secret, which would have to be versioned first. The limit is stated in the docs so that "rotate after compromise" is not read as complete.

## Secrets handling

- **Restore marker (S7).** The marker in the backup bucket is stored as `{"marker": ..., "mac": ...}`. The MAC is
  HMAC-SHA256 over the marker JSON, keyed by HKDF over the backup passphrase with info
  `ichor.maesh.dev/etcd-backup/marker/v1`, so it follows whatever key material the backup passphrase derives from.
  A signature mismatch is an error. Snapshot rollback by replaying an older valid snapshot is not addressed here.
  Amendment (before v0.1.0): an unsigned marker is refused as well (`ErrMarkerSignature`, "take a new backup with this
  version"); the earlier compatibility path for unsigned markers served builds that never shipped, so it was removed.
- **Scrubber (S8).** `bundleSecretMaterial` is derived by reflection over every exported string and `[]byte` field of
  the Talos secrets bundle (trimmed, so a PEM that lost its trailing newline still matches), so keys the bundle gains
  or that were missing (etcd, aggregator and service-account keys, encryption secrets) are redacted. Rotation CAs were
  already added by the rotation code. Base64 re-encodings are not scrubbed.
- **Local files (S9).** `saveKubeconfigFile` now chmods an existing directory to 0700 like `saveBundleFile`. The local
  `secrets.yaml` stays plaintext (0600, atomic); encrypting it is deferred, but `loadBundleFile` now warns on stderr when
  the file is group or world accessible. Downloaded etcd snapshots were already written 0600; the CLI now states that
  they are not encrypted.
- **Reclaim Secret name (S10).** Uses `crypto/rand` (`rand.Text`, lowercased, 16 characters) instead of the
  `math/rand` based `utilrand`.
- **Other (S11).** The manager logs in production mode (JSON, no stack traces on warn); `--zap-devel` turns development
  mode back on, other zap flags are unchanged. The kubetrust trust file is written through a unique `os.CreateTemp`
  file in the same directory and renamed (mode 0644, public CAs only). Refusing `http://` S3 endpoints is left to the
  endpoint gating work.

## Supply chain and workloads

Audit: workload hardening and the supply chain. Fixed here, all of it without a CRD change.

### Decisions

- One release path. The images that ship are built by `test/harness/ui-demo/images.sh` (crane, no container
  runtime), as ADR 0104 unit 7 already had it. The Dockerfiles are for local `docker build` and carry the same
  digests and flags, so there is no second release path to drift. `release-images.yml` and `release-chart.yml` are
  replaced by one `release.yml`.
- Reproducible and pinned. The distroless base, `golang` and `node` images are pinned by digest (the tag stays
  in front for Renovate). Binaries build with `-trimpath` and `-ldflags "-s -w -X ...Version"`; the layer tar has
  fixed owner, mtime and order; the push uses `--set-base-image-annotations`. Pushing the same binaries twice gives
  the same digests (checked locally). goreleaser stamps `internal/version.Version` (it set `main.version`, which
  nothing reads).
- Release gating and split. A `v*` tag runs `verify` (module drift, lint, tests, UI CI, govulncheck, npm audit),
  then `build` (dependencies fetched, `contents: read`, no registry token), then `publish` in the `release`
  environment, the only job with `packages: write` and `id-token: write`. `publish` runs crane, cosign, syft and
  the attest actions and fetches no Go or npm dependency.
- Signing and attestation. Keyless cosign (GitHub OIDC) signs the three images and the OCI chart by digest; syft
  writes an SPDX SBOM per image, attested with `actions/attest-sbom`; `actions/attest-build-provenance` attests
  images and chart. Every new action is pinned by commit SHA. Verification is documented in
  `docs/guides/install-with-helm.md`. Ichor does not verify a signature before it runs the installer Job; digest
  resolution of the installer image (`spec.installerDigest`, audit M3) is deferred.
- CI scanning and updates. `govulncheck ./...` and `npm audit --omit=dev --audit-level=high` run on every PR and
  tag (oras-go is bumped to v2.6.2 for GO-2026-6497 and GO-2026-6499). `renovate.json` covers gomod, npm, GitHub
  Actions and Dockerfiles with digest pinning, the images.sh base, and the Nix flake lock; grouped, three open
  PRs at most, Monday mornings, a three-day release age. Renovate rather than dependabot because it handles Nix and
  digests. `trivy` and OSSF scorecard from the audit are not added (govulncheck and npm audit cover the
  dependencies; the images are distroless static).
- Pod Security Admission. The install manifest labels `ichor-system` `enforce`, `audit` and `warn` at
  `restricted` (enforce pinned to v1.32, the others `latest`). Every Ichor pod passes: non-root, seccomp
  RuntimeDefault, no privilege escalation, all capabilities dropped, only emptyDir, secret and projected volumes,
  no host namespaces. The Cilium Helm Job needs `hostNetwork` and runs in `kube-system`, so it is outside. The
  chart does not own the namespace; NOTES.txt and the guide give the `kubectl label` command.
- Pods and Jobs. Controller and UI run as 65532:65532 and under a new PriorityClass `ichor-critical` (value
  1000000, below every `system-*` class, which a user namespace cannot use), shipped in the manifest and the chart
  (`priorityClass.create`). Installer and Cilium Jobs name user and group 65532 and carry requests 50m/64Mi and
  limits 1 CPU/512Mi. The service-account token stays mounted: both pods and the Jobs need the API. The installer
  ClusterRole gets `scheduling.k8s.io/priorityclasses` create/get/patch/update (the manifest now contains one).
- Replicas and PDB. The raw manifest stays at one replica each and ships no PDB (a `minAvailable: 1` budget on one
  replica blocks node drains); the chart runs two and keeps its PDBs. Documented, not changed.
- Network. The Cilium default stays opt-in. The chart's policies gain `externalCIDRs`, `talosCIDRs` and
  `uiIngressCIDRs`; when set they replace the `world` entity for those rules. The installer's static copy keeps
  `world`, because it has no values. FQDN rules and a metrics-namespace restriction are deferred.
- Network (amended). Supersedes the bullet above for the defaults: the policies are on by default wherever Cilium is
  the CNI (chart: `networkPolicy.cilium.enabled` unset renders them when the `cilium.io/v2` API is served; `false` is the
  escape hatch, with a NOTICE on upgrade). The controller and ichor-ui lose their static `world` egress to Talos nodes,
  API servers and S3; the controller maintains `ichor-controller-talos-egress` with exactly the addresses and names its
  CRs name, updated before a discovery probe is dialled. DNS egress goes through Cilium's DNS proxy so `toFQDNs` works.
  The flow table and the Cilium semantics relied on are in ADR 0100's amendment of 2026-11-02. `externalCIDRs`,
  `talosCIDRs` and `uiIngressCIDRs` remain as static fallbacks; `world` stays only for ichor-ui's 443 (OIDC issuer,
  pull request host) and its serving port.
- Workflow hygiene. `go mod tidy` no longer rewrites files in CI (`go mod tidy -diff` and `go mod verify` fail on
  drift); kind is pinned by version and sha256; every job has `timeout-minutes`; `test-chart` no longer runs on
  every branch push; `actionlint` (in the Nix shell) runs in the lint workflow; `npm ci --ignore-scripts` in the
  Makefile and `Dockerfile.ichor-ui` (the UI build and tests pass without install scripts).

### As built

- `Dockerfile`, `Dockerfile.ichor-ui`, `Dockerfile.installer`, `test/harness/ui-demo/images.sh` (modes `build` and
  `push`, `DIGEST_FILE`), Makefile targets `demo-images-build` and `demo-images-push`, `.goreleaser.yaml`.
- `.github/workflows/release.yml` (new; the two old release workflows are gone), `test.yml`, `test-e2e.yml`,
  `test-chart.yml`, `lint.yml`, `pr-title.yml`, `renovate.json`, `flake.nix` (cosign, syft, actionlint).
- `config/manager/manager.yaml` (namespace labels, PriorityClass), `config/manager-ui/manager.yaml`,
  `config/rbac/ichor-installer/role.yaml`, `internal/controller/job.go` with `job_test.go`.
- Chart: `values.yaml`, `templates/extras/priority-class.yaml`, `templates/network-policy/cilium.yaml`,
  `templates/rbac/ichor-installer.yaml`, `NOTES.txt`, `hardening_test.go`.
- Not verifiable without GitHub or a cluster: the whole `release.yml` run (cosign, attest, environment approval),
  Renovate's reading of `renovate.json`, and a server-side PSA dry-run against a live cluster. The restricted
  profile was checked by reading the pod specs.
- Maintainer steps in the repository settings (not in files): create the `release` environment with required
  reviewers, and a tag ruleset for `v*`.
- ADR 0104 unit 7 named the two removed workflows; `release.yml` replaces them.

## Typed hardening options for managed clusters

Decided: every hardening of a cluster Ichor creates is a typed field of `TalosCluster.spec.security`, off by default, for new and existing clusters. Unset means the rendered machine config and Cilium values are byte-identical to before (proved by rendering a control plane and a worker from one fixed secrets bundle before and after the change, and by a test that an empty or disabled `spec.security` renders the same bytes). Machine config stays typed: Ichor builds the Talos documents itself, below every patch, and no raw YAML is accepted. `docs/guides/hardening.md` lists every option, what it does and what applying it costs.

- **Firewall** (`security.firewall`). Enabled renders `NetworkDefaultActionConfig` `block` and one `NetworkRuleConfig` per service (`ichor-fw-*`) following the Talos v1.14 recommended control-plane and worker rule sets: apid from the controller, operators and control planes; trustd between nodes; kubelet from control planes and the pod subnets; etcd between control planes; 6443 from the controller, configured sources, nodes and the pod and service subnets; Cilium tunnel (8472, or 6081 for geneve), health (4240) and WireGuard (51871); KubeSpan (51820). `allow` entries open host ports for host-network services. The field set needs the node CIDRs as input because render sees one machine at a time. The preflight is the admission webhook: it refuses a firewall whose `controllerCIDRs` does not contain the address the webhook's pod routes to the endpoint from (a UDP connect, nothing is sent; skipped when the endpoint does not resolve). No reboot (fieldmeta already classes both kinds reboot-free). Not done: applying in Talos `try` mode with automatic revert, which needs a hardware run.
- **WireGuard** (`security.wireGuard`). Sets `encryption.enabled`, `encryption.type: wireguard` and `encryption.nodeEncryption` in the effective Cilium values, after the user layer, so the typed option wins. The existing values hash re-runs the Helm Job; the agents restart in turn, no node reboot. Strict mode is left to `spec.cni.cilium.values`.
- **Pod Security and audit** (`security.podSecurity`, `security.audit`). Merged over Talos's own `KubeAdmissionControlConfig` PodSecurity document, so only the levels set change and `kube-system` stays exempt; audit is one of three presets (`metadata`, `rbacChanges`, `writes`), none logging Secrets or tokens above Metadata. Both need a v1.14 contract and are refused on an older one rather than silently raising it. The kube-apiserver restarts (fieldmeta `apiServerRestart`), no node reboot.
- **Disk encryption** (`security.diskEncryption`: `tpm`, `nodeID`, `kms`). Renders `VolumeConfig` for STATE and EPHEMERAL (LUKS2, slot 0, EPHEMERAL locked to STATE); `static` is not offered; `kms` must be `https`; `requireSecureBoot` maps to the TPM key's `checkSecurebootStatusOnEnroll`. On a cluster with approved machines the webhook refuses enabling or changing it unless the annotation `ichor.maesh.dev/confirm-disk-encryption-wipe: "true"` is present; Ichor never wipes a node, the guide documents the one-node-at-a-time reset-and-add path. Ichor records no TPM inventory, so it cannot pre-check TPM presence; Talos's own enrolment failure is the check (to be confirmed on hardware).
- **Talos API access from pods.** A `TalosConfigPatch` with `machine.features.kubernetesTalosAPIAccess` is admitted with a warning (stronger wording for `os:admin` in `allowedRoles`); amended 2026-11-02: it was refused without `override`, which the user found too strict for the user's own cluster ("ichor 以外の部分はあまり強く制限しないで"). Render drops the feature from non-control-plane machines. An update that leaves the documents alone is not re-checked. The rule lives in `internal/webhook/v1alpha1/security_options.go`.
- **Not modelled.** SecureBoot and UKI, kernel lockdown, an `os:reader` talosconfig role, kubelet serving-certificate bootstrap: follow-ups.
- **Needs hardware.** That a block-mode firewall with these rules leaves a Talos plus Cilium cluster healthy (including pod to apiserver traffic and DHCP renewal); that the webhook's routed source matches the address Talos sees when the controller runs in a pod; the WireGuard rollout; the apiserver restart for the two Kubernetes documents; the TPM and KMS behaviours.

## Hardware results: cluster options

Run 2026-11-02 on the e2e harness (prefix `ichor-e2e-s2`): Ichor `v0.0.1870000002-gb9fa12af` (HEAD b9fa12af), Talos v1.14.3, Kubernetes v1.37.1, Cilium chart 1.18.0 (geneve tunnel), one control plane and one worker created by `ichor init` plus `ichor machine approve`, no sign-in (`--allow-unauthenticated`). Pod and service subnets 10.248.0.0/16 and 10.249.0.0/16; nodes are outer pod IPs (10.244.1.20 control plane, 10.244.1.164 worker). `ichor init` to both nodes Ready took about 11 minutes (about 2 minutes for the worker after approval). The options were enabled one after another on the running cluster. After each step all nodes Ready, kube-system pods Running, DNS from a pod on each node, a pod reaching the apiserver Service, pod to pod across nodes, and both TalosMachines Ready with no NodeUnreachable were checked, except where noted.

1. **Firewall: works; two findings about the controller's source address.**
   - Converged within 1 to 2 seconds of the patch on both nodes (`NfTablesChain` `ingress`, policy drop, and `prerouting` exist; there is no `networkrulechains` resource in Talos v1.14.3, use `talosctl get nftableschains ingress -o yaml`). Health after the change: all checks passed, TalosMachines stayed Ready, the controller reconciled the machines normally.
   - Preflight: a `controllerCIDRs` of `192.0.2.0/24` is refused: `spec.security.firewall.controllerCIDRs [192.0.2.0/24] does not contain 10.248.0.108, the address Ichor's controller connects from ... if the controller runs in a pod, list the pod network and its node's address`. The webhook's routed source was the controller pod's IP, as the design assumed.
   - What the nodes see (from `/proc/net/nf_conntrack` through `talosctl read`): the controller reaching apid on its own node arrives from the **pod IP** (10.248.0.108); the controller reaching apid on another node arrives from the **controller's node IP** (10.244.1.20), because Cilium masquerades pod traffic to a remote node's address. So both must be listed, as the guide says: the pod range (or the node's pod CIDR) and the node address. The same masquerade means a pod on node A that talks to a host port on node B is seen as node A: pod-level restrictions only hold for host ports of the pod's own node, and any pod on a listed node reaches what that node's address may reach. With `controllerCIDRs` narrowed to the control plane's pod CIDR plus its node IP, a pod on the worker was dropped on the worker's apid (50000), on the control plane's apid, etcd (2379) and kubelet, and allowed on trustd (50001, node CIDRs), 6443 and 4240; a pod on the control plane (same /24 as the controller) still reached apid on both nodes. A port in no rule (9999, 22) is dropped (timeout), not refused.
   - A narrow `controllerCIDRs` (the control plane's pod /24) would lock the controller out if it were scheduled on the worker; not tried.
   - DHCP renewal: not observed. The lease is 23976 hours, so no renewal happens in the run. A DHCP REQUEST/ACK exchange happened before the firewall (hostname change); Talos's DHCP client uses raw sockets, which a netfilter input chain does not see, but this was not exercised under block mode.
   - **Lock-out and recovery, tried and safe.** The webhook only checks the controller's route to the endpoint, so it accepts `controlPlaneCIDRs` that do not contain the control plane's own address: setting `controlPlaneCIDRs: [10.244.1.99/32]` with `controllerCIDRs: [10.248.0.0/24]` was admitted, the worker received the rule, and the controller then reported on the worker `Ready=False CurrentConfigUnavailable` and `NodeReachable=False NodeUnreachable: the node's current configuration cannot be read` (clear). Reverting the spec alone cannot help, because the controller cannot reach the node to deliver it. Recovery that worked: from a source in `talosAPICIDRs` run `talosctl -e <node> -n <node> patch machineconfig --patch @rule.yaml` with a `NetworkRuleConfig` (`portSelector: {ports: [50000], protocol: tcp}`, `ingress: [{subnet: <control plane>/32}]`), then revert the spec: the machine was Ready again 26 seconds after the talosctl patch and the controller removed the temporary rule. Keep a `talosAPICIDRs` entry for an operator workstation before enabling the firewall; without one the way back is the console or a reset.
2. **WireGuard: works after a manual Cilium agent restart; the guide was wrong about the rollout.**
   - The patch rendered UDP 51871 into the ingress chain of both nodes within 2 seconds and re-ran the Cilium Helm Job (revision 2, 33 s, 49 s from patch to `enable-wireguard: "true"` in `cilium-config` and `CNIValuesApplied=True`). **The agent pods did not restart** (the DaemonSet carries no config hash) and `cilium-dbg encrypt status` kept saying `Encryption: Disabled` while `CNIValuesApplied` was True. `kubectl -n kube-system rollout restart ds/cilium` took 14 seconds, after which both agents report `Encryption: Wireguard`, `cilium_wg0`, port 51871, `Number of peers: 1`, the `cilium_wg0` packet counters grow with cross-node pod traffic (79 to 129 rx packets for ten requests), and DNS, apiserver and cross-node pod traffic work. UDP 51871 passed the block-mode firewall between the nodes (listed `nodeCIDRs`). The guide now says the restart is manual.
3. **Pod Security restricted: works; needs `ichor config apply`.**
   - After the patch the control plane is `Ready=False APIServerRestartRequired` and nothing happens until a config rollout is started (the condition says so; the guide did not). `ichor config apply --follow` took 171 s in total with one control plane (kube-apiserver container restarted 49 s after the start; the operation finished the machine at 112 s). During it the API server is unreachable, the controller logs errors and kube-controller-manager and kube-scheduler restarted once each (lost leases); nothing else restarted.
   - `kubectl run --privileged` in `default` is refused (`violates PodSecurity "restricted:latest": privileged ...`), a plain busybox pod is refused, a restricted-compliant pod is admitted; a privileged pod in `kube-system` is admitted (exempt); existing non-compliant pods in `default` kept running.
4. **Audit `writes`: works.** `ichor config apply` took 109 s (apiserver restart 55 s after the start). `/var/log/audit/kube/kube-apiserver.log` (readable with `talosctl read`) had 38 184 events: 208 at `Request` level (the label patch on a Service carried its body), the rest `Metadata`. A Secret and a ConfigMap created with marker values appear at `Metadata` only, the markers (and the base64 of the Secret one) are absent, and no event for `secrets`, `configmaps`, `serviceaccounts/token` or `tokenreviews` is above `Metadata`.
5. **Disk encryption `nodeID` via wipe and rejoin: works, with friction; `tpm` on a TPM-less node fails and Ichor does not say why.**
   - Without the annotation the update is refused (`spec.security.diskEncryption changes for a cluster with 2 approved machine(s): ... Set the annotation ichor.maesh.dev/confirm-disk-encryption-wipe="true"`). With it, both machines went `Ready=False RebootRequired` (VolumeConfig fields) and nothing was applied, as designed.
   - The path: `ichor node reset <node> --override <reason>` (needs the override verb; since ADR 0110 `--override` alone, with an optional `--override-reason`), delete the stale Kubernetes Node, approve the pending machine again. Observed friction, none of it fixed here: (a) `--confirm` must be the machine name (its UUID) for the TalosOperation webhook, but the CLI asks for the node name and then the webhook refuses ("reset needs confirm equal to the machine name"); passing the UUID as NODE and as `--confirm` works; (b) a bare pod (my probe) on the node blocked the drain and the operation did not retry for more than ten minutes after the pod was gone, until an annotation poked the TalosMachine (fixed in 6fa38b78, after 8e749c1e: a blocked drain is requeued every 15s, see ADR 0010's amendment; not yet rechecked on hardware); without a blocking pod the reset took 45 s; (c) the reset leaves the Kubernetes Node, whose InternalIP is excluded from discovery (`knownHosts`), so the machine was not rediscovered (five minutes and a poke) until `kubectl delete node`; the guide's "Reset and add" (ADR 0107) is for machines of another cluster and refuses one already in Ichor.
   - After rejoin: node Ready 119 s after approval; `talosctl get volumestatus` shows `encryptionProvider: luks2`, `configuredEncryptionKeys: [nodeID]`, `encryptionSlot: 0` for STATE (`/dev/vda5`) and EPHEMERAL (`/dev/vda6`). The control plane, not wiped, stays unencrypted and `RebootRequired`. Firewall and WireGuard were applied from the first configuration (Cilium peers 1, ingress chain present).
   - TPM: with `type: tpm` the reset worker installed, rebooted and stuck in stage `booting` with STATE `failed`: Talos logs every 100 ms `failed to call key handler ... TPMKeyHandler: stat /dev/tpm0: no such file or directory` and `error formatting and encrypting volume: no handlers available`. Ichor shows only `Provisioned=False NodeStarting: the node is in stage "booting"` after four minutes: the cause is not reported. `requireSecureBoot` not tried.
6. **`kubernetesTalosAPIAccess` with `os:admin`: refused for a user without `override`.** As `--as=s2-op --as-group=ichor:operator`: `allowedRoles contains os:admin, which gives every pod in [kube-system] a Talos administrator (use os:reader, or hold the "override" verb on this patch to override)`; `os:reader` is admitted with the warning. As cluster-admin (system:masters, which holds every verb) the `os:admin` patch was admitted with the warning, as the rule says; it was deleted at once. A cluster-layer patch with this feature is refused for a worker at render (`can only be enabled on control plane machines`); use a `role: controlplane` target.

Not run: KMS, `requireSecureBoot`, a controller scheduled on the worker, strict mode. Teardown: all `ichor-e2e-s2` objects deleted and confirmed; the cached ISO disk untouched.

### Fixes after the hardware run (2026-11-02)

Not yet rechecked on hardware unless a line says so.

1. WireGuard: the managed Cilium values now carry `rollOutCiliumPods: true` whenever `wireGuard.enabled` is set, so the chart puts a `cilium-config` checksum on the agent pod template and the DaemonSet rolls (rolling update) when the Helm Job changes the config. `CNIValuesApplied` was already set only after `daemonSetRolledOut` (observed generation, updated and ready pods), so it now means the agents run the new values. Chosen over a controller-side annotation patch because Helm owns the DaemonSet and the chart already has the switch. Recheck: patch wireGuard on a running cluster, expect `Encryption: Wireguard` with no manual restart.
2. Firewall lock-out: the TalosCluster webhook refuses a firewall change whose `controlPlaneCIDRs` miss an approved control plane's address (the controller reaches the other nodes from it, so Ichor itself would be cut off; address from the machine's observed or declared endpoint, hostnames are skipped). A `nodeCIDRs` that misses a machine and an empty `talosAPICIDRs` are warnings only, per the 2026-11-02 direction that errors are for what breaks Ichor's own operation or cannot render, and advice about the user's cluster is a warning. The recovery that worked is in the guide.
3. Reset: a reset asked for by hostname is stored with the machine name as `confirm`, which the TalosOperation CRD compares with `machineRef.name`. A removal whose drain fails (a bare pod) now returns the error so the reconcile is retried with backoff, writes the pod into the `Removal` condition and a Warning Event. There is no force-drain option; the guide says to delete the pod. The Kubernetes Node of a reset machine is annotated `ichor.maesh.dev/reset-by-ichor` and discovery ignores its address while it is not Ready; it is not deleted because the kubelet could register it again.
4. A node stuck in stage `booting` reports volumes in phase failed and their error in the `Ready` message.
5. `kubernetesTalosAPIAccess`: render drops it from workers (Talos allows it on control planes only), so a cluster-layer patch works and renders where it is valid. The same direction replaces the admission refusal of `os:admin` (and the override override) with a Warning; the verb is no longer consulted for this patch rule. This amends the earlier rule of this ADR.
6. Confirmed designed: ADR 0080's amendment leaves API-server restarts to a config rollout. The `APIServerRestartRequired` and `RebootRequired` messages now give `ichor config apply --cluster NAME`, and the guide says so.

## Hardware results

### 2026-11-02: Ichor's own hardening, e2e harness (lane s1)

Setup: Talos v1.14.3, Kubernetes v1.37.1, Cilium 1.18 through Ichor, one control plane and one worker made by `ichor init --preset single --cni cilium --cni-tunnel geneve --ui-url https://<LB> --admin-user admin` with the previous main `f3fc27f3` (`v0.0.1870000000-gf3fc27f3`), then `ichor upgrade ichor` to `b9fa12af` (`v0.0.1870000001-gb9fa12af`). A third node (in maintenance mode) and a second worker were added for the discovery step. Hubble is not installed; drops were read with `cilium-dbg monitor --type drop` on every agent.

**1. Upgrade path: FAILED once, passed after two workarounds (two product bugs).**

- First `ichor upgrade ichor` (operation `ichor-9gpg6`): the installer Job failed and rolled back from its snapshot (clean: old pods kept running). Failing step, verbatim: `applying ValidatingAdmissionPolicy ichor-ichor-controller-job-image: ValidatingAdmissionPolicy.admissionregistration.k8s.io "ichor-ichor-controller-job-image" is invalid: spec.paramKind: Forbidden: user &{system:serviceaccount:ichor-system:ichor-installer ...} must have "get" permission on all objects of the referenced paramKind (kind=IchorInstallation, apiVersion=ichor.maesh.dev/v1alpha1)`. **Bug:** the installer ClusterRole (`config/rbac/ichor-installer/role.yaml`, and the chart's copy) holds no `get` (the apiserver asks for it cluster-wide) on `ichorinstallations`, so no upgrade through the installer can create the `ichor-controller-job-image` policy. Workaround used: a ClusterRole/Binding giving `ichor-installer` `get,list,watch` on `ichorinstallations`; the second run (`ichor-8sr7w`) completed in 53 s.
- After the upgrade: PriorityClass `ichor-critical` (1000000) exists and both pods run under it; the namespace carries `pod-security.kubernetes.io/{enforce=restricted (v1.32),audit,warn=restricted}`; the controller, ichor-ui and the installer pods were admitted and Running with no PSA warning, rejection or violation event; the four ValidatingAdmissionPolicies and bindings exist; the Jobs were admitted under the Job policies; ichor-ui and the controller restarted cleanly (a rollout restart of the controller also came up and took the lease within 20 s).
- **Bug (breaks logins after an upgrade or an `ichor init` install):** the impersonation ClusterRole is `ichor-ichor-ui-impersonate` in the cluster (kustomize `namePrefix`), but `controller.ImpersonationRoleName` is the constant `ichor-ui-impersonate`. The controller logged `the impersonation ClusterRole does not exist; nothing to keep`; the installed role had `rules: null`, so every signed-in request failed: `ichorusers.ichor.maesh.dev is forbidden: User "system:serviceaccount:ichor-system:ichor-ichor-ui" cannot impersonate-on:user-info:list resource "ichorusers"`. The controller's own `resourceNames` (`config/rbac/role.yaml`) name the unprefixed role too. Workaround used: an unprefixed ClusterRole `ichor-ui-impersonate` with no rules plus a binding for the ichor-ui ServiceAccount; the controller then wrote `[{users, resourceNames: [ichor:admin]}, {groups, resourceNames: [ichor:admin]}]` ("updated the impersonation ClusterRole", users: 1) and logins worked. The chart (no prefix) is not affected by this reading; not run.
- One `Reconciler error ... creating ichor-controller-talos-egress: ciliumnetworkpolicies.cilium.io "ichor-controller-talos-egress" already exists` was logged once during the rollover (a create race between the old and new pod); the policy existed afterwards.

**2. Cilium default deny: PASSED.**

- CiliumNetworkPolicies present: `ichor-controller-manager`, `ichor-default-deny`, `ichor-installer`, `ichor-ui` and the controller's `ichor-controller-talos-egress`. `cilium-dbg endpoint list` shows ingress and egress enforcement Enabled for the controller and ichor-ui.
- `ichor-controller-talos-egress` held exactly the machines' addresses (`10.244.0.55/32` control plane, `10.244.2.118/32` worker, the same as the machines' `spec.endpoint` and `spec.discovery.addresses`), ports 50000, 6443, 2380.
- Steady state (150 s on both agents): no policy drops for the controller or ichor-ui. The only `xx drop` lines were `Unsupported L3 protocol` IPv6 router solicitations (`fe80::... -> ff02::2`) from the two pods' identities, which no policy rule governs.
- Discovery: `ichor cluster discovery add 10.244.1.68/30` at 11:02:38 covering the third node (`10.244.1.70`); the pending machine existed at 11:02:47, found on the first pass; the policy listed `10.244.1.68/30` and `10.244.1.70/32` within 8 s; no policy drops during the 150 s window. `ichor machine approve ... --role worker` as `ichor-admin` (new `approve` verb) was accepted and the node was Ready 4 min 20 s later; no policy drops on the two monitored agents during the join (the new node's agent was not yet monitored).
- DNS and `toFQDNs`: no resolver errors in the controller or ichor-ui logs; a TalosEtcdBackup whose S3 host is the name `10.108.131.191.sslip.io` got a `toFQDNs` rule (`matchName` plus port 9000), the controller resolved it and reached the store (the store answered `Could not parse the specified URI`, a virtual-host parsing refusal of the throw-away S3 server, so network and DNS worked). In-cluster `toServices` (item 4 of ADR 0100) was not run.
- Observation: in this shape (outer LoadBalancer to an inner NodePort) every external client reaches ichor-ui from the same source address (the node), see item 4.

**3. Credentials: PASSED, with one RBAC defect and a note on the CLI.**

- `ichor cluster credentials ichor-e2e-s1` (and `--ttl 1h --read-only`) work, but the CLI reads the secrets bundle with the caller's own Kubernetes rights and signs locally; it does not go through ichor-ui and creates no `TalosCredentialRequest`. The files worked: `kubectl get nodes` with both; `talosctl version` with the admin and the read-only talosconfig; the read-only one (`Roles: os:reader`, expiry 50 minutes) got `PermissionDenied desc = not authorized` for `get machineconfig` while the admin one listed it.
- Through ichor-ui: `DownloadCARotationCredentials` (the only download path ichor-ui has for a talosconfig and kubeconfig) with `lifetimeSeconds: 3600, readOnly: true`, during the rotation below, created `cred-...` requests with purposes `credentials-kubernetes` and `credentials-talos`, both TTL 1h, in the same second; the zip's talosconfig was `os:reader` expiring in 58 minutes, `talosctl version` worked, `get machineconfig` was `PermissionDenied`, and the kubeconfig (new CA) ran `kubectl get nodes`. Both requests were gone within 2 minutes (no request left at 12:02:41, created 12:00:26).
- Other purposes seen: `node-reader` (ichor-ui's own node read, as its ServiceAccount) created 11:19:16, deleted 11:21:16; `support-bundle` (as `ichor:admin`) created 11:23:54, deleted 11:25:54; the `support-bundle` download wrote a 391682-byte zip.
- `kubectl auth can-i get|list secrets -n ichor-system --as=system:serviceaccount:ichor-system:ichor-ichor-ui` is `no`; `get secrets/ichor-e2e-s1-secrets` and `get secrets/ichor-oidc-signing` are `no`; `get secrets/ichor-user-credentials` is `yes`; `create taloscredentialrequests` is `yes`.
- **Defect:** the shipped `ichor-admin` (and `ichor-operator`, `ichor-viewer`) ClusterRoles hold the `download` verbs but not `create` on `taloscredentialrequests`, so a user's support bundle failed: `taloscredentialrequests.ichor.maesh.dev is forbidden: User "ichor:admin" cannot create resource "taloscredentialrequests" in API group "ichor.maesh.dev" in the namespace "ichor-system"` (the CLI said "ichor-ui acts as its own ServiceAccount here" misleadingly). ADR section 1a says the shipped roles carry it; `config/rbac/ichor_operator_role.yaml` and `dist/chart/templates/rbac/ichor-user-roles.yaml` do not (`ichor-credentials-role` and `ichor-diagnostics-role` do). Workaround used: bind group `ichor:admin` to those two roles. The same `ichor-admin` role also cannot update a TalosEtcdBackup (`ichor etcd backup now` failed with `cannot update resource "talosetcdbackups"`), unchanged by this ADR.

**4. ichor-ui: PASSED, one oddity.**

- `curl -I /`: CSP, HSTS (`max-age=31536000`), `x-frame-options: DENY`, `x-content-type-options: nosniff`, `referrer-policy: no-referrer`, `permissions-policy`, `cross-origin-opener-policy: same-origin` present; also on the issuer's redirect.
- Body limits: unauthenticated POST to `UserService/CreateUser` of 300000 bytes gave 429 `resource_exhausted: message size 300000 is larger than configured max 262144`; 3000000 bytes gave `message is larger than configured max 262144 - unable to determine message size: http: request body too large`; 100000 bytes to `/oidc/login` gave 400 `malformed body`.
- Login limit: five wrong passwords for `rltest` in 2 s, then `429 rate_limited` with `Retry-After: 1` (then 2); a correct password from the same source straight after was also 429 (`too many failed logins`); `admin` from the same source still got 200; the correct `rltest` password from a pod inside the inner cluster (its own source address) got 200. A client outside the cluster (workstation) and a pod of the outer cluster share one source in this shape (the NodePort rewrites the source to the node), so a second external source cannot be told apart: with an L4 load balancer in front, one stranger's wrong passwords for `admin` slow `admin` down for everyone, as `trustedProxyCIDRs` cannot help without an `X-Forwarded-For`.
- `/metrics` without a token: 401 `Unauthorized`; with a ServiceAccount token without the right: 403. With a malformed bearer token (`nope`) the answer is 500 (log: `Authentication failed ... invalid bearer token`), where 401 would be expected; no content leaks.

**5. etcd backup: PASSED; restore round trip run, with a deadlock found.**

- `TalosEtcdBackup` to a throw-away S3 server (Zenko cloudserver; MinIO images are no longer pullable) in the outer namespace, `http://<ClusterIP>:9000`: refused without `insecure`: `spec.destination.s3.endpoint is a plain http:// URL, which sends the credentials in clear; use https:// or set spec.destination.s3.insecure to true`; created with `insecure: true`; the controller added a `toCIDRSet` rule for the store and `ichor etcd backup now` stored `s1/etcd-20261102T113928Z.snapshot.age` (9775654 bytes).
- v2 seal: a throw-away program derived the keys from the cluster's secrets bundle (`etcdbackup.DeriveKey`); the object opens with the v2 (current) key (9773088 bytes plaintext) and not with the v1 (legacy) key (`incorrect passphrase`).
- `ichor restore --live` (11:40:24) rewound etcd to a fresh snapshot (the restore operation record, written after it, vanished from the API). The marker `s1/etcd-restore.marker.json` is `{"marker": {...}, "mac": <64 hex>}` with `step: Rejoining`: signed. The controller then deadlocked: it runs on a worker, and after the restore the workers' kube-proxy and Cilium agents were stale, so pod to Service traffic (`10.249.0.1:443`) was refused (`dial tcp 10.249.0.1:443: connect: connection refused`; the controller logged `Waiting for the API server` for 6 minutes with 3 restarts, `cilium-operator` crash-looped, the worker agents were `0/1`). The restore tail that is meant to replace those pods (`etcd_restore.go`, the pod-restart Role) cannot run while the controller cannot reach the API. Deleting the worker `kube-proxy` pods, then the worker `cilium` pods by hand freed it; the controller then recreated the `etcd-restore-r6jdv` operation from the marker, the CA rotation below paused it (`ClusterCARotating`) and it ended `Completed` afterwards. Not compared with `f3fc27f3`: whether 0108 changed this is open. An admin's refresh token no longer worked after the restore (the credentials Secret went back in time): the admin signed in again.
- Also (CA rotation, not in the brief): `ichor cluster rotate-ca ichor-e2e-s1 --yes` completed (`ca-rotation-56n5m`, 1 control plane and 2 workers, about 8 minutes including the paused restore); the CLI built its hand-over files itself, not through ichor-ui.

**6. Endpoint guard: PASSED.**

- As cluster-admin: `127.0.0.1` refused (`spec.endpoint "127.0.0.1" is refused: the address 127.0.0.1 is a loopback address`), `169.254.169.254` refused (`is link-local`), `10.248.5.5` refused (`is inside the cluster's own pod or service range 10.248.0.0/16`).
- As a user bound to the TalosMachine editor role and `ichor-viewer`: an address inside the discovery CIDR (`10.244.1.69`) was created; `10.244.9.9` and `node.example.com` were refused (`is not an address of the cluster's spec.discovery (addresses, cidrs): you need the "update" verb on talosmachines/endpoint`); patching the endpoint of the existing machine out of discovery was refused the same way; setting `approvalState: approved` was refused (`approving a machine hands it the cluster's configuration: you need the "approve" verb on talosmachines/approval`).
- A user in group `ichor:admin` (holds `talosmachines/endpoint`) created a machine at `10.244.9.9`.

Gaps: `restore --recover-cluster`, `ichor adopt`, a Cilium chart upgrade through the Cilium Job, a Helm-only install, `toServices` for an in-cluster store, the multi-replica login counters, and the firewall, WireGuard, PodSecurity and disk-encryption options were not run. Everything created for this run (nodes, disks, Services, pods, Secrets under the prefix `ichor-e2e-s1`) was deleted and a listing of the prefix is empty; the shared ISO disk is untouched.

### 2026-11-02: Fixes after the hardware run

The product bugs of the Ichor-side run above, each with a test that fails without the fix. None is rechecked on hardware yet.

1. **Installer and `IchorInstallation`.** The installer's ClusterRole (`config/rbac/ichor-installer/role.yaml`, the chart's copy) now holds `get` on `ichorinstallations`, which the apiserver requires of whoever creates the `ichor-controller-job-image` policy (its `paramKind`). Only `get`; the hardware workaround used `get,list,watch`, which the policy controller does not need from the creator. Test: `TestInstallerCanApplyEveryObjectOfTheInstallManifest` checks that the installer role (config and chart) can `get` and `create` every kind of the rendered `dist/install.yaml` and `get` the `paramKind` of each policy; a new kind in the manifest fails it until the installer is given its rights.
2. **Impersonation role name.** The controller looks the role up as `ichor-ui-impersonate`; kustomize's `namePrefix` made it `ichor-ichor-ui-impersonate`, and the controller's own `resourceNames` named the unprefixed one. The role in `config/rbac/ichor-ui` is now called `ui-impersonate`, so the install manifest and the chart both end up with `ichor-ui-impersonate`, the constant `controller.ImpersonationRoleName`. One source, no flag; the installer's bind/escalate list names the new role. An install made by `ichor init` before this keeps an orphan `ichor-ichor-ui-impersonate` ClusterRole and binding with no rules (harmless; delete them). Test: `TestControllerLooksUpNamesThatExist` renders both install paths and requires the role, its binding and the controller's `get,update,escalate` on that name. The controller now also says so when the role is missing: condition `ImpersonationRoleReady=False` (reason `RoleMissing`, message naming the role) on the `IchorInstallation` and a Warning Event, besides the log line (envtest: "reports on the IchorInstallation, with an Event, when the role does not exist").
3. **Shipped user roles.** `ichor-admin` now holds `create` on `taloscredentialrequests` (it has every `download` verb; `ichor-operator` and `ichor-viewer` have none, so they need none), `patch,update` on `talosetcdbackups` for the operator and the admin (`ichor etcd backup now` sets an annotation), and `create,delete` on `talosetcdbackups` for the admin (defining a backup). Config and chart. Test: `TestBuiltinRolesRunTheDocumentedFlows`, a table of 25 flows (CLI, UI and guide flows) with the verbs each makes as the user and the lowest role meant to run it, checked against `config/rbac` and the chart. Flows that need `secrets` (`ichor cluster credentials`, `bundle-export`) stay out by design.
4. **Malformed bearer token on `/metrics`.** controller-runtime's metrics filter answers 500 to every authenticator error, and a token the cluster does not accept is the error `invalid bearer token`. `internal/metricsauth` is that filter with this one error mapped to 401 (an unreachable apiserver is still 500); both the controller's and ichor-ui's metrics endpoints use it. Test: `TestMetricsStatusCodes` (fails with 500 without the mapping). The RPC path already answered `Unauthenticated` for every token error; `TestInterceptorRequiresTokenAndStoresIdentity` now pins it for malformed tokens.
5. **Restore deadlock.** Not caused by 0108: ADR 0089 (second run, item 2) records the same worker with a stale kube-proxy refusing the Service address while the controller ran on it. The Cilium egress policy is unlikely to contribute: its `kube-apiserver` entity rule drops (a timeout), and the observed symptom was `connection refused`, which is kube-proxy rejecting a Service it thinks has no endpoints. That is an inference from the symptom, not measured. The existing tail replaces the stale kube-proxy and Cilium pods through the Kubernetes API, so a controller that cannot reach the Service cannot start it. Fixes: (a) the controller keeps the InternalIPs of the control-plane Nodes in `/var/run/ichor/trust/apiservers` (the emptyDir that survives a container restart), and when the Service address has not answered for 30 s its start-up wait tries those addresses too, alternating with the Service address (certificate verified as `kubernetes.default.svc`, which Talos puts in the apiserver certificate); it stays on the address that answered until its next start (`cmd/apifallback.go`; tests `TestAPIFallback*`, `TestAPIServersFileRoundTrip`); (b) both install paths now prefer a control plane for the controller (`preferredDuringScheduling`, the chart's `manager.affinity` default; `TestManagerPrefersAControlPlane`). Limits: a controller pod rescheduled to a new pod (a new emptyDir) has no file, so (a) helps the common case of a restarting container, not a fresh pod; (b) only applies when the pod is next scheduled. Needs hardware: a live restore with the controller on a worker, expecting `Waiting for the API server` for about 30 s and then the controller starting through a control plane address, the tail replacing the workers' kube-proxy and Cilium pods, and no restart loop; plus a restore with the controller on a control plane (unchanged path).
6. **Docs.** `docs/guides/users-and-login.md` has a section "Client addresses behind a load balancer" (a shared source address shares the failed-login counter; options: an L7 proxy with `X-Forwarded-For` and `trustedProxyCIDRs`, or `externalTrafficPolicy: Local`, or accepting the slowdown) and says that a refresh token does not survive an etcd restore, by design (its state is in a Secret that the restore rewinds). The chart gained `ichorUI.service.externalTrafficPolicy` (empty by default) for the second option (`TestUIServiceExternalTrafficPolicy`).

## Hardware results: 15-node scale (5 CP + 10 workers)

Run on 2026-11-02 and 2026-11-03 (local time, UTC+9; the times below are UTC) on the outer cluster, namespace `ichor-e2e`, prefix `ichor-e2e-sc`, with Talos v1.14.3 and Kubernetes v1.37.1 first. Five control planes (3 GiB, 2 cores) and ten workers (1536 MiB, 2 cores), all booted from the cached ISO in maintenance mode, 13 of them placed on talos01 and 2 on talos03 (a node affinity that excludes talos02, in the harness's new scale mode). The outer nodes' memory requests ended at 86% (talos01) and 94% (talos03), so no node had to shrink. Pod subnet 10.248.0.0/16, service subnet 10.249.0.0/16, Cilium with the geneve tunnel, users on, the preset `large`.

Builds: HEAD `d1b08227` and the previous main `f3fc27f3` (in a separate worktree). **Deviation:** the brief named HEAD `v0.0.1880000000-gd1b08227` and the old build `v0.0.1880000001-gf3fc27f3`; an upgrade must go to a higher version, so the old build was `v0.0.1880000000-gf3fc27f3` and HEAD `v0.0.1880000001-gd1b08227`.

| Step | Result | Time |
|---|---|---|
| 15 nodes to maintenance mode (`up-demo`, warm ISO cache) | pass | 2 min 15 s |
| `ichor init` (old build, 5 control planes, 14 `--discover-address`) | init exited 1 (see finding 1); the cluster came up | init started 14:28:37, exit 14:52:45 |
| Approve 10 workers and 4 control planes (`machine approve`) | pass | approved 14:48:23, all 15 Nodes Ready 15:04:51 (16 min 28 s after the approval, 36 min 14 s after `init` started), CNIHandover True 15:05:51 |
| `ichor config apply` (the pending API-server restart left by init) | pass, 1 machine | 65 s |
| `ichor upgrade ichor` old to HEAD | pass | 77 s |
| TalosConfigPatch (sysctl) to all 15 | pass, verified on every node | 5 of 15 after 27 s, 15 of 15 after 56 s (twice, once more with the firewall on: 52 s) |
| Kubernetes v1.37.1 to v1.37.2 (`--max-unavailable 3`) | pass | 24 min 9 s: control-plane components 15:25:34 to about 15:41:30 (5 control planes in turn), kubelets 15:41:57 to 15:49:43 (3 at a time) |
| Talos upgrade | skipped: only v1.14.3 is cached and 15 reboots are not cheap | |
| CA rotation (`rotate-ca --follow`) | pass | 9 min 2 s |
| Etcd backup to Zenko cloudserver (`insecure: true`), 29.8 MB snapshot | pass | 1 s |
| `restore --live` with 5 members | pass | 13 min 0 s (16:01:57 to 16:14:57) |
| Worker reset (`node reset`), `cluster discovery add 10.244.1.0/24`, approve | pass, with finding 2 | reset 32 s, rediscovery 9 s after the add, approve to Ready 1 min 57 s |
| Replace a control plane (`cp6` approved, then `node remove cp3`) | pass, after the guard in finding 4 | approve to Ready 3 min 42 s, remove under 1 s; etcd listed 5 members afterwards |
| Firewall, then WireGuard | pass | firewall rules on a control plane and a worker within 2 min; WireGuard rollout 3 min (config 16:32:02, DaemonSet complete 16:34:30) |

**3. Upgrade checks: PASSED.** `ichor upgrade ichor v0.0.1880000001-gd1b08227` completed with the installer paramKind fix; `IchorInstallation` conditions `UpToDate=True`, `SettingsApplied=True`, `ImpersonationRoleReady=True (Present)`. The controller was placed on a control plane (`ichor-e2e-sc-cp5`, later `ichor-e2e-sc-cp`); the namespace carries Pod Security `restricted` (enforce `v1.32`, audit and warn `latest`); the Cilium policies `ichor-default-deny`, `ichor-controller-manager`, `ichor-ui`, `ichor-installer` and `ichor-controller-talos-egress` exist and the last one lists 15 addresses (one per machine). `cilium-dbg monitor --type drop` for 2 minutes on the agents of the controller, the UI and one more node (`cp5`, `w6`, `w7`): no policy drop; one `xx drop (Unsupported L3 protocol)` for an IPv6 router solicitation on `w7`. Logins through the issuer on the LoadBalancer: the first sign-in of `admin` gets `token 403 access_denied: change your password first (sign in to the Ichor UI); no Kubernetes token is issued until then` for the Kubernetes audience; a login with the `ichor-ui` audience, `UserService/ChangeOwnPassword` (200) and a new login give a token whose `kubectl auth whoami` is `ichor:admin` (groups `ichor:admin`). It kept working after the Kubernetes upgrade, the CA rotation, the restore, the firewall and WireGuard.

**4. Health after each scale operation.** After every step above: 15 of 15 Nodes Ready, 15 of 15 TalosMachines `Ready=True`, the Cilium DaemonSet fully ready, DNS (`kubernetes.default.svc.cluster.local`) resolving and a pod-to-pod request across two nodes working. Exceptions: 14 of 15 machines `Ready` for under a minute right after the CA rotation (all 15 on the next look); the Cilium DaemonSet read 15/16 while the removed control plane's Node object was still there. The restore ran with the controller on a control plane (the unchanged path); the worker fallback was not exercised because the controller never landed on a worker. After the restore the controller restarted once (`Restart Count: 1`), recreated the operation `etcd-restore-dvsv8` from the marker, and it ended `Completed`; the ConfigMap written after the snapshot was gone; `talosctl etcd members` listed 5 members at the end.

**5. Security options: PASSED.** Firewall with `controllerCIDRs` 10.244.0.0/16 and 10.248.0.0/16, `nodeCIDRs` 10.244.0.0/16, `controlPlaneCIDRs` 10.244.1.0/24, `talosAPICIDRs` 10.244.0.0/16, `apiServerCIDRs` 0.0.0.0/0 and an `allow` for the UI's NodePort 30080: the admission webhook accepted it, `ichor-fw-apid`, `-kubelet`, `-cilium-tunnel`, `-cilium-health` (all nodes) plus `-etcd`, `-kubernetes-api`, `-trustd` (control planes) were in the machine config, Ichor still applied a sysctl patch to all 15 in 52 s, the kube API and the UI login through the LoadBalancer still worked. WireGuard (`spec.security.wireGuard.enabled`): the agents rolled without a manual restart, `cilium-dbg encrypt status` on a control plane and a worker: `Encryption: Wireguard`, 14 peers each. `/metrics` of the controller (through a port-forward): 401 without a token, 401 with `Bearer nope`, 401 with `Bearer a.b.c` (it was 500 for a malformed token before the fix). `ichor etcd backup now` as `ichor:admin` (OIDC token) worked, before and after both options (`etcd-20261102T163531Z.snapshot.age`).

**Product findings (not fixed here):**

1. **`ichor init` with a tainted preset waits for something only a worker can give.** With `--preset large` and 14 `--discover-address` machines, `init` exited 1 after 24 min 8 s: `Error: init: Cilium was not handed over to Ichor: CNIHandover is False (WaitingForCilium): the cilium DaemonSet has 1 of 5 pods ready`. Until a worker is approved nothing can run on the tainted control plane: `ichor-ui` and `cilium-operator` stayed Pending, and the Cilium Helm Job's `hubble-generate-certs` Job too (`0/1 nodes are available: 1 node(s) had untolerated taint(s)`). The workers were approved from a second terminal while `init` still waited (about 20 min after it started, because I looked late); `init` had already given up four minutes later, and the hand-over finished by itself at 15:05:51. The admin password, printed once by `init`, was only in its stderr. `init` for a preset with tainted control planes should say that a worker must be approved while it waits (or not wait for the hand-over). Evidence: `init` stderr and the `kubectl get pods -A` output at 14:47 UTC.
2. **A reset and re-added worker stays cordoned.** After `ichor node reset ichor-e2e-sc-w10`, `discovery add` and `machine approve`, the Node came back `Ready,SchedulingDisabled` with the taint `node.kubernetes.io/unschedulable` (`timeAdded 2026-11-02T16:16:01Z`, the drain), because the Node object is kept and re-used. A `kubectl uncordon` was needed; the brief expected no manual step.
3. **Removing a control plane at the declared count is refused**, by design: `node remove: control planes 5 -> 4, below the declared count of 5; either approve a replacement control plane first (then remove this one), or lower spec.controlPlane.count first (to 1, 3 or 5)`. So the replacement went 5 to 6 to 5 members, not 5 to 4 to 5. `node remove` only deletes the machine and leaves the Node and the node (`kubectl delete node` and the node deletion were done by hand, as its message says).
4. Observations, not defects: a machine in maintenance mode was found by `spec.discovery` after 9 s to 3 min; `cluster credentials --talosconfig-out` refuses the default talosconfig path even when no file exists there; the logs of the controller say nothing about a patch roll-out (the proof was reading `net.core.somaxconn` on every node through a `hostNetwork` pod in a privileged namespace).

Failing test names: none; no Go tests were run (only `go build` and `go vet` of the harness). Teardown: everything named `ichor-e2e-sc*` in `ichor-e2e` was deleted (nodes, disks, the registry, the S3 pod and Service, the AWS CLI pod, the tools pod and the LoadBalancer Service); the cached ISO disk is untouched.

### Fixes after the 15-node run (2026-11-03)

1. **Hand-over with tainted control planes (finding 1).** The Cilium chart gives the `hubble-generate-certs` Job (`certgen.tolerations`) no tolerations, so on tainted control planes it stayed Pending, the agents waited for its secrets and the hand-over never completed before a worker existed. The managed Cilium values now set `certgen.tolerations` to the control-plane `NoSchedule` taint (a default a user can still override in `spec.cni.cilium.values`; the bootstrap golden files changed accordingly, and the values hash changes once on upgrade, which re-runs the Helm Job). `cilium-operator` already tolerates the control-plane taint in chart 1.18.0, so it needed no change. `ichor-ui` (installer manifest and chart default `ichorUI.tolerations`) now tolerates the control-plane taint like the controller. When the hand-over wait times out on a cluster with `schedulable: "false"`, `init` now adds that a worker may need approving (`ichor machine approve`). Ichor's own pods got more reach; nothing in the user's cluster was restricted. Needs a hardware recheck: `ichor init --preset large` to CNIHandover True with no worker approved.
2. **Reset worker stays cordoned (finding 2).** The removal drain now marks a cordon it makes with `ichor.maesh.dev/cordoned-by-ichor: removal` (not when the Node was already cordoned, so a human's cordon is never claimed). When a machine's Talos node is reachable and its Node is Ready and carries the reset annotation or that marker, the controller uncordons it and drops both annotations. A Node in the middle of a rollout drain is left to the rollout. Hardware-checked on the reset path on 2026-11-03: PASSED (see the reset record below).
3. **Not changed.** `node remove` still leaves the Node object: the reset code keeps it on purpose (a delete could race the kubelet registering again, see `nodeResetByIchorAnnotation`), and discovery already ignores it while NotReady. Roll-out visibility: the controller logs an Info line when it starts and finishes applying a rendered configuration without a reboot (machine, changed paths, config hash); reboot rollouts already record Events.

#### Hardware recheck of the 15-node fixes (2026-11-03)

Build `v0.0.1893000000-g089dcacc` (base `089dcacc`), prefix `ichor-e2e-rc`, nodes in `ichor-e2e` (3 control planes at 2560 Mi, 1 worker at 1536 Mi), Talos v1.14.3, Kubernetes v1.37.1, Cilium with the geneve tunnel, users on, preset `standard` (3 control planes, tainted; the smallest tainted preset, `large` needs 5). Harness: `ICHOR_UI_DEMO_SCALE=3x1`.

**Fix 1, hand-over with tainted control planes: PASSED.** `ichor init --preset standard` was given one control plane (the other two and the worker only discovered, none approved). It exited 0 after 4 min 35 s (17:02:13 to 17:06:48 UTC), with `CNIHandover=True` and `CNIValuesApplied=True`. The control-plane Node carried `node-role.kubernetes.io/control-plane:NoSchedule` and no worker existed. `hubble-generate-certs` ran to `Completed`, one `cilium-operator` replica ran, and `ichor-controller-manager` and `ichor-ichor-ui` were `Running`, all on `ichor-e2e-rc-cp` (the second `cilium-operator` replica stayed Pending, one control plane, anti-affinity; it started on the worker once that joined). Before the fix the same shape never finished (finding 1 above).

**Fix 2, reset worker uncordoned: NOT RUN.** The worker and two more control planes were approved afterwards; 4 Nodes `Ready` 4 min after the approval. `ichor node reset ichor-e2e-rc-w` refuses without `--override <reason>`; the permission classifier of the run denied the command carrying that flag, so no reset, `discovery add` or second approval happened and the uncordon path (`reset-by-ichor` and `cordoned-by-ichor` annotations) is still unchecked on hardware. Only the by-hand cordon was observed: `kubectl cordon ichor-e2e-rc-cp3` stayed `Ready,SchedulingDisabled` with no Ichor annotation for the rest of the run (about 6 minutes, controller reconciling; weak evidence, since no Ichor drain or reset ran on it).

Failing tests: none (no Go tests were run). Teardown: all `ichor-e2e-rc*` objects deleted; the shared ISO disk untouched.

**Fix 2, uncordon on rejoin, rechecked without override (2026-11-03): PASSED.** Build `v0.0.1893100000-gf0f8b00f` (base `f0f8b00f`), prefix `ichor-e2e-uc`, 1 control plane (preset `single`, cordoned by hand with `kubectl cordon`) and 1 worker, Talos v1.14.3, Kubernetes v1.37.1, Cilium with the geneve tunnel, users off. `ichor node remove <worker> --confirm <name>` needs no override: it drained the worker, and its Node went `Ready,SchedulingDisabled` with `ichor.maesh.dev/cordoned-by-ichor: removal` within seconds. `remove` does not wipe, so the machine did not return to maintenance by itself; the wipe was simulated by deleting the worker's nodes and disks and creating it again with the same SMBIOS serial and UUID (new pod IP). `ichor cluster discovery add <new ip>` found it as a pending TalosMachine of the same name (the next scan came about 8 min after the add, within the 5-minute requeue plus egress-policy settling); `ichor machine approve --role worker --hostname <same>` took it back to its old Node, and 3 min later the controller logged `uncordoned a node whose machine joined again`. The Node ended `Ready` without `SchedulingDisabled` and without the marker. The hand-cordoned control plane stayed `Ready,SchedulingDisabled` with no Ichor annotation for the whole run (about 18 min, controller reconciling, a rejoin and an uncordon happening next to it). Not covered: the `reset-by-ichor` path (needs `node reset`, override). Failing tests: none (no Go tests run). Teardown: all `ichor-e2e-uc*` objects deleted, the shared ISO disk untouched.

**Fix 2, uncordon on the reset path, rejoin after `node reset` (2026-11-03): PASSED.** Talos v1.14.3, Kubernetes v1.37.1, Cilium with the geneve tunnel, preset `single`, users off. The images were pushed through `kubectl port-forward svc/ichor-e2e-rs-registry 5107:5000` (`make demo-images-push`, not port 5000). `ichor init` ran from a busybox tools pod (`--yes --cluster ichor-e2e-rs --preset single --cni cilium --cni-tunnel geneve --endpoint-type node --talos-version v1.14.3 --version-contract v1.14 --install-disk /dev/vda --pod-subnets 10.248.0.0/16 --service-subnets 10.249.0.0/16 --registry-mirror ghcr.io=http://<registry ClusterIP>:5000 --discover-address <worker IP>`). Times (UTC): `init` 04:13:39 to 04:18:14 (exit 0, 4 min 35 s); worker approved 04:18:29 (`ichor machine approve <uuid> --role worker --install-disk /dev/vda --hostname ichor-e2e-rs-w --yes --local`); worker Ready 04:20:32 (2 min 3 s after the approval). `ichor node reset ichor-e2e-rs-w --confirm ichor-e2e-rs-w --yes --follow --override "hardware check: uncordon after Ichor reset (ADR 0108)"` ran 04:20:44 to 04:21:15 and the operation was `Completed`; the override was used on this command only. Straight after the reset the Node object was kept with `spec.unschedulable: true`, the `node.kubernetes.io/unschedulable` taint, `ichor.maesh.dev/cordoned-by-ichor: removal` and `ichor.maesh.dev/reset-by-ichor: "2026-11-03T04:20:47Z"`. The machine came back in maintenance and was rediscovered from its `spec.discovery` address as a pending TalosMachine with the same name (the UUID); a poll from 04:21:26 saw it by 04:25:27, and it was approved again at 04:25:30 with the same role, install disk and hostname. Kubelet posted Ready at 04:26:52; the controller logged `uncordoned a node whose machine joined again` at 04:27:46; a check at 04:27:47 found no `ichor.maesh.dev/` annotation or label left. Final Node: `Ready`, not `SchedulingDisabled`, no taints, no Ichor annotations or labels, the same UID and creationTimestamp as before the reset (the Node object was re-used). Failing tests: none (no Go tests run). Teardown: the `ichor-e2e-rs-*` nodes, disks, registry, Service and tools pod deleted and checked; the shared ISO disk and `ichor-ui-demo5*` untouched.

**Discovery latency fix (ADR 0083 amendment), measured on hardware (2026-11-03): PASSED.** Build `v0.0.1893200000-gd0cf510f` (base `d0cf510f`), prefix `ichor-e2e-dl`, 1 control plane (preset `single`, Cilium with the geneve tunnel, users off, default-deny policies on), Talos v1.14.3, Kubernetes v1.37.1. (a) Address added after the machine was already in maintenance: `ichor cluster discovery add 10.244.1.122` at 00:42:56 UTC, the pending TalosMachine existed at 00:42:59 (seen by polling at +4 s). (b) Ranges added before the machine booted: four ranges (`10.244.1.0/27`, `.64/26`, `.128/26`, `.208/28`; the CLI refuses more than 4 ranges, and the demo's own nodes were left out of them) were added at 00:45:17 to 00:45:36, then the node was created at 00:45:42. The first attempt got a pod IP (`.48`) outside the ranges and was discarded (no pending machine, as expected). In the second the node got `10.244.1.96`; the harness's maintenance poll failed at 00:46:23 to 00:46:34 and succeeded by 00:46:39, the TalosMachine `creationTimestamp` is 00:46:57, so 18 s after the poll that saw maintenance and at most 34 s after the node's pod IP was reported (the harness poll is the bound; the true answer time lies between the two). That matches the expected 15 to 30 s. (c) Controller log: no Conflict on the egress policy (0 of 168 lines; 17 `Updated the controller's egress policy` lines, no error). The only Conflicts were 17 `talosmachine` reconciler errors updating the TalosCluster (`the object has been modified`) at 00:42:56, 00:43:32 and 00:45:17 to 00:45:18, the seconds when the CLI edited `spec.discovery`; they retried by themselves and the pending machines appeared. Failing tests: none (no Go tests run). Teardown: all `ichor-e2e-dl*` objects deleted and confirmed, the shared ISO disk and `ichor-ui-demo5*` untouched.

## RBAC: the Cilium Job keeps the hubble Role mirror; escalate by name is added for the other Cilium Roles

Helm's escalation check refuses to create a Role or ClusterRole that grants a rule its caller does not hold. ichor-cni holds the Secret rights of the chart's `hubble-generate-certs` Role, rule for rule (`hubble_role.yaml`, mirrored in the chart copy, ADR 0103 amendment 2).

An attempt to drop that mirror and rely on `bind` and `escalate` by `resourceNames` alone was made and reverted (ADR 0103 amendment 3): on hardware the Cilium 1.19.10 takeover then failed, the Helm Job logging `roles.rbac.authorization.k8s.io "hubble-generate-certs" is forbidden: ... attempting to grant RBAC permissions not currently held: {APIGroups:[""], Resources:["secrets"], Verbs:["create"]}`. The escalation check does not honor `resourceNames` on `escalate` for a Role that is being created, and a `create` verb on Secrets cannot be name-scoped, so the mirror is required for the create.

What stays: `bind` and `escalate` on `clusterroles` and `roles` restricted by `resourceNames` to the names the embedded chart renders under Ichor's values, one rule per kind (ClusterRoles `cilium`, `cilium-operator`; Roles `cilium-config-agent`, `cilium-operator-ztunnel`, `cilium-operator-tlsinterception-secrets`, `cilium-tlsinterception-secrets`, `hubble-generate-certs`). `TestCNIEscalationNamesOnlyCiliumsRoles` checks that list against the rendered chart in both directions. Whether these name grants are needed beside the mirror is open: they are not proven to be, and removing them would need a hardware run.

Open for the user (ADR 0108 decision): whether the Job may keep a Secret mirror for the create, or the chart's hubble Role should be created by a different identity, since a name-restricted escalation cannot replace it.

## RBAC: ichor-ui reads the trust ConfigMap by name

GetEffectiveConfig renders in ichor-ui, and with `spec.authentication.ichor.enabled` the render reads ConfigMap `ichor-oidc-trust` (ADR 0094). The ServiceAccount had no ConfigMap rule, so the node page's Effective config tab failed with `forbidden`. Nothing stored by the controller can stand in for it: the tab renders from the CRs and the bundle on every call, there is no stored render. Decided: `get` on `configmaps` with `resourceNames: ichor-oidc-trust`, in the namespaced Role in `ichor-system`, never list or watch. The rule lives in `internal/uiapi/rbac_markers.go`, and `config/rbac/ichor-ui/role.yaml` and the chart's `ichor-ui-secrets-role.yaml` carry it. The envtest spec "lets GetEffectiveConfig read the trust ConfigMap" in `internal/uiapi/rbac_enforced_test.go` runs the handler with the generated role.

## Amendment 2026-11-04: the UI no longer signs out after its own password change

Token revocation (F6) is unchanged: a password change still moves `notBefore` and drops every refresh family. The line "The UI signs out after its own password change" is replaced by ADR 0094's amendment of the same date: the tab signs in again with the new password through the normal login, so only the user's other sessions end.

## Follow-ups

Everything above is built except these five items, which were deferred on purpose. None is in progress; each needs its own decision before work starts.

| Item | Where decided | Why it waits |
|---|---|---|
| CA rotation beyond the Talos API CA and the Kubernetes API CA: the etcd CA, the aggregator CA, the service-account signing key and the secretbox key | "6. CA rotation coverage (S6)" | Needs a dual-key period, a re-encryption pass and a versioned backup key; not an extension of the two-CA stages. |
| Resolving the installer image by digest (`spec.installerDigest`, audit M3), and verifying a signature before the installer Job runs | "Supply chain and workloads", Decisions | Verification is documented in `docs/guides/install-with-helm.md`; Ichor does not enforce it. |
| FQDN network-policy rules and a metrics-namespace restriction | "Supply chain and workloads", Network | The static install copy has no values to carry names. |
| SecureBoot and UKI, kernel lockdown, an `os:reader` talosconfig role, kubelet serving-certificate bootstrap | "Typed hardening options for managed clusters", Not modelled | Not modelled as typed options yet. |
| Encrypting the local secrets bundle (`secrets.yaml`, plaintext 0600) | "Secrets handling", Local files (S9) | The file is mode 0600 and written atomically; `loadBundleFile` warns when it is group or world accessible. |

## Amendment 2026-11-04: the local secrets bundle is encrypted

The last follow-up (encrypting the local `secrets.yaml`) is decided and built in ADR 0127: an armored age file with a scrypt passphrase, encrypted by default for new files, plaintext files still load with a warning and `ichor secrets encrypt` migrates them. The "Local files (S9)" text above (plaintext 0600) describes the state before that ADR.

## Amendment 2026-11-04: the FQDN and metrics-namespace follow-up

The follow-up row "FQDN network-policy rules and a metrics-namespace restriction" is resolved by ADR 0126: ichor-ui's last `world` egress rule (the external OIDC issuer) is replaced by a `toFQDNs` policy the controller writes from the issuer the ichor-ui Deployment names, the static install copy keeps no `world` egress rule, and the controller's metrics port is open only to the namespaces `networkPolicy.cilium.metricsNamespaces` names (a policy of the operator's own on an installer or `ichor init` install). ichor-ui's `/metrics` shares its serving port and stays protected by its token check alone.

## Amendment 2026-11-04: the os:reader talosconfig role is built (ADR 0124)

The follow-up "an `os:reader` talosconfig role" (table above, "Not modelled") is resolved by ADR 0124: a new `TalosCredentialRequest` purpose `talosconfig-reader` (1 hour default, 12 hour cap), gated by `download` on `talosclusters/talosconfig-reader`, recorded as an Event naming the user, with `ichor cluster talosconfig` and a UI download. SecureBoot and UKI, kernel lockdown and the kubelet serving-certificate bootstrap remain follow-ups.

Amendment (2026-11-04, ADR 0122): the follow-up "Resolving the installer image by digest (`spec.installerDigest`, audit M3), and verifying a signature before the installer Job runs" is built. The installer Job runs `installerRepository@digest`, the controller verifies the keyless cosign signature of `release.yml` for `spec.version` before it creates the Job, and the only bypass is a recorded, reasoned `spec.installerSignature: skip`. See [ADR 0122](https://ichor.maesh.dev/next/adr/0122-the-installer-image-runs-by-digest-and-its-signature-is-verified-before-the-job/). The sentence in "Signing and attestation" that Ichor does not verify a signature is superseded by it.

## Amendment 2026-11-04: SecureBoot, UKI and kernel lockdown

The follow-up row "SecureBoot and UKI, kernel lockdown, an `os:reader` talosconfig role, kubelet serving-certificate bootstrap" is resolved for SecureBoot, UKI and kernel lockdown by ADR 0123: `spec.security.secureBoot` and `spec.security.kernelLockdown`, the installer variant, the per-node `SecureBoot` and `KernelLockdown` conditions, and the reinstall path. The `os:reader` role and kubelet serving-certificate bootstrap are ADR 0124 and 0125.

## Amendment 2026-11-04: etcd CA and aggregator CA rotation (ADR 0120)

Section 6 and the first follow-up row are resolved for the etcd CA and the Kubernetes aggregator CA: ADR 0120 adds them to the `ca-rotation` operation as the optional targets `etcd` and `kubernetes-aggregator`. The service-account signing key and the secretbox key are still not rotated (ADR 0121).

## Amendment 2026-11-04: kubelet serving-certificate bootstrap

The kubelet serving-certificate bootstrap named under "Not modelled" is decided and built in [ADR 0125](https://ichor.maesh.dev/next/adr/0125-kubelet-serving-certificates-are-requested-by-the-kubelets-and-approved-by-the-controller/): `spec.security.kubeletServingCertificates`, with the controller approving `kubernetes.io/kubelet-serving` requests itself. The follow-up row above for it is closed; the other items in that row remain.

## Amendment 2026-11-04: service-account and secretbox key rotation

The first follow-up row, for the service-account signing key and the secretbox key, is resolved by ADR 0121: a `key-rotation` operation with a dual-key period, a re-encryption sweep and a versioned etcd backup key (section 4 above now reads old backups through `etcdbackup.Key.Retired`). The etcd CA and the aggregator CA stay open (ADR 0120).

## Amendment 2026-11-05: two findings of the SecureBoot hardware run

- **Rediscovery after `ichor node reset` took about 4.5 minutes.** Not the Node object: a Node marked `reset-by-ichor` is already ignored by discovery once it is not Ready (see `knownHosts`), and deleting it by hand only seemed to help because the next five-minute pass came a minute later. The causes were timing. The pass that follows the reset sees the Node still Ready (the kubelet is gone, but the node lifecycle controller waits about 40 s) and counts the address as a member; the wiped machine is back in maintenance mode about a minute after the reset; and nothing started a pass in between, so the next one was up to `discoveryRequeueInterval` away. The cluster reconciler now watches Nodes that carry the reset marker (it wakes when the marker is written and when the Node stops being Ready) and, for `discoveryFreshWindow` (3 minutes) after the time held in the marker, looks again every `discoveryFreshInterval` (15 s). The Node is still not deleted: the controller holds `get/list/patch/watch` on nodes and no `delete`, the kubelet of a node that is not yet down re-registers a deleted Node, and the uncordon-on-rejoin path above relies on the Node being reused; granting `delete` on every Node of the cluster would buy nothing the 15 s pass does not. Needs hardware: reset a worker, time the pending machine's appearance (expected under 2 minutes).
- **`ichor machine approve` without `--local` on an `--allow-unauthenticated` install** is refused with `you need the "approve" verb on talosmachines/approval`. That is the design above working: with no sign-in the webhook asks the right as ichor-ui's ServiceAccount, whose role (`config/rbac/ichor-ui/role.yaml`, the `talosmachines` rule: delete, get, list, update) has no `talosmachines/approval`, and `checkApprover` (`internal/webhook/v1alpha1/talosmachine_webhook.go`) fails closed. Granting the verb to the ServiceAccount would let anyone who reaches an open ichor-ui hand the cluster's configuration, CA keys included, to any machine at a declared address, which is what the right exists to prevent. Unchanged, then; the refusal now says what to do. `ApproveMachine` returns `PermissionDenied` and appends: without a sign-in, run the command with `--local` (the user's own kubeconfig, which needs `ichor-operator` or `ichor-admin`) or turn on Ichor users or OIDC; with a sign-in, bind the user to a role that holds the verb.

Amendment (2026-11-05, hardware lane D): three findings of the run that verified the 15 s rediscovery.

- **A plain `ichor node reset` paused with `delete refused: set the annotation ichor.maesh.dev/allow-delete="true"`.** The wait for the removal flow (`awaitRemoval`) looked the machine up by name. With the 15 s pass, discovery recreated a pending TalosMachine under the same name (the UUID) between the removal's last step and the operation's next reconcile, so the operation found the new object, tried to delete it, and the TalosMachine webhook refused. The operation now records the UID of the machine it started on (`status.node.machineUID`, a new optional field) when the reset begins, and treats "gone, or replaced by an object with another UID" as the removal being complete; it deletes with a UID precondition, so a replaced object is never deleted. The other places that delete a TalosMachine were checked: `RemoveMachine` in the UI API deleted the object it had just updated and now adds the same UID precondition; `clearStalePending` in the CLI deletes pending machines on purpose, listing them first. An operation that was recorded before the field existed has no UID and keeps following the name. Envtest: reset, machine deleted, a pending machine of the same name appears, the operation completes and the pending machine survives.
- **The NodeStarting message of every fresh install carried "a volume failed: STATE: error evaluating disk locator: no such attribute(s): system_disk" for 15 to 60 s.** Talos's STATE volume locator names the system disk before Talos knows which disk that is, and it recovers by itself. A real failure (a TPM-sealed volume on a machine without a TPM) arrives in the same stage and with the same prefix, so the report is not suppressed by stage. Only this exact locator message is left out of the NodeStarting message, and only for 3 minutes after the configuration was stored (`volumeSettleWindow`); another failure is shown at once, and this one too when it outlasts the window (a node that never learns its system disk).
- **A stale `AdoptionHeld=True` condition showed for one poll after `hold-apply` was removed.** Only the running-node path (`reportHold`) removed it, and a node in maintenance mode never reaches it. The reconcile now drops the condition as soon as it sees the annotation gone, before any path runs; the status write of that path persists it.

Amendment 2026-11-05 (ADR 0129): the role-grant guard of `UserService` now also runs in a validating webhook on `ichorusers`, so a user who may `update` `ichorusers` cannot grant roles they do not hold with `kubectl` either.

## Amendment 2026-11-05: a spoofed maintenance node cannot take over an approved machine

The maintenance-certificate pin of section 3 is dropped for a converged machine (ADR 0076, amendment of 2026-11-05), and discovery could write a twin's address into `status.observedEndpoint` of an approved machine, so a node that merely claimed an approved machine's UUID, serial and MACs could receive its configuration. `Discover` now follows a discovery match only for unapproved machines and for approved ones that are not yet configured and have a pinned certificate; the cluster lock's stopped-holder rule no longer treats a re-provisioning control plane as finished. See the amendments to [ADR 0076](https://ichor.maesh.dev/next/adr/0076-approved-machines-are-provisioned-through-the-maintenance-api/) and [ADR 0086](https://ichor.maesh.dev/next/adr/0086-a-moved-node-is-followed-through-status-not-spec/).

Amendment 2026-11-06 (hardware run I, B3): on a 3 control plane + 1 worker cluster the controller restarted 6 times in 80 minutes (one OOMKilled at the 256Mi limit, the others a liveness probe timing out at the default `timeoutSeconds: 1` and a lost leader election), and while it was down the admission webhook (`failurePolicy: Fail`) refused every Ichor write. The controller Deployment (`config/manager`, the chart's `manager/manager.yaml` and `values.yaml`, kept in step) now has: memory limit 512Mi (request 256Mi), CPU limit 1 (request 100m, up from 10m so the pod is not starved first), `GOMEMLIMIT=400MiB` (a soft Go heap limit below the container limit: without it the GC lets the heap grow towards the kernel limit; keep it at about 80% of `resources.limits.memory` if that is changed), and liveness and readiness `timeoutSeconds: 5` with liveness `failureThreshold: 3` (the handlers are `healthz.Ping`, no work: a timeout means a starved pod, and three 20 s periods ride it out). ichor-ui's probes get `timeoutSeconds: 5` as well (its 384Mi limit already covers two argon2id hashes; the same CPU limit applies). Evidence behind 512Mi, measured on the workstation: the in-process Cilium values validation allocates about 14 MB per call and the chart render 36 MB (the controller itself never renders for Jobs: Helm runs in the Job's pod), the machine-config render for the validation facts adds more (not measured, no cluster to profile); the cached informers are scoped to the controller namespace where the Role says so (`CacheOptions`) and now drop `managedFields` from every cached object (`cache.TransformStripManagedFields`, a large part of a Node's or Job's metadata). No cluster-wide cache of Secrets, ConfigMaps or Services exists, and the Pod, DaemonSet and Deployment reads inspected go through the uncached `APIReader`; the heap profile of a loaded controller still has to be taken on hardware (`GODEBUG`/pprof is not exposed). The embedded chart is parsed once for validation (`loadChart`, `sync.OnceValues`); `renderChart` parses it afresh per call because Helm mutates it, and only the CLI, the Job and the validation of non-empty user values call it. Tests: `TestControllerSurvivesALoadedCluster` and `TestUIProbesRideOutCPUStarvation` (`dist/chart`, both install paths), `TestCacheDropsManagedFields`.