0108. Security hardening

Status: Accepted; built, hardware-verified (see the Hardware results sections); the five items it deferred as follow-ups are all built since (ADR 0120 to 0127, see the amendments after "Follow-ups") · Date: 2026-11-02 · Design section: Trust, Deployment, Nodes · Touches: 0011, 0041, 0058, 0063, 0089, 0103, 0105, 0107

Decided: Ichor is hardened area by area after five read-only audits (2026-11-02): ichor-ui and its auth, the controller's privileges and admission, workloads and the supply chain, the security of the clusters Ichor creates, and secrets and credentials. Fixes to Ichor itself are made directly. For the clusters Ichor creates, every new hardening is a typed option that is off by default, for new and existing clusters alike; disk encryption stays opt-in (the user's decision, 2026-11-02). Each area below records its decisions and what was built.

Context

The user asked for security to be hardened "to the extreme" (2026-11-02). Five audits read the code and manifests and listed findings with file:line evidence, an exploit scenario and a fix. Asked how far new clusters' defaults should go, the user chose that all cluster-side hardening is opt-in, and that disk encryption stays opt-in.

Admission and confused deputy

The controller does things on a CR author's behalf with rights the author does not have: it pushes a rendered machine configuration (the cluster's CA keys) to a machine, resolves a credential Secret, sends snapshots to an S3 endpoint, and creates Jobs whose ServiceAccounts can create roles. The audit (rbac-admission F1 to F9, secrets S5) found places where the author chose the target or the Secret without the matching right.

Decisions

1. A machine's endpoint is an address the cluster declares. The TalosMachine webhook judges spec.endpoint on create and whenever the endpoint or clusterRef changes (an update that leaves both alone is not judged, so a machine stays editable after its discovery entry goes away). Never allowed, for anyone: an unspecified, loopback, link-local or multicast IP, or one inside the pod or service ranges the cluster declares in spec.network (pkg/endpointguard). Talos's undeclared defaults (10.244.0.0/16, 10.96.0.0/12) are deliberately not assumed: the machines of an outer cluster live in those ranges (the e2e harness). Allowed: an address in the cluster's spec.discovery.addresses or cidrs; any other address needs the new right update on talosmachines/endpoint (a hostname cannot be classified here, so it needs the right too).
2. Approving needs a right. Moving spec.approvalState to approved, including at creation, needs approve on talosmachines/approval, asked by SubjectAccessReview as the requesting user like talosclusters/authentication. ichor-operator and ichor-admin hold it (they could approve before, through update); ichor-admin also holds talosmachines/endpoint. The talosmachine-editor-role helper holds neither: an editor can no longer approve. talosmachine-admin-role holds both.
3. The controller re-checks before it dials. provision() runs endpointguard.ForbiddenResolved on the dial endpoint (including status.observedEndpoint, and what a hostname resolves to) before the insecure maintenance probe, and sets Ready=False EndpointRefused without dialling. It does not require discovery membership: ichor init and ichor adopt write machines at addresses they were handed, and the controller cannot ask who wrote the CR.
4. MachineReclaim.spec.address gets the same forbidden classes in the webhook (the cluster's ranges are not known there; the reclaim verb and the foreign talosconfig remain the other gates).
5. Where credentials go is part of the Secret reference. A change to spec.registries[].endpoints of a registry that keeps its authSecretRef, and a change to spec.destination.s3 endpoint, bucket, prefix, region or the new insecure, needs update on <resource>/secret-references, like the Secret name. A plain http:// S3 endpoint is refused unless spec.destination.s3.insecure: true (new optional field; judged when the endpoint or the flag changes, so a stored http:// backup keeps working; the controller does not re-check).
6. Jobs. Two ValidatingAdmissionPolicies bound to the controller's ServiceAccount (config/admission/controller-jobs-policy.yaml, mirrored in the chart): the first allows only the installer Job (ichor-installer, in the Ichor namespace, installer run) and the Cilium Job (ichor-cni, in kube-system, cni helm apply), one container, no command override, only KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT and HOME as environment, dropped capabilities, no privilege escalation, emptyDir volumes only, hostNetwork for the Cilium Job only, an activeDeadlineSeconds of at most an hour (the installer Job now sets 30 minutes; the Cilium Job already set 25). The second ties the image to IchorInstallation.spec.installerRepository:version and does nothing while no IchorInstallation exists (a parameterNotFoundAction: Allow skips the binding). The chart adds admission.jobImageRepository (default ghcr.io/maeshinshin/ichor-installer) for an install without an IchorInstallation; set it to a mirror or empty it. Neither Job uses a hostPath today, so none is allowed.
7. Escalation verbs are named. The installer's bind and escalate list every ClusterRole and Role of the install manifest (the installer applies its own role first, so a release that adds a role extends the list before using it); the Cilium Job's list the roles the pinned chart creates. dist/chart tests compare both lists with dist/install.yaml and with the chart as rendered under Ichor's values. Residual: the installer keeps escalate on its own role by design (ADR 0059 stage 0), so a compromised installer can still rewrite its role; the Job policy and the ichorinstallations/version right are what keep that process from being started with another image.
8. System namespaces. No TalosCluster, TalosMachine, TalosOperation or TalosEtcdBackup may be created in kube-system, kube-public or kube-node-lease: the cluster lock is a Lease named after the cluster, written with the controller's cluster-wide Lease rights. The leader-election role loses its unused ConfigMap rule. Moving the lock to a Role in the Ichor namespace (audit F5c) is not done.
9. Finalizers. The override finalizer check also sees UPDATE of TalosOperation (the etcd-restore and ca-rotation finalizers). Every Ichor finalizer is now covered (machine-removal, etcd-restore, ca-rotation).
10. Smaller. The controller's cluster-wide pods delete is gone; the one direct delete (stale agent pods after an etcd restore) is a Role in kube-system (config/rbac/pod_restart_role.yaml). The *-admin-role helpers list their verbs instead of *, so override is never implied. IchorInstallation deletion needs the allow-delete annotation like a cluster. The kube-proxy switch policy reads its fields behind has(). Every webhook has timeoutSeconds: 5.

As built

• pkg/endpointguard (Forbidden, ForbiddenResolved, InDiscovery); webhook checks in internal/webhook/v1alpha1 (talosmachine_webhook.go, machinereclaim_webhook.go, override_webhook.go, system_namespace.go); constants MachineApproveVerb, MachineApprovalSubresource, MachineEndpointVerb, MachineEndpointSubresource in api/v1alpha1/talosmachine_gates.go.
• New CRD field TalosEtcdBackup.spec.destination.s3.insecure (optional, append-only).
• Tests: envtest webhook suites (machine_gate_test.go, secret_references_gate_test.go, system_namespace_test.go, override_operation_finalizer_test.go), the Job policy against a real apiserver in api/v1alpha1/controller_jobs_policy_test.go and against the real Job builders in internal/controller/controller_jobs_policy_test.go, and the chart tests in dist/chart/hardening_test.go and user_roles_test.go.

Flows that change

• ADR 0011 (discovery). Unchanged for discovery-created machines: the endpoint is a declared address. Approving now needs approve on talosmachines/approval.
• ADR 0103 (adopt, add by IP, reclaim). Add by IP and reclaim (the UI API, and ichor machine reclaim --cluster) add the address to spec.discovery.addresses before anything else, and the controller creates the pending machine from it, so the endpoint is declared. ichor init and ichor adopt write approved machines at addresses outside discovery, so the user needs approve and update on talosmachines/endpoint (ichor-admin has both; cluster-admin has everything). Reclaim addresses in loopback, link-local, unspecified or multicast ranges are refused.
• ADR 0086 (observed endpoint). Unchanged: status.observedEndpoint is written by the controller, and the controller refuses to dial a forbidden one.
• ADR 0106 (S3 store). An in-cluster or external store over http:// now needs insecure: true; https:// is unchanged.
• ADR 0059 (installer). An installer Job that is still running after 30 minutes is killed.

Not verified on hardware

Nothing here ran on a cluster other than envtest. To prove: an Ichor upgrade through the installer Job (named bind/escalate lists, Job policy, pod-restart Role in kube-system), a Cilium chart upgrade through the Cilium Job, an etcd restore (agent pod deletion), ichor init and ichor adopt as ichor-admin, and a Helm-only install with the default admission.jobImageRepository.

ichor-ui and its auth

Decided and built (audit findings F2 to F18 of the ichor-ui audit; F1, the impersonation scope, is under "Credentials and the ichor-ui identity").

• Request sizes (F2). connect.WithReadMaxBytes on every handler: 256 KiB, and 1 MiB for the services that carry pasted documents (patches, Cilium values, reclaim's talosconfig). The limit applies to the decompressed message and is checked before the interceptors, so an unauthenticated caller cannot make the pod allocate. The mux caps any body at 2 MiB, the issuer's endpoints at 64 KiB, the internal listener at 4 KiB. Every RPC takes one request message (the streaming ones are server streams), so the limits do not touch streaming.
• Timeouts (F5). Main listener: ReadHeaderTimeout 10 s, ReadTimeout 60 s, IdleTimeout 120 s, MaxHeaderBytes 64 KiB, and no WriteTimeout because StreamNodeLogs and the downloads run for minutes; a request body is read within ReadTimeout, and a stream is bounded by its token (below). Internal listener (kube-apiserver only): 5 s, 15 s, 15 s write, 60 s idle, 16 KiB headers.
• Security headers (F4). One middleware on the whole listener, issuer pages included: CSP (default-src 'none', script-src 'self', connect-src 'self', img-src 'self' data:, font-src 'self', base-uri 'none', form-action 'self', frame-ancestors 'none'), HSTS one year without preload, X-Frame-Options: DENY, nosniff, Referrer-Policy: no-referrer, a minimal Permissions-Policy, Cross-Origin-Opener-Policy: same-origin. The theme snippet moved from index.html to ui/public/theme-init.js, so scripts need no 'unsafe-inline'. style-src keeps 'unsafe-inline': two stock pieces create a <style> element at runtime (the Radix select viewport and the sonner shim), and editing preactcn components was ruled out; inline CSS cannot run script, so the residual risk is CSS injection only. With an external OIDC provider the browser itself fetches the provider's discovery document and token endpoint, whose origins are not known to ichor-ui, so connect-src widens to 'self' https: in that case only; scripts stay 'self'.
• Token revocation (F6). authenticateLocal compares the token's iat with the credential's notBefore (same cached read as mustChange, 10 s), so RevokeSessions and a password change end tokens already issued, not only refresh grants. notBefore has whole-second resolution: a token issued in the same second as the revoke survives it. The UI signs out after its own password change. The kube-apiserver cannot do this check; a token already given to kubectl lives out its lifetime.
• Refresh rotation (F7). A refresh grant returns a new refresh token and retires the old. State is one entry per live family (families in the user's credentials entry, so all replicas agree and RevokeSessions or a password change drops them): the id of the current token and the family's expiry, at most 16 per user. Writes are optimistic-lock patches of the Secret, retried on conflict. Presenting a retired token deletes the family (the owner signs in again) and logs it. Rotation keeps the session's start and expiry, so the absolute lifetime is not extended. A token from before rotation is accepted once and starts a family. A race between two processes refreshing the same token is serialised by the CLI's file lock, so it does not trip reuse detection.
• Login limits (F8, F9, F10, F11). Failures are counted per (source, user), so a stranger cannot lock admin out; per source with a looser limit (30 free failures, at most 1 min); and per user across sources, where 30 failures in an hour add a 2 s delay to every sign-in of that user and never refuse one. Tables hold 50 000 keys and evict the entries with the fewest failures first, so flooding fresh names cannot push out a slowed key. The source is the TCP peer, or with --trusted-proxy-cidrs the rightmost X-Forwarded-For address outside the trusted list. Counters stay per replica (a Lease per attempt was too expensive): N replicas give N times the free guesses. /oidc/authorize, /login and /token have a per-source token bucket (20, 10 and 20 per second, bursts 100, 50 and 100). A ticket buys one successful login (a typo may retry). Logins hash on their own argon2 slot with a queue of 8 and answer 503 beyond it; password operations of a signed-in user have another slot, so one cannot starve the other (the memory is the same two slots as before).
• Role grants (F12). CreateUser and UpdateUser refuse to grant a role the caller does not hold, unless the caller is admin. This is in code, not a ValidatingAdmissionPolicy; an external (OIDC) identity has no Ichor roles and is still gated by RBAC alone.
• Streams (F13). A stream gets a deadline at its token's expiry and ends with Unauthenticated ("your sign-in expired"). A disable inside the token lifetime is not seen by a running stream.
• Errors (F14). Internal, unknown, data-loss and unavailable errors reach the browser as the code plus a reference; the full error is logged with that reference. Errors the caller caused are unchanged.
• /metrics (F15). It stays on the main port but needs a Kubernetes token that may get the non-resource URL /metrics (the TokenReview and SubjectAccessReview of the controller's secure endpoint). The ichor-ui role gains create on tokenreviews and subjectaccessreviews; the ServiceMonitors send the pod's token; the scraping Prometheus needs the metrics-reader ClusterRole.
• TLS (F16). TLS 1.2 or newer, with only ECDHE and AES-GCM or ChaCha20 suites on 1.2 (1.3 suites are fixed by Go), curves X25519 and P-256, on both listeners. The certreload mutex is unchanged.
• Dependencies and keys (F17, F18). oras-go v2.6.2 (govulncheck: no vulnerability reached); oidctrust refuses an RSA signing key under 2048 bits.
• No sign-in (F3). With neither OIDC nor Ichor users, ichor-ui serves only the procedures in readOnlyProcedures (a new RPC is closed until listed; downloads are not listed). --allow-unauthenticated (ICHOR_ALLOW_UNAUTHENTICATED, chart ichorUI.allowUnauthenticated) restores the open mode explicitly; /auth/config says anonymous: readonly|open and the UI banner shows which. Compatibility: the installer reads a live ichor-ui with no users, no OIDC and no flag as "ran open" and sets the flag with a NOTICE, so an upgrade changes nothing; ichor init and ichor adopt start ichor-ui with the flag, ichor init without --ui-url prints a warning at the end, and --ui-url drops the flag. A Helm upgrade of such an install turns read-only until ichorUI.allowUnauthenticated=true is set (the chart prints a note). Making users the ichor init default was left out of scope.

Not done here: shared login counters across replicas (documented per replica), a ValidatingAdmissionPolicy for role grants, re-checking a disabled user inside a running stream before its token ends, narrowing the Role's secrets: get to names (F19, with the credentials work).

Credentials and the ichor-ui identity

Findings: S1 (ichor-ui holds every CA private key through get secrets), F1/F7 (ichor-ui may impersonate any user and group), F19, S2, S3, S4, S6.

1. ichor-ui holds no CA key (S1)

Decided: the controller is the only reader of the secrets bundle. ichor-ui asks it for a credential by creating a TalosCredentialRequest (new kind, namespaced next to the cluster, short-lived), and the controller answers in the request's status.

• Channel. A CR, not a controller-local endpoint: it keeps "the UI changes the world by writing CRs", it is authorised and audited by the Kubernetes API server (the audit log shows who created the request, as the impersonated user, and which purpose), and it needs no new listener, certificate or NetworkPolicy. An endpoint would also have had to trust identity claims made by the caller, which is what the API server already checks.

• No private key crosses the channel. The requester generates the key pair in memory (Ed25519) and sends a PKCS#10 CSR in spec.csr. The controller checks the CSR signature (proof of possession), discards its subject and SANs, builds the subject from the purpose, signs with the CA, and writes the certificate and the CA(s) to trust into status. A reader of the request sees only public data. Nothing is written to a Secret, so ichor-ui needs no get secrets for credentials. The controller deletes a request 2 minutes after it was created.

• Purposes (spec.purpose) and the Kubernetes check the validating webhook makes with a SubjectAccessReview as the requester (new file internal/webhook/v1alpha1/credentialrequest_webhook.go; the other webhook files are untouched):

  purpose	certificate	default TTL, cap	SAR as the requester
  public	none; returns the cluster ID and the CA certificates	n/a	issue on talosclusters/public (ichor-ui's ServiceAccount)
  node-reader	Talos os:reader, signed by the current OS CA	10 min, 15 min	issue on talosclusters/node-reader (ichor-ui's ServiceAccount)
  support-bundle	Talos os:reader	10 min, 15 min	download on talosmachines/support-bundle (the signed-in user)
  etcd-snapshot	Talos os:etcd:backup	10 min, 15 min	download on talosmachines/etcd-snapshot
  packet-capture	Talos os:operator	10 min, 15 min	download on talosmachines/packet-capture
  credentials-talos	Talos os:admin or os:reader, signed by the rotation's new OS CA during a rotation, trust = old + new	365 d, 365 d	download on talosclusters/credentials
  credentials-kubernetes	admin in system:masters, same rotation rule	365 d, 365 d	download on talosclusters/credentials

  The verbs on the download rows are the ones ADR 0084 and 0093 already use, so installed bindings keep working. The two issue rows are new and granted only to ichor-ui's ServiceAccount; they are for the UI's own reads (node pages, logs, effective configuration), which today already run as the ServiceAccount after the caller proved get on the machine. A request that fails the SAR is denied at admission and never reaches the controller; the controller re-checks purpose, TTL cap and CSR itself.

• Who creates it. The three download purposes and credentials-* are created as the signed-in user (impersonation), so the SAR and the audit log name that user. That needs plain create on taloscredentialrequests: the shipped roles ichor-viewer, ichor-operator and ichor-admin carry it and the webhook is the gate. public and node-reader are created as ichor-ui's ServiceAccount. ichor-ui caches the node-reader credential (one per cluster, in memory, renewed after half its TTL) and the public data (1 minute).

• ichor-ui's Role (namespace of the controller) loses get and create on all Secrets. What remains is by resourceNames: ichor-user-credentials (get, patch, update; Ichor users) and the issuer's own Secrets that it reads by name. The one create that stays is the MachineReclaim credential Secret, narrowed by the existing ValidatingAdmissionPolicy. The bundle Secret, S3 and registry Secrets are no longer readable by it. Where ichor-ui needs a rendering (effective configuration, CNI facts) it renders with a placeholder bundle whose keys are random and whose public parts (cluster ID, CA certificates) come from public; every secret field is masked in the output anyway.

• Residual risk, stated. A compromised ichor-ui can no longer exfiltrate a CA key, mint certificates offline, outlive a CA rotation, or forge Talos identities beyond a 15 minute window. It can still ask the controller for what the identity it acts as may ask for. With users off that is the ServiceAccount (no download verbs by default); with users on, it can only impersonate Ichor users (section 2) and each purpose is checked against that user's RBAC.

• CRD change: a new kind only; no existing field is touched.

1a. What was built, and where it differs from the first design

• TalosCredentialRequest (api/v1alpha1/taloscredentialrequest_types.go; scaffolded with kubebuilder), its reconciler (internal/controller/taloscredentialrequest_controller.go), the signing code (internal/credentials/issue.go: NewCSR, IssueTalos, IssueKubernetes, PurposeLifetime, TalosRoles) and the webhook (internal/webhook/v1alpha1/credentialrequest_webhook.go, a new file; registered in cmd/main.go and the chart's webhook list).
• ichor-ui: internal/uiapi/credissuer.go (create the request, poll its status with the ServiceAccount, cache the reader identity and the public facts). The Talos client factories take a *TalosIdentity (CA, certificate, key) instead of a bundle; loadBundle is gone. GetMyKubeconfig takes the Kubernetes CA from public.
• Registry credentials. Rendering the effective configuration read the registry authSecretRef Secrets by name, which would have kept get secrets on the whole namespace. The backend now renders through maskedSecrets, which answers every Secret read with a placeholder: the view shows them masked anyway. The cost: a missing or malformed registry Secret is no longer noticed in that view (the controller still reports it on the machine).
• Acknowledging a CA rotation's credentials checks that the new Kubernetes CA is accepted with a certificate of purpose credentials-kubernetes that lives ten minutes, asked as the caller. That needs the download verb on talosclusters/credentials, which acknowledging did not need before (it needed only to update the operation). The alternative, a purpose the backend could ask for as itself, would have let a compromised backend obtain a cluster-admin certificate at will, which is what this change removes. The UI shows the download and the continue buttons together to the people who hold the verb; the CLI path (ichor cluster credentials --acknowledge) is unchanged.
• Roles. The ichor-credentials-role and ichor-diagnostics-role ClusterRoles also carry create on taloscredentialrequests; the webhook is the gate, so plain creation is harmless. ichor-ui's ClusterRole gets create and get on the requests and the two issue verbs. Its namespaced Role keeps only create on Secrets (MachineReclaim credentials) and, with Ichor users, the named ichor-user-credentials rules. The installer's ClusterRole lists the new CRD.
• Upgrade notes. An install with ichorUI.oidc.enabled and empty impersonate.groups or impersonate.users no longer renders: list the groups and users, or set ichorUI.oidc.impersonate.anyUser=true for users. The controller image and the chart must be upgraded together (the backend needs the controller to answer its requests; a new backend with an old controller waits 20 seconds and fails each read).
• Not done: a UI choice of the credentials-talos lifetime shorter than a day (a rotation hand-off must outlive the rotation); the controller's and the CLI's own dial certificates keep a year (section 3).

2. Impersonation (F1, F7)

Decided: ichor-ui's ServiceAccount may impersonate only named groups and only named users, written by the chart or the controller; there is no "any name" mode unless the operator says so in a value.

• Groups. resourceNames is always emitted: ichor:viewer, ichor:operator, ichor:admin plus ichorUI.oidc.impersonate.groups. With OIDC on, the chart fails to render when impersonate.groups is empty (before, empty meant any group). A system: group can therefore never be impersonated, structurally.
• Users. Kubernetes cannot match a name by prefix, and a user such as system:kube-controller-manager is as dangerous as a group. With Ichor users (ADR 0094) the controller keeps the list: it maintains the ClusterRole ichor-ui-impersonate with one resourceNames entry ichor:<name> per IchorUser for users and one ichor:<role> per role any of them has for groups (custom roles are allowed, so the three built-in groups alone would not do). An empty list of rules grants nothing; a rule is never written without resourceNames. It reconciles on every IchorUser change and every two minutes, and does not watch ClusterRoles (that would need list and watch on all of them). It holds escalate and update on that one ClusterRole by resourceNames, nothing more, so it can grant an impersonation right it does not hold. The binding is installed by the chart. With OIDC on, ichorUI.oidc.impersonate.users must be listed, or the operator sets ichorUI.oidc.impersonate.anyUser: true to accept the residual risk knowingly; an empty list without the flag fails rendering. A new Ichor user can sign in a few seconds after creation (the controller's reconcile).
• Rejected. A Kubernetes structured authorisation config with CEL: its CEL only filters which requests go to an authorisation webhook, it cannot deny by itself, and it needs a webhook service in the managed cluster. A ValidatingAdmissionPolicy cannot see the impersonator. Forwarding a minted token instead of impersonating would move the problem to the signing key ichor-ui already holds for the issuer.
• Residual risk, stated. With OIDC and anyUser: true, a compromised ichor-ui can act as any user by name (not as a system: group), including users bound to powerful roles. Without the flag (the default after this change) it can act only as the listed users, which are the users Ichor is already trusted to represent.

3. Lifetimes (S4)

Decided: in-process dial certificates live 10 minutes (ichor-ui; minted per call by the controller on request, section 1). The controller's own dial certificate and the CLI's stay as they are: those processes hold the CA key anyway, so a shorter certificate gains nothing. Files handed to operators keep the default of one year (the user's decision: downloads are Ichor-side and a shorter default would break existing workflows) and gain a choice: --ttl on ichor cluster credentials (default and cap 8760h) with --read-only for an os:reader talosconfig, and the same two choices on the UI credentials download (1 year, 30 days or 1 day; read-only).

4. etcd backup key (S2)

Decided: backups are sealed with a key derived (HKDF-SHA256, info ichor.maesh.dev/etcd-backup/v2, salt cluster ID) from the secretbox encryption secret of the bundle, which only control-plane machine configs carry and which Ichor never rotates (so a CA rotation keeps old backups readable, ADR 0063). The v1 key (from Cluster.Secret, present in every worker config) is derived too and tried second when reading, so existing backups restore; every new backup is v2. Rotating the secretbox key is therefore not part of the CA rotation (section 6). Built: etcdbackup.DeriveKey, Key{Current, Legacy}, Decrypt tries both.

5. Maintenance pin (S3)

Decided: the first apply goes through an unauthenticated TLS connection, so Ichor pins what it can. The SPKI fingerprint of the maintenance certificate seen when the machine is discovered (re-recorded while the machine is not approved) is stored in the new status.maintenanceFingerprint, and the controller refuses to send the initial configuration, which carries the CA keys, to a node that presents another one. A machine that never went through discovery (added by IP, reclaimed) is pinned at its first provisioning probe. Within one ichor init run the identity read and the apply share one pin. Talos issues a new maintenance certificate when a node reboots or changes address; an operator who has checked the new fingerprint (named in the machine's Ready condition) accepts it with the annotation ichor.maesh.dev/trust-maintenance-fingerprint. This narrows the window to the time between discovery and apply; it cannot authenticate the first sight, so the provisioning network still must be separate (design, Trust). Built: internal/controller/maintenancepin.go.

6. CA rotation coverage (S6)

Decided: documented as a follow-up. Ichor's rotation covers the Talos API CA and the Kubernetes API CA. The etcd CA, the aggregator CA, the service-account signing key and the secretbox key are not rotated. Kubernetes needs a dual-key period for the last two (the old key must keep verifying or decrypting until every token and Secret is re-written), which is a multi-stage operation with a re-encryption pass, not an extension of the current two-CA stages; the etcd CA rotation needs every member restarted with both CAs trusted in a defined order; and the backup key (section 4) derives from the secretbox secret, which would have to be versioned first. The limit is stated in the docs so that "rotate after compromise" is not read as complete.

Secrets handling

• Restore marker (S7). The marker in the backup bucket is stored as {"marker": ..., "mac": ...}. The MAC is HMAC-SHA256 over the marker JSON, keyed by HKDF over the backup passphrase with info ichor.maesh.dev/etcd-backup/marker/v1, so it follows whatever key material the backup passphrase derives from. A signature mismatch is an error. Snapshot rollback by replaying an older valid snapshot is not addressed here. Amendment (before v0.1.0): an unsigned marker is refused as well (ErrMarkerSignature, "take a new backup with this version"); the earlier compatibility path for unsigned markers served builds that never shipped, so it was removed.
• Scrubber (S8). bundleSecretMaterial is derived by reflection over every exported string and []byte field of the Talos secrets bundle (trimmed, so a PEM that lost its trailing newline still matches), so keys the bundle gains or that were missing (etcd, aggregator and service-account keys, encryption secrets) are redacted. Rotation CAs were already added by the rotation code. Base64 re-encodings are not scrubbed.
• Local files (S9). saveKubeconfigFile now chmods an existing directory to 0700 like saveBundleFile. The local secrets.yaml stays plaintext (0600, atomic); encrypting it is deferred, but loadBundleFile now warns on stderr when the file is group or world accessible. Downloaded etcd snapshots were already written 0600; the CLI now states that they are not encrypted.
• Reclaim Secret name (S10). Uses crypto/rand (rand.Text, lowercased, 16 characters) instead of the math/rand based utilrand.
• Other (S11). The manager logs in production mode (JSON, no stack traces on warn); --zap-devel turns development mode back on, other zap flags are unchanged. The kubetrust trust file is written through a unique os.CreateTemp file in the same directory and renamed (mode 0644, public CAs only). Refusing http:// S3 endpoints is left to the endpoint gating work.

Supply chain and workloads

Audit: workload hardening and the supply chain. Fixed here, all of it without a CRD change.

Decisions

• One release path. The images that ship are built by test/harness/ui-demo/images.sh (crane, no container runtime), as ADR 0104 unit 7 already had it. The Dockerfiles are for local docker build and carry the same digests and flags, so there is no second release path to drift. release-images.yml and release-chart.yml are replaced by one release.yml.
• Reproducible and pinned. The distroless base, golang and node images are pinned by digest (the tag stays in front for Renovate). Binaries build with -trimpath and -ldflags "-s -w -X ...Version"; the layer tar has fixed owner, mtime and order; the push uses --set-base-image-annotations. Pushing the same binaries twice gives the same digests (checked locally). goreleaser stamps internal/version.Version (it set main.version, which nothing reads).
• Release gating and split. A v* tag runs verify (module drift, lint, tests, UI CI, govulncheck, npm audit), then build (dependencies fetched, contents: read, no registry token), then publish in the release environment, the only job with packages: write and id-token: write. publish runs crane, cosign, syft and the attest actions and fetches no Go or npm dependency.
• Signing and attestation. Keyless cosign (GitHub OIDC) signs the three images and the OCI chart by digest; syft writes an SPDX SBOM per image, attested with actions/attest-sbom; actions/attest-build-provenance attests images and chart. Every new action is pinned by commit SHA. Verification is documented in docs/guides/install-with-helm.md. Ichor does not verify a signature before it runs the installer Job; digest resolution of the installer image (spec.installerDigest, audit M3) is deferred.
• CI scanning and updates. govulncheck ./... and npm audit --omit=dev --audit-level=high run on every PR and tag (oras-go is bumped to v2.6.2 for GO-2026-6497 and GO-2026-6499). renovate.json covers gomod, npm, GitHub Actions and Dockerfiles with digest pinning, the images.sh base, and the Nix flake lock; grouped, three open PRs at most, Monday mornings, a three-day release age. Renovate rather than dependabot because it handles Nix and digests. trivy and OSSF scorecard from the audit are not added (govulncheck and npm audit cover the dependencies; the images are distroless static).
• Pod Security Admission. The install manifest labels ichor-system enforce, audit and warn at restricted (enforce pinned to v1.32, the others latest). Every Ichor pod passes: non-root, seccomp RuntimeDefault, no privilege escalation, all capabilities dropped, only emptyDir, secret and projected volumes, no host namespaces. The Cilium Helm Job needs hostNetwork and runs in kube-system, so it is outside. The chart does not own the namespace; NOTES.txt and the guide give the kubectl label command.
• Pods and Jobs. Controller and UI run as 65532:65532 and under a new PriorityClass ichor-critical (value 1000000, below every system-* class, which a user namespace cannot use), shipped in the manifest and the chart (priorityClass.create). Installer and Cilium Jobs name user and group 65532 and carry requests 50m/64Mi and limits 1 CPU/512Mi. The service-account token stays mounted: both pods and the Jobs need the API. The installer ClusterRole gets scheduling.k8s.io/priorityclasses create/get/patch/update (the manifest now contains one).
• Replicas and PDB. The raw manifest stays at one replica each and ships no PDB (a minAvailable: 1 budget on one replica blocks node drains); the chart runs two and keeps its PDBs. Documented, not changed.
• Network. The Cilium default stays opt-in. The chart's policies gain externalCIDRs, talosCIDRs and uiIngressCIDRs; when set they replace the world entity for those rules. The installer's static copy keeps world, because it has no values. FQDN rules and a metrics-namespace restriction are deferred.
• Network (amended). Supersedes the bullet above for the defaults: the policies are on by default wherever Cilium is the CNI (chart: networkPolicy.cilium.enabled unset renders them when the cilium.io/v2 API is served; false is the escape hatch, with a NOTICE on upgrade). The controller and ichor-ui lose their static world egress to Talos nodes, API servers and S3; the controller maintains ichor-controller-talos-egress with exactly the addresses and names its CRs name, updated before a discovery probe is dialled. DNS egress goes through Cilium's DNS proxy so toFQDNs works. The flow table and the Cilium semantics relied on are in ADR 0100's amendment of 2026-11-02. externalCIDRs, talosCIDRs and uiIngressCIDRs remain as static fallbacks; world stays only for ichor-ui's 443 (OIDC issuer, pull request host) and its serving port.
• Workflow hygiene. go mod tidy no longer rewrites files in CI (go mod tidy -diff and go mod verify fail on drift); kind is pinned by version and sha256; every job has timeout-minutes; test-chart no longer runs on every branch push; actionlint (in the Nix shell) runs in the lint workflow; npm ci --ignore-scripts in the Makefile and Dockerfile.ichor-ui (the UI build and tests pass without install scripts).

As built

• Dockerfile, Dockerfile.ichor-ui, Dockerfile.installer, test/harness/ui-demo/images.sh (modes build and push, DIGEST_FILE), Makefile targets demo-images-build and demo-images-push, .goreleaser.yaml.
• .github/workflows/release.yml (new; the two old release workflows are gone), test.yml, test-e2e.yml, test-chart.yml, lint.yml, pr-title.yml, renovate.json, flake.nix (cosign, syft, actionlint).
• config/manager/manager.yaml (namespace labels, PriorityClass), config/manager-ui/manager.yaml, config/rbac/ichor-installer/role.yaml, internal/controller/job.go with job_test.go.
• Chart: values.yaml, templates/extras/priority-class.yaml, templates/network-policy/cilium.yaml, templates/rbac/ichor-installer.yaml, NOTES.txt, hardening_test.go.
• Not verifiable without GitHub or a cluster: the whole release.yml run (cosign, attest, environment approval), Renovate's reading of renovate.json, and a server-side PSA dry-run against a live cluster. The restricted profile was checked by reading the pod specs.
• Maintainer steps in the repository settings (not in files): create the release environment with required reviewers, and a tag ruleset for v*.
• ADR 0104 unit 7 named the two removed workflows; release.yml replaces them.

Typed hardening options for managed clusters

Decided: every hardening of a cluster Ichor creates is a typed field of TalosCluster.spec.security, off by default, for new and existing clusters. Unset means the rendered machine config and Cilium values are byte-identical to before (proved by rendering a control plane and a worker from one fixed secrets bundle before and after the change, and by a test that an empty or disabled spec.security renders the same bytes). Machine config stays typed: Ichor builds the Talos documents itself, below every patch, and no raw YAML is accepted. docs/guides/hardening.md lists every option, what it does and what applying it costs.

• Firewall (security.firewall). Enabled renders NetworkDefaultActionConfig block and one NetworkRuleConfig per service (ichor-fw-*) following the Talos v1.14 recommended control-plane and worker rule sets: apid from the controller, operators and control planes; trustd between nodes; kubelet from control planes and the pod subnets; etcd between control planes; 6443 from the controller, configured sources, nodes and the pod and service subnets; Cilium tunnel (8472, or 6081 for geneve), health (4240) and WireGuard (51871); KubeSpan (51820). allow entries open host ports for host-network services. The field set needs the node CIDRs as input because render sees one machine at a time. The preflight is the admission webhook: it refuses a firewall whose controllerCIDRs does not contain the address the webhook's pod routes to the endpoint from (a UDP connect, nothing is sent; skipped when the endpoint does not resolve). No reboot (fieldmeta already classes both kinds reboot-free). Not done: applying in Talos try mode with automatic revert, which needs a hardware run.
• WireGuard (security.wireGuard). Sets encryption.enabled, encryption.type: wireguard and encryption.nodeEncryption in the effective Cilium values, after the user layer, so the typed option wins. The existing values hash re-runs the Helm Job; the agents restart in turn, no node reboot. Strict mode is left to spec.cni.cilium.values.
• Pod Security and audit (security.podSecurity, security.audit). Merged over Talos's own KubeAdmissionControlConfig PodSecurity document, so only the levels set change and kube-system stays exempt; audit is one of three presets (metadata, rbacChanges, writes), none logging Secrets or tokens above Metadata. Both need a v1.14 contract and are refused on an older one rather than silently raising it. The kube-apiserver restarts (fieldmeta apiServerRestart), no node reboot.
• Disk encryption (security.diskEncryption: tpm, nodeID, kms). Renders VolumeConfig for STATE and EPHEMERAL (LUKS2, slot 0, EPHEMERAL locked to STATE); static is not offered; kms must be https; requireSecureBoot maps to the TPM key's checkSecurebootStatusOnEnroll. On a cluster with approved machines the webhook refuses enabling or changing it unless the annotation ichor.maesh.dev/confirm-disk-encryption-wipe: "true" is present; Ichor never wipes a node, the guide documents the one-node-at-a-time reset-and-add path. Ichor records no TPM inventory, so it cannot pre-check TPM presence; Talos's own enrolment failure is the check (to be confirmed on hardware).
• Talos API access from pods. A TalosConfigPatch with machine.features.kubernetesTalosAPIAccess is admitted with a warning (stronger wording for os:admin in allowedRoles); amended 2026-11-02: it was refused without override, which the user found too strict for the user's own cluster ("ichor 以外の部分はあまり強く制限しないで"). Render drops the feature from non-control-plane machines. An update that leaves the documents alone is not re-checked. The rule lives in internal/webhook/v1alpha1/security_options.go.
• Not modelled. SecureBoot and UKI, kernel lockdown, an os:reader talosconfig role, kubelet serving-certificate bootstrap: follow-ups.
• Needs hardware. That a block-mode firewall with these rules leaves a Talos plus Cilium cluster healthy (including pod to apiserver traffic and DHCP renewal); that the webhook's routed source matches the address Talos sees when the controller runs in a pod; the WireGuard rollout; the apiserver restart for the two Kubernetes documents; the TPM and KMS behaviours.

Hardware results: cluster options

Run 2026-11-02 on the e2e harness (prefix ichor-e2e-s2): Ichor v0.0.1870000002-gb9fa12af (HEAD b9fa12af), Talos v1.14.3, Kubernetes v1.37.1, Cilium chart 1.18.0 (geneve tunnel), one control plane and one worker created by ichor init plus ichor machine approve, no sign-in (--allow-unauthenticated). Pod and service subnets 10.248.0.0/16 and 10.249.0.0/16; nodes are outer pod IPs (10.244.1.20 control plane, 10.244.1.164 worker). ichor init to both nodes Ready took about 11 minutes (about 2 minutes for the worker after approval). The options were enabled one after another on the running cluster. After each step all nodes Ready, kube-system pods Running, DNS from a pod on each node, a pod reaching the apiserver Service, pod to pod across nodes, and both TalosMachines Ready with no NodeUnreachable were checked, except where noted.

1. Firewall: works; two findings about the controller's source address.
   • Converged within 1 to 2 seconds of the patch on both nodes (NfTablesChain ingress, policy drop, and prerouting exist; there is no networkrulechains resource in Talos v1.14.3, use talosctl get nftableschains ingress -o yaml). Health after the change: all checks passed, TalosMachines stayed Ready, the controller reconciled the machines normally.
   • Preflight: a controllerCIDRs of 192.0.2.0/24 is refused: spec.security.firewall.controllerCIDRs [192.0.2.0/24] does not contain 10.248.0.108, the address Ichor's controller connects from ... if the controller runs in a pod, list the pod network and its node's address. The webhook's routed source was the controller pod's IP, as the design assumed.
   • What the nodes see (from /proc/net/nf_conntrack through talosctl read): the controller reaching apid on its own node arrives from the pod IP (10.248.0.108); the controller reaching apid on another node arrives from the controller's node IP (10.244.1.20), because Cilium masquerades pod traffic to a remote node's address. So both must be listed, as the guide says: the pod range (or the node's pod CIDR) and the node address. The same masquerade means a pod on node A that talks to a host port on node B is seen as node A: pod-level restrictions only hold for host ports of the pod's own node, and any pod on a listed node reaches what that node's address may reach. With controllerCIDRs narrowed to the control plane's pod CIDR plus its node IP, a pod on the worker was dropped on the worker's apid (50000), on the control plane's apid, etcd (2379) and kubelet, and allowed on trustd (50001, node CIDRs), 6443 and 4240; a pod on the control plane (same /24 as the controller) still reached apid on both nodes. A port in no rule (9999, 22) is dropped (timeout), not refused.
   • A narrow controllerCIDRs (the control plane's pod /24) would lock the controller out if it were scheduled on the worker; not tried.
   • DHCP renewal: not observed. The lease is 23976 hours, so no renewal happens in the run. A DHCP REQUEST/ACK exchange happened before the firewall (hostname change); Talos's DHCP client uses raw sockets, which a netfilter input chain does not see, but this was not exercised under block mode.
   • Lock-out and recovery, tried and safe. The webhook only checks the controller's route to the endpoint, so it accepts controlPlaneCIDRs that do not contain the control plane's own address: setting controlPlaneCIDRs: [10.244.1.99/32] with controllerCIDRs: [10.248.0.0/24] was admitted, the worker received the rule, and the controller then reported on the worker Ready=False CurrentConfigUnavailable and NodeReachable=False NodeUnreachable: the node's current configuration cannot be read (clear). Reverting the spec alone cannot help, because the controller cannot reach the node to deliver it. Recovery that worked: from a source in talosAPICIDRs run talosctl -e <node> -n <node> patch machineconfig --patch @rule.yaml with a NetworkRuleConfig (portSelector: {ports: [50000], protocol: tcp}, ingress: [{subnet: <control plane>/32}]), then revert the spec: the machine was Ready again 26 seconds after the talosctl patch and the controller removed the temporary rule. Keep a talosAPICIDRs entry for an operator workstation before enabling the firewall; without one the way back is the console or a reset.
2. WireGuard: works after a manual Cilium agent restart; the guide was wrong about the rollout.
   • The patch rendered UDP 51871 into the ingress chain of both nodes within 2 seconds and re-ran the Cilium Helm Job (revision 2, 33 s, 49 s from patch to enable-wireguard: "true" in cilium-config and CNIValuesApplied=True). The agent pods did not restart (the DaemonSet carries no config hash) and cilium-dbg encrypt status kept saying Encryption: Disabled while CNIValuesApplied was True. kubectl -n kube-system rollout restart ds/cilium took 14 seconds, after which both agents report Encryption: Wireguard, cilium_wg0, port 51871, Number of peers: 1, the cilium_wg0 packet counters grow with cross-node pod traffic (79 to 129 rx packets for ten requests), and DNS, apiserver and cross-node pod traffic work. UDP 51871 passed the block-mode firewall between the nodes (listed nodeCIDRs). The guide now says the restart is manual.
3. Pod Security restricted: works; needs ichor config apply.
   • After the patch the control plane is Ready=False APIServerRestartRequired and nothing happens until a config rollout is started (the condition says so; the guide did not). ichor config apply --follow took 171 s in total with one control plane (kube-apiserver container restarted 49 s after the start; the operation finished the machine at 112 s). During it the API server is unreachable, the controller logs errors and kube-controller-manager and kube-scheduler restarted once each (lost leases); nothing else restarted.
   • kubectl run --privileged in default is refused (violates PodSecurity "restricted:latest": privileged ...), a plain busybox pod is refused, a restricted-compliant pod is admitted; a privileged pod in kube-system is admitted (exempt); existing non-compliant pods in default kept running.
4. Audit writes: works. ichor config apply took 109 s (apiserver restart 55 s after the start). /var/log/audit/kube/kube-apiserver.log (readable with talosctl read) had 38 184 events: 208 at Request level (the label patch on a Service carried its body), the rest Metadata. A Secret and a ConfigMap created with marker values appear at Metadata only, the markers (and the base64 of the Secret one) are absent, and no event for secrets, configmaps, serviceaccounts/token or tokenreviews is above Metadata.
5. Disk encryption nodeID via wipe and rejoin: works, with friction; tpm on a TPM-less node fails and Ichor does not say why.
   • Without the annotation the update is refused (spec.security.diskEncryption changes for a cluster with 2 approved machine(s): ... Set the annotation ichor.maesh.dev/confirm-disk-encryption-wipe="true"). With it, both machines went Ready=False RebootRequired (VolumeConfig fields) and nothing was applied, as designed.
   • The path: ichor node reset <node> --override <reason> (needs the override verb; since ADR 0110 --override alone, with an optional --override-reason), delete the stale Kubernetes Node, approve the pending machine again. Observed friction, none of it fixed here: (a) --confirm must be the machine name (its UUID) for the TalosOperation webhook, but the CLI asks for the node name and then the webhook refuses ("reset needs confirm equal to the machine name"); passing the UUID as NODE and as --confirm works; (b) a bare pod (my probe) on the node blocked the drain and the operation did not retry for more than ten minutes after the pod was gone, until an annotation poked the TalosMachine (fixed in 6fa38b78, after 8e749c1e: a blocked drain is requeued every 15s, see ADR 0010's amendment; not yet rechecked on hardware); without a blocking pod the reset took 45 s; (c) the reset leaves the Kubernetes Node, whose InternalIP is excluded from discovery (knownHosts), so the machine was not rediscovered (five minutes and a poke) until kubectl delete node; the guide's "Reset and add" (ADR 0107) is for machines of another cluster and refuses one already in Ichor.
   • After rejoin: node Ready 119 s after approval; talosctl get volumestatus shows encryptionProvider: luks2, configuredEncryptionKeys: [nodeID], encryptionSlot: 0 for STATE (/dev/vda5) and EPHEMERAL (/dev/vda6). The control plane, not wiped, stays unencrypted and RebootRequired. Firewall and WireGuard were applied from the first configuration (Cilium peers 1, ingress chain present).
   • TPM: with type: tpm the reset worker installed, rebooted and stuck in stage booting with STATE failed: Talos logs every 100 ms failed to call key handler ... TPMKeyHandler: stat /dev/tpm0: no such file or directory and error formatting and encrypting volume: no handlers available. Ichor shows only Provisioned=False NodeStarting: the node is in stage "booting" after four minutes: the cause is not reported. requireSecureBoot not tried.
6. kubernetesTalosAPIAccess with os:admin: refused for a user without override. As --as=s2-op --as-group=ichor:operator: allowedRoles contains os:admin, which gives every pod in [kube-system] a Talos administrator (use os:reader, or hold the "override" verb on this patch to override); os:reader is admitted with the warning. As cluster-admin (system:masters, which holds every verb) the os:admin patch was admitted with the warning, as the rule says; it was deleted at once. A cluster-layer patch with this feature is refused for a worker at render (can only be enabled on control plane machines); use a role: controlplane target.

Not run: KMS, requireSecureBoot, a controller scheduled on the worker, strict mode. Teardown: all ichor-e2e-s2 objects deleted and confirmed; the cached ISO disk untouched.

Fixes after the hardware run (2026-11-02)

Not yet rechecked on hardware unless a line says so.

1. WireGuard: the managed Cilium values now carry rollOutCiliumPods: true whenever wireGuard.enabled is set, so the chart puts a cilium-config checksum on the agent pod template and the DaemonSet rolls (rolling update) when the Helm Job changes the config. CNIValuesApplied was already set only after daemonSetRolledOut (observed generation, updated and ready pods), so it now means the agents run the new values. Chosen over a controller-side annotation patch because Helm owns the DaemonSet and the chart already has the switch. Recheck: patch wireGuard on a running cluster, expect Encryption: Wireguard with no manual restart.
2. Firewall lock-out: the TalosCluster webhook refuses a firewall change whose controlPlaneCIDRs miss an approved control plane's address (the controller reaches the other nodes from it, so Ichor itself would be cut off; address from the machine's observed or declared endpoint, hostnames are skipped). A nodeCIDRs that misses a machine and an empty talosAPICIDRs are warnings only, per the 2026-11-02 direction that errors are for what breaks Ichor's own operation or cannot render, and advice about the user's cluster is a warning. The recovery that worked is in the guide.
3. Reset: a reset asked for by hostname is stored with the machine name as confirm, which the TalosOperation CRD compares with machineRef.name. A removal whose drain fails (a bare pod) now returns the error so the reconcile is retried with backoff, writes the pod into the Removal condition and a Warning Event. There is no force-drain option; the guide says to delete the pod. The Kubernetes Node of a reset machine is annotated ichor.maesh.dev/reset-by-ichor and discovery ignores its address while it is not Ready; it is not deleted because the kubelet could register it again.
4. A node stuck in stage booting reports volumes in phase failed and their error in the Ready message.
5. kubernetesTalosAPIAccess: render drops it from workers (Talos allows it on control planes only), so a cluster-layer patch works and renders where it is valid. The same direction replaces the admission refusal of os:admin (and the override override) with a Warning; the verb is no longer consulted for this patch rule. This amends the earlier rule of this ADR.
6. Confirmed designed: ADR 0080's amendment leaves API-server restarts to a config rollout. The APIServerRestartRequired and RebootRequired messages now give ichor config apply --cluster NAME, and the guide says so.

Hardware results

2026-11-02: Ichor's own hardening, e2e harness (lane s1)

Setup: Talos v1.14.3, Kubernetes v1.37.1, Cilium 1.18 through Ichor, one control plane and one worker made by ichor init --preset single --cni cilium --cni-tunnel geneve --ui-url https://<LB> --admin-user admin with the previous main f3fc27f3 (v0.0.1870000000-gf3fc27f3), then ichor upgrade ichor to b9fa12af (v0.0.1870000001-gb9fa12af). A third node (in maintenance mode) and a second worker were added for the discovery step. Hubble is not installed; drops were read with cilium-dbg monitor --type drop on every agent.

1. Upgrade path: FAILED once, passed after two workarounds (two product bugs).

• First ichor upgrade ichor (operation ichor-9gpg6): the installer Job failed and rolled back from its snapshot (clean: old pods kept running). Failing step, verbatim: applying ValidatingAdmissionPolicy ichor-ichor-controller-job-image: ValidatingAdmissionPolicy.admissionregistration.k8s.io "ichor-ichor-controller-job-image" is invalid: spec.paramKind: Forbidden: user &{system:serviceaccount:ichor-system:ichor-installer ...} must have "get" permission on all objects of the referenced paramKind (kind=IchorInstallation, apiVersion=ichor.maesh.dev/v1alpha1). Bug: the installer ClusterRole (config/rbac/ichor-installer/role.yaml, and the chart's copy) holds no get (the apiserver asks for it cluster-wide) on ichorinstallations, so no upgrade through the installer can create the ichor-controller-job-image policy. Workaround used: a ClusterRole/Binding giving ichor-installer get,list,watch on ichorinstallations; the second run (ichor-8sr7w) completed in 53 s.
• After the upgrade: PriorityClass ichor-critical (1000000) exists and both pods run under it; the namespace carries pod-security.kubernetes.io/{enforce=restricted (v1.32),audit,warn=restricted}; the controller, ichor-ui and the installer pods were admitted and Running with no PSA warning, rejection or violation event; the four ValidatingAdmissionPolicies and bindings exist; the Jobs were admitted under the Job policies; ichor-ui and the controller restarted cleanly (a rollout restart of the controller also came up and took the lease within 20 s).
• Bug (breaks logins after an upgrade or an ichor init install): the impersonation ClusterRole is ichor-ichor-ui-impersonate in the cluster (kustomize namePrefix), but controller.ImpersonationRoleName is the constant ichor-ui-impersonate. The controller logged the impersonation ClusterRole does not exist; nothing to keep; the installed role had rules: null, so every signed-in request failed: ichorusers.ichor.maesh.dev is forbidden: User "system:serviceaccount:ichor-system:ichor-ichor-ui" cannot impersonate-on:user-info:list resource "ichorusers". The controller's own resourceNames (config/rbac/role.yaml) name the unprefixed role too. Workaround used: an unprefixed ClusterRole ichor-ui-impersonate with no rules plus a binding for the ichor-ui ServiceAccount; the controller then wrote [{users, resourceNames: [ichor:admin]}, {groups, resourceNames: [ichor:admin]}] ("updated the impersonation ClusterRole", users: 1) and logins worked. The chart (no prefix) is not affected by this reading; not run.
• One Reconciler error ... creating ichor-controller-talos-egress: ciliumnetworkpolicies.cilium.io "ichor-controller-talos-egress" already exists was logged once during the rollover (a create race between the old and new pod); the policy existed afterwards.

2. Cilium default deny: PASSED.

• CiliumNetworkPolicies present: ichor-controller-manager, ichor-default-deny, ichor-installer, ichor-ui and the controller's ichor-controller-talos-egress. cilium-dbg endpoint list shows ingress and egress enforcement Enabled for the controller and ichor-ui.
• ichor-controller-talos-egress held exactly the machines' addresses (10.244.0.55/32 control plane, 10.244.2.118/32 worker, the same as the machines' spec.endpoint and spec.discovery.addresses), ports 50000, 6443, 2380.
• Steady state (150 s on both agents): no policy drops for the controller or ichor-ui. The only xx drop lines were Unsupported L3 protocol IPv6 router solicitations (fe80::... -> ff02::2) from the two pods' identities, which no policy rule governs.
• Discovery: ichor cluster discovery add 10.244.1.68/30 at 11:02:38 covering the third node (10.244.1.70); the pending machine existed at 11:02:47, found on the first pass; the policy listed 10.244.1.68/30 and 10.244.1.70/32 within 8 s; no policy drops during the 150 s window. ichor machine approve ... --role worker as ichor-admin (new approve verb) was accepted and the node was Ready 4 min 20 s later; no policy drops on the two monitored agents during the join (the new node's agent was not yet monitored).
• DNS and toFQDNs: no resolver errors in the controller or ichor-ui logs; a TalosEtcdBackup whose S3 host is the name 10.108.131.191.sslip.io got a toFQDNs rule (matchName plus port 9000), the controller resolved it and reached the store (the store answered Could not parse the specified URI, a virtual-host parsing refusal of the throw-away S3 server, so network and DNS worked). In-cluster toServices (item 4 of ADR 0100) was not run.
• Observation: in this shape (outer LoadBalancer to an inner NodePort) every external client reaches ichor-ui from the same source address (the node), see item 4.

3. Credentials: PASSED, with one RBAC defect and a note on the CLI.

• ichor cluster credentials ichor-e2e-s1 (and --ttl 1h --read-only) work, but the CLI reads the secrets bundle with the caller's own Kubernetes rights and signs locally; it does not go through ichor-ui and creates no TalosCredentialRequest. The files worked: kubectl get nodes with both; talosctl version with the admin and the read-only talosconfig; the read-only one (Roles: os:reader, expiry 50 minutes) got PermissionDenied desc = not authorized for get machineconfig while the admin one listed it.
• Through ichor-ui: DownloadCARotationCredentials (the only download path ichor-ui has for a talosconfig and kubeconfig) with lifetimeSeconds: 3600, readOnly: true, during the rotation below, created cred-... requests with purposes credentials-kubernetes and credentials-talos, both TTL 1h, in the same second; the zip's talosconfig was os:reader expiring in 58 minutes, talosctl version worked, get machineconfig was PermissionDenied, and the kubeconfig (new CA) ran kubectl get nodes. Both requests were gone within 2 minutes (no request left at 12:02:41, created 12:00:26).
• Other purposes seen: node-reader (ichor-ui's own node read, as its ServiceAccount) created 11:19:16, deleted 11:21:16; support-bundle (as ichor:admin) created 11:23:54, deleted 11:25:54; the support-bundle download wrote a 391682-byte zip.
• kubectl auth can-i get|list secrets -n ichor-system --as=system:serviceaccount:ichor-system:ichor-ichor-ui is no; get secrets/ichor-e2e-s1-secrets and get secrets/ichor-oidc-signing are no; get secrets/ichor-user-credentials is yes; create taloscredentialrequests is yes.
• Defect: the shipped ichor-admin (and ichor-operator, ichor-viewer) ClusterRoles hold the download verbs but not create on taloscredentialrequests, so a user's support bundle failed: taloscredentialrequests.ichor.maesh.dev is forbidden: User "ichor:admin" cannot create resource "taloscredentialrequests" in API group "ichor.maesh.dev" in the namespace "ichor-system" (the CLI said "ichor-ui acts as its own ServiceAccount here" misleadingly). ADR section 1a says the shipped roles carry it; config/rbac/ichor_operator_role.yaml and dist/chart/templates/rbac/ichor-user-roles.yaml do not (ichor-credentials-role and ichor-diagnostics-role do). Workaround used: bind group ichor:admin to those two roles. The same ichor-admin role also cannot update a TalosEtcdBackup (ichor etcd backup now failed with cannot update resource "talosetcdbackups"), unchanged by this ADR.

4. ichor-ui: PASSED, one oddity.

• curl -I /: CSP, HSTS (max-age=31536000), x-frame-options: DENY, x-content-type-options: nosniff, referrer-policy: no-referrer, permissions-policy, cross-origin-opener-policy: same-origin present; also on the issuer's redirect.
• Body limits: unauthenticated POST to UserService/CreateUser of 300000 bytes gave 429 resource_exhausted: message size 300000 is larger than configured max 262144; 3000000 bytes gave message is larger than configured max 262144 - unable to determine message size: http: request body too large; 100000 bytes to /oidc/login gave 400 malformed body.
• Login limit: five wrong passwords for rltest in 2 s, then 429 rate_limited with Retry-After: 1 (then 2); a correct password from the same source straight after was also 429 (too many failed logins); admin from the same source still got 200; the correct rltest password from a pod inside the inner cluster (its own source address) got 200. A client outside the cluster (workstation) and a pod of the outer cluster share one source in this shape (the NodePort rewrites the source to the node), so a second external source cannot be told apart: with an L4 load balancer in front, one stranger's wrong passwords for admin slow admin down for everyone, as trustedProxyCIDRs cannot help without an X-Forwarded-For.
• /metrics without a token: 401 Unauthorized; with a ServiceAccount token without the right: 403. With a malformed bearer token (nope) the answer is 500 (log: Authentication failed ... invalid bearer token), where 401 would be expected; no content leaks.

5. etcd backup: PASSED; restore round trip run, with a deadlock found.

• TalosEtcdBackup to a throw-away S3 server (Zenko cloudserver; MinIO images are no longer pullable) in the outer namespace, http://<ClusterIP>:9000: refused without insecure: spec.destination.s3.endpoint is a plain http:// URL, which sends the credentials in clear; use https:// or set spec.destination.s3.insecure to true; created with insecure: true; the controller added a toCIDRSet rule for the store and ichor etcd backup now stored s1/etcd-20261102T113928Z.snapshot.age (9775654 bytes).
• v2 seal: a throw-away program derived the keys from the cluster's secrets bundle (etcdbackup.DeriveKey); the object opens with the v2 (current) key (9773088 bytes plaintext) and not with the v1 (legacy) key (incorrect passphrase).
• ichor restore --live (11:40:24) rewound etcd to a fresh snapshot (the restore operation record, written after it, vanished from the API). The marker s1/etcd-restore.marker.json is {"marker": {...}, "mac": <64 hex>} with step: Rejoining: signed. The controller then deadlocked: it runs on a worker, and after the restore the workers' kube-proxy and Cilium agents were stale, so pod to Service traffic (10.249.0.1:443) was refused (dial tcp 10.249.0.1:443: connect: connection refused; the controller logged Waiting for the API server for 6 minutes with 3 restarts, cilium-operator crash-looped, the worker agents were 0/1). The restore tail that is meant to replace those pods (etcd_restore.go, the pod-restart Role) cannot run while the controller cannot reach the API. Deleting the worker kube-proxy pods, then the worker cilium pods by hand freed it; the controller then recreated the etcd-restore-r6jdv operation from the marker, the CA rotation below paused it (ClusterCARotating) and it ended Completed afterwards. Not compared with f3fc27f3: whether 0108 changed this is open. An admin's refresh token no longer worked after the restore (the credentials Secret went back in time): the admin signed in again.
• Also (CA rotation, not in the brief): ichor cluster rotate-ca ichor-e2e-s1 --yes completed (ca-rotation-56n5m, 1 control plane and 2 workers, about 8 minutes including the paused restore); the CLI built its hand-over files itself, not through ichor-ui.

6. Endpoint guard: PASSED.

• As cluster-admin: 127.0.0.1 refused (spec.endpoint "127.0.0.1" is refused: the address 127.0.0.1 is a loopback address), 169.254.169.254 refused (is link-local), 10.248.5.5 refused (is inside the cluster's own pod or service range 10.248.0.0/16).
• As a user bound to the TalosMachine editor role and ichor-viewer: an address inside the discovery CIDR (10.244.1.69) was created; 10.244.9.9 and node.example.com were refused (is not an address of the cluster's spec.discovery (addresses, cidrs): you need the "update" verb on talosmachines/endpoint); patching the endpoint of the existing machine out of discovery was refused the same way; setting approvalState: approved was refused (approving a machine hands it the cluster's configuration: you need the "approve" verb on talosmachines/approval).
• A user in group ichor:admin (holds talosmachines/endpoint) created a machine at 10.244.9.9.

Gaps: restore --recover-cluster, ichor adopt, a Cilium chart upgrade through the Cilium Job, a Helm-only install, toServices for an in-cluster store, the multi-replica login counters, and the firewall, WireGuard, PodSecurity and disk-encryption options were not run. Everything created for this run (nodes, disks, Services, pods, Secrets under the prefix ichor-e2e-s1) was deleted and a listing of the prefix is empty; the shared ISO disk is untouched.

2026-11-02: Fixes after the hardware run

The product bugs of the Ichor-side run above, each with a test that fails without the fix. None is rechecked on hardware yet.

1. Installer and IchorInstallation. The installer's ClusterRole (config/rbac/ichor-installer/role.yaml, the chart's copy) now holds get on ichorinstallations, which the apiserver requires of whoever creates the ichor-controller-job-image policy (its paramKind). Only get; the hardware workaround used get,list,watch, which the policy controller does not need from the creator. Test: TestInstallerCanApplyEveryObjectOfTheInstallManifest checks that the installer role (config and chart) can get and create every kind of the rendered dist/install.yaml and get the paramKind of each policy; a new kind in the manifest fails it until the installer is given its rights.
2. Impersonation role name. The controller looks the role up as ichor-ui-impersonate; kustomize's namePrefix made it ichor-ichor-ui-impersonate, and the controller's own resourceNames named the unprefixed one. The role in config/rbac/ichor-ui is now called ui-impersonate, so the install manifest and the chart both end up with ichor-ui-impersonate, the constant controller.ImpersonationRoleName. One source, no flag; the installer's bind/escalate list names the new role. An install made by ichor init before this keeps an orphan ichor-ichor-ui-impersonate ClusterRole and binding with no rules (harmless; delete them). Test: TestControllerLooksUpNamesThatExist renders both install paths and requires the role, its binding and the controller's get,update,escalate on that name. The controller now also says so when the role is missing: condition ImpersonationRoleReady=False (reason RoleMissing, message naming the role) on the IchorInstallation and a Warning Event, besides the log line (envtest: "reports on the IchorInstallation, with an Event, when the role does not exist").
3. Shipped user roles. ichor-admin now holds create on taloscredentialrequests (it has every download verb; ichor-operator and ichor-viewer have none, so they need none), patch,update on talosetcdbackups for the operator and the admin (ichor etcd backup now sets an annotation), and create,delete on talosetcdbackups for the admin (defining a backup). Config and chart. Test: TestBuiltinRolesRunTheDocumentedFlows, a table of 25 flows (CLI, UI and guide flows) with the verbs each makes as the user and the lowest role meant to run it, checked against config/rbac and the chart. Flows that need secrets (ichor cluster credentials, bundle-export) stay out by design.
4. Malformed bearer token on /metrics. controller-runtime's metrics filter answers 500 to every authenticator error, and a token the cluster does not accept is the error invalid bearer token. internal/metricsauth is that filter with this one error mapped to 401 (an unreachable apiserver is still 500); both the controller's and ichor-ui's metrics endpoints use it. Test: TestMetricsStatusCodes (fails with 500 without the mapping). The RPC path already answered Unauthenticated for every token error; TestInterceptorRequiresTokenAndStoresIdentity now pins it for malformed tokens.
5. Restore deadlock. Not caused by 0108: ADR 0089 (second run, item 2) records the same worker with a stale kube-proxy refusing the Service address while the controller ran on it. The Cilium egress policy is unlikely to contribute: its kube-apiserver entity rule drops (a timeout), and the observed symptom was connection refused, which is kube-proxy rejecting a Service it thinks has no endpoints. That is an inference from the symptom, not measured. The existing tail replaces the stale kube-proxy and Cilium pods through the Kubernetes API, so a controller that cannot reach the Service cannot start it. Fixes: (a) the controller keeps the InternalIPs of the control-plane Nodes in /var/run/ichor/trust/apiservers (the emptyDir that survives a container restart), and when the Service address has not answered for 30 s its start-up wait tries those addresses too, alternating with the Service address (certificate verified as kubernetes.default.svc, which Talos puts in the apiserver certificate); it stays on the address that answered until its next start (cmd/apifallback.go; tests TestAPIFallback*, TestAPIServersFileRoundTrip); (b) both install paths now prefer a control plane for the controller (preferredDuringScheduling, the chart's manager.affinity default; TestManagerPrefersAControlPlane). Limits: a controller pod rescheduled to a new pod (a new emptyDir) has no file, so (a) helps the common case of a restarting container, not a fresh pod; (b) only applies when the pod is next scheduled. Needs hardware: a live restore with the controller on a worker, expecting Waiting for the API server for about 30 s and then the controller starting through a control plane address, the tail replacing the workers' kube-proxy and Cilium pods, and no restart loop; plus a restore with the controller on a control plane (unchanged path).
6. Docs. docs/guides/users-and-login.md has a section "Client addresses behind a load balancer" (a shared source address shares the failed-login counter; options: an L7 proxy with X-Forwarded-For and trustedProxyCIDRs, or externalTrafficPolicy: Local, or accepting the slowdown) and says that a refresh token does not survive an etcd restore, by design (its state is in a Secret that the restore rewinds). The chart gained ichorUI.service.externalTrafficPolicy (empty by default) for the second option (TestUIServiceExternalTrafficPolicy).

Hardware results: 15-node scale (5 CP + 10 workers)

Run on 2026-11-02 and 2026-11-03 (local time, UTC+9; the times below are UTC) on the outer cluster, namespace ichor-e2e, prefix ichor-e2e-sc, with Talos v1.14.3 and Kubernetes v1.37.1 first. Five control planes (3 GiB, 2 cores) and ten workers (1536 MiB, 2 cores), all booted from the cached ISO in maintenance mode, 13 of them placed on talos01 and 2 on talos03 (a node affinity that excludes talos02, in the harness's new scale mode). The outer nodes' memory requests ended at 86% (talos01) and 94% (talos03), so no node had to shrink. Pod subnet 10.248.0.0/16, service subnet 10.249.0.0/16, Cilium with the geneve tunnel, users on, the preset large.

Builds: HEAD d1b08227 and the previous main f3fc27f3 (in a separate worktree). Deviation: the brief named HEAD v0.0.1880000000-gd1b08227 and the old build v0.0.1880000001-gf3fc27f3; an upgrade must go to a higher version, so the old build was v0.0.1880000000-gf3fc27f3 and HEAD v0.0.1880000001-gd1b08227.

3. Upgrade checks: PASSED. ichor upgrade ichor v0.0.1880000001-gd1b08227 completed with the installer paramKind fix; IchorInstallation conditions UpToDate=True, SettingsApplied=True, ImpersonationRoleReady=True (Present). The controller was placed on a control plane (ichor-e2e-sc-cp5, later ichor-e2e-sc-cp); the namespace carries Pod Security restricted (enforce v1.32, audit and warn latest); the Cilium policies ichor-default-deny, ichor-controller-manager, ichor-ui, ichor-installer and ichor-controller-talos-egress exist and the last one lists 15 addresses (one per machine). cilium-dbg monitor --type drop for 2 minutes on the agents of the controller, the UI and one more node (cp5, w6, w7): no policy drop; one xx drop (Unsupported L3 protocol) for an IPv6 router solicitation on w7. Logins through the issuer on the LoadBalancer: the first sign-in of admin gets token 403 access_denied: change your password first (sign in to the Ichor UI); no Kubernetes token is issued until then for the Kubernetes audience; a login with the ichor-ui audience, UserService/ChangeOwnPassword (200) and a new login give a token whose kubectl auth whoami is ichor:admin (groups ichor:admin). It kept working after the Kubernetes upgrade, the CA rotation, the restore, the firewall and WireGuard.

4. Health after each scale operation. After every step above: 15 of 15 Nodes Ready, 15 of 15 TalosMachines Ready=True, the Cilium DaemonSet fully ready, DNS (kubernetes.default.svc.cluster.local) resolving and a pod-to-pod request across two nodes working. Exceptions: 14 of 15 machines Ready for under a minute right after the CA rotation (all 15 on the next look); the Cilium DaemonSet read 15/16 while the removed control plane's Node object was still there. The restore ran with the controller on a control plane (the unchanged path); the worker fallback was not exercised because the controller never landed on a worker. After the restore the controller restarted once (Restart Count: 1), recreated the operation etcd-restore-dvsv8 from the marker, and it ended Completed; the ConfigMap written after the snapshot was gone; talosctl etcd members listed 5 members at the end.

5. Security options: PASSED. Firewall with controllerCIDRs 10.244.0.0/16 and 10.248.0.0/16, nodeCIDRs 10.244.0.0/16, controlPlaneCIDRs 10.244.1.0/24, talosAPICIDRs 10.244.0.0/16, apiServerCIDRs 0.0.0.0/0 and an allow for the UI's NodePort 30080: the admission webhook accepted it, ichor-fw-apid, -kubelet, -cilium-tunnel, -cilium-health (all nodes) plus -etcd, -kubernetes-api, -trustd (control planes) were in the machine config, Ichor still applied a sysctl patch to all 15 in 52 s, the kube API and the UI login through the LoadBalancer still worked. WireGuard (spec.security.wireGuard.enabled): the agents rolled without a manual restart, cilium-dbg encrypt status on a control plane and a worker: Encryption: Wireguard, 14 peers each. /metrics of the controller (through a port-forward): 401 without a token, 401 with Bearer nope, 401 with Bearer a.b.c (it was 500 for a malformed token before the fix). ichor etcd backup now as ichor:admin (OIDC token) worked, before and after both options (etcd-20261102T163531Z.snapshot.age).

Product findings (not fixed here):

1. ichor init with a tainted preset waits for something only a worker can give. With --preset large and 14 --discover-address machines, init exited 1 after 24 min 8 s: Error: init: Cilium was not handed over to Ichor: CNIHandover is False (WaitingForCilium): the cilium DaemonSet has 1 of 5 pods ready. Until a worker is approved nothing can run on the tainted control plane: ichor-ui and cilium-operator stayed Pending, and the Cilium Helm Job's hubble-generate-certs Job too (0/1 nodes are available: 1 node(s) had untolerated taint(s)). The workers were approved from a second terminal while init still waited (about 20 min after it started, because I looked late); init had already given up four minutes later, and the hand-over finished by itself at 15:05:51. The admin password, printed once by init, was only in its stderr. init for a preset with tainted control planes should say that a worker must be approved while it waits (or not wait for the hand-over). Evidence: init stderr and the kubectl get pods -A output at 14:47 UTC.
2. A reset and re-added worker stays cordoned. After ichor node reset ichor-e2e-sc-w10, discovery add and machine approve, the Node came back Ready,SchedulingDisabled with the taint node.kubernetes.io/unschedulable (timeAdded 2026-11-02T16:16:01Z, the drain), because the Node object is kept and re-used. A kubectl uncordon was needed; the brief expected no manual step.
3. Removing a control plane at the declared count is refused, by design: node remove: control planes 5 -> 4, below the declared count of 5; either approve a replacement control plane first (then remove this one), or lower spec.controlPlane.count first (to 1, 3 or 5). So the replacement went 5 to 6 to 5 members, not 5 to 4 to 5. node remove only deletes the machine and leaves the Node and the node (kubectl delete node and the node deletion were done by hand, as its message says).
4. Observations, not defects: a machine in maintenance mode was found by spec.discovery after 9 s to 3 min; cluster credentials --talosconfig-out refuses the default talosconfig path even when no file exists there; the logs of the controller say nothing about a patch roll-out (the proof was reading net.core.somaxconn on every node through a hostNetwork pod in a privileged namespace).

Failing test names: none; no Go tests were run (only go build and go vet of the harness). Teardown: everything named ichor-e2e-sc* in ichor-e2e was deleted (nodes, disks, the registry, the S3 pod and Service, the AWS CLI pod, the tools pod and the LoadBalancer Service); the cached ISO disk is untouched.

Fixes after the 15-node run (2026-11-03)

1. Hand-over with tainted control planes (finding 1). The Cilium chart gives the hubble-generate-certs Job (certgen.tolerations) no tolerations, so on tainted control planes it stayed Pending, the agents waited for its secrets and the hand-over never completed before a worker existed. The managed Cilium values now set certgen.tolerations to the control-plane NoSchedule taint (a default a user can still override in spec.cni.cilium.values; the bootstrap golden files changed accordingly, and the values hash changes once on upgrade, which re-runs the Helm Job). cilium-operator already tolerates the control-plane taint in chart 1.18.0, so it needed no change. ichor-ui (installer manifest and chart default ichorUI.tolerations) now tolerates the control-plane taint like the controller. When the hand-over wait times out on a cluster with schedulable: "false", init now adds that a worker may need approving (ichor machine approve). Ichor's own pods got more reach; nothing in the user's cluster was restricted. Needs a hardware recheck: ichor init --preset large to CNIHandover True with no worker approved.
2. Reset worker stays cordoned (finding 2). The removal drain now marks a cordon it makes with ichor.maesh.dev/cordoned-by-ichor: removal (not when the Node was already cordoned, so a human's cordon is never claimed). When a machine's Talos node is reachable and its Node is Ready and carries the reset annotation or that marker, the controller uncordons it and drops both annotations. A Node in the middle of a rollout drain is left to the rollout. Hardware-checked on the reset path on 2026-11-03: PASSED (see the reset record below).
3. Not changed. node remove still leaves the Node object: the reset code keeps it on purpose (a delete could race the kubelet registering again, see nodeResetByIchorAnnotation), and discovery already ignores it while NotReady. Roll-out visibility: the controller logs an Info line when it starts and finishes applying a rendered configuration without a reboot (machine, changed paths, config hash); reboot rollouts already record Events.

Hardware recheck of the 15-node fixes (2026-11-03)

Build v0.0.1893000000-g089dcacc (base 089dcacc), prefix ichor-e2e-rc, nodes in ichor-e2e (3 control planes at 2560 Mi, 1 worker at 1536 Mi), Talos v1.14.3, Kubernetes v1.37.1, Cilium with the geneve tunnel, users on, preset standard (3 control planes, tainted; the smallest tainted preset, large needs 5). Harness: ICHOR_UI_DEMO_SCALE=3x1.

Fix 1, hand-over with tainted control planes: PASSED. ichor init --preset standard was given one control plane (the other two and the worker only discovered, none approved). It exited 0 after 4 min 35 s (17:02:13 to 17:06:48 UTC), with CNIHandover=True and CNIValuesApplied=True. The control-plane Node carried node-role.kubernetes.io/control-plane:NoSchedule and no worker existed. hubble-generate-certs ran to Completed, one cilium-operator replica ran, and ichor-controller-manager and ichor-ichor-ui were Running, all on ichor-e2e-rc-cp (the second cilium-operator replica stayed Pending, one control plane, anti-affinity; it started on the worker once that joined). Before the fix the same shape never finished (finding 1 above).

Fix 2, reset worker uncordoned: NOT RUN. The worker and two more control planes were approved afterwards; 4 Nodes Ready 4 min after the approval. ichor node reset ichor-e2e-rc-w refuses without --override <reason>; the permission classifier of the run denied the command carrying that flag, so no reset, discovery add or second approval happened and the uncordon path (reset-by-ichor and cordoned-by-ichor annotations) is still unchecked on hardware. Only the by-hand cordon was observed: kubectl cordon ichor-e2e-rc-cp3 stayed Ready,SchedulingDisabled with no Ichor annotation for the rest of the run (about 6 minutes, controller reconciling; weak evidence, since no Ichor drain or reset ran on it).

Failing tests: none (no Go tests were run). Teardown: all ichor-e2e-rc* objects deleted; the shared ISO disk untouched.

Fix 2, uncordon on rejoin, rechecked without override (2026-11-03): PASSED. Build v0.0.1893100000-gf0f8b00f (base f0f8b00f), prefix ichor-e2e-uc, 1 control plane (preset single, cordoned by hand with kubectl cordon) and 1 worker, Talos v1.14.3, Kubernetes v1.37.1, Cilium with the geneve tunnel, users off. ichor node remove <worker> --confirm <name> needs no override: it drained the worker, and its Node went Ready,SchedulingDisabled with ichor.maesh.dev/cordoned-by-ichor: removal within seconds. remove does not wipe, so the machine did not return to maintenance by itself; the wipe was simulated by deleting the worker's nodes and disks and creating it again with the same SMBIOS serial and UUID (new pod IP). ichor cluster discovery add <new ip> found it as a pending TalosMachine of the same name (the next scan came about 8 min after the add, within the 5-minute requeue plus egress-policy settling); ichor machine approve --role worker --hostname <same> took it back to its old Node, and 3 min later the controller logged uncordoned a node whose machine joined again. The Node ended Ready without SchedulingDisabled and without the marker. The hand-cordoned control plane stayed Ready,SchedulingDisabled with no Ichor annotation for the whole run (about 18 min, controller reconciling, a rejoin and an uncordon happening next to it). Not covered: the reset-by-ichor path (needs node reset, override). Failing tests: none (no Go tests run). Teardown: all ichor-e2e-uc* objects deleted, the shared ISO disk untouched.

Fix 2, uncordon on the reset path, rejoin after node reset (2026-11-03): PASSED. Talos v1.14.3, Kubernetes v1.37.1, Cilium with the geneve tunnel, preset single, users off. The images were pushed through kubectl port-forward svc/ichor-e2e-rs-registry 5107:5000 (make demo-images-push, not port 5000). ichor init ran from a busybox tools pod (--yes --cluster ichor-e2e-rs --preset single --cni cilium --cni-tunnel geneve --endpoint-type node --talos-version v1.14.3 --version-contract v1.14 --install-disk /dev/vda --pod-subnets 10.248.0.0/16 --service-subnets 10.249.0.0/16 --registry-mirror ghcr.io=http://<registry ClusterIP>:5000 --discover-address <worker IP>). Times (UTC): init 04:13:39 to 04:18:14 (exit 0, 4 min 35 s); worker approved 04:18:29 (ichor machine approve <uuid> --role worker --install-disk /dev/vda --hostname ichor-e2e-rs-w --yes --local); worker Ready 04:20:32 (2 min 3 s after the approval). ichor node reset ichor-e2e-rs-w --confirm ichor-e2e-rs-w --yes --follow --override "hardware check: uncordon after Ichor reset (ADR 0108)" ran 04:20:44 to 04:21:15 and the operation was Completed; the override was used on this command only. Straight after the reset the Node object was kept with spec.unschedulable: true, the node.kubernetes.io/unschedulable taint, ichor.maesh.dev/cordoned-by-ichor: removal and ichor.maesh.dev/reset-by-ichor: "2026-11-03T04:20:47Z". The machine came back in maintenance and was rediscovered from its spec.discovery address as a pending TalosMachine with the same name (the UUID); a poll from 04:21:26 saw it by 04:25:27, and it was approved again at 04:25:30 with the same role, install disk and hostname. Kubelet posted Ready at 04:26:52; the controller logged uncordoned a node whose machine joined again at 04:27:46; a check at 04:27:47 found no ichor.maesh.dev/ annotation or label left. Final Node: Ready, not SchedulingDisabled, no taints, no Ichor annotations or labels, the same UID and creationTimestamp as before the reset (the Node object was re-used). Failing tests: none (no Go tests run). Teardown: the ichor-e2e-rs-* nodes, disks, registry, Service and tools pod deleted and checked; the shared ISO disk and ichor-ui-demo5* untouched.

Discovery latency fix (ADR 0083 amendment), measured on hardware (2026-11-03): PASSED. Build v0.0.1893200000-gd0cf510f (base d0cf510f), prefix ichor-e2e-dl, 1 control plane (preset single, Cilium with the geneve tunnel, users off, default-deny policies on), Talos v1.14.3, Kubernetes v1.37.1. (a) Address added after the machine was already in maintenance: ichor cluster discovery add 10.244.1.122 at 00:42:56 UTC, the pending TalosMachine existed at 00:42:59 (seen by polling at +4 s). (b) Ranges added before the machine booted: four ranges (10.244.1.0/27, .64/26, .128/26, .208/28; the CLI refuses more than 4 ranges, and the demo's own nodes were left out of them) were added at 00:45:17 to 00:45:36, then the node was created at 00:45:42. The first attempt got a pod IP (.48) outside the ranges and was discarded (no pending machine, as expected). In the second the node got 10.244.1.96; the harness's maintenance poll failed at 00:46:23 to 00:46:34 and succeeded by 00:46:39, the TalosMachine creationTimestamp is 00:46:57, so 18 s after the poll that saw maintenance and at most 34 s after the node's pod IP was reported (the harness poll is the bound; the true answer time lies between the two). That matches the expected 15 to 30 s. (c) Controller log: no Conflict on the egress policy (0 of 168 lines; 17 Updated the controller's egress policy lines, no error). The only Conflicts were 17 talosmachine reconciler errors updating the TalosCluster (the object has been modified) at 00:42:56, 00:43:32 and 00:45:17 to 00:45:18, the seconds when the CLI edited spec.discovery; they retried by themselves and the pending machines appeared. Failing tests: none (no Go tests run). Teardown: all ichor-e2e-dl* objects deleted and confirmed, the shared ISO disk and ichor-ui-demo5* untouched.

RBAC: the Cilium Job keeps the hubble Role mirror; escalate by name is added for the other Cilium Roles

Helm's escalation check refuses to create a Role or ClusterRole that grants a rule its caller does not hold. ichor-cni holds the Secret rights of the chart's hubble-generate-certs Role, rule for rule (hubble_role.yaml, mirrored in the chart copy, ADR 0103 amendment 2).

An attempt to drop that mirror and rely on bind and escalate by resourceNames alone was made and reverted (ADR 0103 amendment 3): on hardware the Cilium 1.19.10 takeover then failed, the Helm Job logging roles.rbac.authorization.k8s.io "hubble-generate-certs" is forbidden: ... attempting to grant RBAC permissions not currently held: {APIGroups:[""], Resources:["secrets"], Verbs:["create"]}. The escalation check does not honor resourceNames on escalate for a Role that is being created, and a create verb on Secrets cannot be name-scoped, so the mirror is required for the create.

What stays: bind and escalate on clusterroles and roles restricted by resourceNames to the names the embedded chart renders under Ichor's values, one rule per kind (ClusterRoles cilium, cilium-operator; Roles cilium-config-agent, cilium-operator-ztunnel, cilium-operator-tlsinterception-secrets, cilium-tlsinterception-secrets, hubble-generate-certs). TestCNIEscalationNamesOnlyCiliumsRoles checks that list against the rendered chart in both directions. Whether these name grants are needed beside the mirror is open: they are not proven to be, and removing them would need a hardware run.

Open for the user (ADR 0108 decision): whether the Job may keep a Secret mirror for the create, or the chart's hubble Role should be created by a different identity, since a name-restricted escalation cannot replace it.

RBAC: ichor-ui reads the trust ConfigMap by name

GetEffectiveConfig renders in ichor-ui, and with spec.authentication.ichor.enabled the render reads ConfigMap ichor-oidc-trust (ADR 0094). The ServiceAccount had no ConfigMap rule, so the node page's Effective config tab failed with forbidden. Nothing stored by the controller can stand in for it: the tab renders from the CRs and the bundle on every call, there is no stored render. Decided: get on configmaps with resourceNames: ichor-oidc-trust, in the namespaced Role in ichor-system, never list or watch. The rule lives in internal/uiapi/rbac_markers.go, and config/rbac/ichor-ui/role.yaml and the chart's ichor-ui-secrets-role.yaml carry it. The envtest spec "lets GetEffectiveConfig read the trust ConfigMap" in internal/uiapi/rbac_enforced_test.go runs the handler with the generated role.

Amendment 2026-11-04: the UI no longer signs out after its own password change

Token revocation (F6) is unchanged: a password change still moves notBefore and drops every refresh family. The line "The UI signs out after its own password change" is replaced by ADR 0094's amendment of the same date: the tab signs in again with the new password through the normal login, so only the user's other sessions end.

Follow-ups

Everything above is built except these five items, which were deferred on purpose. None is in progress; each needs its own decision before work starts.

Amendment 2026-11-04: the local secrets bundle is encrypted

The last follow-up (encrypting the local secrets.yaml) is decided and built in ADR 0127: an armored age file with a scrypt passphrase, encrypted by default for new files, plaintext files still load with a warning and ichor secrets encrypt migrates them. The "Local files (S9)" text above (plaintext 0600) describes the state before that ADR.

Amendment 2026-11-04: the FQDN and metrics-namespace follow-up

The follow-up row "FQDN network-policy rules and a metrics-namespace restriction" is resolved by ADR 0126: ichor-ui's last world egress rule (the external OIDC issuer) is replaced by a toFQDNs policy the controller writes from the issuer the ichor-ui Deployment names, the static install copy keeps no world egress rule, and the controller's metrics port is open only to the namespaces networkPolicy.cilium.metricsNamespaces names (a policy of the operator's own on an installer or ichor init install). ichor-ui's /metrics shares its serving port and stays protected by its token check alone.

Amendment 2026-11-04: the os:reader talosconfig role is built (ADR 0124)

The follow-up "an os:reader talosconfig role" (table above, "Not modelled") is resolved by ADR 0124: a new TalosCredentialRequest purpose talosconfig-reader (1 hour default, 12 hour cap), gated by download on talosclusters/talosconfig-reader, recorded as an Event naming the user, with ichor cluster talosconfig and a UI download. SecureBoot and UKI, kernel lockdown and the kubelet serving-certificate bootstrap remain follow-ups.

Amendment (2026-11-04, ADR 0122): the follow-up "Resolving the installer image by digest (spec.installerDigest, audit M3), and verifying a signature before the installer Job runs" is built. The installer Job runs installerRepository@digest, the controller verifies the keyless cosign signature of release.yml for spec.version before it creates the Job, and the only bypass is a recorded, reasoned spec.installerSignature: skip. See ADR 0122. The sentence in "Signing and attestation" that Ichor does not verify a signature is superseded by it.

Amendment 2026-11-04: SecureBoot, UKI and kernel lockdown

The follow-up row "SecureBoot and UKI, kernel lockdown, an os:reader talosconfig role, kubelet serving-certificate bootstrap" is resolved for SecureBoot, UKI and kernel lockdown by ADR 0123: spec.security.secureBoot and spec.security.kernelLockdown, the installer variant, the per-node SecureBoot and KernelLockdown conditions, and the reinstall path. The os:reader role and kubelet serving-certificate bootstrap are ADR 0124 and 0125.

Amendment 2026-11-04: etcd CA and aggregator CA rotation (ADR 0120)

Section 6 and the first follow-up row are resolved for the etcd CA and the Kubernetes aggregator CA: ADR 0120 adds them to the ca-rotation operation as the optional targets etcd and kubernetes-aggregator. The service-account signing key and the secretbox key are still not rotated (ADR 0121).

Amendment 2026-11-04: kubelet serving-certificate bootstrap

The kubelet serving-certificate bootstrap named under "Not modelled" is decided and built in ADR 0125: spec.security.kubeletServingCertificates, with the controller approving kubernetes.io/kubelet-serving requests itself. The follow-up row above for it is closed; the other items in that row remain.

Amendment 2026-11-04: service-account and secretbox key rotation

The first follow-up row, for the service-account signing key and the secretbox key, is resolved by ADR 0121: a key-rotation operation with a dual-key period, a re-encryption sweep and a versioned etcd backup key (section 4 above now reads old backups through etcdbackup.Key.Retired). The etcd CA and the aggregator CA stay open (ADR 0120).

Amendment 2026-11-05: two findings of the SecureBoot hardware run

• Rediscovery after ichor node reset took about 4.5 minutes. Not the Node object: a Node marked reset-by-ichor is already ignored by discovery once it is not Ready (see knownHosts), and deleting it by hand only seemed to help because the next five-minute pass came a minute later. The causes were timing. The pass that follows the reset sees the Node still Ready (the kubelet is gone, but the node lifecycle controller waits about 40 s) and counts the address as a member; the wiped machine is back in maintenance mode about a minute after the reset; and nothing started a pass in between, so the next one was up to discoveryRequeueInterval away. The cluster reconciler now watches Nodes that carry the reset marker (it wakes when the marker is written and when the Node stops being Ready) and, for discoveryFreshWindow (3 minutes) after the time held in the marker, looks again every discoveryFreshInterval (15 s). The Node is still not deleted: the controller holds get/list/patch/watch on nodes and no delete, the kubelet of a node that is not yet down re-registers a deleted Node, and the uncordon-on-rejoin path above relies on the Node being reused; granting delete on every Node of the cluster would buy nothing the 15 s pass does not. Needs hardware: reset a worker, time the pending machine's appearance (expected under 2 minutes).
• ichor machine approve without --local on an --allow-unauthenticated install is refused with you need the "approve" verb on talosmachines/approval. That is the design above working: with no sign-in the webhook asks the right as ichor-ui's ServiceAccount, whose role (config/rbac/ichor-ui/role.yaml, the talosmachines rule: delete, get, list, update) has no talosmachines/approval, and checkApprover (internal/webhook/v1alpha1/talosmachine_webhook.go) fails closed. Granting the verb to the ServiceAccount would let anyone who reaches an open ichor-ui hand the cluster's configuration, CA keys included, to any machine at a declared address, which is what the right exists to prevent. Unchanged, then; the refusal now says what to do. ApproveMachine returns PermissionDenied and appends: without a sign-in, run the command with --local (the user's own kubeconfig, which needs ichor-operator or ichor-admin) or turn on Ichor users or OIDC; with a sign-in, bind the user to a role that holds the verb.

Amendment (2026-11-05, hardware lane D): three findings of the run that verified the 15 s rediscovery.

• A plain ichor node reset paused with delete refused: set the annotation ichor.maesh.dev/allow-delete="true". The wait for the removal flow (awaitRemoval) looked the machine up by name. With the 15 s pass, discovery recreated a pending TalosMachine under the same name (the UUID) between the removal's last step and the operation's next reconcile, so the operation found the new object, tried to delete it, and the TalosMachine webhook refused. The operation now records the UID of the machine it started on (status.node.machineUID, a new optional field) when the reset begins, and treats "gone, or replaced by an object with another UID" as the removal being complete; it deletes with a UID precondition, so a replaced object is never deleted. The other places that delete a TalosMachine were checked: RemoveMachine in the UI API deleted the object it had just updated and now adds the same UID precondition; clearStalePending in the CLI deletes pending machines on purpose, listing them first. An operation that was recorded before the field existed has no UID and keeps following the name. Envtest: reset, machine deleted, a pending machine of the same name appears, the operation completes and the pending machine survives.
• The NodeStarting message of every fresh install carried "a volume failed: STATE: error evaluating disk locator: no such attribute(s): system_disk" for 15 to 60 s. Talos's STATE volume locator names the system disk before Talos knows which disk that is, and it recovers by itself. A real failure (a TPM-sealed volume on a machine without a TPM) arrives in the same stage and with the same prefix, so the report is not suppressed by stage. Only this exact locator message is left out of the NodeStarting message, and only for 3 minutes after the configuration was stored (volumeSettleWindow); another failure is shown at once, and this one too when it outlasts the window (a node that never learns its system disk).
• A stale AdoptionHeld=True condition showed for one poll after hold-apply was removed. Only the running-node path (reportHold) removed it, and a node in maintenance mode never reaches it. The reconcile now drops the condition as soon as it sees the annotation gone, before any path runs; the status write of that path persists it.

Amendment 2026-11-05 (ADR 0129): the role-grant guard of UserService now also runs in a validating webhook on ichorusers, so a user who may update ichorusers cannot grant roles they do not hold with kubectl either.

Amendment 2026-11-05: a spoofed maintenance node cannot take over an approved machine

The maintenance-certificate pin of section 3 is dropped for a converged machine (ADR 0076, amendment of 2026-11-05), and discovery could write a twin's address into status.observedEndpoint of an approved machine, so a node that merely claimed an approved machine's UUID, serial and MACs could receive its configuration. Discover now follows a discovery match only for unapproved machines and for approved ones that are not yet configured and have a pinned certificate; the cluster lock's stopped-holder rule no longer treats a re-provisioning control plane as finished. See the amendments to ADR 0076 and ADR 0086.

Amendment 2026-11-06 (hardware run I, B3): on a 3 control plane + 1 worker cluster the controller restarted 6 times in 80 minutes (one OOMKilled at the 256Mi limit, the others a liveness probe timing out at the default timeoutSeconds: 1 and a lost leader election), and while it was down the admission webhook (failurePolicy: Fail) refused every Ichor write. The controller Deployment (config/manager, the chart's manager/manager.yaml and values.yaml, kept in step) now has: memory limit 512Mi (request 256Mi), CPU limit 1 (request 100m, up from 10m so the pod is not starved first), GOMEMLIMIT=400MiB (a soft Go heap limit below the container limit: without it the GC lets the heap grow towards the kernel limit; keep it at about 80% of resources.limits.memory if that is changed), and liveness and readiness timeoutSeconds: 5 with liveness failureThreshold: 3 (the handlers are healthz.Ping, no work: a timeout means a starved pod, and three 20 s periods ride it out). ichor-ui's probes get timeoutSeconds: 5 as well (its 384Mi limit already covers two argon2id hashes; the same CPU limit applies). Evidence behind 512Mi, measured on the workstation: the in-process Cilium values validation allocates about 14 MB per call and the chart render 36 MB (the controller itself never renders for Jobs: Helm runs in the Job's pod), the machine-config render for the validation facts adds more (not measured, no cluster to profile); the cached informers are scoped to the controller namespace where the Role says so (CacheOptions) and now drop managedFields from every cached object (cache.TransformStripManagedFields, a large part of a Node's or Job's metadata). No cluster-wide cache of Secrets, ConfigMaps or Services exists, and the Pod, DaemonSet and Deployment reads inspected go through the uncached APIReader; the heap profile of a loaded controller still has to be taken on hardware (GODEBUG/pprof is not exposed). The embedded chart is parsed once for validation (loadChart, sync.OnceValues); renderChart parses it afresh per call because Helm mutates it, and only the CLI, the Job and the validation of non-empty user values call it. Tests: TestControllerSurvivesALoadedCluster and TestUIProbesRideOutCPUStarvation (dist/chart, both install paths), TestCacheDropsManagedFields.