0105. ichor-ui is TLS-only with an Ichor-made serving certificate and a CA the edge can verify, and Ichor's own runtime settings live in IchorInstallation, editable from a Settings tab

Status: Accepted; built (unit 10, the plaintext removal, done early by the user's decision on 2026-10-30); hardware-verified (T1 to T8) 2026-10-30 · Date: 2026-10-29 · Design section: Trust, Deployment, CLI and UI · Extends: 0059, 0094, 0100, 0104 · Touches: 0065, 0103 (reclaim)

Decided: ichor-ui never serves plaintext to the network: it listens for HTTPS only on a new port 8443, with the certificate the operator supplies (ichorUI.tls.secretName, for example a cert-manager Secret) or, when none is supplied, a leaf from the Ichor CA that the controller already keeps for the 8444 issuer listener (ichor-oidc-ca, ichor-oidc-tls), issued for every install and not only with users enabled; the controller publishes that CA, public part only, as a ConfigMap and a Secret named ichor-ui-ca so an Ingress-NGINX, a Gateway API BackendTLSPolicy or a client holding --ca-file can verify the backend, while a plain L4 load balancer needs the operator's own certificate; an edge re-encrypts to ichor-ui; the kube-apiserver's 8444 fetch is unchanged and keeps the Ichor leaf even when the operator supplies a certificate for 8443; an existing install keeps a deprecated plaintext listener on 8080 for exactly one minor release (never on a fresh init or Helm install) and then loses it; certificate hot reload is the work of ADR 0094's amendment (branch cert-reload) and is used here, not redesigned; Ichor's runtime settings are new optional fields of the existing singleton IchorInstallation (not a new kind), live-read by the controller, the webhooks and ichor-ui, edited from a Settings tab and by ichor settings, guarded by RBAC verbs checked in the webhook; a Helm install gets an IchorInstallation too, created once by the controller if absent, and a new field spec.deployment.manager (installer or helm) is how ichor upgrade ichor, the UI, the webhook and ADR 0104's ichorPresent tell the installer Job path from Helm

Context

ichor-ui is highly privileged: it holds a read-only Talos identity, impersonates users against the Kubernetes API, is Ichor's OIDC issuer (ADR 0094) and carries the override reclaim form (ADR 0103). Today:

• cmd/ichor-ui/main.go serves plain HTTP on 8080 unless --tls-cert-file/--tls-key-file are given; with them it calls srv.ListenAndServeTLS, which reads the pair once and never again. docs/guides/users-and-login.md ("Expose ichor-ui over https") tells operators to put TLS "at the edge", which leaves edge to pod in cleartext.
• The 8444 listener already has the machinery this decision needs: the IssuerTrustReconciler (internal/controller/issuer_trust.go) keeps a 10 year CA in ichor-oidc-ca and a one year leaf in ichor-oidc-tls (SANs: the Service ClusterIP and <service>.<ns>.svc), renews at two thirds, and publishes the public parts in ichor-oidc-trust (key ca.crt). But it is started only with --user-issuer (cmd/main.go), so an install without users has no CA.
• ichorUI.tls.secretName (chart) and --ui-tls-secret/--ui-tls-cert (ichor init) mount a Secret at /etc/ichor/tls; internal/installer (CustomizeUI, ReadUISettings) re-applies that on every upgrade (ADR 0059 amendment).
• Ichor's own settings are command-line flags on the controller (--reclaim, --no-cilium-policies, --user-issuer) and on ichor-ui (--token-ttl, --refresh-lifetime, --external-url). Changing one means editing a Deployment, which ichor init and the installer own, and which a Helm release owns on a Helm install. There is no Settings page.
• IchorInstallation (api/v1alpha1/ichorinstallation_types.go) is the cluster-scoped singleton named ichor with spec.version and spec.installerRepository; ichor init creates it. ADR 0104 decision 4.3 says a Helm-installed Ichor has none and that ichorPresent (IchorInstallation, or a deployed Helm release of the chart, or the controller Deployment) must tolerate that.

Library facts checked in the module cache at the go.mod version (controller-runtime v0.24.1): pkg/certwatcher.CertWatcher offers New, WithWatchInterval, RegisterCallback, GetCertificate, Start; webhook.Server already builds a certwatcher and sets GetCertificate whenever TLSOpts does not set one (pkg/webhook/server.go), so the webhook already follows its certificate. The metrics server and the 8444 listener read the key pair on every handshake (lazyKeyPair, internalServer); only ichor-ui's main listener and ListenAndServeTLS do not follow rotation.

Decisions

Part A. TLS to ichor-ui

1. ichor-ui has no plaintext mode. One listener serves the SPA, the RPC API, /metrics and the probes, on port 8443, HTTPS only. The listener starts even when no certificate exists yet and answers every handshake with an error until one does (the readiness probe, an HTTPS httpGet, keeps the pod out of the Service); it never falls back to HTTP. The flags --tls-cert-file and --tls-key-file stay and now mean "the operator's pair". New flags: --tls-dir (default /etc/ichor/ui-tls, the managed pair). --addr defaults to :8443. Rejected: keeping a --insecure escape hatch for development (a flag that disables the control is the first thing a copy-pasted manifest carries; development uses the managed certificate and --ca-file); same port 8080 serving TLS (every probe, Service, NetworkPolicy, port-forward and bookmark would silently change meaning, and a client that still speaks HTTP gets a confusing TLS error; a new port makes both versions coexist, decision 6); mTLS between edge and backend (the edge proxies for browsers and cannot present a client identity ichor-ui could map to anything; the user is authenticated by the OIDC token inside the request).

2. Which certificate.

The Ichor leaf gets more SANs than today: the Service ClusterIP, <svc>.<ns>.svc, <svc>.<ns>.svc.cluster.local, <svc>, localhost, 127.0.0.1, and the host of --external-url when users are enabled (so a client given the CA verifies a direct connection through an L4 load balancer; a browser will not trust it, which is the operator-certificate row's reason to exist). The Secret and CA names stay ichor-oidc-* (renaming a Secret that an installed apiserver trust chain and the installer's resourceNames reference buys nothing); the docs call the pair "the Ichor serving CA". The IssuerTrustReconciler splits: the TLS half (CA, leaf, trust ConfigMap, ichor-ui-ca) runs for every install; the signing-key and user-credential halves stay behind --user-issuer. --user-issuer remains a flag with its meaning narrowed to those halves. The pod mounts ichor-oidc-tls with optional: true so ichor-ui starts before the controller has issued the first leaf; whether the kubelet then populates the volume in place, and how long it takes, is a hardware test (T3) because the reload latency of every Secret-backed certificate rests on it. Rejected: a second CA and leaf for the edge-facing listener (two trust anchors and two renewals for one ClusterIP; the leaf may serve both listeners because the 8444 CA is already pinned only for the apiserver's issuer fetch and trusts nothing else); cert-manager as a dependency (ADR 0006 and 0101: Ichor ships no certificate dependency; cert-manager stays an operator choice through secretName); a self-signed certificate generated in the pod at start (each replica would differ and nothing could publish a CA for the edge).

3. How the edge verifies the backend. The controller publishes the CA certificate only (never a key) as the ConfigMap ichor-ui-ca and the Secret ichor-ui-ca, both with key ca.crt, in ichor-system, updated on CA change. Service port 8443 is named https. The guide gets one recipe each; exact annotations and field names are looked up in the controllers' own documentation when the guide is written (work unit 9), not taken from memory:

• Ingress-NGINX: backend protocol HTTPS, proxy-ssl-verify on, proxy-ssl-secret: ichor-system/ichor-ui-ca, proxy-ssl-name/server name ichor-ui-service.ichor-system.svc. The Ingress lives in ichor-system, because it must reference the Service.
• Gateway API BackendTLSPolicy: targetRefs the ichor-ui Service, validation.caCertificateRefs the ConfigMap ichor-ui-ca, validation.hostname ichor-ui-service.ichor-system.svc. Needs a Gateway API release that serves BackendTLSPolicy at the cluster's version; the guide states which and falls back to the Ingress recipe's equivalent for the implementation in use.
• Plain L4 load balancer (Service type LoadBalancer, or TCP passthrough): the load balancer terminates nothing, so clients see ichor-ui's certificate and the host name they typed. Use the operator-certificate row. With the Ichor leaf only clients given ichor-ui-ca (ichor --ca-file) verify.
• An L7 cloud load balancer that cannot verify its backend is out of scope; the guide says "use a controller that can".

4. The 8444 fetch. Unchanged in shape: the kube-apiserver reads https://<ClusterIP>:8444/.well-known/openid-configuration and verifies it with ichor-oidc-trust's CA (ADR 0094). What changes: the listener and 8443 share the reload helper of the cert-reload amendment, so a renewed leaf is served by both; the NetworkPolicy of ADR 0100 gains ingress on 8443 (the rule that listed 8080) and keeps 8444 for kube-apiserver. Port 8444 is never routed by an edge (the guide already says so; the policy enforces it for Cilium installs).

5. Hot reload. Not designed here. ADR 0094's amendment on branch cert-reload makes ichor-ui's listeners, the webhook and the metrics server follow their certificate files without a restart. This ADR depends on it for the main 8443 listener and for the cert-manager case (secretName Secrets are mounted, so a cert-manager renewal reaches the pod through the kubelet's Secret volume sync and then the reload). Work unit 2 consumes it; the tests for the reload itself belong to that branch. If it does not land first, unit 2 uses lazyKeyPair (per-handshake read) as the stand-in, which is correct and merely unobserved.

6. Migration without downtime. Version N (this change) serves 8443 TLS and, only where the install already served plaintext, 8080 plaintext marked deprecated. Version N+1 deletes the plaintext listener.

The installer derives the case from the live Deployment as it already does and never needs a new state store (ADR 0059 amendment). The skew rule (the CLI refuses writes more than one minor away) bounds the window to one minor in practice; the N+1 installer refuses to run on an install whose live ichor-ui still has --http-addr set without printing the same advice. During the N rollout the old pods serve 8080 and the new pods both; the Service has port 8443 only on new endpoints (readiness keeps old pods out of it), and the edge is re-pointed at the operator's pace. ichor ui (port-forward) and the CLI use 8443 and https://localhost:<port> with the CA fetched from ichor-ui-ca (the kubeconfig already grants the read), falling back to 8080 against a pre-N server.

Part B. Settings

7. Settings live in IchorInstallation, as new optional fields; there is no new kind. The singleton already exists on every ichor init install, is already guarded by a CEL name rule and the webhook, is already read by the CLI, UI and installer, and is already the place "how Ichor itself is installed" is recorded. Added to spec (all optional; none removed or renamed; defaults keep today's behaviour):

spec.deployment.manager      installer | helm      (omitted = installer)
spec.reclaim.enabled         *bool                  (override, ADR 0103)
spec.users.tokenTTL          duration, 1m..1h       (ADR 0094)
spec.users.refreshLifetime   duration, 1h..720h
spec.defaults.driftPolicy    enforce | warnOnly     (a new TalosCluster's default)

Initial scope is what is read cheaply at run time and has no Deployment shape: reclaim.enabled (the controller and the reclaim webhook read the setting per call instead of a startup bool), the two user lifetimes (ichor-ui reads them per issued token from its cache), and the default drift policy (the existing defaulting webhook fills TalosCluster.spec.driftPolicy at create; an existing cluster is never touched, so the controller still never mutates a spec). Further settings are added as optional fields in later ADRs; each must be live-readable.

Precedence: a set field wins; an unset field falls back to the flag or chart value, so a Helm install without any spec.* field behaves exactly as before. status.effective (written by the controller; ichor-ui reports its own two values over the Settings RPC) shows each value and its source (spec or deployment), and the condition SettingsApplied shows the controller has seen the generation. A deployment that wants settings owned by Git or Helm sets the controller flag --settings-read-only (chart settings.readOnly): the webhook then denies writes to the settings fields, every consumer ignores them, the Settings tab is read-only with a "managed by the deployment" Alert. No live-reload gap is accepted: a change takes effect on the next call (ichor-ui and webhook within the informer's delay); nothing in settings needs a rollout. Deployment-shaped values (images, replicas, resources, the TLS Secret name, externalURL, the Cilium-policy opt-out, which are applied by objects Ichor does not render at run time) stay with the installer and Helm and appear on the tab as read-only facts (TLS: source, subject, expiry; Ichor version; manager).

Rejected: a new cluster-scoped IchorSettings kind. It would give settings their own RBAC and their own lifecycle, but: every consumer, the CLI, the installer (CRD list in resourceNames, TestInstallerRBAC), the chart and the snapshot would learn a second singleton whose existence must be handled in the absent case on top of the first; the Helm question (who creates it) is the same question and is answered once for IchorInstallation below; and the one real benefit, splitting edit rights, is available on one object with the webhook pattern ADR 0094 already uses (decision 9). Rejected: a ConfigMap (no schema, no CEL, no printer columns, no typed UI). Rejected: flags only, with the UI patching Deployments (the UI would write objects it does not own and the installer or Helm would revert the edit). Rejected: putting settings on TalosCluster (they are installation-wide, not per cluster; the default drift policy is the one per-cluster fact and is a default there).

8. A Helm install gets an IchorInstallation, and spec.deployment.manager says who deploys. The chart does not render it: a rendered object would be overwritten by helm upgrade, reverted by a GitOps tool tracking the release, and would invite the installer to fight Helm (the reason ADR 0104 4.3 skipped it). Instead the controller, on start and only when the singleton does not exist, creates it once with spec.version = its own version and spec.deployment.manager taken from a new flag --deployed-by (helm rendered by the chart; installer is the default and what ichor init writes itself). It never updates spec afterwards. This is a narrow, stated exception to "the controller never mutates spec": creating a missing singleton with computed defaults changes no user intent, and nothing is lost if it fails (every consumer tolerates absence, as 0104 required). The alternative that keeps the rule whole, the first save in the Settings tab creating the object, was rejected: the version and manager would then be unknown until someone saved, and ichor upgrade ichor could not tell Helm from the installer on a cluster nobody has opened the tab on.

For manager: helm: spec.version records the version at creation and is not compared with the running controller (UpToDate is True with reason ManagedByHelm, the UI's "Update available" badge and OperationService.GetIchorInstallation's update_available are suppressed); the validating webhook denies a TalosOperation of kind ichor (message: "this Ichor is installed by Helm; run helm upgrade"), the executor and the installer Job refuse as a second line, and the chart's installer.enabled stays false. ichor upgrade ichor reads the field first and prints the Helm command. The Settings tab shows "Installed by Helm" and still edits runtime settings, which Helm does not render.

ADR 0104 unit 5, as changed. ichorPresent is unchanged in its three signals (IchorInstallation, deployed Helm release of the chart, controller Deployment) because the first two windows remain: before the controller seeds, and on installs older than this ADR whose controller has not yet been upgraded. What changes: (a) when the singleton exists, spec.deployment.manager is authoritative over the other two; (b) adopt skips the install step on manager: helm, creates no IchorInstallation itself (the controller does) and, when the object is still absent, does not wait for it; (c) installer-Job commands refuse on manager: helm and, while the object is absent, on a Helm release; (d) the line "no IchorInstallation... fixes that code rather than having the chart create the object" becomes "...tolerates an absent object in the seed window; the chart still never renders it". Unit 5's audit list gains GetIchorInstallation, the CLI skew check (status.installedVersion is still written by the controller) and the PR-mode editor of ADR 0059 amendment 4 (it edits only spec.version in a tracked file, so a tracked manager: helm object is simply refused there).

9. RBAC, validation, audit. Reading is part of ichor-viewer (the spec holds no secret and the page shows no secret). Writing ichorinstallations stays ichor-admin's; the finer split uses the webhook pattern of ADR 0094 (a SubjectAccessReview on the request's user): changing spec.version or spec.deployment needs the verb update on ichorinstallations/version, changing spec.reclaim needs update on ichorinstallations/reclaim, and everything else under spec.users/spec.defaults needs update on ichorinstallations/settings; the built-in ichor-admin role holds all three on separate rule lines so a custom role can hold fewer. spec.reclaim.enabled: true additionally requires the typed confirmation (the UI's dialog; the CLI's --confirm-reclaim) that the API enforces by an annotation the request must carry (ichor.maesh.dev/confirm-reclaim: enable), so a stray kubectl apply cannot turn override on by accident. Validation is CEL on the CRD (duration bounds, tokenTTL <= refreshLifetime, the enums, the name rule that exists) plus the webhook for the SAR checks and --settings-read-only. Audit: the Kubernetes audit log names the user (ichor-ui impersonates, ADR 0058), and the webhook emits an Event on the IchorInstallation per accepted change listing the changed field paths, old and new values (none is secret by construction) and the user; enabling reclaim also emits the Warning event ADR 0103 already defines for its use.

10. UI and CLI. A "Settings" entry in the sidebar, a page of stock preactcn components: Tabs (General, Sign-in, Override, Defaults), Card with CardHeader/Title/Description, Field, Input, Select, Alert (managed by Helm or the deployment, plaintext still on), Badge (source: spec or deployment), AlertDialog (reclaim confirmation), Button. Switch is not in ui/src/components/ui yet; it is added with npx shadcn@latest add @preactcn/switch, never by hand. English strings in ui/src/lib/i18n.ts. A new SettingsService (Get, Update) in the UI API; Update is a merge patch of the settings fields only, as the user. CLI: ichor settings get and ichor settings set KEY=VALUE... write the same CR.

Work units

Tests

Unit and envtest: listener answers no plaintext on 8443 and fails handshakes (not the process) with no certificate; selection table of decision 2; SAN set; ichor-ui-ca carries no key; installer migration cases from fixture Deployments; settings webhook (SAR matrix, read-only flag, CEL bounds, confirm annotation); seed creates once and never updates; manager: helm denies kind: ichor; precedence set versus unset.

Hardware and e2e (batched per the hardware rule): T1 fresh ichor init serves HTTPS only on 8443 and nothing on 8080; T2 Ingress-NGINX re-encrypting with proxy-ssl-verify against ichor-ui-ca, then a BackendTLSPolicy run, then an L4 passthrough with an operator certificate; T3 an optional ichor-oidc-tls volume created after the pod starts is populated in place, and the time to first good handshake; T4 a cert-manager renewal reaches a running ichor-ui with no restart (with the cert-reload branch); T5 the apiserver keeps fetching 8444 across a leaf renewal and an operator certificate on 8443; T6 an upgrade from the last release with plaintext 8080: no failed request through a looping client during the rollout, edge moved to 8443, then N+1 removes 8080; T7 a helm install gets a seeded IchorInstallation with manager: helm, ichor upgrade ichor refuses, helm upgrade works, a Settings edit survives it; T8 reclaim toggled from the tab takes effect with no restart and leaves an Event, refused for a role without the verb.

Risks

• A breaking edge: an operator who upgrades to N+1 without moving the edge loses the UI. Mitigated by the visible deprecation, the one-minor window and the N+1 installer's refusal.
• Readiness depends on the controller having issued the leaf; a controller outage at first install leaves ichor-ui unready (it was ready with plaintext). ichor init already bounds its waits and names the reason (ADR 0104).
• A seeded object is created by a component that otherwise only reads spec; a bug there would hide behind "absent is fine". Create-only, covered by test.
• Anything that reads settings live widens what a mistaken edit can change at once; the three RBAC lines, the confirm annotation and the Event are the compensation, and reclaim stays default off.
• Browser trust: the managed leaf is not browser-trusted. This is intended (the edge presents the public certificate) but a user on L4 passthrough must read the guide.

Open questions

1. Rotation of the Ichor CA itself with an overlap period (publishing two CAs in ichor-ui-ca and the apiserver's certificateAuthority); today the CA is replaced when near its end and everything re-trusts at once. Not needed for this change, needed before the 10 year mark.
2. Whether the ichor-ui login settings (--oidc-*, the external IdP) should become runtime settings too; they need a provider re-discovery path and are left to a later ADR.

As built (2026-10-29)

Units 1 to 9 are built; the hardware tests T1 to T8 are not run. Differences from the text above:

• Settings page: stock Cards, not Tabs (General, Sign-in, Override, Defaults became cards: Deployment, Override reclaim, Sign-in, Defaults). Switch was added with npx shadcn@latest add @preactcn/switch.
• No "unset": neither the UI nor ichor settings can clear a field once set; a field set stays set. Precedence is as designed for fields never set.
• defaults.driftPolicy is warnOnly | autoCorrect (the enum of TalosCluster.spec.driftPolicy), not enforce | warnOnly.
• Plaintext on 8080: the installer pins any live --addr that is not 8443 (not only a TLS-on-8080 install). The chart has no automatic detection; the opt-in value is ichorUI.plaintextPort (default 0), and ichorUI.port keeps a former 8080 TLS install in place.
• CLI trust: ichor ui and the API client stay on 8080 while ichor-ui-ca is missing.
• Helm denial: the denial of kind: ichor on manager: helm lives in the TalosOperation validating webhook; the installer refuses by reading the live Deployment, as a second line.
• Drift default: the CRD default of TalosCluster.spec.driftPolicy was removed in favour of a create-time defaulting webhook (failurePolicy: Ignore; an empty value is read as warnOnly, so a down webhook changes nothing). The installer's RBAC gained mutatingwebhookconfigurations. The CRD field itself is kept (CRD fields are append-only).
• Token lifetime: users.tokenTTL is capped at 1h (MaxTokenTTL), as the CEL rule says.
• Monitoring: the ServiceMonitor verifies the HTTPS port against ichor-ui-ca (ichorUI.tls.caConfigMap/caSecret/caKey) and never skips verification.
• Reload: certificates reload on change through certwatcher (ADR 0094 amendment, built).
• Docs: the Ingress-NGINX recipe uses backend-protocol, proxy-ssl-verify, proxy-ssl-secret (namespace/name), proxy-ssl-name and proxy-ssl-server-name; the reference describes proxy-ssl-secret as holding a client pair that "should also contain" ca.crt, so whether a CA-only Secret is accepted is part of T2. BackendTLSPolicy is gateway.networking.k8s.io/v1, Standard channel since Gateway API v1.4, same namespace as the Service, ca.crt ConfigMap as Core support. The guide for settings is docs/guides/settings.md.

Hardware results (2026-10-30)

• Passed: T1 (HTTPS only, leaf, ichor-ui-ca); certificate hot reload in about 10 s; T3 (the optional volume is populated in place after about 100 s; liveness is now a TCP probe, so no restart loop); T6 migration from plaintext to 8443, with 8080 deprecated, no traffic loss; T8 (settings RBAC, read-only mode, confirm-reclaim, fixed to require a fresh confirmation).
• A demo upgrade across the migration pinned 8080; fixed in 67aa97c4 (policy-name bug).
• T2 and T4: see "T2 and T4" below.

T7 and T5 (2026-10-30, hardware)

Build v0.0.1761800000-ga163d822, Talos v1.14.3, Kubernetes v1.37.1, Cilium 1.18.0, geneve.

• T7 passed (TestT7HelmManager in test/e2e/foreign/t7_test.go, 251 s, prefix ichor-e2e-fq, a foreign cilium-helm cluster, packaged chart from make helm-package, images from the in-cluster registry through the ghcr.io mirror). helm install --wait seeded IchorInstallation ichor with spec.deployment.manager: helm and spec.version equal to the running version. ichor upgrade ichor v0.0.1-g0000000 --local exited 1 with: ichor is installed by Helm (release "ichor", namespace ichor-system); upgrade it with `helm upgrade ichor oci://ghcr.io/maeshinshin/charts/ichor --version VERSION --namespace ichor-system`, not with the installer and created no TalosOperation. ichor settings set users.tokenTTL=10m stored spec.users.tokenTTL: 10m0s; after helm upgrade --reuse-values --set manager.replicas=1 --wait the value and manager: helm were unchanged. Notes: ichor settings has no --local or -n flag (it fell back on its own from the tools pod); the API normalises 10m to 10m0s. One earlier run of the test failed on my own test bugs (a -n/--local flag the settings command does not take, 10m vs 10m0s, a --no-headers stderr line), no product failure.
• T5 passed (prefix ichor-e2e-v, one Ichor-made control plane, ichor init --cni cilium --cni-tunnel geneve --ui-url https://<cp ip> --admin-user admin, then ichor config apply, an inner NodePort for 8443 and socat relay pods for the workstation). Non-interactive sign-in by POST /oidc/authorize (Accept: application/json) then /oidc/login then /oidc/token. Findings: (1) the browser client ichor-ui never receives the Kubernetes audience (by design, audienceFor), so the token for kubectl comes from the same flow with client_id=ichor-cli and a loopback redirect_uri (http://127.0.0.1:<port>/callback); the ichor-ui token is rejected by the kube-apiserver (expected audience "ichor-kube" got ["ichor-ui"]). (2) The first admin must change the generated password first: /oidc/token for ichor-kube answers 403 access_denied "change your password first"; the change was made with the ichor-ui UserService/ChangeOwnPassword call using an ichor-ui token. (3) With kubectl --token against a kubeconfig that also holds a client certificate, the certificate wins (the user showed as the cluster admin); the checks used a kubeconfig-less call with only the CA and --token. Results with a fresh token each time: baseline auth whoami = ichor:admin, groups ichor:admin, get nodes works; after (a) deleting ichor-oidc-tls (the controller reissued the leaf within seconds, serial 037E2E2E... to 6C0A1239..., 8443 served the new one) the same calls succeeded; after (b) pointing 8443 at an operator certificate (Secret operator-ui-tls, the ui-tls volume of the ichor-ui Deployment, rollout; 8443 then served CN=operator-ui while ichor-oidc-tls kept its Ichor leaf) the same calls succeeded. The kube-apiserver log shows no authentication error after the two changes. Limit of the evidence: go-oidc caches the signing keys, so these runs show that validation keeps working, not that a fresh key fetch from 8444 happened during the window.

T2 and T4 (2026-10-30, hardware)

Build v0.0.1793312776-gbe6d9bf8, Talos v1.14.3, Kubernetes v1.37.1, Cilium 1.18.0, geneve. Prefix ichor-e2e-w, one Ichor-made control plane and one worker (ichor init --cni cilium --cni-tunnel geneve --ui-url https://<cp ip> --admin-user admin, config apply, machine approve). The Ingress, Gateway and cert-manager installs went into this inner cluster only; the nodes are deleted and nothing of the ichor-e2e-w prefix is left. Requests went from a tools pod to the NodePort of the ingress or gateway Service, with the Host header or --resolve.

• T2 Ingress-NGINX passed (chart 4.15.1, controller 1.15.1, NodePort). The guide's Ingress with backend-protocol: HTTPS, proxy-ssl-verify: on, proxy-ssl-secret: ichor-system/ichor-ui-ca, proxy-ssl-name and proxy-ssl-server-name: on returned 200 for /readyz and the UI page. A CA-only Secret is accepted: ichor-ui-ca has only ca.crt; nginx.conf showed proxy_ssl_trusted_certificate .../ca-ichor-system-ichor-ui-ca.pem and proxy_ssl_verify on. Verification is real: proxy-ssl-secret pointing at a Secret with another CA gave 502 (upstream SSL certificate verify error: (21:unable to verify the first certificate)), proxy-ssl-name: wrong.example.com gave 502 (upstream SSL certificate does not match "wrong.example.com"), no proxy-ssl-name and proxy-ssl-server-name: off gave 502 (the name checked is upstream_balancer), and proxy-ssl-verify: off with the wrong CA gave 200 (the control). Ingress-NGINX needs up to half a minute to reload: a first run read stale results (200 for a wrong name after 15 s, 502 right after a restore); the repeat with 20 s of settle time was consistent.
• Finding, fixed in the guide: an ichor init install names the Service ichor-ichor-ui-service (the leaf has ichor-ichor-ui-service[.ichor-system.svc[.cluster.local]]), not ichor-ui-service as the guide said for init installs. The guide now says so; proxy-ssl-name is stated as required.
• T2 Gateway API passed with a caveat (Envoy Gateway v1.9.2 from oci://docker.io/envoyproxy/gateway-helm; its chart installs the Gateway API v1.6.1 CRDs in the experimental channel, not standard; BackendTLSPolicy is served as v1, with v1alpha3 also served). The guide's BackendTLSPolicy (ConfigMap ichor-ui-ca, hostname the Service name) was Accepted=True and ResolvedRefs=True; through the Envoy Service the route returned 200 and the UI page. With the policy pointing at a ConfigMap with another CA, or hostname: wrong.example.com, the route returned 503 (remote connection failure); with the policy deleted, Envoy spoke plain HTTP to 8443 and ichor-ui answered 400. Envoy applied changes in 15 s to 1 min; one early run showed 200 for the wrong CA after 30 s and 503 consistently after a longer wait. Not confirmed: after the final change back from the wrong CA to ichor-ui-ca the request was still 503 at 45 s and I deleted the objects before reading it again (200 had been seen before the break and after re-creating the policy). The Gateway showed Programmed=False only because the LoadBalancer Service had no address; its NodePort worked. A NodePort reached through the control plane's address failed while the Envoy pod ran on the worker (000), and through the worker's address worked; this is a property of the harness network, not of Ichor. L4 passthrough was covered by T1 and skipped.
• T4 passed (cert-manager v1.21.3, self-signed Issuer, Certificate ichor-ui-cert, duration: 1h, renewBefore: 59m, ECDSA P-256, rotation Always, so a renewal about every minute). The operator certificate was set by patching the ichor-ui Deployment as internal/installer CustomizeUI does (volume tls from Secret ichor-ui-cert at /etc/ichor/tls, --tls-cert-file and --tls-key-file); ichor init --ui-tls-secret and ichorUI.tls.secretName do the same at install time, and ichor upgrade has no flag for it. Over 3 minutes of sampling, two of three renewals reached the wire with the pod uid unchanged and restartCount 0: the Secret serial changed at 23:03:47, 23:04:47 and 23:05:47 and the served serial followed at 23:04:12 (25 s later), 23:05:33 (46 s) ; the third (Secret at 23:05:47) had not reached the wire when the run ended, 2 s later. Most of the delay is the kubelet copying the Secret into the pod (the sampling loop itself took a few seconds per round); the 10 s reload is measured from the file change, as in the earlier certificate test.
• No failing tests (no Go tests were run for T2 and T4; they were run by hand). Own mistakes: the first Gateway manifest used listeners[].hostnames (the field is hostname), and two early curl loops lacked a timeout.

Amendment 2026-10-30: Unit 10 done early by the user's decision (2026-10-30)

The user decided to remove the transitional plaintext listener now instead of in the next minor release. As built:

• ichor-ui: the plaintext server is gone. --http-addr stays as a deprecated no-op for one release (flags are an interface): a hand-edited Deployment that still passes it starts and logs --http-addr=... ignored: plaintext was removed (ADR 0105). Delete the flag in the release after.
• Installer: ReadUISettings no longer produces a plaintext listener. A plaintext install (live --addr other than 8443, or a leftover --http-addr) is moved to HTTPS on 8443; the Service has no http port and the policy opens only 8443. UISettings.WasPlaintext records the old address and drives the notice "ichor-ui no longer serves plain HTTP; point your edge at https://:8443 (trust ichor-ui-ca)". A TLS install on 8080 is still pinned to 8080 (and its policy port opened). The installer no longer strips or re-adds --http-addr; it deletes it from the live args.
• Chart: ichorUI.plaintextPort is removed. An old values file that sets it is ignored by Helm and renders no plaintext port (tested).
• The migration table row for plain HTTP above now reads: moves to 8443 TLS, 8080 is closed. The Settings/Overview Alert about plain HTTP was never built.
• No T6 test exists under test/e2e; the hardware T6 result above (8080 kept, deprecated) describes the pre-amendment behaviour. After this change, a plaintext install must show no 8080 after the upgrade; run on hardware 2026-10-30: the listener and the policy are right but an ichor init-made Service keeps its http port (see "Hardware result 2026-10-30" below).

Hardware result 2026-10-30: unit 10 upgrade of a plaintext install (partly failed) and the T2 Gateway leftover (passed)

Cluster: ichor init 1 control plane + 1 worker (prefix ichor-e2e-y, Talos v1.14.3, Cilium geneve, images from an in-cluster registry through the ghcr.io mirror). Old build v0.0.1840000000-g2b67caf6 (commit eedfb9f0^, built in its own git worktree, images and CLI from there; ichor init with that CLI, no --ui-url, users off), new build v0.0.1840000001-g0f09c67a (HEAD, ichor upgrade ichor v0.0.1840000001-g0f09c67a --local with the HEAD CLI). No Go tests were run; everything below was done by hand.

Unit 10, plaintext install upgraded: FAILED on one expectation

• Before. Service ichor-ichor-ui-service had only 8080/TCP, ichor-ui ran --addr=:8080, the CiliumNetworkPolicy ichor-ui opened 8080 and 8444. A kubectl port-forward to the Service port 8080 returned 200 for /readyz and 200 text/html for /.
• Upgrade. The operation ichor-9vpkf reported Completed after about 3 min; the installer Job printed NOTICE: ichor-ui no longer serves plain HTTP (it did on :8080); point your edge at https://<ichor-ui>:8443 (trust ichor-ui-ca) (docs/adr/0105). The new ichor-ui pod stayed 0/1 for about 2.5 min with certificate not available yet: open /etc/ichor/ui-tls/tls.crt and the readiness failure remote error: tls: internal error, until the controller (restarted by the upgrade) issued ichor-oidc-ca, ichor-oidc-tls, ichor-ui-ca and ichor-oidc-trust (issued at 00:16:00 UTC, the Job had started at 00:13:03; the new controller pod had been running since about 00:13:30 but its issuer only logged "Issuer serving certificate issued" at 00:16:00, so the 2.5 minute gap is the controller's, which I did not investigate). It then became Ready on its own; the Job ended Complete at 3 min 18 s.
• Passed. ichor-ui runs --addr=:8443 with the container port https/8443 and an HTTPS readiness probe. The CiliumNetworkPolicy ichor-ui opens 8443 (world, cluster) and 8444 (kube-apiserver, remote-node) and no longer 8080. From a pod in the inner cluster: curl https://ichor-ichor-ui-service.ichor-system.svc:8443/readyz --cacert <ichor-ui-ca> gave 200; without the CA, curl failed with SSL certificate problem: unable to get local issuer certificate; plain http://...:8443/readyz gave 400; http://...:8080/readyz timed out (000 after 5 s). ichor-ui-ca exists as a Secret and as a ConfigMap and the leaf verifies against it.
• Failed: the Service still has the http/8080 port. After the upgrade kubectl get svc ichor-ichor-ui-service shows 8080/TCP,8443/TCP (http 8080 -> 8080 and https 8443 -> 8443) and the Endpoints list :8080 next to :8443, although nothing listens on 8080. Cause, from metadata.managedFields: ichor init created the Service with manager ichor, operation Update (a plain create, not server-side apply), and that manager owns k:{"port":8080,"protocol":"TCP"}. The installer applies as field manager ichor-installer with server-side apply; it correctly sets only the 8443 port for itself but cannot remove a list item that another manager owns, so the 8080 entry stays. Every install made by ichor init before the plaintext removal is affected; an upgrade that started from an installer-made Service is not (not tested). This contradicts "the Service has no http port" in the unit 10 amendment above. Not fixed here (product bug). A fix needs the installer to take over the stale field (apply with the old manager's ownership dropped, or delete the http port in a migration step) and a test that starts from an ichor init-made Service.
• Remaining detail: docs/guides/users-and-login.md names the Service ichor-ui-service in the BackendTLSPolicy recipe, while an ichor init install names it ichor-ichor-ui-service. I used the real name and the real Service DNS name in the policy hostname (the leaf has both as SANs).

T2, Gateway API leftover: passed, with a better explanation of the earlier flip

Envoy Gateway v1.9.2 (oci://docker.io/envoyproxy/gateway-helm) installed only in the inner cluster; Gateway (HTTP :80, listeners[].hostname), HTTPRoute to ichor-ichor-ui-service:8443, and the guide's BackendTLSPolicy (ConfigMap ichor-ui-ca, hostname: ichor-ichor-ui-service.ichor-system.svc). Every request had -m 4 and went to the Envoy Service from a pod in the cluster.

• Policy Accepted=True, ResolvedRefs=True; 200 for /readyz from the 16th second after the apply (the Envoy pod needed that long).
• A ConfigMap wrong-ca (another self-signed CA) swapped into caCertificateRefs: the policy stayed Accepted/ResolvedRefs True. The first test showed a single 503 after 5 s and then 200 again, and 200 stayed for 100 s more (checked in Envoy's admin config_dump: the SDS secret ichor-ui/ichor-system-ca did contain the wrong CA, updated at the time of the change). Cause: Envoy keeps its pooled upstream TLS connection, which was opened with the old CA and is reused by the steady stream of test requests; only new upstream connections see the new CA. That is the "200 for a wrong CA after 30 s, 503 later" of the earlier run.
• After a rollout restart of the Envoy Deployment (new upstream connections), the wrong CA gave 503 on every request (9 requests over 42 s).
• Back to ichor-ui-ca: 200 on the first poll, 1 s after the patch (09:22:56 patch, 09:22:57 first 200), and 200 on 6 more polls over 26 s. The "still 503 at 45 s" of the first run is therefore not reproduced; the likely cause there was a stale connection or the pod restart race, not the policy. Conclusion: BackendTLSPolicy changes apply within seconds, but a wrong CA only shows up on new upstream connections; test a negative case after restarting the proxy or with no steady traffic.

No product defect in T2.

Amendment 2026-10-30: the stale http Service port of an ichor init install is removed

Fixes the failed expectation in "Unit 10, plaintext install upgraded" above. The cause was general: ichor init created every object of Ichor's manifest with Create (field manager ichor, operation Update), and the installer's server-side apply as ichor-installer cannot remove a list item (or any field) another manager owns, so anything a release drops from the manifest stayed on an init-made install.

• Installer. Before it applies any object other than a Deployment, applyOne reads the live object and, if manager ichor holds Update entries, hands those fields to ichor-installer with k8s.io/client-go/util/csaupgrade (UpgradeManagedFieldsPatch, the documented client-side to server-side upgrade; the patch guards on the resourceVersion) and logs it. The apply then owns the whole object it knew and drops what the manifest no longer has: the http 8080 port of the ichor-ui Service, and equally a dropped field on a CRD, ClusterRole, ConfigMap or CiliumNetworkPolicy. Fields set by other managers (an operator's kubectl edit port) are untouched. Deployments need nothing: the installer replaces them whole.
• Objects affected (all objects ichor init creates from the manifest except Deployments): in practice only the Service had a dropped item on this upgrade; the others had no fields removed, but the same hand-over now protects them on later releases.
• ichor init. New installs create every non-Deployment object with a server-side apply as ichor-installer (a Deployment is still created, with that field manager). An existing object is still skipped, so re-running init is unchanged. Init's own later Updates (the internal port, the Deployment args) remain manager ichor, and the next upgrade hands them over like any older install.
• Tests. TestApplyOneRemovesAPortOwnedByAnOlderInit (envtest): a Service created as ichor with an http 8080 port and an operator port extra added by kubectl-edit; after applyOne of a manifest with https only, the ports are extra and https.
• Docs. docs/guides/users-and-login.md named the Service ichor-ui-service where an ichor init install has ichor-ichor-ui-service; the certificate-names bullet is corrected.

Amendment (ADR 0107)

spec.reclaim.enabled, the ichorinstallations/reclaim subresource, the confirmation annotation and the Settings reclaim card are gone: the field is deprecated and ignored, and "Reset and add" is decided by the reclaim verb on machinereclaims alone.

Amendment (2026-10-30): why the upgraded ichor-ui waited about 2.5 minutes

The Secrets are issued by the controller leader (IssuerTrustReconciler, event driven, it runs once at start). After an upgrade the new controller pod was not the leader: LeaseDuration is 120 s (ADR 0063) and the old pod did not release the lease when it stopped, so the new one waited out the lease before issuing ichor-oidc-ca, ichor-oidc-tls and ichor-ui-ca. That is the 00:13:30 to 00:16:00 gap. The manager now hands the Lease over itself after a plain voluntary stop (ADR 0063 amendment of 2026-10-30), so an upgrade hands over at once; after it the remaining wait is the kubelet copying the new Secrets into ichor-ui's volume, up to about a minute (an optional Secret volume is refreshed on the kubelet sync period). A crash, or a stop that goes into the etcd restore tail, still waits out the lease.

Hardware result 2026-10-30 (rerun at c549005e): plaintext upgrade and lease hand-over

Cluster: ichor init 1 control plane + 1 worker (prefix ichor-e2e-z, Talos v1.14.3, Cilium geneve, images from an in-cluster registry through the ghcr.io mirror). Old build v0.0.1850000000-g2b67caf6 (commit 2b67caf6, built in its own worktree, ichor init with its CLI, no --ui-url, users off), new build v0.0.1850000001-gc549005e (this tree) installed with ichor upgrade ichor ... --local from the new CLI. No Go tests were run; everything by hand.

• Before. Service ichor-ichor-ui-service had only http 8080, ichor-ui --addr=:8080; from a pod in the inner cluster http://...:8080/readyz gave 200 and / gave 200 text/html.
• Upgrade passed. The operation took 3 min 30 s (02:54:21 to 02:57:50 UTC, Job complete 02:58:25). The Job logged NOTICE: ichor-ui no longer serves plain HTTP (it did on :8080); ... and handing the fields ichor init wrote on Service ichor-ichor-ui-service over to ichor-installer (the same line for every other object ichor init wrote, including the three CiliumNetworkPolicies and the CRDs).
• The earlier failure is fixed. The Service has exactly one port, https 8443 (kubectl get svc ... -o jsonpath={.spec.ports}), and its EndpointSlice lists port 8443 only. ichor-ui runs --addr=:8443; the CiliumNetworkPolicy ichor-ui opens 8443 (world, cluster) and 8444 (kube-apiserver, remote-node). From a pod in the default namespace of the inner cluster: https://...:8443/readyz --cacert <ichor-ui-ca> gave 200; without the CA curl exited 60 (unable to get local issuer certificate); http://...:8443/readyz gave 400; http://...:8080/readyz now fails at once with curl exit 7 (connection refused; before, with the port still in the Service, it timed out). ichor-ui-ca exists as a Secret and as a ConfigMap. (A test pod in ichor-system cannot reach anything: the namespace is default-deny; use another namespace.)
• Timing of that upgrade (old controller, no hand-over code). New controller pod started 02:55:11; it logged Attempting to acquire leader lease at 02:55:21 and Successfully acquired lease at 02:57:50, 2 min 29 s later (the lease the old pod, a build without the hand-over, did not release); Issuer internal CA created and the serving certificate followed in the same second; the new ichor-ui pod (started 02:55:34) became Ready at 02:58:23. So 3 min 12 s from the new controller pod to a Ready ichor-ui, as in the first run: the fix lives in the stopping pod, so the upgrade away from a build that has it cannot show it.
• Lease hand-over, second upgrade (build to build, both with the fix). Built v0.0.1850000002-gc549005e from the same tree and ran ichor upgrade ichor again (operation done in 47 s, 03:00:47 to 03:01:34). The old controller (log followed from outside) logged Stopping and waiting for ... at 03:01:26 and no Could not hand the leader lease over error; the new pod (started 03:01:15) logged Attempting to acquire leader lease at 03:01:21 and Successfully acquired lease at 03:01:33, 7 s after the old pod began to stop (before: 2 min 29 s). The new ichor-ui pod was Ready at 03:01:27, 12 s after the controller pod started, but this number does not measure issuing: the Secrets from the first upgrade already existed, so the kubelet had them at once. There is no controller log line for a successful hand-over (only the error line on failure, cmd/main.go); the evidence is the missing error and the immediate acquire. The old pod's last line was error received after stop sequence was engaged: leader election lost, which is the stop sequence, not a failure of the hand-over.
• No failing tests (none run). Own mistakes: a first curl pod in ichor-system could not reach the Service (default-deny), moved to default.

Amendment (2026-11-03): an operator's certificate does not stop in-cluster clients verifying ichor-ui by its Service name

Found on the UI demo: ichor upgrade ichor warned that ichor-ui's certificate could not be verified and computed the plan from the Kubernetes API. The demo supplies its own self-signed certificate for the load balancer's address (init --ui-tls-cert), so 8443 served only that certificate. The CLI dials the port-forward with the Service name (<service>.<namespace>.svc) and verifies against ichor-ui-ca, which is Ichor's CA: the operator's certificate has neither the name nor that CA. Ichor's own leaf, which has the name, stayed on disk beside the operator's pair (/etc/ichor/ui-tls, mounted by the chart and by the installer), but no handshake was ever answered with it.

Decision: ichor-ui serves the Ichor-made leaf to a handshake whose SNI is a Service DNS name (*.svc or *.svc.cluster.local) that the leaf covers, and the operator's pair to every other handshake: an external name, an address, or no SNI. The CLI and in-cluster clients keep verifying the Ichor leaf against ichor-ui-ca; no client gets a new trust anchor. The Ichor leaf names the Service as <name>, <name>.<namespace>, <name>.<namespace>.svc and <name>.<namespace>.svc.cluster.local, plus localhost and the external host (the names the controller already put there, now with the <name>.<namespace> form). Adding a SAN makes the controller reissue the leaf once.

Rejected: the CLI trusting the operator's certificate (it has no in-cluster way to learn its CA or its names, and a self-signed certificate for an address needs --ca-file and --tls-server-name on every command); publishing the operator's CA as ichor-ui-ca (the controller does not hold that CA and the ConfigMap is the Ichor CA by name); a second in-cluster port (the 8444 listener serves only the issuer's discovery and key set).

The fallback is unchanged in shape and stays for real failures: an unreachable ichor-ui, or a certificate that does not verify even by Service name (for example after a CA rotation the ConfigMap has not caught up with). The warning no longer appears for a healthy ichor-ui with an operator certificate.

Consequences: an edge that verifies the backend by the Service name still needs Ichor's CA (ichor-ui-ca), as before; an edge with an operator certificate verifies that certificate for its external name. The first ichor upgrade run with a CLI from this change still uses the old ichor-ui and so still warns; later commands are verified once the upgraded ichor-ui serves. Not yet verified on hardware: the unit tests cover the SNI selection (cmd/ichor-ui), the leaf names (internal/controller) and the CLI's verification and fallback (internal/cli).