# 0105. ichor-ui is TLS-only with an Ichor-made serving certificate and a CA the edge can verify, and Ichor's own runtime settings live in `IchorInstallation`, editable from a Settings tab

Status: Accepted; built (unit 10, the plaintext removal, done early by the user's decision on 2026-10-30); hardware-verified (T1 to T8) 2026-10-30 · Date: 2026-10-29 · Design section: Trust, Deployment, CLI and UI · Extends: 0059, 0094, 0100, 0104 · Touches: 0065, 0103 (reclaim)

Decided: ichor-ui never serves plaintext to the network: it listens for HTTPS only on a new port 8443, with the certificate the operator supplies (`ichorUI.tls.secretName`, for example a cert-manager Secret) or, when none is supplied, a leaf from the Ichor CA that the controller already keeps for the 8444 issuer listener (`ichor-oidc-ca`, `ichor-oidc-tls`), issued for every install and not only with users enabled; the controller publishes that CA, public part only, as a ConfigMap and a Secret named `ichor-ui-ca` so an Ingress-NGINX, a Gateway API `BackendTLSPolicy` or a client holding `--ca-file` can verify the backend, while a plain L4 load balancer needs the operator's own certificate; an edge re-encrypts to ichor-ui; the kube-apiserver's 8444 fetch is unchanged and keeps the Ichor leaf even when the operator supplies a certificate for 8443; an existing install keeps a deprecated plaintext listener on 8080 for exactly one minor release (never on a fresh `init` or Helm install) and then loses it; certificate hot reload is the work of ADR 0094's amendment (branch `cert-reload`) and is used here, not redesigned; Ichor's runtime settings are new optional fields of the existing singleton `IchorInstallation` (not a new kind), live-read by the controller, the webhooks and ichor-ui, edited from a Settings tab and by `ichor settings`, guarded by RBAC verbs checked in the webhook; a Helm install gets an `IchorInstallation` too, created once by the controller if absent, and a new field `spec.deployment.manager` (`installer` or `helm`) is how `ichor upgrade ichor`, the UI, the webhook and ADR 0104's `ichorPresent` tell the installer Job path from Helm

## Context

ichor-ui is highly privileged: it holds a read-only Talos identity, impersonates users against the Kubernetes API, is Ichor's OIDC issuer (ADR 0094) and carries the override reclaim form (ADR 0103). Today:

- `cmd/ichor-ui/main.go` serves plain HTTP on 8080 unless `--tls-cert-file`/`--tls-key-file` are given; with them it calls `srv.ListenAndServeTLS`, which reads the pair once and never again. `docs/guides/users-and-login.md` ("Expose ichor-ui over https") tells operators to put TLS "at the edge", which leaves edge to pod in cleartext.
- The 8444 listener already has the machinery this decision needs: the `IssuerTrustReconciler` (`internal/controller/issuer_trust.go`) keeps a 10 year CA in `ichor-oidc-ca` and a one year leaf in `ichor-oidc-tls` (SANs: the Service ClusterIP and `<service>.<ns>.svc`), renews at two thirds, and publishes the public parts in `ichor-oidc-trust` (key `ca.crt`). But it is started only with `--user-issuer` (`cmd/main.go`), so an install without users has no CA.
- `ichorUI.tls.secretName` (chart) and `--ui-tls-secret`/`--ui-tls-cert` (`ichor init`) mount a Secret at `/etc/ichor/tls`; `internal/installer` (`CustomizeUI`, `ReadUISettings`) re-applies that on every upgrade (ADR 0059 amendment).
- Ichor's own settings are command-line flags on the controller (`--reclaim`, `--no-cilium-policies`, `--user-issuer`) and on ichor-ui (`--token-ttl`, `--refresh-lifetime`, `--external-url`). Changing one means editing a Deployment, which `ichor init` and the installer own, and which a Helm release owns on a Helm install. There is no Settings page.
- `IchorInstallation` (`api/v1alpha1/ichorinstallation_types.go`) is the cluster-scoped singleton named `ichor` with `spec.version` and `spec.installerRepository`; `ichor init` creates it. ADR 0104 decision 4.3 says a Helm-installed Ichor has none and that `ichorPresent` (IchorInstallation, or a deployed Helm release of the chart, or the controller Deployment) must tolerate that.

Library facts checked in the module cache at the go.mod version (controller-runtime v0.24.1): `pkg/certwatcher.CertWatcher` offers `New`, `WithWatchInterval`, `RegisterCallback`, `GetCertificate`, `Start`; `webhook.Server` already builds a `certwatcher` and sets `GetCertificate` whenever `TLSOpts` does not set one (`pkg/webhook/server.go`), so the webhook already follows its certificate. The metrics server and the 8444 listener read the key pair on every handshake (`lazyKeyPair`, `internalServer`); only ichor-ui's main listener and `ListenAndServeTLS` do not follow rotation.

## Decisions

### Part A. TLS to ichor-ui

**1. ichor-ui has no plaintext mode.** One listener serves the SPA, the RPC API, `/metrics` and the probes, on port **8443**, HTTPS only. The listener starts even when no certificate exists yet and answers every handshake with an error until one does (the readiness probe, an HTTPS `httpGet`, keeps the pod out of the Service); it never falls back to HTTP. The flags `--tls-cert-file` and `--tls-key-file` stay and now mean "the operator's pair". New flags: `--tls-dir` (default `/etc/ichor/ui-tls`, the managed pair). `--addr` defaults to `:8443`. Rejected: **keeping a `--insecure` escape hatch for development** (a flag that disables the control is the first thing a copy-pasted manifest carries; development uses the managed certificate and `--ca-file`); **same port 8080 serving TLS** (every probe, Service, NetworkPolicy, port-forward and bookmark would silently change meaning, and a client that still speaks HTTP gets a confusing TLS error; a new port makes both versions coexist, decision 6); **mTLS between edge and backend** (the edge proxies for browsers and cannot present a client identity ichor-ui could map to anything; the user is authenticated by the OIDC token inside the request).

**2. Which certificate.**

| Operator supplies | 8443 serves | 8444 serves |
|---|---|---|
| a Secret (`ichorUI.tls.secretName`, `--ui-tls-secret`, cert-manager) | the operator's pair from `/etc/ichor/tls` | the Ichor leaf |
| nothing | the Ichor leaf, from `ichor-oidc-tls` mounted at `/etc/ichor/ui-tls` | the Ichor leaf |

The Ichor leaf gets more SANs than today: the Service ClusterIP, `<svc>.<ns>.svc`, `<svc>.<ns>.svc.cluster.local`, `<svc>`, `localhost`, `127.0.0.1`, and the host of `--external-url` when users are enabled (so a client given the CA verifies a direct connection through an L4 load balancer; a browser will not trust it, which is the operator-certificate row's reason to exist). The Secret and CA names stay `ichor-oidc-*` (renaming a Secret that an installed apiserver trust chain and the installer's `resourceNames` reference buys nothing); the docs call the pair "the Ichor serving CA". The `IssuerTrustReconciler` splits: the TLS half (CA, leaf, trust ConfigMap, `ichor-ui-ca`) runs for every install; the signing-key and user-credential halves stay behind `--user-issuer`. `--user-issuer` remains a flag with its meaning narrowed to those halves. The pod mounts `ichor-oidc-tls` with `optional: true` so ichor-ui starts before the controller has issued the first leaf; whether the kubelet then populates the volume in place, and how long it takes, is a hardware test (T3) because the reload latency of every Secret-backed certificate rests on it. Rejected: **a second CA and leaf for the edge-facing listener** (two trust anchors and two renewals for one ClusterIP; the leaf may serve both listeners because the 8444 CA is already pinned only for the apiserver's issuer fetch and trusts nothing else); **cert-manager as a dependency** (ADR 0006 and 0101: Ichor ships no certificate dependency; cert-manager stays an operator choice through `secretName`); **a self-signed certificate generated in the pod at start** (each replica would differ and nothing could publish a CA for the edge).

**3. How the edge verifies the backend.** The controller publishes the CA certificate only (never a key) as the ConfigMap `ichor-ui-ca` and the Secret `ichor-ui-ca`, both with key `ca.crt`, in `ichor-system`, updated on CA change. Service port 8443 is named `https`. The guide gets one recipe each; exact annotations and field names are looked up in the controllers' own documentation when the guide is written (work unit 9), not taken from memory:

- **Ingress-NGINX**: backend protocol HTTPS, `proxy-ssl-verify` on, `proxy-ssl-secret: ichor-system/ichor-ui-ca`, `proxy-ssl-name`/server name `ichor-ui-service.ichor-system.svc`. The Ingress lives in `ichor-system`, because it must reference the Service.
- **Gateway API `BackendTLSPolicy`**: `targetRefs` the ichor-ui Service, `validation.caCertificateRefs` the ConfigMap `ichor-ui-ca`, `validation.hostname` `ichor-ui-service.ichor-system.svc`. Needs a Gateway API release that serves `BackendTLSPolicy` at the cluster's version; the guide states which and falls back to the Ingress recipe's equivalent for the implementation in use.
- **Plain L4 load balancer** (`Service` type `LoadBalancer`, or TCP passthrough): the load balancer terminates nothing, so clients see ichor-ui's certificate and the host name they typed. Use the operator-certificate row. With the Ichor leaf only clients given `ichor-ui-ca` (`ichor --ca-file`) verify.
- An L7 cloud load balancer that cannot verify its backend is out of scope; the guide says "use a controller that can".

**4. The 8444 fetch.** Unchanged in shape: the kube-apiserver reads `https://<ClusterIP>:8444/.well-known/openid-configuration` and verifies it with `ichor-oidc-trust`'s CA (ADR 0094). What changes: the listener and 8443 share the reload helper of the `cert-reload` amendment, so a renewed leaf is served by both; the NetworkPolicy of ADR 0100 gains ingress on 8443 (the rule that listed 8080) and keeps 8444 for `kube-apiserver`. Port 8444 is never routed by an edge (the guide already says so; the policy enforces it for Cilium installs).

**5. Hot reload.** Not designed here. ADR 0094's amendment on branch `cert-reload` makes ichor-ui's listeners, the webhook and the metrics server follow their certificate files without a restart. This ADR depends on it for the main 8443 listener and for the cert-manager case (`secretName` Secrets are mounted, so a cert-manager renewal reaches the pod through the kubelet's Secret volume sync and then the reload). Work unit 2 consumes it; the tests for the reload itself belong to that branch. If it does not land first, unit 2 uses `lazyKeyPair` (per-handshake read) as the stand-in, which is correct and merely unobserved.

**6. Migration without downtime.** Version N (this change) serves 8443 TLS **and**, only where the install already served plaintext, 8080 plaintext marked deprecated. Version N+1 deletes the plaintext listener.

| Existing install | N | N+1 |
|---|---|---|
| plain HTTP on 8080 (no `--tls-cert-file`) | 8443 TLS (Ichor leaf) added, Service gets port `https`; 8080 stays, flagged by `--http-addr=:8080` (deprecated) and an Alert on the Settings tab and Overview ("plain HTTP is still on; move your edge to 8443, removed in N+1") | `--http-addr` ignored with a startup warning; Service port `http` removed |
| TLS on 8080 with the operator's Secret | `--addr` pinned to `:8080` by the installer (it already reads `--tls-cert-file` from the live Deployment, `ReadUISettings`), nothing moves; 8443 is not added | `--addr` default flips only on fresh installs; these keep their pinned value, and gain the reload |
| fresh `ichor init`, fresh `helm install` | 8443 TLS only, no plaintext | same |

The installer derives the case from the live Deployment as it already does and never needs a new state store (ADR 0059 amendment). The skew rule (the CLI refuses writes more than one minor away) bounds the window to one minor in practice; the N+1 installer refuses to run on an install whose live ichor-ui still has `--http-addr` set without printing the same advice. During the N rollout the old pods serve 8080 and the new pods both; the Service has port 8443 only on new endpoints (readiness keeps old pods out of it), and the edge is re-pointed at the operator's pace. `ichor ui` (port-forward) and the CLI use 8443 and `https://localhost:<port>` with the CA fetched from `ichor-ui-ca` (the kubeconfig already grants the read), falling back to 8080 against a pre-N server.

### Part B. Settings

**7. Settings live in `IchorInstallation`, as new optional fields; there is no new kind.** The singleton already exists on every `ichor init` install, is already guarded by a CEL name rule and the webhook, is already read by the CLI, UI and installer, and is already the place "how Ichor itself is installed" is recorded. Added to `spec` (all optional; none removed or renamed; defaults keep today's behaviour):

```text
spec.deployment.manager      installer | helm      (omitted = installer)
spec.reclaim.enabled         *bool                  (override, ADR 0103)
spec.users.tokenTTL          duration, 1m..1h       (ADR 0094)
spec.users.refreshLifetime   duration, 1h..720h
spec.defaults.driftPolicy    enforce | warnOnly     (a new TalosCluster's default)
```

Initial scope is what is read cheaply at run time and has no Deployment shape: `reclaim.enabled` (the controller and the reclaim webhook read the setting per call instead of a startup bool), the two user lifetimes (ichor-ui reads them per issued token from its cache), and the default drift policy (the existing defaulting webhook fills `TalosCluster.spec.driftPolicy` at create; an existing cluster is never touched, so the controller still never mutates a spec). Further settings are added as optional fields in later ADRs; each must be live-readable.

Precedence: a set field wins; an unset field falls back to the flag or chart value, so a Helm install without any `spec.*` field behaves exactly as before. `status.effective` (written by the controller; ichor-ui reports its own two values over the Settings RPC) shows each value and its source (`spec` or `deployment`), and the condition `SettingsApplied` shows the controller has seen the generation. A deployment that wants settings owned by Git or Helm sets the controller flag `--settings-read-only` (chart `settings.readOnly`): the webhook then denies writes to the settings fields, every consumer ignores them, the Settings tab is read-only with a "managed by the deployment" Alert. No live-reload gap is accepted: a change takes effect on the next call (ichor-ui and webhook within the informer's delay); nothing in settings needs a rollout. Deployment-shaped values (images, replicas, resources, the TLS Secret name, `externalURL`, the Cilium-policy opt-out, which are applied by objects Ichor does not render at run time) stay with the installer and Helm and appear on the tab as read-only facts (TLS: source, subject, expiry; Ichor version; manager).

Rejected: **a new cluster-scoped `IchorSettings` kind**. It would give settings their own RBAC and their own lifecycle, but: every consumer, the CLI, the installer (CRD list in `resourceNames`, `TestInstallerRBAC`), the chart and the snapshot would learn a second singleton whose existence must be handled in the absent case on top of the first; the Helm question (who creates it) is the same question and is answered once for `IchorInstallation` below; and the one real benefit, splitting edit rights, is available on one object with the webhook pattern ADR 0094 already uses (decision 9). Rejected: **a ConfigMap** (no schema, no CEL, no printer columns, no typed UI). Rejected: **flags only, with the UI patching Deployments** (the UI would write objects it does not own and the installer or Helm would revert the edit). Rejected: **putting settings on `TalosCluster`** (they are installation-wide, not per cluster; the default drift policy is the one per-cluster fact and is a default there).

**8. A Helm install gets an `IchorInstallation`, and `spec.deployment.manager` says who deploys.** The chart does not render it: a rendered object would be overwritten by `helm upgrade`, reverted by a GitOps tool tracking the release, and would invite the installer to fight Helm (the reason ADR 0104 4.3 skipped it). Instead the controller, on start and only when the singleton does not exist, **creates it once** with `spec.version` = its own version and `spec.deployment.manager` taken from a new flag `--deployed-by` (`helm` rendered by the chart; `installer` is the default and what `ichor init` writes itself). It never updates `spec` afterwards. This is a narrow, stated exception to "the controller never mutates `spec`": creating a missing singleton with computed defaults changes no user intent, and nothing is lost if it fails (every consumer tolerates absence, as 0104 required). The alternative that keeps the rule whole, the first save in the Settings tab creating the object, was rejected: the version and manager would then be unknown until someone saved, and `ichor upgrade ichor` could not tell Helm from the installer on a cluster nobody has opened the tab on.

For `manager: helm`: `spec.version` records the version at creation and is not compared with the running controller (`UpToDate` is `True` with reason `ManagedByHelm`, the UI's "Update available" badge and `OperationService.GetIchorInstallation`'s `update_available` are suppressed); the validating webhook denies a `TalosOperation` of kind `ichor` (message: "this Ichor is installed by Helm; run helm upgrade"), the executor and the installer Job refuse as a second line, and the chart's `installer.enabled` stays false. `ichor upgrade ichor` reads the field first and prints the Helm command. The Settings tab shows "Installed by Helm" and still edits runtime settings, which Helm does not render.

**ADR 0104 unit 5, as changed.** `ichorPresent` is unchanged in its three signals (IchorInstallation, deployed Helm release of the chart, controller Deployment) because the first two windows remain: before the controller seeds, and on installs older than this ADR whose controller has not yet been upgraded. What changes: (a) when the singleton exists, `spec.deployment.manager` is authoritative over the other two; (b) adopt skips the install step on `manager: helm`, creates no `IchorInstallation` itself (the controller does) and, when the object is still absent, does not wait for it; (c) installer-Job commands refuse on `manager: helm` and, while the object is absent, on a Helm release; (d) the line "no `IchorInstallation`... fixes that code rather than having the chart create the object" becomes "...tolerates an absent object in the seed window; the chart still never renders it". Unit 5's audit list gains `GetIchorInstallation`, the CLI skew check (`status.installedVersion` is still written by the controller) and the PR-mode editor of ADR 0059 amendment 4 (it edits only `spec.version` in a tracked file, so a tracked `manager: helm` object is simply refused there).

**9. RBAC, validation, audit.** Reading is part of `ichor-viewer` (the spec holds no secret and the page shows no secret). Writing `ichorinstallations` stays `ichor-admin`'s; the finer split uses the webhook pattern of ADR 0094 (a `SubjectAccessReview` on the request's user): changing `spec.version` or `spec.deployment` needs the verb `update` on `ichorinstallations/version`, changing `spec.reclaim` needs `update` on `ichorinstallations/reclaim`, and everything else under `spec.users`/`spec.defaults` needs `update` on `ichorinstallations/settings`; the built-in `ichor-admin` role holds all three on separate rule lines so a custom role can hold fewer. `spec.reclaim.enabled: true` additionally requires the typed confirmation (the UI's dialog; the CLI's `--confirm-reclaim`) that the API enforces by an annotation the request must carry (`ichor.maesh.dev/confirm-reclaim: enable`), so a stray `kubectl apply` cannot turn override on by accident. Validation is CEL on the CRD (duration bounds, `tokenTTL <= refreshLifetime`, the enums, the name rule that exists) plus the webhook for the SAR checks and `--settings-read-only`. Audit: the Kubernetes audit log names the user (ichor-ui impersonates, ADR 0058), and the webhook emits an Event on the `IchorInstallation` per accepted change listing the changed field paths, old and new values (none is secret by construction) and the user; enabling reclaim also emits the Warning event ADR 0103 already defines for its use.

**10. UI and CLI.** A "Settings" entry in the sidebar, a page of stock preactcn components: `Tabs` (General, Sign-in, Override, Defaults), `Card` with `CardHeader/Title/Description`, `Field`, `Input`, `Select`, `Alert` (managed by Helm or the deployment, plaintext still on), `Badge` (source: spec or deployment), `AlertDialog` (reclaim confirmation), `Button`. `Switch` is not in `ui/src/components/ui` yet; it is added with `npx shadcn@latest add @preactcn/switch`, never by hand. English strings in `ui/src/lib/i18n.ts`. A new `SettingsService` (Get, Update) in the UI API; Update is a merge patch of the settings fields only, as the user. CLI: `ichor settings get` and `ichor settings set KEY=VALUE...` write the same CR.

## Work units

| # | Unit | Files | Size | Depends on |
|---|---|---|---|---|
| 1 | CRD fields (`spec.deployment`, `spec.reclaim`, `spec.users`, `spec.defaults`, `status.effective`, condition `SettingsApplied`), CEL, regenerate, add nothing to the installer CRD list (same CRD) | `api/v1alpha1/ichorinstallation_types.go`, `config/crd`, `dist/chart/templates/crd` | S | |
| 2 | ichor-ui: HTTPS-only 8443, `--tls-dir`, `--http-addr` transitional listener, managed/operator pair selection, reload via the `cert-reload` helper | `cmd/ichor-ui/main.go`, `users.go` | M | `cert-reload` branch (or `lazyKeyPair`) |
| 3 | Controller: TLS half of `IssuerTrustReconciler` for every install, wider SANs, `ichor-ui-ca` ConfigMap and Secret, RBAC `resourceNames`, expiry metric | `internal/controller/issuer_trust.go`, `cmd/main.go`, `config/rbac`, chart roles | M | |
| 4 | Installer, init and chart: Deployment/Service/probe changes, optional `ichor-oidc-tls` volume, the three migration cases from `ReadUISettings`, NetworkPolicy 8443, chart values and tests | `internal/installer/customize.go`, `internal/cli/init_ichor.go`, `dist/chart/templates/ui`, `config/cilium-network-policy`, `dist/chart/*_test.go` | M | 2, 3 |
| 5 | CLI client trust: `ichor ui` and `auth` use 8443, CA from `ichor-ui-ca`, fallback to 8080 | `internal/cli/ui_run.go`, `api.go` | S | 3 |
| 6 | Controller seed of the singleton, `--deployed-by`, `ManagedByHelm` handling, webhook denial of `kind: ichor`, `ichor upgrade ichor` message; ADR 0104 unit 5 detection as changed above | `internal/controller/ichorinstallation_controller.go`, `internal/webhook`, `internal/cli/adopt_*.go`, `internal/uiapi` | M | 1 |
| 7 | Settings consumption: reclaim read per call (controller, reclaim webhook), token lifetimes in ichor-ui, default drift policy in the defaulting webhook, `--settings-read-only`, SAR checks, Events | `cmd/main.go`, `internal/webhook`, `internal/uiapi`, `cmd/ichor-ui` | M | 1 |
| 8 | `SettingsService` proto and handler, Settings page, i18n, `switch` component, `ichor settings` | `proto`, `internal/uiapi`, `ui/src/pages/Settings.tsx`, `internal/cli` | L | 1, 7 |
| 9 | Docs: users-and-login "Expose ichor-ui over https" (recipes, with controller docs checked), install-with-helm, architecture Trust/Deployment, amendment lines in 0094/0100/0104, `make adr-index` | `docs/` | S | 2-8 |
| 10 | N+1: delete the plaintext listener and Service port (done early, see the amendment below) | `cmd/ichor-ui`, installer, chart | S | one minor after N |

## Tests

Unit and envtest: listener answers no plaintext on 8443 and fails handshakes (not the process) with no certificate; selection table of decision 2; SAN set; `ichor-ui-ca` carries no key; installer migration cases from fixture Deployments; settings webhook (SAR matrix, read-only flag, CEL bounds, confirm annotation); seed creates once and never updates; `manager: helm` denies `kind: ichor`; precedence set versus unset.

Hardware and e2e (batched per the hardware rule): **T1** fresh `ichor init` serves HTTPS only on 8443 and nothing on 8080; **T2** Ingress-NGINX re-encrypting with `proxy-ssl-verify` against `ichor-ui-ca`, then a `BackendTLSPolicy` run, then an L4 passthrough with an operator certificate; **T3** an optional `ichor-oidc-tls` volume created after the pod starts is populated in place, and the time to first good handshake; **T4** a cert-manager renewal reaches a running ichor-ui with no restart (with the `cert-reload` branch); **T5** the apiserver keeps fetching 8444 across a leaf renewal and an operator certificate on 8443; **T6** an upgrade from the last release with plaintext 8080: no failed request through a looping client during the rollout, edge moved to 8443, then N+1 removes 8080; **T7** a `helm install` gets a seeded `IchorInstallation` with `manager: helm`, `ichor upgrade ichor` refuses, `helm upgrade` works, a Settings edit survives it; **T8** reclaim toggled from the tab takes effect with no restart and leaves an Event, refused for a role without the verb.

## Risks

- A breaking edge: an operator who upgrades to N+1 without moving the edge loses the UI. Mitigated by the visible deprecation, the one-minor window and the N+1 installer's refusal.
- Readiness depends on the controller having issued the leaf; a controller outage at first install leaves ichor-ui unready (it was ready with plaintext). `ichor init` already bounds its waits and names the reason (ADR 0104).
- A seeded object is created by a component that otherwise only reads `spec`; a bug there would hide behind "absent is fine". Create-only, covered by test.
- Anything that reads settings live widens what a mistaken edit can change at once; the three RBAC lines, the confirm annotation and the Event are the compensation, and reclaim stays default off.
- Browser trust: the managed leaf is not browser-trusted. This is intended (the edge presents the public certificate) but a user on L4 passthrough must read the guide.

## Open questions

1. Rotation of the Ichor CA itself with an overlap period (publishing two CAs in `ichor-ui-ca` and the apiserver's `certificateAuthority`); today the CA is replaced when near its end and everything re-trusts at once. Not needed for this change, needed before the 10 year mark.
2. Whether the ichor-ui login settings (`--oidc-*`, the external IdP) should become runtime settings too; they need a provider re-discovery path and are left to a later ADR.

## As built (2026-10-29)

Units 1 to 9 are built; the hardware tests T1 to T8 are not run. Differences from the text above:

- **Settings page**: stock `Card`s, not `Tabs` (General, Sign-in, Override, Defaults became cards: Deployment, Override reclaim, Sign-in, Defaults). `Switch` was added with `npx shadcn@latest add @preactcn/switch`.
- **No "unset"**: neither the UI nor `ichor settings` can clear a field once set; a field set stays set. Precedence is as designed for fields never set.
- **`defaults.driftPolicy`** is `warnOnly | autoCorrect` (the enum of `TalosCluster.spec.driftPolicy`), not `enforce | warnOnly`.
- **Plaintext on 8080**: the installer pins any live `--addr` that is not 8443 (not only a TLS-on-8080 install). The chart has no automatic detection; the opt-in value is `ichorUI.plaintextPort` (default 0), and `ichorUI.port` keeps a former 8080 TLS install in place.
- **CLI trust**: `ichor ui` and the API client stay on 8080 while `ichor-ui-ca` is missing.
- **Helm denial**: the denial of `kind: ichor` on `manager: helm` lives in the `TalosOperation` validating webhook; the installer refuses by reading the live Deployment, as a second line.
- **Drift default**: the CRD default of `TalosCluster.spec.driftPolicy` was removed in favour of a create-time defaulting webhook (`failurePolicy: Ignore`; an empty value is read as `warnOnly`, so a down webhook changes nothing). The installer's RBAC gained `mutatingwebhookconfigurations`. The CRD field itself is kept (CRD fields are append-only).
- **Token lifetime**: `users.tokenTTL` is capped at 1h (`MaxTokenTTL`), as the CEL rule says.
- **Monitoring**: the ServiceMonitor verifies the HTTPS port against `ichor-ui-ca` (`ichorUI.tls.caConfigMap/caSecret/caKey`) and never skips verification.
- **Reload**: certificates reload on change through `certwatcher` (ADR 0094 amendment, built).
- **Docs**: the Ingress-NGINX recipe uses `backend-protocol`, `proxy-ssl-verify`, `proxy-ssl-secret` (`namespace/name`), `proxy-ssl-name` and `proxy-ssl-server-name`; the reference describes `proxy-ssl-secret` as holding a client pair that "should also contain" `ca.crt`, so whether a CA-only Secret is accepted is part of T2. `BackendTLSPolicy` is `gateway.networking.k8s.io/v1`, Standard channel since Gateway API v1.4, same namespace as the Service, `ca.crt` ConfigMap as Core support. The guide for settings is `docs/guides/settings.md`.

## Hardware results (2026-10-30)

- Passed: T1 (HTTPS only, leaf, `ichor-ui-ca`); certificate hot reload in about 10 s; T3 (the optional volume is populated in place after about 100 s; liveness is now a TCP probe, so no restart loop); T6 migration from plaintext to 8443, with 8080 deprecated, no traffic loss; T8 (settings RBAC, read-only mode, confirm-reclaim, fixed to require a fresh confirmation).
- A demo upgrade across the migration pinned 8080; fixed in 67aa97c4 (policy-name bug).
- T2 and T4: see "T2 and T4" below.

### T7 and T5 (2026-10-30, hardware)

Build `v0.0.1761800000-ga163d822`, Talos v1.14.3, Kubernetes v1.37.1, Cilium 1.18.0, geneve.

- **T7 passed** (`TestT7HelmManager` in `test/e2e/foreign/t7_test.go`, 251 s, prefix `ichor-e2e-fq`, a foreign cilium-helm cluster, packaged chart from `make helm-package`, images from the in-cluster registry through the `ghcr.io` mirror). `helm install --wait` seeded `IchorInstallation` `ichor` with `spec.deployment.manager: helm` and `spec.version` equal to the running version. `ichor upgrade ichor v0.0.1-g0000000 --local` exited 1 with: ``ichor is installed by Helm (release "ichor", namespace ichor-system); upgrade it with `helm upgrade ichor oci://ghcr.io/maeshinshin/charts/ichor --version VERSION --namespace ichor-system`, not with the installer`` and created no `TalosOperation`. `ichor settings set users.tokenTTL=10m` stored `spec.users.tokenTTL: 10m0s`; after `helm upgrade --reuse-values --set manager.replicas=1 --wait` the value and `manager: helm` were unchanged. Notes: `ichor settings` has no `--local` or `-n` flag (it fell back on its own from the tools pod); the API normalises `10m` to `10m0s`. One earlier run of the test failed on my own test bugs (a `-n`/`--local` flag the settings command does not take, `10m` vs `10m0s`, a `--no-headers` stderr line), no product failure.
- **T5 passed** (prefix `ichor-e2e-v`, one Ichor-made control plane, `ichor init --cni cilium --cni-tunnel geneve --ui-url https://<cp ip> --admin-user admin`, then `ichor config apply`, an inner NodePort for 8443 and socat relay pods for the workstation). Non-interactive sign-in by POST `/oidc/authorize` (Accept: application/json) then `/oidc/login` then `/oidc/token`. Findings: (1) the browser client `ichor-ui` never receives the Kubernetes audience (by design, `audienceFor`), so the token for `kubectl` comes from the same flow with `client_id=ichor-cli` and a loopback `redirect_uri` (`http://127.0.0.1:<port>/callback`); the `ichor-ui` token is rejected by the kube-apiserver (`expected audience "ichor-kube" got ["ichor-ui"]`). (2) The first admin must change the generated password first: `/oidc/token` for `ichor-kube` answers 403 `access_denied` "change your password first"; the change was made with the ichor-ui `UserService/ChangeOwnPassword` call using an `ichor-ui` token. (3) With `kubectl --token` against a kubeconfig that also holds a client certificate, the certificate wins (the user showed as the cluster admin); the checks used a kubeconfig-less call with only the CA and `--token`. Results with a fresh token each time: baseline `auth whoami` = `ichor:admin`, groups `ichor:admin`, `get nodes` works; after (a) deleting `ichor-oidc-tls` (the controller reissued the leaf within seconds, serial `037E2E2E...` to `6C0A1239...`, 8443 served the new one) the same calls succeeded; after (b) pointing 8443 at an operator certificate (Secret `operator-ui-tls`, the `ui-tls` volume of the ichor-ui Deployment, rollout; 8443 then served `CN=operator-ui` while `ichor-oidc-tls` kept its Ichor leaf) the same calls succeeded. The kube-apiserver log shows no authentication error after the two changes. Limit of the evidence: go-oidc caches the signing keys, so these runs show that validation keeps working, not that a fresh key fetch from 8444 happened during the window.
### T2 and T4 (2026-10-30, hardware)

Build `v0.0.1793312776-gbe6d9bf8`, Talos v1.14.3, Kubernetes v1.37.1, Cilium 1.18.0, geneve. Prefix `ichor-e2e-w`, one Ichor-made control plane and one worker (`ichor init --cni cilium --cni-tunnel geneve --ui-url https://<cp ip> --admin-user admin`, `config apply`, `machine approve`). The Ingress, Gateway and cert-manager installs went into this inner cluster only; the nodes are deleted and nothing of the `ichor-e2e-w` prefix is left. Requests went from a tools pod to the NodePort of the ingress or gateway Service, with the `Host` header or `--resolve`.

- **T2 Ingress-NGINX passed** (chart 4.15.1, controller 1.15.1, NodePort). The guide's Ingress with `backend-protocol: HTTPS`, `proxy-ssl-verify: on`, `proxy-ssl-secret: ichor-system/ichor-ui-ca`, `proxy-ssl-name` and `proxy-ssl-server-name: on` returned 200 for `/readyz` and the UI page. **A CA-only Secret is accepted**: `ichor-ui-ca` has only `ca.crt`; nginx.conf showed `proxy_ssl_trusted_certificate .../ca-ichor-system-ichor-ui-ca.pem` and `proxy_ssl_verify on`. Verification is real: `proxy-ssl-secret` pointing at a Secret with another CA gave 502 (`upstream SSL certificate verify error: (21:unable to verify the first certificate)`), `proxy-ssl-name: wrong.example.com` gave 502 (`upstream SSL certificate does not match "wrong.example.com"`), no `proxy-ssl-name` and `proxy-ssl-server-name: off` gave 502 (the name checked is `upstream_balancer`), and `proxy-ssl-verify: off` with the wrong CA gave 200 (the control). Ingress-NGINX needs up to half a minute to reload: a first run read stale results (200 for a wrong name after 15 s, 502 right after a restore); the repeat with 20 s of settle time was consistent.
- **Finding, fixed in the guide**: an `ichor init` install names the Service `ichor-ichor-ui-service` (the leaf has `ichor-ichor-ui-service[.ichor-system.svc[.cluster.local]]`), not `ichor-ui-service` as the guide said for init installs. The guide now says so; `proxy-ssl-name` is stated as required.
- **T2 Gateway API passed with a caveat** (Envoy Gateway v1.9.2 from `oci://docker.io/envoyproxy/gateway-helm`; its chart installs the Gateway API v1.6.1 CRDs in the **experimental** channel, not standard; `BackendTLSPolicy` is served as `v1`, with `v1alpha3` also served). The guide's `BackendTLSPolicy` (ConfigMap `ichor-ui-ca`, `hostname` the Service name) was `Accepted=True` and `ResolvedRefs=True`; through the Envoy Service the route returned 200 and the UI page. With the policy pointing at a ConfigMap with another CA, or `hostname: wrong.example.com`, the route returned 503 (`remote connection failure`); with the policy deleted, Envoy spoke plain HTTP to 8443 and ichor-ui answered 400. Envoy applied changes in 15 s to 1 min; one early run showed 200 for the wrong CA after 30 s and 503 consistently after a longer wait. **Not confirmed**: after the final change back from the wrong CA to `ichor-ui-ca` the request was still 503 at 45 s and I deleted the objects before reading it again (200 had been seen before the break and after re-creating the policy). The Gateway showed `Programmed=False` only because the LoadBalancer Service had no address; its NodePort worked. A NodePort reached through the control plane's address failed while the Envoy pod ran on the worker (000), and through the worker's address worked; this is a property of the harness network, not of Ichor. L4 passthrough was covered by T1 and skipped.
- **T4 passed** (cert-manager v1.21.3, self-signed Issuer, Certificate `ichor-ui-cert`, `duration: 1h`, `renewBefore: 59m`, ECDSA P-256, rotation `Always`, so a renewal about every minute). The operator certificate was set by patching the ichor-ui Deployment as `internal/installer` `CustomizeUI` does (volume `tls` from Secret `ichor-ui-cert` at `/etc/ichor/tls`, `--tls-cert-file` and `--tls-key-file`); `ichor init --ui-tls-secret` and `ichorUI.tls.secretName` do the same at install time, and `ichor upgrade` has no flag for it. Over 3 minutes of sampling, two of three renewals reached the wire with the pod uid unchanged and restartCount 0: the Secret serial changed at 23:03:47, 23:04:47 and 23:05:47 and the served serial followed at 23:04:12 (25 s later), 23:05:33 (46 s) ; the third (Secret at 23:05:47) had not reached the wire when the run ended, 2 s later. Most of the delay is the kubelet copying the Secret into the pod (the sampling loop itself took a few seconds per round); the 10 s reload is measured from the file change, as in the earlier certificate test.
- No failing tests (no Go tests were run for T2 and T4; they were run by hand). Own mistakes: the first Gateway manifest used `listeners[].hostnames` (the field is `hostname`), and two early curl loops lacked a timeout.

## Amendment 2026-10-30: Unit 10 done early by the user's decision (2026-10-30)

The user decided to remove the transitional plaintext listener now instead of in the next minor release. As built:

- **ichor-ui**: the plaintext server is gone. `--http-addr` stays as a deprecated no-op for one release (flags are an interface): a hand-edited Deployment that still passes it starts and logs `--http-addr=... ignored: plaintext was removed (ADR 0105)`. Delete the flag in the release after.
- **Installer**: `ReadUISettings` no longer produces a plaintext listener. A plaintext install (live `--addr` other than 8443, or a leftover `--http-addr`) is moved to HTTPS on 8443; the Service has no `http` port and the policy opens only 8443. `UISettings.WasPlaintext` records the old address and drives the notice "ichor-ui no longer serves plain HTTP; point your edge at https://<ichor-ui>:8443 (trust ichor-ui-ca)". A TLS install on 8080 is still pinned to 8080 (and its policy port opened). The installer no longer strips or re-adds `--http-addr`; it deletes it from the live args.
- **Chart**: `ichorUI.plaintextPort` is removed. An old values file that sets it is ignored by Helm and renders no plaintext port (tested).
- The migration table row for plain HTTP above now reads: moves to 8443 TLS, 8080 is closed. The Settings/Overview Alert about plain HTTP was never built.
- No T6 test exists under `test/e2e`; the hardware T6 result above (8080 kept, deprecated) describes the pre-amendment behaviour. After this change, a plaintext install must show no 8080 after the upgrade; run on hardware 2026-10-30: the listener and the policy are right but an `ichor init`-made Service keeps its `http` port (see "Hardware result 2026-10-30" below).

## Hardware result 2026-10-30: unit 10 upgrade of a plaintext install (partly failed) and the T2 Gateway leftover (passed)

Cluster: `ichor init` 1 control plane + 1 worker (prefix `ichor-e2e-y`, Talos v1.14.3, Cilium geneve, images from an in-cluster registry through the `ghcr.io` mirror). Old build `v0.0.1840000000-g2b67caf6` (commit `eedfb9f0^`, built in its own `git worktree`, images and CLI from there; `ichor init` with that CLI, no `--ui-url`, users off), new build `v0.0.1840000001-g0f09c67a` (HEAD, `ichor upgrade ichor v0.0.1840000001-g0f09c67a --local` with the HEAD CLI). No Go tests were run; everything below was done by hand.

### Unit 10, plaintext install upgraded: FAILED on one expectation

- **Before.** Service `ichor-ichor-ui-service` had only `8080/TCP`, ichor-ui ran `--addr=:8080`, the CiliumNetworkPolicy `ichor-ui` opened 8080 and 8444. A `kubectl port-forward` to the Service port 8080 returned 200 for `/readyz` and 200 `text/html` for `/`.
- **Upgrade.** The operation `ichor-9vpkf` reported `Completed` after about 3 min; the installer Job printed `NOTICE: ichor-ui no longer serves plain HTTP (it did on :8080); point your edge at https://<ichor-ui>:8443 (trust ichor-ui-ca) (docs/adr/0105)`. The new ichor-ui pod stayed 0/1 for about 2.5 min with `certificate not available yet: open /etc/ichor/ui-tls/tls.crt` and the readiness failure `remote error: tls: internal error`, until the controller (restarted by the upgrade) issued `ichor-oidc-ca`, `ichor-oidc-tls`, `ichor-ui-ca` and `ichor-oidc-trust` (issued at 00:16:00 UTC, the Job had started at 00:13:03; the new controller pod had been running since about 00:13:30 but its issuer only logged "Issuer serving certificate issued" at 00:16:00, so the 2.5 minute gap is the controller's, which I did not investigate). It then became Ready on its own; the Job ended `Complete` at 3 min 18 s.
- **Passed.** ichor-ui runs `--addr=:8443` with the container port `https`/8443 and an HTTPS readiness probe. The CiliumNetworkPolicy `ichor-ui` opens 8443 (world, cluster) and 8444 (kube-apiserver, remote-node) and no longer 8080. From a pod in the inner cluster: `curl https://ichor-ichor-ui-service.ichor-system.svc:8443/readyz --cacert <ichor-ui-ca>` gave 200; without the CA, curl failed with `SSL certificate problem: unable to get local issuer certificate`; plain `http://...:8443/readyz` gave 400; `http://...:8080/readyz` timed out (000 after 5 s). `ichor-ui-ca` exists as a Secret and as a ConfigMap and the leaf verifies against it.
- **Failed: the Service still has the `http`/8080 port.** After the upgrade `kubectl get svc ichor-ichor-ui-service` shows `8080/TCP,8443/TCP` (`http` 8080 -> 8080 and `https` 8443 -> 8443) and the Endpoints list `:8080` next to `:8443`, although nothing listens on 8080. Cause, from `metadata.managedFields`: `ichor init` created the Service with manager `ichor`, operation `Update` (a plain create, not server-side apply), and that manager owns `k:{"port":8080,"protocol":"TCP"}`. The installer applies as field manager `ichor-installer` with server-side apply; it correctly sets only the 8443 port for itself but cannot remove a list item that another manager owns, so the 8080 entry stays. Every install made by `ichor init` before the plaintext removal is affected; an upgrade that started from an installer-made Service is not (not tested). This contradicts "the Service has no `http` port" in the unit 10 amendment above. Not fixed here (product bug). A fix needs the installer to take over the stale field (apply with the old manager's ownership dropped, or delete the `http` port in a migration step) and a test that starts from an `ichor init`-made Service.
- Remaining detail: `docs/guides/users-and-login.md` names the Service `ichor-ui-service` in the `BackendTLSPolicy` recipe, while an `ichor init` install names it `ichor-ichor-ui-service`. I used the real name and the real Service DNS name in the policy `hostname` (the leaf has both as SANs).

### T2, Gateway API leftover: passed, with a better explanation of the earlier flip

Envoy Gateway v1.9.2 (`oci://docker.io/envoyproxy/gateway-helm`) installed only in the inner cluster; Gateway (HTTP :80, `listeners[].hostname`), HTTPRoute to `ichor-ichor-ui-service:8443`, and the guide's `BackendTLSPolicy` (ConfigMap `ichor-ui-ca`, `hostname: ichor-ichor-ui-service.ichor-system.svc`). Every request had `-m 4` and went to the Envoy Service from a pod in the cluster.

- Policy `Accepted=True`, `ResolvedRefs=True`; 200 for `/readyz` from the 16th second after the apply (the Envoy pod needed that long).
- A ConfigMap `wrong-ca` (another self-signed CA) swapped into `caCertificateRefs`: the policy stayed `Accepted`/`ResolvedRefs` True. The first test showed a single 503 after 5 s and then 200 again, and 200 stayed for 100 s more (checked in Envoy's admin `config_dump`: the SDS secret `ichor-ui/ichor-system-ca` did contain the wrong CA, updated at the time of the change). Cause: Envoy keeps its pooled upstream TLS connection, which was opened with the old CA and is reused by the steady stream of test requests; only new upstream connections see the new CA. That is the "200 for a wrong CA after 30 s, 503 later" of the earlier run.
- After a rollout restart of the Envoy Deployment (new upstream connections), the wrong CA gave 503 on every request (9 requests over 42 s).
- Back to `ichor-ui-ca`: 200 on the first poll, 1 s after the patch (09:22:56 patch, 09:22:57 first 200), and 200 on 6 more polls over 26 s. The "still 503 at 45 s" of the first run is therefore not reproduced; the likely cause there was a stale connection or the pod restart race, not the policy. Conclusion: BackendTLSPolicy changes apply within seconds, but a wrong CA only shows up on new upstream connections; test a negative case after restarting the proxy or with no steady traffic.

No product defect in T2.

## Amendment 2026-10-30: the stale `http` Service port of an `ichor init` install is removed

Fixes the failed expectation in "Unit 10, plaintext install upgraded" above. The cause was general: `ichor init` created every object of Ichor's manifest with `Create` (field manager `ichor`, operation `Update`), and the installer's server-side apply as `ichor-installer` cannot remove a list item (or any field) another manager owns, so anything a release drops from the manifest stayed on an init-made install.

- **Installer.** Before it applies any object other than a Deployment, `applyOne` reads the live object and, if manager `ichor` holds `Update` entries, hands those fields to `ichor-installer` with `k8s.io/client-go/util/csaupgrade` (`UpgradeManagedFieldsPatch`, the documented client-side to server-side upgrade; the patch guards on the resourceVersion) and logs it. The apply then owns the whole object it knew and drops what the manifest no longer has: the `http` 8080 port of the ichor-ui Service, and equally a dropped field on a CRD, ClusterRole, ConfigMap or CiliumNetworkPolicy. Fields set by other managers (an operator's `kubectl edit` port) are untouched. Deployments need nothing: the installer replaces them whole.
- **Objects affected** (all objects `ichor init` creates from the manifest except Deployments): in practice only the Service had a dropped item on this upgrade; the others had no fields removed, but the same hand-over now protects them on later releases.
- **`ichor init`.** New installs create every non-Deployment object with a server-side apply as `ichor-installer` (a Deployment is still created, with that field manager). An existing object is still skipped, so re-running init is unchanged. Init's own later `Update`s (the internal port, the Deployment args) remain manager `ichor`, and the next upgrade hands them over like any older install.
- **Tests.** `TestApplyOneRemovesAPortOwnedByAnOlderInit` (envtest): a Service created as `ichor` with an `http` 8080 port and an operator port `extra` added by `kubectl-edit`; after `applyOne` of a manifest with `https` only, the ports are `extra` and `https`.
- **Docs.** `docs/guides/users-and-login.md` named the Service `ichor-ui-service` where an `ichor init` install has `ichor-ichor-ui-service`; the certificate-names bullet is corrected.

## Amendment (ADR 0107)

`spec.reclaim.enabled`, the `ichorinstallations/reclaim` subresource, the confirmation annotation and the Settings reclaim card are gone: the field is deprecated and ignored, and "Reset and add" is decided by the `reclaim` verb on `machinereclaims` alone.

## Amendment (2026-10-30): why the upgraded ichor-ui waited about 2.5 minutes

The Secrets are issued by the controller leader (`IssuerTrustReconciler`, event driven, it runs once at start). After an upgrade the new controller pod was not the leader: `LeaseDuration` is 120 s (ADR 0063) and the old pod did not release the lease when it stopped, so the new one waited out the lease before issuing `ichor-oidc-ca`, `ichor-oidc-tls` and `ichor-ui-ca`. That is the 00:13:30 to 00:16:00 gap. The manager now hands the Lease over itself after a plain voluntary stop (ADR 0063 amendment of 2026-10-30), so an upgrade hands over at once; after it the remaining wait is the kubelet copying the new Secrets into ichor-ui's volume, up to about a minute (an optional Secret volume is refreshed on the kubelet sync period). A crash, or a stop that goes into the etcd restore tail, still waits out the lease.

## Hardware result 2026-10-30 (rerun at c549005e): plaintext upgrade and lease hand-over

Cluster: `ichor init` 1 control plane + 1 worker (prefix `ichor-e2e-z`, Talos v1.14.3, Cilium geneve, images from an in-cluster registry through the `ghcr.io` mirror). Old build `v0.0.1850000000-g2b67caf6` (commit `2b67caf6`, built in its own worktree, `ichor init` with its CLI, no `--ui-url`, users off), new build `v0.0.1850000001-gc549005e` (this tree) installed with `ichor upgrade ichor ... --local` from the new CLI. No Go tests were run; everything by hand.

- **Before.** Service `ichor-ichor-ui-service` had only `http` 8080, ichor-ui `--addr=:8080`; from a pod in the inner cluster `http://...:8080/readyz` gave 200 and `/` gave 200 `text/html`.
- **Upgrade passed.** The operation took 3 min 30 s (02:54:21 to 02:57:50 UTC, Job complete 02:58:25). The Job logged `NOTICE: ichor-ui no longer serves plain HTTP (it did on :8080); ...` and `handing the fields ichor init wrote on Service ichor-ichor-ui-service over to ichor-installer` (the same line for every other object `ichor init` wrote, including the three CiliumNetworkPolicies and the CRDs).
- **The earlier failure is fixed.** The Service has exactly one port, `https` 8443 (`kubectl get svc ... -o jsonpath={.spec.ports}`), and its EndpointSlice lists port 8443 only. ichor-ui runs `--addr=:8443`; the CiliumNetworkPolicy `ichor-ui` opens 8443 (world, cluster) and 8444 (kube-apiserver, remote-node). From a pod in the `default` namespace of the inner cluster: `https://...:8443/readyz --cacert <ichor-ui-ca>` gave 200; without the CA curl exited 60 (`unable to get local issuer certificate`); `http://...:8443/readyz` gave 400; `http://...:8080/readyz` now fails at once with curl exit 7 (connection refused; before, with the port still in the Service, it timed out). `ichor-ui-ca` exists as a Secret and as a ConfigMap. (A test pod in `ichor-system` cannot reach anything: the namespace is default-deny; use another namespace.)
- **Timing of that upgrade (old controller, no hand-over code).** New controller pod started 02:55:11; it logged `Attempting to acquire leader lease` at 02:55:21 and `Successfully acquired lease` at 02:57:50, 2 min 29 s later (the lease the old pod, a build without the hand-over, did not release); `Issuer internal CA created` and the serving certificate followed in the same second; the new ichor-ui pod (started 02:55:34) became Ready at 02:58:23. So 3 min 12 s from the new controller pod to a Ready ichor-ui, as in the first run: the fix lives in the stopping pod, so the upgrade away from a build that has it cannot show it.
- **Lease hand-over, second upgrade (build to build, both with the fix).** Built `v0.0.1850000002-gc549005e` from the same tree and ran `ichor upgrade ichor` again (operation done in 47 s, 03:00:47 to 03:01:34). The old controller (log followed from outside) logged `Stopping and waiting for ...` at 03:01:26 and no `Could not hand the leader lease over` error; the new pod (started 03:01:15) logged `Attempting to acquire leader lease` at 03:01:21 and `Successfully acquired lease` at 03:01:33, 7 s after the old pod began to stop (before: 2 min 29 s). The new ichor-ui pod was Ready at 03:01:27, 12 s after the controller pod started, but this number does not measure issuing: the Secrets from the first upgrade already existed, so the kubelet had them at once. There is no controller log line for a successful hand-over (only the error line on failure, `cmd/main.go`); the evidence is the missing error and the immediate acquire. The old pod's last line was `error received after stop sequence was engaged: leader election lost`, which is the stop sequence, not a failure of the hand-over.
- No failing tests (none run). Own mistakes: a first `curl` pod in `ichor-system` could not reach the Service (default-deny), moved to `default`.

## Amendment (2026-11-03): an operator's certificate does not stop in-cluster clients verifying ichor-ui by its Service name

Found on the UI demo: `ichor upgrade ichor` warned that ichor-ui's certificate could not be verified and computed the plan from the Kubernetes API. The demo supplies its own self-signed certificate for the load balancer's address (`init --ui-tls-cert`), so 8443 served only that certificate. The CLI dials the port-forward with the Service name (`<service>.<namespace>.svc`) and verifies against `ichor-ui-ca`, which is Ichor's CA: the operator's certificate has neither the name nor that CA. Ichor's own leaf, which has the name, stayed on disk beside the operator's pair (`/etc/ichor/ui-tls`, mounted by the chart and by the installer), but no handshake was ever answered with it.

Decision: ichor-ui serves the Ichor-made leaf to a handshake whose SNI is a Service DNS name (`*.svc` or `*.svc.cluster.local`) that the leaf covers, and the operator's pair to every other handshake: an external name, an address, or no SNI. The CLI and in-cluster clients keep verifying the Ichor leaf against `ichor-ui-ca`; no client gets a new trust anchor. The Ichor leaf names the Service as `<name>`, `<name>.<namespace>`, `<name>.<namespace>.svc` and `<name>.<namespace>.svc.cluster.local`, plus `localhost` and the external host (the names the controller already put there, now with the `<name>.<namespace>` form). Adding a SAN makes the controller reissue the leaf once.

Rejected: the CLI trusting the operator's certificate (it has no in-cluster way to learn its CA or its names, and a self-signed certificate for an address needs `--ca-file` and `--tls-server-name` on every command); publishing the operator's CA as `ichor-ui-ca` (the controller does not hold that CA and the ConfigMap is the Ichor CA by name); a second in-cluster port (the 8444 listener serves only the issuer's discovery and key set).

The fallback is unchanged in shape and stays for real failures: an unreachable ichor-ui, or a certificate that does not verify even by Service name (for example after a CA rotation the ConfigMap has not caught up with). The warning no longer appears for a healthy ichor-ui with an operator certificate.

Consequences: an edge that verifies the backend by the Service name still needs Ichor's CA (`ichor-ui-ca`), as before; an edge with an operator certificate verifies that certificate for its external name. The first `ichor upgrade` run with a CLI from this change still uses the old ichor-ui and so still warns; later commands are verified once the upgraded ichor-ui serves. Not yet verified on hardware: the unit tests cover the SNI selection (`cmd/ichor-ui`), the leaf names (`internal/controller`) and the CLI's verification and fallback (`internal/cli`).