0094. Ichor users, its own OIDC issuer, roles and per-user kubeconfigs

Status: Accepted · Date: 2026-10-27 · Design section: Trust, Secrets, UI backend, Deployment · Extends: 0058 · Touches: 0084, 0091, 0065 (override) Decided: Ichor keeps its own users as IchorUser CRs (namespace ichor-system, spec = display name, e-mail, roles[], disabled; never a password) with argon2id hashes in one Secret ichor-user-credentials, and ichor-ui becomes a minimal stateless OIDC issuer (hand-written on go-jose/v4, RS256, authorization code + PKCE for the browser and a loopback redirect for the CLI, 5 minute ID tokens, 12 hour non-rotating refresh tokens, signing keys kept and rotated by the controller in a Secret); the controller renders both Ichor's issuer (reached by the kube-apiserver through discoveryURL on a ClusterIP with an Ichor-made CA) and the existing external IdP as two jwt[] entries of Talos's KubeAuthenticationConfig document (Talos v1.14 contract and later), usernames ichor:<name> and oidc:<name>; permissions are plain Kubernetes RBAC: three built-in ClusterRoles ichor-viewer, ichor-operator, ichor-admin bound to the groups ichor:viewer|operator|admin, with new verbs for destructive operation starts, user administration and the trust configuration; a user's kubeconfig holds no secret and runs ichor auth token as a client-go exec plugin; no user ever receives a talosconfig, and the two paths that hand out Talos credentials (ADR 0091's CLI hand-off, ADR 0093's UI download) stay admin-only; the whole feature is off by default and the secrets-bundle admin certificate stays as override

Context

ADR 0058 lets ichor-ui verify an ID token from one external IdP and call Kubernetes as that user. It has no users of its own, so a site without an IdP has no way to give two people different rights, and nothing makes the cluster's kube-apiserver accept those people: the user's own kubectl still needs a certificate or a token someone minted by hand. The decision asked for: Ichor manages its own users and the external OIDC login keeps working beside them; permissions are roles; each user can get a kubeconfig with no secret in it; users never receive a talosconfig; one cluster only (self-managed, no hub mode).

Everything below was checked against the module cache: Talos pkg/machinery v1.14.3, k8s.io/apiserver v0.36.0, k8s.io/client-go v0.36.0, github.com/coreos/go-oidc/v3 v3.21.0, golang.org/x/crypto v0.58.0. Paths are relative to each module root.

Decision 1: users

CR. IchorUser (ichor.maesh.dev/v1alpha1, namespaced, only in ichor-system, scaffolded with kubebuilder create api). The name is the identity: lower-case DNS label, at most 63 characters, immutable, and it becomes sub and the Kubernetes user ichor:<name>. Spec: displayName, email (informational), roles []string (each ^[a-z][a-z0-9-]{0,30}$, at most 16), disabled bool. Status (written by the controller only): conditions Ready, PasswordSet. There is no password, hash, token or session field, so a user CR can live in Git next to the other CRs. A CR from Git without a password is a user who cannot log in (PasswordSet=False) until an admin sets one; that is the GitOps story, not a gap.

Hash. One Secret ichor-user-credentials in ichor-system, key = user name, value = JSON {"v":1,"hash":"$argon2id$v=19$m=65536,t=3,p=4$…","notBefore":"RFC3339","mustChange":bool}. argon2id with the parameters golang.org/x/crypto/argon2 itself gives as RFC 9106's second recommendation (argon2/argon2.go:29-33: time=3, memory=64 MiB, threads=4; IDKey at argon2.go:101). golang.org/x/crypto v0.58.0 is already in go.mod, as // indirect (go.mod:161); it becomes a direct requirement, nothing new to download. The PHC string carries the parameters, so a login with older parameters rehashes. 64 MiB per hash means ichor-ui admits at most two hashes at once (a semaphore; the rest wait), and a wrong or unknown user costs the same (verify against a dummy hash) so response time does not reveal which users exist. Minimum password length 12, maximum 128 bytes, no composition rules.

Who writes what. The UI backend and the CLI write the CR (as the caller, so Kubernetes RBAC decides) and the Secret entry (as ichor-ui's ServiceAccount, after a SelfSubjectAccessReview as the caller, ADR 0084's pattern). The controller never writes spec; it writes IchorUser.status, deletes Secret entries whose CR is gone, and owns the signing Secret (Decision 2). ichor-ui's Role gets get, update, patch on exactly the Secret named ichor-user-credentials (resourceNames); it cannot create other Secrets. A Secret entry changes through a JSON merge patch of one key, so two replicas do not overwrite each other.

Operations. UserService (Connect, proto/ichor/v1alpha1/user.proto): ListUsers, GetUser, CreateUser, UpdateUser (roles, disabled, display data), DeleteUser, SetPassword (admin, optional generated password returned once, sets mustChange), ChangeOwnPassword (old and new password; needs no RBAC, it is the user's own account), RevokeSessions (sets notBefore to now), GetMyKubeconfig (Decision 5), WhoAmI. RBAC for the admin ones: ichorusers verbs through the impersonated client; SetPassword and RevokeSessions additionally need verb update on subresource ichorusers/password. Changing a password sets notBefore, which kills every refresh token issued earlier. Disable is spec.disabled: true; nothing else is needed. Refusing to disable, delete or demote the last enabled admin is a guard in the RPC, not a security boundary (the override certificate exists).

First admin. ichor init --admin-user NAME (default admin; the flag --admin-password-file reads one instead) runs after Ichor is installed. It writes the IchorUser (role admin) and the Secret entry with mustChange:true, using the same package (pkg/ichorauth) ichor-ui uses, with the operator's own kubeconfig. A generated password (24 characters, crypto/rand) is printed once to stdout and stored nowhere; the first login forces a change. Without --ui-url (Decision 3) init creates nothing and users stay off.

Override. The secrets-bundle admin client certificate (O=system:masters, what ichor init writes into ~/.ichor/clusters/<name>/kubeconfig) and the Talos os:admin identity are untouched. They do not depend on any JWT authenticator, so a broken issuer, a stopped ichor-ui or a deleted user cannot lock the operator out of Kubernetes, and the controller dials Talos with the bundle, not through Kubernetes. The docs call this the override credential and say to keep it offline once users exist.

Decision 2: the issuer

Which component. ichor-ui, because it already authenticates browsers, is stateless, runs two replicas and has the login UI. It gets a second package internal/oidcissuer and two listeners: the existing one (port 8080) gains /.well-known/openid-configuration, /oidc/jwks, /oidc/authorize, /oidc/login, /oidc/token; a new HTTPS listener (port 8444) serves only the discovery document and the JWKS for the kube-apiserver (Decision 3). The login page is a route of the preactcn SPA (/login), not server-rendered HTML.

Library: hand-written on github.com/go-jose/go-jose/v4, not zitadel/oidc. go-jose v4.1.4 is already a direct requirement (go.mod:10) and go-oidc v3.21.0 (client side, unchanged) already uses it. zitadel/oidc v3 is the credible OP library, but it is not in go.mod or the module cache, and its OP is built on a Storage that persists authorization requests, codes, access tokens and refresh tokens and resolves them by lookup (CreateAuthRequest, AuthRequestByCode, SaveAuthCode, CreateAccessToken, TokenRequestByRefreshToken, RevokeToken, TerminateSession; docs: https://github.com/zitadel/oidc/blob/main/_autodocs/api-reference/op-storage.md). Behind two replicas that means a shared store, so every login and every five-minute token refresh would write to etcd, and the stateless backend of ADR 0058 would stop being stateless. The surface needed here is small and fixed (two clients, one flow, one refresh grant), so the hand-written issuer is stateless by design:

• authorization code = a JWS signed by the issuer key (typ ichor-code+jwt, 60 s) carrying client_id, redirect_uri, sub, nonce, code_challenge, audience; the token endpoint checks PKCE S256 (plain is refused) and the exact redirect URI. Single use is enforced per replica from an in-memory jti set only; a replay to the other replica within 60 s still needs the code_verifier, which only the client holds. This is a stated weakening of strict OAuth, accepted.
• the authorize request is carried to the login form as a signed 5 minute ticket, so no cookies and no server session exist and a cross-site POST has no ambient authority.
• refresh token = a JWS (typ ichor-refresh+jwt) with sub, client, iat (session start) and exp (absolute lifetime). It is valid while the user exists, is not disabled and iat >= notBefore. It is not rotated: nothing has to be stored, and concurrent kubectl processes cannot race each other into a revoked token.
• no implicit or hybrid flow, no password grant, no client secret, no dynamic registration, no userinfo, introspection, revocation or end-session endpoint (logout discards the tokens; "sign out everywhere" is RevokeSessions).
• the risk of hand-writing a security component is covered by tests that run the real consumers against it: go-oidc's NewProvider and Verify, and k8s.io/apiserver/plugin/pkg/authenticator/token/oidc.New (the code the kube-apiserver runs; oidc.go:269) in an httptest TLS server, plus negative tests (wrong audience, expired, tampered, replayed code, wrong PKCE verifier, unknown redirect URI, disabled user, iat < notBefore).

Clients (fixed, configured in the chart, not CRs). ichor-ui: public, redirect URI exactly <externalURL>/auth/callback. ichor-cli: public, redirect http://127.0.0.1:<any port>/callback (RFC 8252 section 7.3: the server MUST allow any port for loopback redirects; the literal 127.0.0.1, not localhost, section 8.3). Both use response_type=code, S256, state, nonce, scope openid (+ offline_access for a refresh token).

Audiences. The ID token's aud is chosen by the request: ichor-ui for the SPA, and for the CLI either ichor-kube (the one the kube-apiserver accepts) or ichor-ui, asked for by an audience parameter on the refresh-token request. The kube-apiserver trusts only ichor-kube, so the token that sits in the browser's sessionStorage (ADR 0058 point 6) is not a Kubernetes credential, which keeps ADR 0058's statement true.

Claims and prefixes. iss = the external URL, sub = user name, aud, exp, iat, groups = the user's roles (for example ["admin"]). The kube-apiserver maps sub with prefix ichor: and groups with prefix ichor:, giving user ichor:alice and group ichor:admin. An external IdP is mapped with prefix oidc: (ADR 0058's default), so the two cannot collide, and system: is blocked by a user validation rule on both. The webhook refuses an external prefix that equals ichor:, starts with it, or is empty.

TTLs. ID token 5 minutes (chart ichorUI.users.tokenTTL, 1 to 15 min). Refresh token absolute lifetime 12 hours (refreshLifetime, 1 hour to 30 days). The kube-apiserver does not check anything against Ichor after a token is issued, so the token TTL is the worst-case delay for "disable".

Disable, role change, password change. ichor-ui re-reads the IchorUser for every RPC (an in-memory cache of at most 10 s), refuses a disabled user, and impersonates with the roles from the CR, not from the token. So in the UI a disable or a role change takes effect within 10 s. For kubectl it takes effect at the next token refresh, at most 5 minutes later, because the token's groups claim is old until then and the refresh endpoint re-reads the CR and notBefore. That is "promptly" and is stated as a number rather than "immediately".

Signing keys. RSA-3072, RS256 only. RS256 is the one algorithm OIDC requires and the kube-apiserver authenticator defaults to (k8s.io/apiserver/plugin/pkg/authenticator/token/oidc/oidc.go:281-286); whether structured configuration widens that list is a claim in Talos's own doc comment (config/types/k8s/authentication.go:59-61) that was not verified in kube-apiserver source, so nothing here depends on it. Keys live in the Secret ichor-oidc-signing (keys.json: kid, PEM, createdAt, activeFrom, retireAfter). The controller (leader-elected, so one writer) creates the first key, adds a new key every 30 days, makes it the signing key 10 minutes after it is published (covering the kubelet's Secret-volume sync lag in ichor-ui), and drops a key 2 hours after the last token signed with it can have expired. ichor-ui mounts the Secret read-only and reloads on change; it never writes keys. The kube-apiserver does not need to be told about a rotation: go-oidc's key set caches keys in memory and re-fetches the JWKS when it sees an unknown kid (oidc/jwks.go:57-58, 168-174), so no machine-config change and no restart follow a rotation.

Decision 3: making the kube-apiserver trust both issuers

What Talos can carry (verified). In the v1.14 contract Talos has a typed document KubeAuthenticationConfig (config/types/k8s/authentication.go) whose configuration is the literal Kubernetes AuthenticationConfiguration, i.e. a list jwt[] of authenticators, each with its own issuer. Its doc comment says kube-apiserver refuses any --oidc-* flag when --authentication-config is set "and Talos always sets it", and KubeAPIServerConfig.extraArgs rejects oidc-*, anonymous-auth and authentication-config (config/types/k8s/apiserver.go:178-180). Talos's generator adds the default document with an empty jwt: [] and anonymous access limited to /livez, /readyz, /healthz (config/generate/kubernetes.go:120, authentication.go:84-107). It is used only with the multi-document form: the legacy v1alpha1 shim returns UseAuthenticationConfig() == false (config/types/v1alpha1/v1alpha1_k8s_bridge.go:141-144), the modern document returns true (apiserver.go:293-296), and multi-document Kubernetes configuration is generated when the contract is greater than 1.13 (config/contract.go:282-285). So:

• contract v1.14 and later: multiple issuers, typed. This is the only supported mode for Ichor's issuer.
• earlier contracts: only the legacy --oidc-* flag set, i.e. exactly one issuer, no discoveryURL, no per-issuer CA. Ichor's issuer cannot be used there (the apiserver would need a CA file and a reachable external URL), and a cluster below v1.14 keeps ADR 0058 as it is. The webhook refuses spec.authentication unless versionContract >= v1.14.

How Ichor stays typed. KubeAuthenticationConfig is not on the mirror generator's kind allow-list (internal/tools/mirrorgen/main.go, roots) and its field is a meta.Unstructured, so no TalosConfigPatch can set it (ADR 0003). That is right: the document is owned by the renderer, as the control-plane metrics documents are (pkg/render/controlplanemetrics.go). The renderer builds k8s.io/apiserver/pkg/apis/apiserver/v1.AuthenticationConfiguration (a typed Go struct, types.go:56, already a dependency through the ADR 0003 overrides), validates it with the apiserver's own validation.ValidateAuthenticationConfiguration (pkg/apis/apiserver/validation/validation.go:47) before it ever reaches a node (an invalid file makes kube-apiserver fail to start, and this is the single document whose breakage affects every control plane), converts it through the existing mirrorToMachineryUnstructured helper and applies it as a strategic merge patch. The field is merge:"replace" (authentication.go:70), so the rendered configuration must contain the default anonymous block as well, or health probes lose anonymous access. No raw YAML is accepted anywhere.

CR. TalosCluster.spec.authentication (immutable-name fields, CEL-validated; changes to it need update on talosclusters/authentication, Decision 4):

• ichor: {enabled, url} — url is the external https URL of ichor-ui and is the issuer (iss).
• external[]: {issuerURL, audiences[], usernameClaim (email), groupsClaim (groups), usernamePrefix ("oidc:"), groupsPrefix ("oidc:"), certificateAuthority (PEM, public) } — at most 3. The ichor-ui ichorUI.oidc.* values (ADR 0058) are what the UI uses; this list is what the apiserver uses; ichor init and the chart docs say to keep the two equal, and the UI shows a warning when they differ. Each entry becomes one jwt[] element: issuer URL, audiences, claim mappings with the prefixes above, userValidationRules: [!user.username.startsWith('system:')] and !user.groups.exists(g, g.startsWith('system:')). Issuer URLs must be unique across entries (validation.go:61-70).

Reaching the issuer from a host-network kube-apiserver. The apiserver fetches discovery and JWKS from the issuer URL, but the spec allows discoveryURL for "the well-known and jwks endpoints hosted at a different location than the issuer (such as locally in the cluster)", with certificateAuthority to verify that connection, and requires the document's issuer to equal url (apiserver/v1/types.go:139-166; discoveryURL must be https and different from url, validation.go:149-154, 170). Ichor uses it:

• url = https://ichor.example.com (external, what browsers and the CLI use, and the iss of every token).
• discoveryURL = https://<ClusterIP of ichor-ui-service>:8444/.well-known/openid-configuration, served by the internal listener, whose discovery document says issuer = the external URL and jwks_uri = the internal address. The cluster does not need the external URL to be reachable from the control planes, nor DNS, nor a load balancer that itself runs on the cluster it authenticates.
• certificateAuthority = an Ichor-made CA (RSA, 10 years) kept by the controller in ichor-oidc-ca with a leaf (1 year, renewed at two thirds) in ichor-oidc-tls, SANs: the ClusterIP and ichor-ui-service.ichor-system.svc. The controller publishes the public parts (discoveryURL, the CA PEM) in the ConfigMap ichor-oidc-trust, which is a render input (ADR 0061 watches render inputs), so a changed CA or ClusterIP is a desired-state change like any other. A leaf renewal touches nothing in machine config. It is not the Kubernetes or Talos CA, so ADR 0063/0091 rotations do not interact with it.
• egressSelectorType stays unset (no custom dialer; types.go:190-200). That a control plane's host network can reach a Service ClusterIP is not verified (see "Not verified").

Bootstrap and when ichor-ui is down. The authenticator is built per entry asynchronously and polls every 10 s until discovery works; "this enables self-hosted providers, providers that run on top of Kubernetes itself" (oidc.go:423-430). The apiserver starts without the issuer, and until discovery succeeds, tokens of that issuer are rejected (oidc.go:412, 867) while certificate authentication is unaffected. Once running, keys are cached in memory and only re-fetched on an unknown kid (jwks.go:57-58), so with ichor-ui down, already-issued tokens keep working until they expire (5 minutes) and nothing new can be issued. After an apiserver restart with ichor-ui still down, Ichor-issuer tokens fail until ichor-ui is back, and the external IdP's entries are independent. First boot has no jwt entries: ichor init creates the cluster, installs Ichor, creates the admin, then sets spec.authentication (Decision 7).

Rollout. The rendered document changes for every control plane. pkg/fieldmeta has no rule for KubeAuthenticationConfig, so the default applies: reboot-required, and it is rolled out by the config operation (ADR 0080), one control plane at a time with etcd checks. Whether Talos v1.14.3 applies this document live (and so a rule {prefix: "KubeAuthenticationConfig", safe: true} like the one for KubeAPIServerConfig, fieldmeta.go:212) was not checked, because Talos controller source is not in pkg/machinery; the first implementation unit must read it and add the rule only if it is true. The kube-apiserver is assumed to restart on a change; the reload behaviour of --authentication-config is in k8s.io/kubernetes, not in the module cache, and was not verified. Adopt. A cluster adopted with non-empty jwt in its KubeAuthenticationConfig would have those entries replaced by the render. ADR 0075's adopt names the document as unsupported today; it must refuse (not drop) a non-empty jwt, and say to move the IdP into spec.authentication.external.

Decision 4: roles

No new binding CR. Roles are Kubernetes RBAC objects, so they are the same for the UI (impersonation, ADR 0058), for kubectl with the user's kubeconfig, and for a Git-managed setup. Three ClusterRoles ship in the chart and kustomize (config/rbac/ichor_*_role.yaml), each bound with a ClusterRoleBinding to the group of the same name (ichor-viewer to Group ichor:viewer, and so on); IchorUser.spec.roles names the groups a user is in, so a custom role is a ClusterRole plus a binding to ichor:<name>, and a one-off grant is a binding to User ichor:<name>. Role names viewer, operator, admin are reserved. ClusterRoleBindings, because the product uses one namespace and the roles also need nodes and pods; a site that wants less writes its own namespaced binding.

Destructive starts (new). A TalosOperation of kind shutdown, ca-rotation, etcd-restore or node with action reset or shutdown needs verb start on the subresource of that name, asked with a SelfSubjectAccessReview by OperationService.CreateOperation as the caller and by the validating webhook (a SubjectAccessReview on the request's user, the way the override webhook already gates a reset and an etcd restore, internal/webhook/v1alpha1/override_webhook.go) so a hand-written CR obeys it too. The controller's own identity is exempt, as in the override webhook. The override requirements of ADR 0065/0089 stay on top. The same webhook pattern gates edits to spec.authentication (update on talosclusters/authentication), because an operator who could repoint the IdP could break sign-in for everyone.

Not in any built-in role: secrets (so nobody gets the secrets bundle or the credentials Secret through Ichor's roles), impersonate, *. cluster-admin is not part of ichor-admin; a site that wants its Ichor admins to run arbitrary kubectl binds cluster-admin to Group ichor:admin itself (chart value ichorUI.users.clusterAdminRoles: [], empty by default). Doing so also lets them read the secrets bundle, which is the source of Talos credentials, so the statement "users never get Talos access" in Decision 6 holds for the built-in roles only.

Enforcement. Unchanged from ADR 0058, extended: the Authenticator accepts tokens from the local issuer (verified in process against its own keys, aud=ichor-ui, iss = the configured URL) and from the external one (go-oidc); it routes by the unverified iss, and each path then verifies fully. Both produce Identity{User, Groups} and the same impersonating client. The impersonate grant is rendered when either source is on; groups narrow to the built-in group names plus configured ones when only Ichor users are on.

Decision 5: kubeconfig issuance

No secret in the file. The kubeconfig has the API server URL, the cluster CA (public), and a user with an exec block: apiVersion: client.authentication.k8s.io/v1 (in client-go's accepted list, plugin/pkg/client/auth/exec/exec.go:76-79), command: ichor, args: [auth, token, --issuer, <url>, --client-id, ichor-cli, --audience, ichor-kube], interactiveMode: IfAvailable (required for the v1 API; tools/clientcmd/api/types.go:260-270), provideClusterInfo: false. The plugin prints an ExecCredential whose status.token is the ID token and status.expirationTimestamp is its exp; client-go caches it until then (exec.go:386-390, 496-497) and drops it on a 401 (exec.go:378).

ichor auth token. (1) a cached ID token with 30 s left: print it. (2) a cached refresh token: call the token endpoint, store the new ID token, print. (3) otherwise log in: listen on 127.0.0.1:<random>, open the authorize URL (S256 PKCE, state, nonce, offline_access), exchange the code, store, print. Diagnostics go to stderr; stdout carries only the ExecCredential. A file lock on the cache serialises concurrent kubectl runs. Cache: ~/.ichor/auth/<sha256(issuer,client)>.json, directory 0700, file 0600, written atomically by bundlefile.go's writer; it holds a refresh token, so it is a credential at rest on that user's machine, as any CLI login cache is, and the kubeconfig itself stays secret-free. ichor auth login, ichor auth logout (deletes the cache entry) and ichor auth status complete the set. Not built: device authorization grant, so a headless box without a browser cannot log in (it can copy a refresh cache from another machine); the password is never handled by the CLI, only by the browser's login form.

Getting the file. ichor auth kubeconfig [--issuer URL] [-o FILE]: logs in if needed, calls UserService.GetMyKubeconfig on ichor-ui (authenticated with the aud=ichor-ui token), which fills the server URL from TalosCluster.spec.endpoint and the cluster CA from the bundle (public certificate only) and returns the YAML; the UI has "Download my kubeconfig" on the account menu calling the same RPC. It requires ichor-viewer or better, and never contains anything but public data. After a Kubernetes CA rotation (ADR 0091) an old file has the old CA and fails TLS until it is fetched again; the rotation plan text says so.

CLI to ichor-ui. With the Ichor issuer the CLI can now present a token to the backend (ichor auth token --audience ichor-ui), which is the "flag that would carry an ID token" ADR 0058 point 5a left for later. It does this only when the current kubeconfig user is an Ichor exec-plugin user; it still never forwards kubeconfig credentials, and an external-IdP user keeps the in-process fallback.

Decision 6: Talos

Users never get a talosconfig, and nothing in this ADR issues one: no per-user Talos identity exists, so there is nothing to revoke when a user is disabled. Every Talos action stays a CR written by someone with the role for it and carried out by the controller, or one of ADR 0084's three read-only downloads, which the backend performs with a per-call certificate after Kubernetes RBAC allows the signed-in user. The remaining paths that put Talos credentials in someone's hands:

• ichor init writes the operator's kubeconfig and talosconfig locally and ~/.ichor/clusters/<name>/ holds the bundle (the override credentials; unchanged).
• ichor cluster credentials, bundle-export and rotate-ca (ADR 0091) read the bundle Secret with the operator's own kubeconfig. They stay admin-only without a new gate because no built-in role can get secrets; only override or cluster-admin holders can run them.
• The UI download ADR 0093 builds is new surface: it hands an os:admin identity to a browser. This ADR requires of it: verb download on a talosclusters subresource (placeholder credentials) held by ichor-admin only; a Warning Event written before any byte is read and a refusal if the Event cannot be written (ADR 0084's rule); a short-lived certificate (24 hours) rather than ADR 0091's one year; and a confirm dialog that says the file is the cluster's root identity and is not attributable to a person. Where ADR 0093 differs, the differing point is for that ADR to change or to supersede this paragraph. The CLI's and the backend's Talos code paths are unchanged.

Decision 7: ADR 0058's flags, migration, defaults

ichorUI.oidc.* and the --oidc-* flags keep their meaning and defaults; an install that uses neither feature is byte-for-byte as before apart from the new CRD (ichorusers). New values: ichorUI.users.enabled (default false), .externalURL (required when enabled, https), .tokenTTL, .refreshLifetime, .clusterAdminRoles, and ichorUI.service.type (ClusterIP by default; exposing ichor-ui over https is the operator's job through an Ingress, Gateway or LoadBalancer, or ichorUI.tls.secretName to terminate TLS in ichor-ui; Ichor installs no ingress controller). Built-in users are off by default because the feature needs three things a default install does not have: an https URL reachable by browsers and CLIs, a Talos v1.14 contract, and a config rollout on the control planes. ichor init --ui-url https://… turns it on: it sets ichorUI.users.*, creates the first admin, sets spec.authentication.ichor and prints that ichor config apply rolls the control planes. With both ichorUI.oidc.enabled and users.enabled, /auth/config returns providers: [{id,issuer,clientId}] and keeps the old enabled/issuer/clientId keys, so an older CLI (oidcEnabled, internal/cli/api.go:379) still sees "enabled" and the SPA shows both sign-in choices. With neither, ADR 0058 point 4's warning and banner remain. Upgrading from an OIDC-off install changes nothing until users.enabled is set; turning it on makes every RPC require a token, as turning on OIDC does, so the install notes say to create the first admin before the switch.

Not built

Hub mode or several clusters; a device-code login; a user directory (LDAP), SSO session cookie, MFA, password reset by e-mail, lockout state shared between replicas (failed-login back-off is per replica and in memory: five failures per user and per source IP back off exponentially, so the effective limit is twice that with two replicas); per-token revocation; a refresh-token rotation; group nesting; an Ichor-run IdP for other applications; Ichor installing an ingress; per-user Talos identities; hardware-checked behaviour (below).

• (Built; see "Amendment: the apiserver restart rolls out one control plane at a time" below.) An apiserver-restart rollout for KubeAuthenticationConfig.

Not verified

• (Resolved by the hardware amendment below.) That a host-network kube-apiserver on a Talos control plane reaches a Service ClusterIP (kube-proxy or Cilium on the node). The design relies on it for discoveryURL; the first e2e must read the authenticator's state, and if it fails the fall-back is the external URL for discovery too.
• (Resolved by the hardware amendment below.) Whether a stopped ichor-ui (authenticator uninitialised, oidc.go:412) can ever make the kube-apiserver's /readyz fail. HealthCheck() exists on the authenticator (oidc.go:974) but its caller is in k8s.io/kubernetes, which is not in the module cache. If it does, the load balancer pulls the control plane out when ichor-ui is down; the e2e must stop ichor-ui and read /readyz.
• Whether kube-apiserver accepts algorithms other than RS256 under structured configuration (irrelevant here; RS256 is used).
• ADR 0093's names and certificate lifetime (Decision 6); it was not on this branch.

Implementation units

Each is one PR, with the listed tests, and scaffolded with the tools named in CLAUDE.md. Order: 1 first; 2, 3, 6 and 9 after 1 and in parallel; 4 after 2 and 3; 5 after 1 and 3; 7 after 4; 8 after 4 and 5; 10 after 7 and 8; 11 last.

1. pkg/ichorauth and the IchorUser API. kubebuilder create api for IchorUser (api/v1alpha1/ichoruser_types.go, CEL for names and roles, samples, CRD in config/crd and the chart); pkg/ichorauth (argon2id hash/verify with a dummy-hash path and rehash, PHC encode/decode, credentials-Secret entry type, password policy, token and ticket claim structs). Tests: hash round-trip, parameter upgrade, constant-time shape, envtest for the CRD validation.
2. Issuer core internal/oidcissuer: discovery (external and internal variants), JWKS, authorize/login ticket, code, token (code and refresh grants), key loading and rotation-aware signing, per-replica code jti set and login back-off. Tests: the go-oidc client and the real k8s.io/apiserver/.../oidc.New authenticator against an httptest TLS server, plus the negative list in Decision 2. Depends on 1.
3. Controller: signing keys and internal TLS (internal/controller/issuer_trust.go): Secrets ichor-oidc-signing, ichor-oidc-ca, ichor-oidc-tls, ConfigMap ichor-oidc-trust, rotation timeline, IchorUser.status, credentials-Secret garbage collection, RBAC markers and chart Roles (resourceNames), metrics (issuer certificate expiry is exported as a metric only; no TalosCluster condition, because both certificates renew themselves). Tests: envtest reconcile, fake clock for rotation and renewal, ClusterIP change reissues the leaf.
4. ichor-ui integration: new listeners and flags, Authenticator for two sources, 10 s IchorUser cache, UserService (RBAC-checked with SelfSubjectAccessReview, Secret writes through the SA), /auth/config providers, chart values and ichor-ui role changes (ichorusers get, the credentials Secret by name, Secret volumes). Extends rbac_enforced_test.go so the generated role is the only thing under test. Depends on 2, 3.
5. Render and webhook (pkg/render/authentication.go, api/v1alpha1/taloscluster_types.go, internal/webhook): spec.authentication, the typed AuthenticationConfiguration builder, ValidateAuthenticationConfiguration, the contract gate, the prefix rules, update on talosclusters/authentication, the start check on talosoperations/* in the webhook and in OperationService.CreateOperation, fieldmeta classification (after reading Talos source), adopt refusing non-empty jwt. Tests: render golden files per contract, envtest webhook with real roles (the shape of override_reset_test.go), fieldmeta rule test. Depends on 1, 3.
6. Roles: config/rbac/ichor_{viewer,operator,admin}_role.yaml, chart templates, bindings to the three groups, ichorUI.users.clusterAdminRoles; dist/chart/chart_test.go pins that no role contains secrets, impersonate or *, and that the defaults render nothing. Independent of the rest except the names in Decision 4.
7. CLI ichor auth {token,login,logout,status,kubeconfig} and ichor user {list,create,update,delete,set-password,revoke-sessions}: loopback + PKCE, cache and lock, ExecCredential output, bearer to ichor-ui when the user is an exec-plugin user (internal/cli/api.go). Tests: fake issuer, golden ExecCredential, 0600/0700 checks, no token in logs. Depends on 4.
8. UI: login route, account menu (change password, download kubeconfig), Users page (list, create, roles, disable, reset password), both sign-in choices from /auth/config; preactcn components only, scaffolded with npx shadcn@latest add @preactcn/…; English and Japanese strings (ADR 0064). Tests: vitest for the PKCE helper and the login state machine. Depends on 4, 5 for the final shape.
9. ichor init bootstrap (internal/cli/init_ichor.go): --ui-url, --admin-user, --admin-password-file, first admin with mustChange, spec.authentication.ichor. Tests: init_order_test.go-style ordering, the password is never written to a file or a log (init_leak_test.go pattern). Depends on 1 and the CR in 5.
10. E2E: Talos v1.14 cluster, users on, log in with the CLI, kubectl as an operator and as a viewer, disable and see the 5 minute bound, stop ichor-ui and check /readyz and cached-key behaviour, restart an apiserver with ichor-ui down, rotate a signing key without restart, a rollout through ichor config apply. This is where the "Not verified" items are decided. Depends on 7, 8.
11. Docs: architecture.md "Trust" (replace "OIDC login... comes later" with this), "Deployment", the Amendment in ADR 0058 (done with this ADR), the user guide for exposing ichor-ui over https.

Consequences

• A compromised ichor-ui pod can now read password hashes and the signing keys and so mint a token for any user, including an ichor:admin that the apiserver accepts. ADR 0058 already gives that pod impersonate on any non-system: user, so the class of risk is the same, but the pod is now also the root of a login system. Two replicas, resourceNames on every Secret rule, no secrets list, and short tokens bound it; they do not remove it.
• The issuer needs an https URL for browsers and CLIs that Ichor does not provide. A lab without one stays on ADR 0058 or on certificates.
• The controller gains Secret writes in ichor-system for four named objects and a reconciler; the Talos config for control planes gains one more rendered document, and every change to it is a control-plane rollout.
• The ichor-operator role can start the routine kinds of TalosOperation (upgrades, config, CNI, reboot, service restart) but not shutdown, CA rotation, restore, reset: those are separate, named, auditable grants.
• Kubeconfigs downloaded before a Kubernetes CA rotation must be fetched again.

Amendments

• Unit 2: the audience parameter (only ichor-kube or ichor-ui; ichor-ui client only ichor-ui) is accepted on the authorize request and on the refresh request. It is carried in the code and in the refresh token, so a refresh without the parameter keeps the audience chosen at login; absent everywhere, ichor-ui client gets ichor-ui and ichor-cli gets ichor-kube. A refresh grant returns no new refresh token (not rotated).

• Unit 2: the kube-apiserver authenticator uses go-oidc v2.5.0, not v3 as the ADR cites for rotation: it refetches the JWKS on an unknown kid only after the cache expiry given by the response headers. Discovery and JWKS are therefore served with Cache-Control: no-cache (no positive max-age), which makes rotation pickup immediate; a test pins it.

• Unit 2/3 integration: the issuer reads the controller's keys.json ({"v":1,"keys":[...]}, internal/oidctrust) through the shared oidctrust.ParseKeys, Signing and Published; it has no format of its own. Keys are PKCS#8 only. The retire boundary is the controller's: a key is dropped at retireAfter exactly, not after it. A test feeds the issuer a document made by the controller's own rotation code.

• Unit 7: the ichor user commands and ichor auth status|kubeconfig do not use the port-forward or the in-process fallback of ADR 0058/0047. With users on, ichor-ui refuses an unauthenticated tunnel, and ichor auth kubeconfig has to work for a person with no cluster access yet. They call UserService on ichor-ui at --issuer (the external URL is ichor-ui's own URL) over https with a bearer ID token for audience ichor-ui from the login cache, and refuse plain http except to the loopback. -o FILE of the ADR text is --out FILE (-o is the global output format). The --audience default of ichor auth login|token is ichor-kube.

• Unit 7: --ca-file (env ICHOR_CA_FILE) adds a PEM CA to the system roots for the issuer and ichor-ui connections. GetMyKubeconfigRequest.ca_file makes the server add --ca-file <path> to the exec args; unset adds nothing.

• Unit 9: ichor init installs from the embedded kustomize manifest, not the chart, so --ui-url edits the ichor-ui Deployment (flags, internal port, optional signing and tls volumes) and Service in place and creates the empty ichor-user-credentials Secret, in the order Secret, first admin, switch, then sets spec.authentication.ichor by merge patch once the cluster exists. --admin-password-file is not built; the password is generated and printed once to stderr, also when a later step fails.

• Unit 9: ichor user create NAME --role admin --generate --bootstrap is the first admin on an existing install: the caller's own kubeconfig writes the IchorUser and the Secret entry (merge patch of one key, hashing with pkg/ichorauth). It refuses an existing user and a missing Secret.

Amendment: Hardware results

A spike on real hardware (Talos v1.14.3, Kubernetes v1.37.1, one control plane; the change was talosctl patch machineconfig --mode=auto, with nginx as the issuer on ClusterIP 10.251.0.50:8443) settled the two "Not verified" items about Decision 3 and replaces the reboot-required default of its "Rollout" paragraph.

1. Applying KubeAuthenticationConfig needs no reboot, but restarts the kube-apiserver. --mode=auto answered "Applied configuration without a reboot" and the boot ID stayed the same. Talos re-renders the kube-apiserver static pod and the apiserver restarts on every change; with a single control plane /readyz was down for 15 to 45 s per change. pkg/fieldmeta therefore has {prefix: "KubeAuthenticationConfig", safe: true}. The intended rollout is an apiserver restart, not a reboot: one control plane at a time, wait for that node's /readyz before the next, no drain. The machine controller today applies every safe change to every machine at once, and the config operation (ADR 0080) only rolls reboot-required changes, so the ordering and the /readyz wait were not built at this point; the amendment below builds them, and KubeAuthenticationConfig is no longer safe but the new APIServerRestart class.
2. Talos does not validate the document. An invalid one was accepted ("applied without a reboot") and then crash-looped kube-apiserver for about 6 minutes, which also put kube-controller-manager into CrashLoopBackOff. The trigger was two audiences without audienceMatchPolicy: jwt[0].issuer.audienceMatchPolicy: Invalid value: "": audienceMatchPolicy must be MatchAny for multiple audiences. So validation.ValidateAuthenticationConfiguration is mandatory before rendering and any error stops the render, and the renderer always sets audienceMatchPolicy: MatchAny when an issuer has more than one audience.
3. Verified, no change needed. A ClusterIP plus discoveryURL plus an Ichor CA with an IP SAN works from the host-network kube-apiserver; /readyz stays ok while the issuer is down, before and after an apiserver restart; two jwt[] issuers work together; an unknown kid makes the apiserver re-fetch the JWKS at once (about 3 s).

Amendment: the apiserver restart rolls out one control plane at a time

Replaces the "Not built" line about the apiserver-restart rollout. pkg/fieldmeta has a third class beside safe and reboot-required, APIServerRestart (ADR 0080's amendment has the whole design and the audit of which fields belong to it). KubeAuthenticationConfig is in it, so a change to spec.authentication is no longer applied by the machine controller to every control plane at once: the machines report Ready=False/APIServerRestartRequired and ichor config apply (or the cluster page) rolls it, one control plane at a time with no drain and no reboot, waiting for that node's kube-apiserver /readyz before the next. ichor init --ui-url prints that command, as before. The plan says "Restarts the Kubernetes API server on each control plane, one at a time; no reboot." On a cluster with one control plane the plan also warns that the API is unavailable while it restarts. Not verified on hardware.

Amendment: must-change is enforced server-side; the kubeconfig server is configurable

Hardware testing found that mustChange only stopped the SPA. Now AuthInterceptor refuses every RPC except WhoAmI and ChangeOwnPassword with PermissionDenied "change your password first" for an Ichor user whose credentials entry has mustChange (read through UserCache.MustChange, cached at most 10 s like the IchorUser, and dropped on this replica when the password is written; a read failure refuses the token). The issuer, via oidcissuer.User.MustChange, issues no ichor-kube token (code or refresh) to such a user and says why; ichor-ui tokens still work, so the user can change the password. GetMyKubeconfig used the cluster spec.endpoint, which is a node address, not what users reach through a load balancer: ichorUI.users.kubeAPIServerURL (--kube-api-server-url) replaces the server, and ichorUI.users.kubeAPITLSServerName (--kube-api-tls-server-name) sets tls-server-name when that host is not in the apiserver certificate SANs. Both are empty by default, which keeps the old behaviour.

Amendment (names): object names differ by install path (kustomize namePrefix: ichor- gives ichor-ichor-ui and ichor-ichor-ui-service), so ichor init and the controller find the ichor-ui Deployment and Service by the label app.kubernetes.io/component=ui (--ui-service-name stays as an override), init also adds --user-issuer and --oidc-refresh-lifetime to the controller before creating the admin, and gives ichor-ui --internal-jwks-uri from the Service ClusterIP instead of the kubelet *_SERVICE_HOST variable, whose name follows the Service name.

Amendment: accepted after the hardware run

Amendment (2026-10-28): verified end to end on hardware; ichor init --ui-url works as shipped, and must-change, roles, disable, issuer-down and the one-at-a-time apiserver rollout were each checked, so Status is Accepted.

Amendment (CLI bearer): when ichor-ui's /auth/config says auth is required, the CLI sends Authorization: Bearer <ID token> on its Connect calls through the port-forward, using the cached ichor-ui-audience login for one of the advertised issuers (refreshed with the cached refresh token when near expiry, never printed); with none cached it keeps the in-process fallback and prints ichor auth login --issuer <url> --audience ichor-ui. A TLS verification failure now says so in the fallback line. ichor auth logout removes every cached audience for the issuer unless --audience is given.

Amendment (events): the operation page's activity log lists Events as the signed-in user, so the namespaced ichor-user-access Role (not the cluster-wide roles) also grants get list watch on events (core and events.k8s.io) in ichor-system; "no events" in the viewer row means no cluster-wide events.

Amendment (ClusterIP changes): a deleted and recreated ichor-ui Service gets a new ClusterIP. Everything on the controller side already followed: IssuerTrustReconciler watches the Service and reissues only the leaf (IP SAN) and the ichor-oidc-trust discoveryURL, and the machine reconciler maps that ConfigMap to every machine of an Ichor-issuer cluster, so the apiserver config re-renders through the normal rollout. What went stale was the internal discovery document: ichor init pinned --internal-jwks-uri to the ClusterIP in the ichor-ui Deployment, so the apiserver was sent to the old address for the JWKS. Decision: ichor-ui now advertises the jwks_uri on the host the caller connected to (oidcissuer.Config.InternalJWKSFromHost), and init no longer passes --internal-jwks-uri (a re-run removes it from an existing Deployment; the flag stays as an explicit override). The apiserver reaches the internal listener only through an address in the serving certificate SAN, so that host is always one the controller has issued for. A fixed ClusterIP was rejected: it needs a chosen free address in the Service CIDR on every install path and does not help the Service-recreated-by-someone-else case.

Amendment (2026-10-29): in-page sign-in. The SPA shows one card, "Sign in to Ichor", with the user name and password form and, when the external provider of ADR 0058 is also configured, a "Sign in with your identity provider" button below a separator; there is no provider-choice screen and no page transition for the password sign-in. GET /oidc/authorize answers {"ticket": "..."} (200) instead of the 302 to /login when the request carries Accept: application/json; every check is the same as on the redirect path (client, exact redirect URI, S256 PKCE, state, nonce, audience), and a failure is a 400 JSON error with no redirect. The SPA then posts the ticket and credentials to /oidc/login, checks that the returned redirect_uri is its own /auth/callback with its own state, and exchanges the code at the token endpoint with fetch (verifier and nonce kept in memory, not in sessionStorage). Why: the second card and two navigations added nothing the PKCE and ticket checks needed, and the full-page redirects made the flow feel broken. The redirect path and the /login?ticket= page stay for ichor auth login (the CLI opens a browser at the issuer); that page renders the same card. The external IdP button still redirects, necessarily.

Amendment 2026-10-29: served certificates are reloaded on change. ichor-ui's HTTPS listener (--tls-cert-file), its internal listener on 8444 and the controller's metrics server follow their certificate files through controller-runtime's certwatcher (file events plus a 10 s poll), so a cert-manager renewal reaches new connections without a restart; a pair that fails to load keeps the previous certificate serving and is logged without key material. The webhook server already used certwatcher through its CertDir.

Amendment 2026-10-29 (ADR 0105): ichor-ui's main listener is HTTPS only, on 8443, with the operator's certificate or the Ichor leaf (ichor-oidc-tls, now issued for every install, not only with --user-issuer); the 8444 issuer listener is unchanged and keeps the Ichor leaf. The controller publishes the CA as ichor-ui-ca. Token and refresh lifetimes are runtime settings in IchorInstallation (spec.users), read per issued token; the flags are the fallback. The guide's "Expose ichor-ui over https" now carries the Ingress-NGINX, BackendTLSPolicy and passthrough recipes.

Amendment 2026-11-04: a password change keeps the browser tab signed in. ADR 0108 made a password change move notBefore and drop every refresh family, which ends the tab's own ID token too, and the SPA then called signOut(), so a user forced through the must-change screen (or choosing a new password in the account menu) had to sign in again by hand. Decision: after ChangeOwnPassword succeeds the SPA signs in again with the new password through the in-page flow above (signInAgain in ui/src/lib/login.ts: /auth/config for the ichor provider, the user name from the sub claim of the stored token, then authorize, login and token as for any sign-in), and reloads. No new server surface and no new token-minting path: the password is in memory for that one call, the new token is issued by the normal login, and notBefore and the families stay as ADR 0108 left them, so every other session and refresh-token family of the user is still revoked. The new login's iat is not before notBefore (whole seconds, compared with <), so it is accepted. If the silent sign-in fails for any reason the tab clears its token and shows the sign-in card, as before. Rejected: returning tokens from ChangeOwnPassword (a second minting path next to the issuer's) and sparing the caller's own family (the SPA has no refresh token, and the server could not tell which family is the caller's). The CLI has no change-own-password command (ichor user set-password is the administrator's and revokes the user's sessions by design), so it is unaffected.

Amendment 2026-11-04 (browser session): the SPA signs in with a refresh token, so a tab stays signed in up to the issuer's absolute refresh lifetime instead of 5 minutes. The cause of the short session was in the SPA alone: it asked for openid email profile groups, and the issuer issues a refresh token only for offline_access (codeGrant), so the tab held an ID token of DefaultTokenTTL (5 minutes) and nothing to renew it with. The issuer already allowed the refresh grant for ichor-ui (same rotation, reuse detection and absolute lifetime as the CLI's, ADR 0108 F7; audience stays ichor-ui, never ichor-kube), so it has no change; a test now pins the browser client's rotation.

Decision.

• The ichor sign-in (redirect and in-page) adds offline_access. The external IdP request is unchanged: its refresh behaviour is the provider's, not ours.
• The refresh token is kept in sessionStorage (ichor.refresh, with the token endpoint and client id) beside the ID token, not in localStorage, not in a cookie. Reasoning against XSS: the ID token is already there and is the bearer for every Ichor call, so a script that can read the storage can already act as the user for its 5 minutes; the refresh token extends that to the session lifetime, which is the new exposure. It is bounded by what is already in place: the CSP is script-src 'self' with no inline script, there is no HTML injection surface in the pages, the token is tab-scoped and gone when the tab closes, it is single-use with rotation and reuse detection (a thief's first use makes the owner's next refresh fail and kills the family, so the theft is visible as a sign-out), the lifetime is absolute (rotation never extends it), the audience is ichor-ui only so it is not a Kubernetes credential, and RevokeSessions, a password change and disabling the user end the family. In-memory only was rejected: a reload would sign the user in again, which is the problem being fixed. A cookie (httpOnly) would need a server-side session, which the issuer deliberately does not have, and would add CSRF surface.
• Refresh runs one at a time per tab (single flight), a minute before the ID token ends (a third of its life for tokens under 3 minutes, never more often than every 10 s), when a hidden tab becomes visible, before a request that would carry an ID token about to end, and once on a 401 for a unary call, which is retried once. A refresh the issuer refuses (400/401/403) ends the session: tokens dropped, tab back to the sign-in card. A network or 5xx failure keeps the session and retries every 15 s. A server stream that ends with "your sign-in expired" (the log view) is reopened from its resume token after a refresh, within its reconnect budget.
• Two tabs: each tab signs in separately and has its own family (sessionStorage is per tab, so no tokens are shared and no cross-tab protocol is needed). The one way two tabs end up with one family is the browser's "duplicate tab" and window.open without noopener, which copy sessionStorage; the first rotation would then make the copy present a retired token and revoke the family for both. A tab that has a family holds a Web Lock named after it (ichor-session-<fam>) for as long as its page lives; a copy cannot take it within 1.5 s, drops the copied tokens and shows the sign-in card, so the original keeps its session. Without Web Locks, or if the lock call fails for another reason, the tab goes ahead (the failure is then the reuse detection's: both tabs sign in again). Coordinating one family across tabs (BroadcastChannel plus a lock) was rejected as more code and more exposure (tokens on a channel) for no gain over a second sign-in.
• Password change: signInAgain goes through the normal flow, so it asks for offline_access and stores the new refresh token; the old family died with the change (ADR 0108), as before.
• No idle timeout. An open tab refreshes by itself, so the session ends at the absolute lifetime (ichorUI.users.refreshLifetime, runtime-settable, default 12h, 1h to 30d) and not before, like the CLI. An idle timeout would need either a server-side last-seen (the issuer is stateless apart from the family entry) or browser activity tracking that a hidden, throttled tab defeats; an administrator who wants a shorter exposure lowers refreshLifetime. Not added; the guide says so.
• Sign-out drops the tokens in the tab; the issuer has no revocation endpoint, so a copy taken earlier lives to the end of its lifetime unless the user's sessions are revoked. Same as ichor auth logout. A revocation endpoint (RFC 7009) would need a RefreshStore.Revoke and is not built.
• Superseded: the "A refresh grant returns no new refresh token (not rotated)" amendment of Unit 2 (rotation came with ADR 0108), and the sentence in 2026-11-04's password-change amendment that the SPA has no refresh token.
• Known limit: if the answer to a refresh is lost after the issuer rotated, the next try presents the retired token and the family is revoked: the tab signs in again. Rotation without a grace window makes this unavoidable.

Follow-up 2026-11-04 (sign-out revokes): the open issue above is closed. The issuer has POST /oidc/revoke, the RFC 7009 revocation endpoint for refresh tokens, advertised as revocation_endpoint in the external discovery document (not on the internal listener, which serves only discovery and the JWKS). Request: form token (required), client_id (ichor-ui or ichor-cli, public clients, no other authentication; same checks and the same per-source rate limit as /oidc/token), optional token_type_hint=refresh_token. Behaviour: a valid refresh token deletes its whole family (RefreshStore.Revoke, an optimistic-lock patch of the user's credentials entry like rotation), so that token, its predecessors and its successor all stop refreshing; a token that is not a valid refresh token (garbage, expired, an ID token, family already gone) answers 200 with no body, as RFC 7009 section 2.2 requires, so the endpoint is no oracle; a valid refresh token issued to another client is refused with 400 invalid_grant and not revoked (section 2.1); a missing or repeated token, or a body that is not a form, is 400 invalid_request; an unknown client_id is 401 invalid_client; token_type_hint=access_token is 400 unsupported_token_type, because ID tokens (the bearer) are not revocable and end within TokenTTL; a store failure is 500 server_error. Sign-out: the SPA's signOut posts the refresh token to the revocation endpoint it saved from discovery with the session (revocationEndpoint beside the token endpoint in ichor.refresh; keepalive, 3 s timeout) before it clears the tokens, and ichor auth logout posts every cached login's refresh token (discovering the endpoint, 10 s timeout) before it deletes the cache. Both are best effort: a failure, a refusal or an issuer without the endpoint only warns (CLI) or is ignored (SPA), and the sign-out completes, so the token then lasts to its lifetime as before. A tab signed in before this change has no stored endpoint and only discards its tokens. A tab that lost a family to the duplicate-tab check or a refused refresh has already cleared its tokens and posts nothing, so a copied tab never revokes the original's family.

Amendment 2026-11-05 (ADR 0129): the last-enabled-admin guard and the role-grant guard also hold for a kubectl write, through a validating webhook on ichorusers. System identities and system:masters (the override certificate) are exempt, so the override credential stays the recovery path described above.