Users and login

Ichor can keep its own users, so people can sign in to ichor-ui and get their own kubeconfig with no shared secret. This guide covers the steps. The design is, which builds for logins from an external identity provider.

Status: accepted and was verified on hardware. The steps below are what it decides, plus the CLI flags the implementation added (--ca-file, --out, --bootstrap).

Users are off by default. Nothing here applies until you turn them on.

Without any sign-in

With neither Ichor users nor an external OIDC provider (ichorUI.oidc), ichor-ui has no way to tell callers apart. It then serves read-only calls only and refuses every change with "ichor-ui has no sign-in configured"; the UI shows a "Read-only" banner saying so. Downloads need an identity and are refused too.

The old open mode, where anyone who can reach ichor-ui writes with its ServiceAccount's permissions and is audited as ichor-ui, still exists but you have to ask for it: --allow-unauthenticated on ichor-ui, ichorUI.allowUnauthenticated=true in the chart. The flag does nothing once users or OIDC are on.

• ichor init without --ui-url keeps its behaviour: it starts ichor-ui with the flag and prints a warning at the end. --ui-url turns users on and drops the flag.
• ichor-installer upgrading an install that ran with no sign-in sets the flag for you and prints a NOTICE, so nothing stops working. Turn users or OIDC on, and the flag goes away at the next upgrade.
• A Helm upgrade does not guess: set ichorUI.allowUnauthenticated=true to keep writing, or enable ichorUI.users or ichorUI.oidc. helm prints a note when no sign-in is configured.

Before you start

Turning users on needs three things:

• An https URL that browsers and the ichor CLI can reach. Ichor does not provide one.
• A Talos v1.14 contract or later. Older contracts cannot carry the issuer configuration.
• A rollout of the control planes with ichor config apply. The kube-apiserver reads its login settings from the machine config.

Turn users on

On a new cluster, pass the URL to ichor init:

ichor init --ui-url https://ichor.example.com --admin-user admin ...

The first install's version contract is the release set's (v1.14 in this build), so --version-contract accepts only that value on ichor init; init refuses any other before it touches the node. ichor init installs Ichor from the manifest in the binary, not from the Helm chart, so it applies in place what ichorUI.users.enabled and ichorUI.users.externalURL render: the flags, the internal port and the key volumes of the ichor-ui Deployment, the internal port of its Service, and the empty ichor-user-credentials Secret. It then creates the first admin (--admin-user, default admin, role admin) and sets spec.authentication.ichor on the TalosCluster. The order is: Secret, admin, then the switch, so nobody is locked out. At the end it prints, to stderr, the Helm values that match and the generated password of the admin. The password is printed once, also when a later step failed, and is stored nowhere; the first login forces a change. Save it when it is printed. The kube-apiserver login settings reach the control planes with ichor config apply, which restarts the kube-apiserver on each control plane in turn (see the end of this guide). Init retries transient API errors (such as an etcd timeout right after the API server comes up) for a few minutes before it gives up.

ichor init takes four more flags, all of which need --ui-url. They are checked before anything is written, and each has a chart value:

ichor-ui reloads its certificate files when they change, within about ten seconds of the file changing, without a restart: a kubernetes.io/tls Secret that cert-manager renews works as is. The kubelet copies a changed Secret into the pod on its own schedule, so the new certificate reached the wire 25 to 46 seconds after cert-manager wrote the Secret in a test (cert-manager v1.21.3, pod restart count 0). If a new pair is invalid, ichor-ui keeps serving the old certificate and logs the error.

ichor-ui serves HTTPS only, on port 8443. Without --ui-tls-secret it uses a leaf from the Ichor serving CA. A browser does not trust that leaf; see "Expose ichor-ui over https". Making ichor-ui reachable at the --ui-url (a LoadBalancer, an Ingress or a Gateway) is your job: init installs none of them.

On an existing install, set ichorUI.users.enabled to true and ichorUI.users.externalURL to the https URL in the chart values, and create the first admin before every request starts to need a login:

ichor user create admin --role admin --generate --bootstrap

--bootstrap needs no login, because nobody can log in yet. It uses your own kubeconfig, so Kubernetes decides: you need create on ichorusers and get and patch on the Secret ichor-user-credentials. The CLI creates the IchorUser and writes the argon2id hash of the password into that Secret with the same code ichor-ui uses. It refuses a name that already exists (it never resets a password) and a cluster where the Secret does not exist, which means users are not turned on yet (the chart creates it with ichorUI.users.enabled). The generated password is printed once to stderr and must be changed at the first login. Instead of --generate you can type the password at a prompt with no echo, or pipe it with --password-stdin. Then set spec.authentication.ichor (enabled: true, url) on the TalosCluster, which needs update on talosclusters/authentication, and run ichor config apply if a change waits for a reboot. Once users are on, every request to ichor-ui needs a login.

Expose ichor-ui over https

ichor-ui never serves plain HTTP to the network. One listener on port 8443 serves the UI, the API, /metrics and the probes over TLS, with one of two certificates:

• The pair you supply with ichorUI.tls.secretName (or --ui-tls-secret), for example a cert-manager Secret. Browsers trust it if the issuer is public.
• Otherwise a leaf from the Ichor serving CA (Secret ichor-oidc-tls, issued for every install). Its names are the Service ClusterIP, ichor-ui-service.ichor-system.svc (ichor-ichor-ui-service... on an ichor init install) (and .cluster.local), localhost and, when users are on, the host of the external URL. Browsers do not trust it, so an edge that terminates the browser's TLS has to verify it, and a client that talks to it directly needs the CA.

The controller publishes the public part of the Ichor serving CA, never a key, as the ConfigMap ichor-ui-ca and the Secret ichor-ui-ca in ichor-system (key ca.crt). Service port 8443 is named https. Pick one recipe. The names below are for the Helm chart install (ichor-ui-service). An ichor init install and a kustomize install prefix them: the Service is ichor-ichor-ui-service (checked on a cluster made by ichor init, with the same prefix in the leaf's names). Use that name for name: and for the verified host name below.

If you supplied your own certificate, the edge verifies your CA instead of ichor-ui-ca, and the name it checks must be in your certificate. The ichor CLI and other clients inside the cluster are not affected: ichor-ui answers a handshake that asks for the Service name (SNI <service>.<namespace>.svc) with Ichor's own certificate, so they still verify it against ichor-ui-ca. Every other name and address gets your certificate (2026-11-03).

Ingress-NGINX (re-encrypt to the backend)

The Ingress must be in ichor-system, because it names the Service. Annotations (Ingress-NGINX annotations reference, "Backend Protocol" and "Backend Certificate Authentication"):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ichor-ui
  namespace: ichor-system
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-secret: "ichor-system/ichor-ui-ca"
    nginx.ingress.kubernetes.io/proxy-ssl-name: "ichor-ui-service.ichor-system.svc"
    nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on"
spec:
  ingressClassName: nginx
  tls:
    - hosts: [ichor.example.com]
      secretName: ichor-example-com-tls
  rules:
    - host: ichor.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ichor-ui-service
                port:
                  name: https

• backend-protocol: "HTTPS" makes NGINX speak TLS to the pod.
• proxy-ssl-verify is off by default; without it NGINX does not check the pod's certificate.
• proxy-ssl-secret takes namespace/secretName. The reference docs describe a Secret with tls.crt and tls.key (a client certificate, which ichor-ui does not ask for) that "should also contain" ca.crt. A Secret with only ca.crt, which is what ichor-ui-ca is, is accepted: tested with Ingress-NGINX chart 4.15.1 (controller 1.15.1), the request reached ichor-ui and nginx used the CA as proxy_ssl_trusted_certificate. Pointing the annotation at a Secret that holds another CA gives 502 (upstream SSL certificate verify error).
• proxy-ssl-name overrides the name NGINX verifies; proxy-ssl-server-name sends it as SNI. Use a name in the certificate. It is required: without it NGINX verifies the literal name upstream_balancer and every request is a 502. A name that is not in the certificate is a 502 too (upstream SSL certificate does not match). Ingress-NGINX takes up to about half a minute to reload after an annotation change, so wait before you read a result.
• The public certificate for browsers is the Ingress's own tls Secret.
• If you supplied your own certificate, point proxy-ssl-secret at a Secret that holds its CA as ca.crt.

Gateway API: BackendTLSPolicy

BackendTLSPolicy is gateway.networking.k8s.io/v1, in the Standard channel since Gateway API v1.4 (it was v1alpha3 before). Your Gateway implementation must support it, and the CRDs installed in the cluster must be v1.4 or later. Release 1.4.1 notes say a policy supports a single targetRefs entry.

apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: ichor-ui
  namespace: ichor-system
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: ichor-ui-service
  validation:
    hostname: ichor-ui-service.ichor-system.svc
    caCertificateRefs:
      - group: ""
        kind: ConfigMap
        name: ichor-ui-ca

• It lives in the Service's namespace; caCertificateRefs may name only objects in the policy's own namespace. That is why Ichor publishes ichor-ui-ca in ichor-system.
• The Core-support reference is one ConfigMap with the key ca.crt, which is what ichor-ui-ca is. A Secret reference is implementation-specific.
• validation.hostname is the SNI and the name the certificate must match.
• The HTTPRoute that sends traffic to the Service references port 8443 (named https). A route in another namespace needs a ReferenceGrant.
• Check the policy's Accepted and ResolvedRefs conditions.
• Tested with Envoy Gateway v1.9.2 (its chart installs the Gateway API v1.6.1 CRDs, BackendTLSPolicy served as v1): the policy was Accepted and ResolvedRefs, requests reached ichor-ui, a ConfigMap with another CA or a hostname that is not in the certificate gave 503 from Envoy, and without the policy Envoy spoke plain HTTP to port 8443 (ichor-ui answered 400). Envoy took from 15 s to a minute to apply a change, so wait and repeat the request. A Gateway needs a listeners[].hostname (singular), not hostnames.

Load balancer passthrough (operator certificate)

A Service of type LoadBalancer (ichorUI.service.type: LoadBalancer) or a TCP passthrough terminates nothing, so clients see ichor-ui's own certificate and the name they typed. Supply your certificate:

ichor init --ui-tls-secret ichor-ui-cert --ui-url https://ichor.example.com ...
# or in the chart
helm upgrade ichor ... --set ichorUI.tls.secretName=ichor-ui-cert

The Secret is kubernetes.io/tls and its certificate must cover the name clients use. Without it, only clients given the Ichor CA verify: ichor --ca-file ca.crt ... (or ICHOR_CA_FILE, from kubectl -n ichor-system get configmap ichor-ui-ca -o jsonpath='{.data.ca\.crt}'). An L7 cloud load balancer that cannot verify its backend is out of scope: use a controller that can.

Port 8080 and migration

A new install serves nothing on 8080. ichor-ui no longer serves plain HTTP at all. An install that served plain HTTP before is moved to HTTPS on 8443 by ichor upgrade ichor, which prints "ichor-ui no longer serves plain HTTP; point your edge at https://...:8443 (trust ichor-ui-ca)". Point the edge at 8443 and trust ichor-ui-ca (see above). The --http-addr flag is accepted for one more release as a no-op that logs "ignored", so a hand-edited Deployment still starts; the chart value ichorUI.plaintextPort is gone. An install that served TLS on 8080 keeps --addr pinned to 8080. See "Install Ichor with Helm" for the chart values.

Port 8444 is an internal listener. The kube-apiserver uses it to fetch the login keys. Never route it from an edge.

Create users and roles

Each user is an IchorUser object in ichor-system. You can create one with the CLI or apply one yourself. Its name is the login name and cannot change. It has:

• a display name and an e-mail (for information only);
• roles, a list of role names;
• disabled, which is false for a new user.

The built-in roles are:

A role is a group in Kubernetes. A custom permission is a ClusterRole bound to the group ichor:<name> of the user. No built-in role can read secrets.

Changing the Cilium Helm values (spec.cni.cilium.values: the Cilium page, ichor cni values set|edit, ichor adopt --cilium-values, kubectl) needs the verb update on talosclusters/cilium-values, which only admin holds: the chart can run privileged pods on every node. An operator sees the values read-only, and a write of theirs that changes the values is refused; an edit of the cluster that leaves them as they are is not. To let someone else change them, bind a ClusterRole with that verb to their group.

A new user has no password until an admin sets one. The admin can set a password, or ask for a generated one, which is shown once. The user changes it at the next login. A password is 12 to 128 bytes long, with no other rules.

Users can change their own password in the account menu.

The ichor user commands

They talk to ichor-ui directly at the issuer URL over https, with your login (a token for ichor-ui, from the browser sign-in; the first command opens it). They do not use a port-forward and need no kubeconfig, so they work from a laptop. Give the URL with --issuer or ICHOR_ISSUER, and --ca-file (or ICHOR_CA_FILE) when a private CA signed ichor-ui's certificate. The commands print for people; -o yaml or -o json print the full response for scripts.

ichor user list
ichor user get alice
ichor user create alice --display-name "Alice" --role viewer --generate
ichor user update alice --role operator        # only the flags you pass change
ichor user update alice --disabled=true
ichor user set-password alice --password-stdin
ichor user revoke-sessions alice
ichor user delete alice                        # asks; --yes skips

list is a table, and a write prints one line naming what it did:

$ ichor user list
NAME   DISPLAY NAME  EMAIL  ROLES   STATE        AGE
admin  -             -      admin   active       2d
alice  Alice         -      viewer  no password  4s

$ ichor user create bob --role viewer --generate
✓ created user bob (viewer)
! Password for bob (shown once, stored nowhere; they must change it at the first sign-in): Zq3x...

$ ichor user update bob --role operator
✓ updated user bob

The state is active, no password (nobody can sign in as the user yet) or disabled.

create and set-password take the password from a prompt with no echo (asked twice), from the first line of stdin with --password-stdin, or generate one with --generate; a generated password is printed once to stderr, under the result line, and nowhere else. A password is never a flag value. The password set by an admin is marked must-change. Kubernetes RBAC decides who may do what: the admin role for everything, update on ichorusers/password for set-password and revoke-sessions.

Sign in and get a kubeconfig

To sign in from the terminal:

ichor auth login --issuer https://ichor.example.com --audience ichor-ui

This opens a browser for the login page. The password is typed only into that page; the CLI never sees it. The default audience is ichor-kube, the login kubectl needs; --audience ichor-ui is the one the ichor user commands use. ichor auth status shows who you are and when the cached tokens expire; it prints no token. ichor auth logout revokes the refresh token of each cached login at the issuer, then removes the local login cache; if the issuer cannot be reached it warns, and still removes the cache. If the certificate of ichor-ui comes from a private CA, add --ca-file ca.pem (or export ICHOR_CA_FILE=ca.pem) to every ichor auth and ichor user command.

To get a kubeconfig:

ichor auth kubeconfig --issuer https://ichor.example.com --out ~/.kube/ichor.yaml

--out - prints it to stdout; without --out the file is named as ichor-ui suggests and written to the current directory. The file is written with mode 0600 and the command refuses to overwrite an existing file. With --ca-file ca.pem, the kubeconfig carries --ca-file <absolute path> in the ichor auth token arguments, so kubectl trusts the same CA; without the flag the arguments do not change.

You can also use "Download my kubeconfig" in the account menu of ichor-ui. The file holds the API server address and the cluster CA. It holds no secret. When kubectl runs, it calls ichor auth token, which uses the login cache. You need at least the viewer role to download it.

The login cache is a credential on your machine. It is stored in ~/.ichor/auth/ with restricted permissions.

After a Kubernetes CA rotation, a kubeconfig downloaded earlier fails TLS. Get a new one.

A read-only talosconfig for talosctl

operator and admin can take a talosconfig whose client certificate has the Talos role os:reader: talosctl can read a node (get members, logs, dmesg, processes) and the node refuses everything that changes it (reboot, upgrade, reset). It needs no access to the secrets bundle.

ichor cluster talosconfig my-cluster --ttl 4h
TALOSCONFIG=~/.talos/config.ichor-reader talosctl get members

It lives one hour by default and at most twelve; there is no way to ask for longer, and it cannot be revoked before it expires, so ask for what you need. The key is made on your machine. The UI offers the same under the cluster's "More actions" menu. Each request leaves an Event on the TalosCluster that names you (kubectl get events --field-selector reason=TalosconfigReader).

The permission is the Kubernetes verb download on talosclusters/talosconfig-reader. viewer does not hold it, because a talosconfig is the node's own API, without the filtering and masking of the UI. To give it to a group, bind a ClusterRole with that rule to it. Ichor offers no os:operator talosconfig: it could reboot and upgrade nodes outside the operations Ichor plans and records. For admin access to the node API use ichor cluster credentials.

What a sign-in is, and what ends it

• An ID token lasts 5 minutes by default (1 minute to 1 hour). ichor-ui checks it against the user's notBefore on every request: revoking sessions or changing the password ends tokens issued before it, here within 10 seconds per replica (a cache). The kube-apiserver cannot do this check, so a token already given to kubectl lives out its 5 minutes there.
• The browser stays signed in for as long as the CLI would, not for 5 minutes. The sign-in form asks for a refresh token, and the page trades it for a new ID token about a minute before the current one ends (and when you come back to a tab, and when a request is refused), until the session lifetime (ichorUI.users.refreshLifetime, 12 hours by default, 1 hour to 30 days) since you typed the password. It does not slide: after that you sign in again, even in a tab you are using. Sign out (which revokes the refresh token at the issuer), a revoke-sessions, a password change or a disabled user end it as before (the next refresh is refused and the tab returns to the sign-in card). The refresh token lives in the tab's sessionStorage next to the ID token, so closing the tab ends the browser session; each tab has its own sign-in, and a tab duplicated by the browser signs in again instead of sharing the original's session. There is no idle timeout: an open tab on an unattended screen stays signed in until the session lifetime, so lock the screen, or shorten refreshLifetime. The sign-in with an external identity provider is unchanged.
• A refresh token is single-use: every refresh returns a new one and retires the old. Presenting a retired token ends the whole chain, which is what a stolen copy looks like; the owner signs in again. The state is kept in the user's entry of the ichor-user-credentials Secret, so all replicas agree, and it never extends the session: the chain dies at the original lifetime. Tokens issued before the upgrade that added this are accepted once and then rotated.
• Log and download streams end with "your sign-in expired" when the token does.
• Wrong passwords slow down, they do not lock out. The counter is per (client address, user name), so a stranger cannot lock admin out for you. After 5 failures that pair waits 1 s, 2 s, 4 s and so on up to 15 minutes; one address waits too after 30 failures across names (up to 1 minute), and once a user has 30 failures within an hour from any addresses, every sign-in of that user takes 2 s longer (never refused). Counters are kept per replica, so with N replicas the free guesses are N times as many. Behind a proxy or load balancer the client address is the proxy's unless you list it in ichorUI.users.trustedProxyCIDRs (--trusted-proxy-cidrs), when the rightmost X-Forwarded-For address outside the list is used (see "Client addresses behind a load balancer" below). Password hashing for sign-in has its own queue, so a flood of sign-ins gets 503 Retry-After and cannot block a password change.
• A refresh token does not survive an etcd restore. Its state is in the ichor-user-credentials Secret, which ichor restore rewinds with the rest of Kubernetes, so a token that was refreshed after the snapshot was taken is a retired one in the restored Secret, and presenting it ends the chain (above). This is expected, not a fault: sign in again (ichor auth login, or the browser). Measured: after a live restore the admin's refresh token was refused and a new sign-in worked. Users, passwords and roles are restored to the snapshot's state as well, so a password changed after the snapshot is the old one again.

Client addresses behind a load balancer

The failed-login counters key on the address ichor-ui sees. If every client arrives with the same address, they share one counter: a stranger's wrong passwords for admin then slow admin down for everyone. This is the case whenever something between the client and the pod replaces the source address:

• a LoadBalancer or NodePort Service with the default externalTrafficPolicy: Cluster (the node SNATs the connection, so every client looks like the node; measured on a NodePort behind an L4 load balancer: a workstation and a pod of the outer cluster were indistinguishable),
• an L4 load balancer that does not keep the client address (no PROXY protocol support in ichor-ui, and none is planned).

Pick one of these:

1. An L7 proxy that sets X-Forwarded-For (an Ingress controller, a Gateway, nginx, HAProxy in http mode). Set ichorUI.users.trustedProxyCIDRs (the flag --trusted-proxy-cidrs of ichor-ui; ichor init has no option for it, so on an ichor init install add it to the arguments of the ichor-ichor-ui Deployment) to the proxy's address range, and only to that: ichor-ui then takes the rightmost X-Forwarded-For entry that is not in the list. Do not list a range that clients can reach directly, or they can write the header themselves and pick their own counter.
2. externalTrafficPolicy: Local on the ichor-ui Service (the chart value ichorUI.service.externalTrafficPolicy, or the same field on the Service of an ichor init install, which kubectl patch sets), so the node does not SNAT and the pod sees the real source. The load balancer must send only to nodes that run an ichor-ui pod (it health-checks them), and it must itself keep the client address.
3. If neither is possible, accept the shared counter and know its effect: it is a slowdown (up to 15 minutes for one user name), never a lock-out, and an administrator can still be reached with kubectl and the override credential below. Pods inside the cluster have their own addresses and are not affected.

• A user may grant only roles they hold (admin: any); Kubernetes RBAC alone used to decide this. A validating webhook on ichorusers applies the same rule to a kubectl write made as an Ichor user; an external identity stays gated by RBAC alone.
• The last enabled admin cannot be deleted, disabled or demoted, in the UI, the CLI or with kubectl. The system:masters override certificate below is exempt, and so are the controller and the Kubernetes namespace deletion, so deleting the ichor-system namespace still works.

Disable a user

Set disabled: true on the IchorUser, through the CLI or with kubectl. The effect has two parts:

• In ichor-ui, the change applies within 10 seconds.
• For kubectl, the change applies at the next token refresh, at most 5 minutes later.

Ichor has no way to take back a token that is already issued. The 5 minute token lifetime is the limit.

To end every existing session of a user, revoke their sessions. Changing a user's password also ends their earlier sessions. A session lasts at most the session lifetime (12 hours by default). Signing out in the browser (or ichor auth logout) revokes that session's refresh token at the issuer (RFC 7009, POST /oidc/revoke) and then discards the tokens, so a copy of the refresh token someone took earlier stops working too. The ID token already issued is not revocable and ends within its 5 minutes. The revocation is best effort: if the issuer cannot be reached when you sign out, the sign-out still completes, and the refresh token then lasts to the end of its lifetime unless you revoke the user's sessions. A tab signed in before this was added has no revocation endpoint stored and only discards its tokens.

Override

Ichor keeps an override credential that does not depend on login. It is the admin client certificate in ~/.ichor/clusters/<name>/kubeconfig, and the Talos os:admin identity in the same folder. If ichor-ui is down, the issuer is broken, or every user is locked out, use this credential.

Once users are on, keep this folder offline and use it only in an incident. The bundle in that folder (secrets.yaml) is encrypted with a passphrase (see Getting started); the kubeconfig is not, because kubectl reads it directly. The bundle is what matters: with it ichor cluster credentials --bundle makes a new kubeconfig and talosconfig at any time. So you can move or delete the kubeconfig once login works, and keep only the encrypted bundle and its passphrase. Do not confuse it with the override write, which is an RBAC verb that guards destructive operations.

Control plane rollout

Talos applies the login settings without a reboot, but restarts the kube-apiserver on each change (15 to 45 seconds of /readyz down per control plane). The machine controller does not apply the change by itself: the machines report APIServerRestartRequired, and ichor config apply (or Apply in the attention list of the UI's Overview) rolls it out one control plane at a time with no drain, waiting for that node's /readyz before the next. On a cluster with one control plane the API is unavailable while it restarts.