# 0081. The UI demo cluster is built by `ichor init`, pulling Ichor's images from a throwaway in-namespace registry Status: Accepted · Date: 2026-10-26 · Design section: Bootstrap Decided: The demo cluster's control plane is created by `ichor init` and its worker approved through `ichor machine approve --install-disk`, so every node's configuration is rendered by Ichor; images are pushed with `crane` to a `registry:2` Deployment in `ichor-e2e` and reached through an anonymous `spec.registries` mirror of `ghcr.io` that `ichor init --registry-mirror` writes; `init` also gains `--pod-subnets` / `--service-subnets`, waits for the node, the API server and the webhooks, and applies the install manifest before the bundle Secret; the controller's shipped RBAC and cache are made consistent. ## Context The previous demo (`up-demo`) installed Talos with a config built by the test harness, so Ichor's render differed from the node's config and its control plane could never converge (ADR 0077's drift). The user wanted the demo built with the product's own bootstrap. Doing that on a workstation with no container runtime and no registry exposed product gaps. ## Decision 1. **Harness.** `up-demo` only creates the two nodes (name prefix from `ICHOR_UI_DEMO_PREFIX`, default `ichor-ui-demo`) and leaves both in maintenance mode. `ichor init` runs from a busybox pod in `ichor-e2e` because the workstation cannot reach pod IPs. 2. **Images.** `crane` (added to `flake.nix`) appends the statically built `manager` / `ichor-ui` binaries as a layer on `gcr.io/distroless/static:nonroot` and pushes them through a `kubectl port-forward` to a `registry:2` Deployment+Service (`-registry`, emptyDir, in `ichor-e2e` only). The install manifest is built with `make build-ichor-embed IMG=ghcr.io/maeshinshin/ichor-controller: IMG_UI=ghcr.io/maeshinshin/ichor-ui:`, so the image references stay real `ghcr.io` names and the nodes reach the registry only through `TalosCluster.spec.registries`: host `ghcr.io`, endpoint `http://:5000`. containerd tries the mirror first and falls back to `ghcr.io` for everything else (Talos's own images). No raw YAML and no patch: the typed field already existed. 3. **`ichor init --registry-mirror HOST=URL`** (repeatable) fills `spec.registries` with anonymous mirrors. `init` used to refuse any registry because a new cluster has no Secret to resolve credentials from; it now refuses only an entry carrying `authSecretRef`, which the flag cannot produce. 4. **`ichor init --pod-subnets` / `--service-subnets`** set `spec.network` (ADR 0077); each is validated as at most two CIDRs. The demo uses 10.248.0.0/16 and 10.249.0.0/16 because the outer cluster owns 10.244.0.0/16. 5. **`init` ordering and waiting**, all found by running it against a real node. Bootstrapping etcd retries while the node answers `Unavailable` (still serving its maintenance certificate) or `FailedPrecondition` ("bootstrap is not available yet"), up to 15 minutes. `init` waits for the API server before the first write. It applies Ichor's manifest before the bundle Secret, because the Secret lives in the namespace the manifest creates (it failed with "namespace not found"). The CRs are created with a retry while the API server cannot call Ichor's `failurePolicy: Fail` webhooks, whose controller `init` has only just started. The bootstrap `TalosConfigPatch` (it carries `machine.type` and `certSANs`) moves from the `cluster` layer to the `role` layer targeting the first node's role: at the cluster layer it forced every later machine, a worker included, to render as a control plane. An existing cluster made by an older `init` keeps the cluster-layer patch; recreate it with `layer: role` and `target.role: controlplane`. 6. **Controller RBAC and cache.** The shipped Role grants Secret `list`/`watch` in `ichor-system` only and the webhook configuration only by `resourceNames`, but the manager's cache listed Secrets cluster-wide and cert-controller's informer lists every `ValidatingWebhookConfiguration`; both were forbidden, the cert rotator's cache never synced and the pod crash-looped (the old demo hid this behind `cluster-admin` and `ENABLE_WEBHOOKS=false`). The Secret informer is now confined to `POD_NAMESPACE`, and `list`/`watch` on `validatingwebhookconfigurations` are granted without `resourceNames` (an informer cannot name its object); `get`/`update`/`patch` stay restricted to Ichor's one configuration. This widens the controller's read access to webhook configurations cluster-wide, read-only. The same bug hit real hardware for `Job`: the self-upgrade executor reads installer Jobs through the cached client, whose cluster-wide informer was forbidden (`jobs.batch is forbidden ... at the cluster scope`), so the installer Job was never created. `controller.CacheOptions` (moved out of `cmd/main.go`) now scopes `Job` as well as `Secret` to `POD_NAMESPACE`, the Role grants `jobs` `list`/`watch`, and `cache_rbac_test.go` builds the manager's cache as the controller's identity bound to the generated `role.yaml` under envtest and requires an informer to sync for every kind read through the cached client. Pods, DaemonSets, ConfigMaps and Deployments are read through the uncached `APIReader` or the direct client and need only `get`. 7. **ichor-ui needs `POD_NAMESPACE`.** `cmd/ichor-ui` resolves the secrets bundle in `POD_NAMESPACE`, but neither the chart nor `config/manager-ui` set it (the old hand-built demo did, hiding it), so every machine page failed with "controller namespace is empty". Both now set it from the downward API, and `TestPodNamespaceFromDownwardAPI` pins it on the manager and ichor-ui containers. ## Not done - The registry is plain HTTP and unauthenticated, reachable by anything in the outer cluster; it is a demo fixture. - `crane` pushes the base image's layers too, so the registry holds a copy of the distroless base. - The demo uses the `node` endpoint type with the node's pod IP, which changes if the node restarts; a restart needs a re-init.