# 0068. The UI and CLI start, follow, pause and delete TalosOperations through one OperationService, and write the spec first

Status: Accepted · Date: 2026-10-26 · Design section: Upgrades, CLI and UI

Decided: an `OperationService` (list, get, plan, create, pause/resume, delete) is the one entry point for `TalosOperation`s, used by `ichor upgrade|cni|cluster shutdown|operation` and the UI; every call is made as the signed-in user, `CreateOperation` changes the spec first and creates the operation second after refusing everything it can beforehand (including a dry-run create as the caller), the kube-proxy confirmation annotation is set for that one write and removed again, destructive starts show the server's plan and ask for confirmation (a shutdown also needs the cluster name typed in the UI)

## Context

The executors for `talos` (0043), `kubernetes` (0057), `ichor` (0059), `cni` (0066) and `shutdown` (0067) existed,
but nothing could start one: no code in `internal/cli` touched `TalosOperation`, `proto/` had no RPC for it and the UI
had no screen. The design promises a progress view shared by the three upgrades and a CLI/UI that both write CRs
through the same Connect API (the CLI falling back to the same handlers in-process, ADR 0049).

## Decision 1: one service, the same handler on both paths

`proto/ichor/v1alpha1/operation.proto` adds `OperationService`: `ListOperations`, `GetOperation`, `PlanOperation`,
`CreateOperation`, `SetOperationPaused`, `DeleteOperation`. `uiapi.Server` implements it; the CLI's in-process
fallback (`localClients`) uses the same `uiapi.Server`, so every refusal below runs on both paths. Progress is
`TalosOperation.status` as it is (`phase`, `failureReason`, the machines with their phases); nothing is derived or
guessed server-side. There is no stream: `get --watch` and the UI poll (2 s in the CLI; 3 s running, 10 s paused,
stop on completed or failed in the UI). The status is a few fields and a poll is one Get.

## Decision 2: which comes first, the spec change or the operation

**The spec first, then the operation.** The executors take their target from the spec when a machine's turn comes
(`TalosCluster.spec.talosVersion` / `kubernetesVersion` / `cni.kubeProxy`, `IchorInstallation.spec.version`; ADR 0059
says outright that the `ichor` operation carries no version). An operation created first would run against the old
value and finish having done nothing, while the cluster lock was held; and the spec would then change under no
operation, which is the state the controller treats as drift.

The risk of the other order is that the second write fails after the first succeeded. What that leaves is
harmless: a spec that names a new version with no operation is only drift (the controller dry-runs it, it does not
apply a reboot-requiring change), and running the command again is idempotent because a spec that already names
the target is not written twice. To make that failure rare, `CreateOperation` refuses before the first write:

1. the plan's blockers: another operation on the cluster that has not `Completed` (the lock would refuse it
   anyway, but only after the spec had changed), and a target spec that Argo CD or Flux tracks;
2. a dry-run `Create` of the operation made as the caller (`client.DryRunAll`), which exercises RBAC and
   admission for creating operations before the spec is touched. A caller who may edit a cluster but may not
   create operations changes nothing.

Left: the spec update succeeding and the real `Create` failing (a race with another operation, an apiserver
error). The message says what was done; a retry finishes it.

A tracked spec (ADR 0051) is refused with a message saying to merge the change in the repository and then start the
operation with the version the spec now names, which is a start without a spec write. Opening a pull request for
the spec change and creating the operation later is not done here (below).

## Decision 3: the kube-proxy confirmation annotation

ADR 0013's `ValidatingAdmissionPolicy` denies a `spec.cni.kubeProxy` change without
`ichor.maesh.dev/confirm-kube-proxy-switch`. `CreateOperation` for `kind: cni` sets the annotation in the same
`Update` as the field (retried on a conflict with a fresh read), creates the operation, then removes it in a second `Update` (amended 2026-10-29, ADR 0066). The
confirmation is the user's, given in the CLI prompt or the UI dialog; leaving the annotation behind would turn it
into a standing consent for a later hand edit. If the removal fails the call reports it after the operation exists (naming the
annotation; remove it by hand). `--yes` / the dialog's Start is the confirmation; there is no way to set the
annotation without the switch.

## Decision 4: permissions

Every `Get/List/Create/Update/Delete` in `operation_service.go` is `s.X`, the shadowed calls that go through the
impersonated client when a caller identity exists (ADR 0058), so Kubernetes RBAC decides who may list, start,
pause and delete operations and edit the cluster spec, and the API server's audit log names the user. The only
`s.Client` use is the tracking check, which decides whether a write may happen and returns nothing to the caller
(ADR 0062's rule). `AuditRecord.actor` is the user, or `ichor-cli` on the fallback. The ServiceAccount's ClusterRole
gets `talosclusters update`, `talosoperations get/list/create/update/delete` and `ichorinstallations get/update`
(markers, `make manifests-ichor-ui`, the chart template); with OIDC on they are not used for these calls, with
OIDC off they are what "anyone who reaches the UI may do anything" (ADR 0058, point 4) means. The unit tests
inject a client that refuses `Create`/`Update`/`Delete` as the caller and assert `PermissionDenied` and that nothing
was written.

## Decision 5: confirmation and the plan

`PlanOperation` returns, without writing: the effect (`REBOOT`, `RESTART_COMPONENTS`, `STOP`, `NONE`), the
approved machines in the order they will be handled and in steps (control planes one at a time, then workers in
groups of `maxUnavailable`, label order as the controller sorts them; for a shutdown the workers together, then the
control planes), the machines skipped by the hold annotation, the current and target values and a `blocked` reason.
The CLI (`planText`) and the UI build their confirmation from it, so both say how many machines, in which order and
whether they reboot, stop or only restart components. Destructive commands confirm unless `--yes` (`confirm`, as
`machine approve`); with `--yes` the plan is not asked for (the create does the same refusals). The UI's dialog is
two steps, "Review" (which calls the plan and writes nothing) and a separate "Start"; a shutdown also asks for the
cluster name to be typed.

Two shapes were weighed for the plan: computing it in the client from `ListMachines` (duplicates the controller's
ordering in TypeScript and Go) or one server function. The ordering is one function, `upgrade.SortByOrder` in
`pkg/upgrade`, called by the executor (rollouts and shutdown) and by the plan, so the order a person confirms is the
order that runs. `TestPlanOperationOrderMatchesTheExecutorsCandidateOrder` holds the two together. The plan had first
sorted a shutdown by name alone; the executor orders it by the same label, so the plan now does too.

## Decision 6: commands

`ichor upgrade talos|kubernetes|ichor VERSION`, `ichor cni kube-proxy replaced|enabled`, `ichor cluster shutdown
NAME`, `ichor operation list|get|pause|resume|delete` (alias `op`). The version may be given without the leading
`v`. `--cluster` (default: the only cluster in the namespace), `--max-unavailable`, `--follow` (poll until it ends;
a pause caused by a failed machine is an error exit), `get --watch`. The start, pause, resume and delete commands
are marked as writes for the version-skew check (ADR 0059, point 6). Scaffolded with `cobra-cli add` in a scratch
module and moved (ADR 0014). `ichor upgrade ichor` also names a cluster: the operation backbone needs one for its
lock, though the target is `IchorInstallation.spec.version`.

## Decision 7: the `ca-rotation` kind (ADR 0063)

*Amended by ADR 0091: the stages gain `AwaitingCredentials`, `rotate-ca` performs the credential hand-off after the start, and the plan says the operator's credentials stop working.*

`OperationKind` gains `OPERATION_KIND_CA_ROTATION`, `OperationEffect` gains `OPERATION_EFFECT_CA_ROTATION`, `Operation` carries
`ca_rotation_stage` (`status.caRotation.stage`, empty before the new CAs are saved), and `PlanOperationResponse` carries `stages`
(`Accepting`, `Issuing`, `Dropping`) and `replaces_secrets_bundle`. The rest is the discipline of the other kinds: `CreateOperation`
writes as the caller, refuses while an operation on the cluster is unfinished, runs the dry-run create first, and takes no target
(a target is `InvalidArgument`) and writes no spec, so a tracked cluster is not refused (nothing in the spec changes).

The plan lists every approved machine, control planes first and then workers, one at a time, in the executor's order
(`upgrade.SortByOrder`). A machine with `upgrade-hold` is not skipped and `held` is empty: a machine left out would be locked out
when the old CA is dropped (ADR 0063). The confirmation says that each machine is reconfigured once per pass without a reboot, that
the kubelet restarts on each machine in the passes that change the Kubernetes CA, and that when the rotation completes the secrets
bundle in the cluster is replaced and the copy on the user's machine is stale: run `ichor cluster bundle-export NAME`. The CLI and
the UI build this from the same plan fields; the CLI also prints the bundle-export reminder after the start, and `--follow` and
`operation get --watch` print the stage.

`ichor cluster rotate-ca` now calls `runStartOperation` like `ichor cluster shutdown`; the earlier direct write
through the kubeconfig (and its own already-running check) is removed, so the same refusals run on the CLI, the UI and the
in-process fallback. The UI dialog is the two-step Review/Start of the other kinds and, like a shutdown, asks for the cluster name to
be typed before Start. The progress page shows "Pass n of 5: <stage>" from `caStageProgress`.

Not done: the operation page does not read the cluster's `BundleBackupStale` condition (a completed rotation says what to run);
the stage is shown, not a per-machine view inside a stage (`status.machines` already shows the stage under way).

## UI

`/operations` (list: kind, cluster, state, progress, pause/resume/delete) and `/operations/:namespace/:name` (the
state of each machine, the failure reason with the statement that nothing retries and nothing is rolled back). The
start buttons are on the cluster page. All wording is in `en`/`ja` (the type forces both). Pure logic is in
`ui/src/lib/operations.ts` with tests.

## Not done

- Opening a pull request for the spec change of a tracked cluster (PR mode, ADR 0056) and creating the operation
  after the merge. Today the CLI/UI refuses and says to change Git first.
- Override for these writes (ADR 0064).
- A stream instead of polling; a per-machine log of what the executor is doing (the status has phase only).
- Clearing a failed machine so the operation resumes (ADR 0026 left that open); the screens say to delete and
  start again.
- No end-to-end run against a cluster; the service is tested over a fake client and the CLI over both the
  in-process and the httptest backends.

## E2E observation (2026-10-26)

All three commands were run through the CLI's in-process fallback (the UI was not deployed): `upgrade kubernetes`,
`cni kube-proxy replaced`, `cluster shutdown`, with `--yes`, `--follow` and without. `operation get`, `operation
delete` and `operation resume` also ran. See ADRs 0057, 0066 and 0067 for what each did.

- The plan text and the `n` answer worked for all three; nothing was written on `n`.
- `--follow` printed the status and exited with an error when the operation paused after a failure, as designed.
- `operation resume` on an operation paused after a failed machine did nothing (it stayed `Paused` with the same
  failure for ten minutes), which matches "delete and start again", but the command accepts it silently.
- The spec-first order showed its cost: after a failed first attempt the spec already named the new version.
  Starting the command again created a new operation without rewriting the spec, as described.
- The UI screens and `operation list` / `get --watch` were not exercised.

- `--follow` / `--watch` (one shared helper, `retryTransient`) retry transient API errors (connection refused/reset, Unavailable, EOF, timeouts) with backoff for up to 10 minutes, printing `API unreachable, retrying`, and re-open a dead ichor-ui tunnel. NotFound, PermissionDenied and the like still end the command.
- `PlanOperation` blocks a version target older than the current one (Talos, Kubernetes, Ichor), so `CreateOperation` never writes a downgrade into the spec before the executor refuses it.

## Amendment (2026-11-04): the service refuses early, the controller serializes

`blockedReason` / `unfinishedOperation` (Decision 2, point 1) only guard operations created through `OperationService`. The controller now keeps operations apart by itself: each takes the cluster lock under its own identity, a second one waits with `ClusterLockHeld` naming the holder, and a lease held by an operation that is gone, `Completed` or paused on a failed step is taken over at once (ADR 0026, amendment 2026-11-04). The service's refusal stays, since it stops the spec from being written for an operation that cannot start. The admission webhook deliberately does not reject a second operation, so operations created by `kubectl` or GitOps queue instead of failing.