0006. Webhook serving certificates without cert-manager, failing closed

Status: Accepted · Date: 2026-10-25 · Design section: 4.5, 5.2.7

Decided: Run cert-controller in-process for the webhook serving certificate instead of depending on cert-manager

Context

Ichor needs validating webhooks from phase 1 on — deletion protection first, then topology and immutability checks. Two project rules bear on how their serving certificates work: the default install must not require cert-manager, and nothing in the chart may produce a diff on every Argo CD sync. kubebuilder 4.15.0 scaffolds a config/certmanager/ Issuer and assumes cert-manager at runtime; it offers no alternative.

What the investigation established:

• controller-runtime v0.24.1 only reloads certificates. certwatcher watches files and webhook.Server points at CertDir; nothing in it generates a CA or a serving certificate.
• caBundle cannot be avoided. It is optional in the API, but when it is absent the API server falls back to its own system trust roots — not the cluster CA. None of the built-in CSR signers in 1.36 (kube-apiserver-client, kube-apiserver-client-kubelet, kubelet-serving, legacy-unknown) issues a serving certificate the API server will trust for an admission webhook. So for a self-signed setup some component has to write caBundle at runtime.
• github.com/open-policy-agent/cert-controller (Apache-2.0) does exactly that job as a library: generates a self-signed CA and serving certificate into a Secret, rotates ahead of expiry, and injects caBundle into validating and mutating webhook configurations, CRD conversion webhooks and APIServices. It builds against controller-runtime v0.24.1. Its last feature release is v0.16.0 (March 2025); since then it is dependency bumps only.
• Gatekeeper's chart — cert-controller's own reference consumer — ships the webhook configuration with no caBundle key, and caBundle is written at runtime. It does, however, ship the certificate Secret, as metadata with no data, and that turns out to be load-bearing: cert-controller v0.16.0 contains no Create call anywhere. refreshCertIfNeeded does reader.Get on the Secret and returns a wrapped error if that fails, so on a cluster where the Secret does not exist the rotator errors out, the manager runnable fails and the process exits — on every restart. The empty Secret is what makes Get succeed; secret.Data == nil is then what triggers generation, and Update fills it in.
• On the Argo CD side, ignoreDifferences alone only hides the diff; self-heal still overwrites the field unless the Application also sets RespectIgnoreDifferences=true. Keying the exception on managedFieldsManagers requires the writer to set a field manager name.

Decision

• Run open-policy-agent/cert-controller in-process in ichor-controller, with CertRotator.FieldOwner set to ichor-controller, the field manager the naming conventions already assign to it. Serving certificates are read from the mounted Secret by controller-runtime's certwatcher, so the two compose and Ichor owns no cryptography.
• The chart and the Kustomize base ship the ValidatingWebhookConfiguration without a caBundle key, and ship the certificate Secret as metadata only, with no data. No key material is templated, so no generated value can differ on a re-sync, but the object exists — which is what lets the rotator's Get succeed and its Update fill it in. Shipping it in Git has a second benefit: if the Secret is ever deleted, Argo CD puts the empty one back and the controller refills it, instead of the operator having to.
• The volume that mounts it is optional: true as well. With the Secret shipped this is belt and braces, but it changes the failure mode if the Secret goes missing: the pod starts and crash-loops with the reason in its logs, rather than sitting in FailedMount where the cause is far less visible.
• failurePolicy: Fail, which is where Ichor departs from Gatekeeper's Ignore, deliberately. Gatekeeper validates the whole cluster, so a webhook that is down must not block everything. Ichor's webhooks match only ichor.maesh.dev resources, so failing closed blocks Ichor operations and nothing else — and the thing they mostly protect is deletion of TalosCluster and TalosMachine, where failing open lets an Argo CD prune cascade into node resets. A deletion guard that opens when the controller is down is not a guard.
• Set RequireLeaderElection on the rotator. The controller runs two replicas, and without it both run the rotator: on a cold start both can see no Secret and generate a different CA each. FieldOwner only decides who writes last, so the losing replica would serve a certificate signed by a CA the live caBundle does not name until the mounted Secret resyncs. Letting only the leader rotate is the same instinct as principle 6 — one writer for a disruptive operation — and costs nothing, since both replicas need the Secret before they can serve anyway.
• Ship, with the chart, an ignoreDifferences entry for the webhook configuration's caBundle and one for the certificate Secret, both keyed on managedFieldsManagers: [ichor-controller], together with the RespectIgnoreDifferences=true sync option, and say in the docs that both are needed — the first one alone leaves self-heal to clobber the field. Example: dist/chart/examples/argocd-application.yaml.
• cert-manager stays available as an opt-in: a chart value turns the in-process rotator off. The design requires only that it not be the default.

Consequences

• Until the controller is running and has written caBundle, writes to Ichor's CRs are refused. That is the right order regardless — the sync waves already place Ichor's CRDs and controller ahead of Ichor's CRs — but ichor init has to wait for the webhook to answer before it creates the first CRs, and the bootstrap path must treat "webhook not ready yet" as something to wait on rather than an error.
• If the Secret is deleted while the webhook configuration still carries the old caBundle, the rotator's Get fails, Start returns an error and the manager exits, so the pod crash-loops rather than serving with a stale certificate. Under Argo CD the empty Secret comes back on the next sync and the controller refills it. With failurePolicy: Fail that means Ichor operations stop until it recovers. That is the intended direction of failure, and ichor CLI recovery paths must not depend on the webhook.
• Neither library exposes certificate expiry as a metric or a condition. Ichor's own expiry gauge lands with the rest of the metrics in phase 9; until then expiry is visible only in logs. Recorded as a known gap, not a surprise.
• cert-controller is maintained but not developed. The surface Ichor uses is one function and one struct, so if it stops building against a future controller-runtime, replacing it with an in-house rotator is a contained change rather than a rewrite.

Alternatives considered

cert-manager as the default. Ruled out by the design, which wants the default install to carry no such dependency.

Our own rotator. Roughly 200 lines of certificate generation, rotation windows and failure handling that cert-controller already has, in exchange for one fewer dependency and metrics from day one. Kept as the fallback.

A Helm-hook Job (kube-webhook-certgen-style). Well established, used by ingress-nginx, and needs no Go code from us — but it is a point-in-time Job with no continuous rotation, and it adds hook-ordering bookkeeping under Argo CD that the in-process rotator does not need.

Amendment: cold-start ordering, no crash restarts

Found on hardware: a fresh install never converged. The rotator is leader-only (see the earlier amendment), but controller-runtime's webhook server started at once, read tls.crt, found nothing and stopped the manager 15 ms after the lease was acquired, before the rotator wrote anything. With the 120 s lease and no release on cancel (ADR 0063, ADR 0089), every dead holder then blocked the next container for up to 120 s and the pod backoff grew to 5 min. The "one or two expected restarts" of ADR 0031 won by luck while the lease was short; it lost every time afterwards.

Decision: the webhook server does not start until its certificate exists.

• controller-runtime v0.24.1 adds the webhook server to the manager lazily, on the first mgr.GetWebhookServer() (webhookServerOnce). So that call, every Setup*WebhookWithManager and the readiness check all move behind one gate (webhooksUp in cmd/main.go), in a goroutine.
• The gate opens on either trigger: the rotator closes IsReady (the leader), or tls.crt and tls.key exist and are non-empty in the cert directory (polled every second, waitForCertFiles). The second is how a non-leader replica, which runs no rotator, serves once the leader has written the Secret and the kubelet has synced the volume. Either replica becomes Ready on its own.
• Readiness stays false until the gate is open (gatedChecker) and then delegates to the server's StartedChecker, so Ready still means "the webhook answers".
• Only the leader writes the certificate, so a single writer is kept and the cold-start CA race stays closed.
• The metrics server, when --metrics-cert-path is set, has the same problem and cannot be delayed (it is built into the manager). It now takes its certificate from a GetCertificate that reads the files per handshake, so it starts first and follows rotation; scrapes fail until the files exist.
• Whether a start-up failure before leading should hold the lease is moot: with no failure before the certificate exists, nothing exits holding it. A later failure still holds it for up to the lease duration, which is the accepted ADR 0063 trade-off.
• logCertStatusBeforeStart now says the server waits, not that restarts are expected. ADR 0031's "one or two restarts" no longer holds.

ichor init waits for the webhooks for 30 min by default (WebhookWait, was the 15 min node wait), and reports the last real server answer when that wait ends rather than the client's own "rate limiter ... deadline exceeded".