0119. A daily CI run turns a new Cilium patch into a pull request

Status: Accepted; built · Date: 2026-11-04 · Design section: Upgrades, Cilium · Touches: 0098, 0108, 0113, 0115

Decided: Ichor pins the Cilium version, and the pin moves to a new upstream patch of the same minor through a pull request that CI opens: .github/workflows/cilium-patch.yml runs daily (and on demand) hack/bump-cilium-patch.sh (make bump-cilium-patch), which takes the newest final patch of the pinned minor from the Cilium Helm index, verifies the cosign signatures of the agent, operator-generic and hubble-relay images that chart pins against the Cilium release workflow of that tag (the identity internal/imagesig uses), checks that Cilium tests the minor with the same Kubernetes versions, and only then moves the pins (Makefile version and SHA-256, the go:embed line, ChartVersion, the chart archive, the bootstrap goldens, the version named in the README, the guides and the foreign hardware tests, the compat.go citation); the workflow runs make test and make lint and opens the pull request chore(cni): bump the embedded Cilium chart to X.Y.Z with peter-evans/create-pull-request. A merged bump reaches clusters only with the next Ichor release. The script never crosses a minor; a new minor is a person's decision.

Context

On 2026-11-04 the user decided that a cluster's Cilium only moves to the chart embedded in the running Ichor, and that Cilium patches reach users through Ichor releases (ADR 0115, amendment of the same day, removes the path that ran a patch's images without a new chart). That makes the delay between an upstream patch and an Ichor change the only lag users feel, so the user asked for CI to make the change quickly. A Cilium patch carries security and datapath fixes; ADR 0113's bump of the chart to 1.20.4 was done by hand and needed the same steps each time: fetch the chart and verify its digest, move the pin in four places, regenerate the goldens, check the signatures.

Decision

• What the script moves. The CILIUM_CHART_VERSION and CILIUM_CHART_SHA256 pin in the Makefile (the digest comes from the Helm index and make cilium-chart verifies the download against it); //go:embed chart/cilium-X.tgz in pkg/cni/cilium/render.go; ChartVersion in pkg/cni/cilium/values.go; the chart archive (the old one is removed, unless it is the foreign-cluster chart); pkg/cni/cilium/testdata/bootstrap-*.golden.yaml, regenerated by make cilium-goldens (internal/tools/ciliumgolden); the version named in README.md, docs/guides/getting-started.md, docs/guides/adopt-and-reclaim.md and the foreign-cluster hardware tests (test/e2e/foreign, test/internal/foreign); and the tag in the citation above "1.20" in pkg/release/compat.go. It does not touch CHANGELOG.md (a release section is written when the release is cut), ADRs, or tests that use a Cilium version as a fixture.
• What it checks before writing. The new version is a final release (no -rc, -pre) of the same X.Y; the downloaded chart matches the index digest and its Chart.yaml names the version; cosign verify accepts the three images the chart pins (the same three Ichor verifies at runtime) with issuer https://token.actions.githubusercontent.com and identity https://github.com/cilium/cilium/.github/workflows/build-images-releases.yaml@refs/tags/vX.Y.Z; and the Kubernetes versions in .github/actions/ginkgo/main-k8s-versions.yaml at the new tag equal those at the old tag, because compat.go states the range from that file. A failed check exits non-zero and changes nothing. "No newer patch" exits 0.
• What the workflow does. It runs the script in nix develop, then make test and make lint, and opens or updates the pull request on the branch chore/cilium-patch. A red run, whether from a signature, a changed Kubernetes matrix or a test that pinned the old patch, means the patch needs a person; no pull request is opened for it. The commit is signed through the API (sign-commits). The pull request body names the upstream release notes (https://github.com/cilium/cilium/releases/tag/vX.Y.Z).
• Merging is not releasing. The pull request changes what a first install gets and what the next Ichor build offers an existing cluster; nothing moves a cluster until the user installs that Ichor and runs ichor cni upgrade (ADR 0098). A Cilium security patch is shipped by cutting a patch release of Ichor after the merge.
• Token. A pull request opened with GITHUB_TOKEN does not start other workflows. The workflow uses the secret CILIUM_PATCH_PR_TOKEN (a fine-grained token with contents and pull requests write) when it exists, so that Tests, Lint and PR Title also run on the pull request.

Rejected

• Merging the bump automatically. A patch changes a CNI that runs on every node; a person reads the release notes and merges.
• Following a new minor. A minor changes values and removes options (ADR 0113's review of the 1.19 and 1.20 removals), and Cilium upgrades one minor at a time.
• Verifying only the agent image. Ichor runs and verifies three images, so the script verifies those three.

Test

hack/bump-cilium-patch.sh run on the current tree reports "no newer patch". Run on a copy whose pins were lowered to 1.20.3 it verified the three signatures, moved every pin to 1.20.4, fetched the chart with its digest checked, and left a tree byte-identical to the tree that ADR 0113's manual bump produced.