Skip to content

Commit 1a2b1da

Browse files
AzathothasAzathothas
committed
Notice when the publish stops, verify the signed tag, and publish releases
Three schedules had no watcher, the pacman release manager fingerprints were decorative, and this project published no artefact a consumer without container tooling could use. A publish that stops is now a red mark. scripts/check-publish-recency reads the newest dated index tag from the registry and scripts/check-schedules-fired reads every scheduled workflow's last scheduled run, discovered from the tree rather than listed. freshness-publish.yml and build-deploy.yml each run the check that sees the other go quiet. Every threshold is derived from the cron it judges, by scripts/cron-tolerance, so changing a schedule changes its threshold. The date in a tag is recognised by shape: an anchored year matches nothing from January and matching nothing reads exactly like never having published. The signed pacman tag is verified before the build starts. The pin gains the tag object sha and two keyservers, the fetch is deepened by one ref, and a VALIDSIG naming a pinned signer is required. Making the fingerprints load-bearing found that the two names beside them were swapped. release.yml owns the v* tag and publishes a rootfs tarball, an OCI archive, a resolved package set and an evidence file per architecture, a static pacman per architecture, and one manifest of every published tag read from the registry. A release missing one architecture publishes nothing. The keyserver fetch had no timeout and the fetch policy read only curl.
1 parent 4992326 commit 1a2b1da

45 files changed

Lines changed: 6308 additions & 158 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/build-deploy.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -550,3 +550,49 @@ jobs:
550550
else
551551
echo "::notice::publish_hub is false, so no Docker Hub tag was created or verified"
552552
fi
553+
554+
#----------------------------------------------------------------------------------#
555+
# The other half of the publish watchdog.
556+
#
557+
# ⛔ freshness-publish.yml watches whether this workflow's schedule is still
558+
# firing. Nothing would watch whether that one's is, so this runs the same
559+
# check and the two become each other's witness. A schedule that stops
560+
# produces no run, so the only place its silence can be seen is a run of
561+
# something else.
562+
#
563+
# ⛔ needs: nothing, and nothing needs it. A broken watchdog must not be able
564+
# to stop a publish: the run goes red while every tag is still created. That
565+
# is the correct shape for an alarm that is not a gate.
566+
#
567+
# ⚠ What this cannot see is itself. If the schedule on this workflow stops,
568+
# this job stops with it, which is exactly why the check also lives in a
569+
# workflow on a different schedule.
570+
#----------------------------------------------------------------------------------#
571+
watchdog:
572+
name: Check the schedules are still firing
573+
runs-on: ubuntu-latest
574+
permissions:
575+
contents: read
576+
actions: read
577+
steps:
578+
- name: Checkout
579+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
580+
with:
581+
persist-credentials: false
582+
583+
- name: Every schedule in the repository
584+
env:
585+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
586+
run: |
587+
set -uo pipefail
588+
scripts/check-schedules-fired
589+
rc=$?
590+
case "$rc" in
591+
0) echo "::notice::every schedule has fired within its own tolerance" ;;
592+
3)
593+
echo "::error::at least one schedule has stopped firing"
594+
echo "⛔ The publish itself is unaffected: this job gates nothing." >&2
595+
exit 1
596+
;;
597+
*) echo "the schedule check could not run" >&2; exit 1 ;;
598+
esac

‎.github/workflows/freshness-image-pins.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,12 @@ jobs:
8787
-t pin-check:amd64 .
8888
8989
echo "::group::image suite"
90+
# ⛔ CONTAINER_RUNTIME is not optional. The runner image carries both
91+
# docker and podman, and gen-evidence prefers podman when it finds
92+
# one. The image was built with docker and lives in docker's store, so
93+
# podman tries to pull it from Docker Hub and quay.io and fails.
9094
SOURCE_COMMIT="${{ github.sha }}" BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
95+
CONTAINER_RUNTIME=docker \
9196
scripts/gen-evidence amd64 pin-check:amd64 linux/amd64 /tmp/evidence-amd64.json
9297
IMAGE=pin-check:amd64 PLATFORM=linux/amd64 CONTAINER_RUNTIME=docker \
9398
EVIDENCE=/tmp/evidence-amd64.json tests/run.sh image

‎.github/workflows/freshness-keyring.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,8 +156,12 @@ jobs:
156156
--output "type=local,dest=/tmp/dbsnapshot" .
157157
158158
echo "::group::image suite"
159+
# ⛔ CONTAINER_RUNTIME is not optional. The runner image carries both
160+
# docker and podman, and gen-evidence prefers podman when it finds
161+
# one. The image was built with docker and lives in docker's store, so
162+
# podman tries to pull it from Docker Hub and quay.io and fails.
159163
SOURCE_COMMIT="${{ github.sha }}" BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
160-
DB_SNAPSHOT=/tmp/dbsnapshot \
164+
DB_SNAPSHOT=/tmp/dbsnapshot CONTAINER_RUNTIME=docker \
161165
scripts/gen-evidence "$DOCKER_ARCH" "freshness-check:$DOCKER_ARCH" "$PLATFORM" \
162166
"/tmp/evidence-$DOCKER_ARCH.json"
163167
IMAGE="freshness-check:$DOCKER_ARCH" PLATFORM="$PLATFORM" CONTAINER_RUNTIME=docker \

‎.github/workflows/freshness-publish.yml‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
name: Freshness, the publish itself
2+
3+
# ⛔ A scheduled workflow that stops firing produces no run, so no failure, no
4+
# annotation and no red mark. It has already happened here: scheduled
5+
# build-deploy.yml runs were daily and unbroken to 2026-07-13, then 45 days of
6+
# nothing, then one on 2026-08-27 that failed.
7+
# HISTORY/reviews/19-a-job-that-stops-running.md has the measurement.
8+
#
9+
# ⚠ Every workflow reported state "active" throughout those 45 days, so a check
10+
# that reads the state field concludes the schedule is healthy. Two things are
11+
# read instead, and both are outcomes rather than settings:
12+
#
13+
# scripts/check-publish-recency the newest dated index tag on the registry
14+
# scripts/check-schedules-fired the newest scheduled run of every schedule
15+
#
16+
# ⭐ This workflow and build-deploy.yml watch each other. build-deploy.yml runs
17+
# check-schedules-fired in a job of its own, so this one going quiet turns that
18+
# run red; this one runs the same check, so build-deploy.yml going quiet turns
19+
# this run red.
20+
#
21+
# ⚠ Written down rather than solved: neither sees every schedule in the
22+
# repository stopping at once. That is what GitHub does to a repository with 60
23+
# days of no activity, and it leaves no run for either check to be run by. Both
24+
# workflows are dispatchable, so a human asking gets the answer.
25+
#
26+
# ⛔ This opens no pull request. The other freshness workflows watch a pinned
27+
# thing and can propose the new pin; there is nothing to apply here. A publish
28+
# that stopped is fixed by finding out why, and the red run is the whole output.
29+
on:
30+
workflow_dispatch:
31+
schedule:
32+
- cron: "30 07 * * *" # daily, 07:30 UTC, two hours after the publish cron
33+
34+
defaults:
35+
run:
36+
shell: bash
37+
38+
permissions:
39+
contents: read
40+
41+
concurrency:
42+
group: freshness-publish
43+
cancel-in-progress: false
44+
45+
jobs:
46+
check:
47+
name: Check the publish is still happening
48+
runs-on: ubuntu-latest
49+
permissions:
50+
contents: read
51+
actions: read
52+
steps:
53+
- name: Checkout
54+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
55+
with:
56+
persist-credentials: false
57+
58+
# ⛔ Both checks run before either verdict is taken. Stopping at the first
59+
# failure would hide the second, and the two answer different questions:
60+
# one asks whether tags are still being written, the other whether the
61+
# schedules are still firing. A publish job that runs daily and publishes
62+
# nothing is green to the second and red to the first, which is the 59
63+
# days of green runs that policy 6 exists to prevent.
64+
- name: The newest dated index tag
65+
id: recency
66+
run: |
67+
set -uo pipefail
68+
scripts/check-publish-recency
69+
rc=$?
70+
echo "rc=$rc" >> "$GITHUB_OUTPUT"
71+
case "$rc" in
72+
0) echo "::notice::the publish is current" ;;
73+
3) echo "::error::the newest dated index tag is older than the schedule allows" ;;
74+
*) echo "the recency check could not run" >&2; exit 1 ;;
75+
esac
76+
77+
- name: Every schedule in the repository
78+
id: schedules
79+
env:
80+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
81+
run: |
82+
set -uo pipefail
83+
scripts/check-schedules-fired
84+
rc=$?
85+
echo "rc=$rc" >> "$GITHUB_OUTPUT"
86+
case "$rc" in
87+
0) echo "::notice::every schedule has fired within its own tolerance" ;;
88+
3) echo "::error::at least one schedule has stopped firing" ;;
89+
*) echo "the schedule check could not run" >&2; exit 1 ;;
90+
esac
91+
92+
- name: Take the verdict
93+
env:
94+
RECENCY: ${{ steps.recency.outputs.rc }}
95+
SCHEDULES: ${{ steps.schedules.outputs.rc }}
96+
run: |
97+
set -euo pipefail
98+
if [ "$RECENCY" = "0" ] && [ "$SCHEDULES" = "0" ]; then
99+
echo "the publish is happening and every schedule is firing"
100+
exit 0
101+
fi
102+
echo "publish recency: $RECENCY, schedules: $SCHEDULES (0 is healthy, 3 is stopped)" >&2
103+
echo "⛔ This red mark is the whole point of this workflow: the failure it" >&2
104+
echo "reports produces no run of its own to be red." >&2
105+
echo "reproduce: scripts/check-publish-recency; scripts/check-schedules-fired" >&2
106+
exit 1

‎.github/workflows/pacman-static.yml‎

Lines changed: 64 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -7,14 +7,13 @@ name: Static pacman
77
# pin and does not fetch these binaries. This produces release assets only, and
88
# tests/static/85-pacman-static-pin.sh fails if that ever stops being true.
99
#
10-
# ⭐ A dispatch builds and uploads for inspection and stops. Only a v* tag whose
11-
# commit is on the default branch can create a release, so a manual run cannot
12-
# publish one by accident.
10+
# ⭐ A dispatch builds and uploads for inspection and stops. Nothing here creates
11+
# a release: release.yml owns the v* tag and calls this workflow for the eight
12+
# binaries, so a manual run cannot publish one by accident and there is one
13+
# place that publishes rather than two.
1314
on:
1415
workflow_dispatch:
15-
push:
16-
tags:
17-
- 'v*'
16+
workflow_call:
1817

1918
defaults:
2019
run:
@@ -63,13 +62,49 @@ jobs:
6362
# The build runs natively; the emulator is only used to run the finished
6463
# binary and print its version. ⛔ Without one the run column reads
6564
# NOT MEASURED, so the emulator is what turns a link into a proof.
65+
# ⛔ meson comes from pip at a pinned version, not from apt.
66+
#
67+
# meson detects a linker by its banner, and the branch that recognises
68+
# zig's `zig ld` arrived in meson 1.6.0. ubuntu-latest ships 1.3.2 through
69+
# apt, so every architecture built all eleven libraries and then stopped
70+
# at the pacman step with "Unable to detect linker". Measured 2026-08-29
71+
# in run 33208408451.
72+
#
73+
# ⛔ Pinned rather than floated, for the same reason every other input
74+
# here is: the runner image's apt version is upstream's mood, and this
75+
# build's output must not depend on it. scripts/build-pacman-static
76+
# asserts the minimum separately, so a host with an old meson fails with
77+
# the version named rather than with meson's own message.
78+
#
79+
# ⚠ This pin has no freshness job watching it. HISTORY/pacman-static.md.
6680
- name: Install the build tools and the emulators
81+
env:
82+
MESON_VERSION: "1.10.1"
6783
run: |
6884
set -euo pipefail
6985
sudo apt-get update -qq
7086
sudo apt-get install -y -qq --no-install-recommends \
71-
cmake ninja-build meson gperf diffutils qemu-user-static
72-
echo "meson $(meson --version), ninja $(ninja --version)"
87+
cmake ninja-build gperf diffutils qemu-user-static gnupg
88+
89+
# An explicit branch on a capability, not a suppressed failure: the
90+
# runner image has pipx, and a host without it still gets a pinned
91+
# meson rather than whatever apt holds.
92+
if command -v pipx > /dev/null; then
93+
pipx install "meson==${MESON_VERSION}"
94+
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
95+
export PATH="$HOME/.local/bin:$PATH"
96+
else
97+
python3 -m pip install --break-system-packages --quiet "meson==${MESON_VERSION}"
98+
fi
99+
100+
got="$(meson --version)"
101+
if [ "$got" != "${MESON_VERSION}" ]; then
102+
echo "meson is $got, the pin says ${MESON_VERSION}" >&2
103+
echo "an unpinned meson is the runner image's apt version, which was 1.3.2" >&2
104+
command -v meson
105+
exit 1
106+
fi
107+
echo "meson $got, ninja $(ninja --version)"
73108
74109
# ⛔ Asserted, not assumed. The emulator for this target has to be on PATH
75110
# before the build starts, because the build treats a missing one as
@@ -107,6 +142,27 @@ jobs:
107142
WORK="${RUNNER_TEMP}/pacman-static" OUT="${RUNNER_TEMP}/dist" \
108143
scripts/build-pacman-static "$DOCKER_ARCH"
109144
145+
# ⛔ The pin names two pacman release manager fingerprints and the build
146+
# requires a signature from one of them. NOT VERIFIED is what an evidence
147+
# file carries when that never ran, and SKIPPED is what it carries when
148+
# somebody set PACMAN_TAG_VERIFY=skip. Neither may reach a release asset.
149+
- name: Refuse an asset whose signed tag was not verified
150+
env:
151+
DOCKER_ARCH: ${{ matrix.docker_arch }}
152+
run: |
153+
set -euo pipefail
154+
ev="${RUNNER_TEMP}/dist/pacman-static-${DOCKER_ARCH}.json"
155+
verified="$(jq -r .pacman_tag_verified_by "$ev")"
156+
signer="$(jq -r .pacman_tag_signer "$ev")"
157+
case "$verified" in
158+
"NOT VERIFIED" | SKIPPED | "" | null)
159+
echo "the ${DOCKER_ARCH} asset carries tag verification: $verified" >&2
160+
echo "an asset whose upstream signature was never checked is not evidence" >&2
161+
exit 1
162+
;;
163+
esac
164+
echo "::notice::${DOCKER_ARCH} signed tag verified by ${signer} (${verified})"
165+
110166
# ⛔ The build says NOT MEASURED when it could not run the binary. That is
111167
# the honest value and it must not reach a release, so it is refused here
112168
# rather than shipped with a caveat nobody reads.
@@ -133,64 +189,3 @@ jobs:
133189
${{ runner.temp }}/dist/pacman-static-${{ matrix.docker_arch }}.json
134190
if-no-files-found: error
135191
retention-days: 7
136-
137-
#----------------------------------------------------------------------------------#
138-
# Publish. Runs only for a v* tag, and only when every architecture built.
139-
#----------------------------------------------------------------------------------#
140-
release:
141-
name: Release
142-
needs: build
143-
if: startsWith(github.ref, 'refs/tags/v')
144-
runs-on: ubuntu-latest
145-
permissions:
146-
contents: write
147-
steps:
148-
# fetch-depth 0 because the tag has to be checked against the default
149-
# branch, and a shallow checkout cannot answer that.
150-
- name: Checkout
151-
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
152-
with:
153-
fetch-depth: 0
154-
persist-credentials: false
155-
156-
# ⛔ A tag on a commit that never reached main would publish assets built
157-
# from code nobody reviewed and nothing would say so afterwards.
158-
- name: Refuse a tag whose commit is not on the default branch
159-
env:
160-
DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
161-
run: |
162-
set -euo pipefail
163-
if ! git merge-base --is-ancestor HEAD "origin/${DEFAULT_BRANCH}"; then
164-
echo "this tag is not an ancestor of origin/${DEFAULT_BRANCH}" >&2
165-
echo "assets would be built from a commit that never reached the default branch" >&2
166-
exit 1
167-
fi
168-
echo "::notice::tag commit is on ${DEFAULT_BRANCH}"
169-
170-
- name: Download every architecture
171-
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
172-
with:
173-
path: assets
174-
merge-multiple: true
175-
176-
# ⛔ The body is generated from the evidence files, so the release notes
177-
# and the assets cannot disagree. A hand written body is a second record
178-
# that goes stale on the first bump.
179-
- name: Generate the release body
180-
run: |
181-
set -euo pipefail
182-
ls -1 assets
183-
scripts/release-notes assets "${GITHUB_REF_NAME}" > body.md
184-
cat body.md
185-
186-
- name: Publish
187-
env:
188-
GH_TOKEN: ${{ github.token }}
189-
TAG: ${{ github.ref_name }}
190-
run: |
191-
set -euo pipefail
192-
gh release create "$TAG" \
193-
--repo "$GITHUB_REPOSITORY" \
194-
--title "$TAG" \
195-
--notes-file body.md \
196-
assets/pacman-static-* assets/SHA256SUMS

0 commit comments

Comments
 (0)