From 711b2ac55d1f9fc458d28570021ee9f205244908 Mon Sep 17 00:00:00 2001 From: "SUSE Observability AI (POC)" Date: Wed, 9 Sep 2026 09:31:51 +0000 Subject: [PATCH 1/2] Copy the datadog-agent checkout with cp instead of installing rsync Every job in this repository has failed since 2026-09-07 in the first CI step, on both architectures, with: E: Release file for http://deb.debian.org/debian-security/dists/ bullseye-security/InRelease is expired Process completed with exit code 100 The prebuild images quay.io/stackstate/datadog_build_system-probe_x64 and _arm64 at tag 61b4ad67 are Debian 11 bullseye, which is EOL. Its bullseye-security Release file expired on 2026-09-07, so apt-get update now exits 100 and no package can be installed from those images. That is the only reason this step ran apt at all: rsync is absent from the image and was installed just to copy the checkout into the work directory. Reproduced directly in quay.io/stackstate/datadog_build_system-probe_x64 :61b4ad67 rather than inferred from the log: apt-get update exits 100 on that one expired Release file, and /etc/apt/sources.list points at deb.debian.org, which no longer carries bullseye. cp -au is equivalent here and needs no network. Both are non-deleting copies of the directory contents that skip files not newer than the destination, so re-running is still cheap. cp -a additionally preserves hard links, which rsync -a does not. Verified in the same image (GNU coreutils 8.32): identical file count and modes, and a second run copies nothing. This is not a CVE fix, but it blocks every candidate in this repository, including the CVE-2026-84445 grpc bump in #278. Pinning the prebuild images to a supported Debian release is the durable fix and belongs with whoever owns those images; this only removes the dependency on installing packages inside them. --- .../run-datadog-agent-prebuild.sh | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/prebuild-datadog-agent-scripts/run-datadog-agent-prebuild.sh b/prebuild-datadog-agent-scripts/run-datadog-agent-prebuild.sh index 167cd2ac..e1cca97e 100755 --- a/prebuild-datadog-agent-scripts/run-datadog-agent-prebuild.sh +++ b/prebuild-datadog-agent-scripts/run-datadog-agent-prebuild.sh @@ -11,13 +11,11 @@ then echo "$WORKDIR is a symlink to a directory. It is your responsibility to ensure that the directory has the up-to-date code." else - if ! type "rsync" > /dev/null; then - apt-get update - DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends rsync - fi - + # coreutils cp, not rsync: the prebuild images are Debian bullseye, which is + # EOL, so apt-get update exits 100 on the expired bullseye-security Release + # file and installing rsync is no longer possible. cp -au needs no network. mkdir -p $WORKDIR - rsync -au "$SOURCEDIR"/. $WORKDIR + cp -au "$SOURCEDIR"/. $WORKDIR chown -R root:root $WORKDIR fi From 901935ce7f31ad28bc7183014d19b3cbdfe5eacc Mon Sep 17 00:00:00 2001 From: "SUSE Observability AI (POC)" Date: Wed, 9 Sep 2026 11:23:06 +0000 Subject: [PATCH 2/2] Replace destinations whose type changed before copying the checkout Review of 711b2ac5 found that cp -au is not equivalent to the rsync it replaced once WORKDIR is reused. WORKDIR is datadog-agent-workdir, which is constant, while SOURCEDIR is prebuild_artifacts/checkout/$VERSION, which is version-scoped, so a dependency bump copies a different tree into the same workdir. That is exactly when a path can change type. --clean does not remove the workdir either, so a developer keeps it across versions indefinitely. Reproduced in quay.io/stackstate/datadog_build_system-probe_x64:61b4ad67 with GNU coreutils 8.32, replaying two consecutive runs over one workdir where an upstream file becomes a directory and a symlink becomes a directory: cp: cannot overwrite non-directory '/tmp/workdir/./pkg/ebpf/legacy_helper' with directory Every later run fails the same way in 173ms, so the workdir stays wedged until someone deletes it by hand. Hosted CI starts from an empty workspace, which is why run 34335257004 was green. The defect is wider than the two hard errors. With the destination no older, cp -u also left a stale file where the source had become a symlink and a stale symlink where it had become a file, exiting 0 both times, so the build would have run against a tree that does not match the source. Fixed by unlinking just those destinations before the copy. Measured against real rsync 3.2.7 rather than assumed, because rsync is absent from both prebuild images: rsync replaces a non-directory with a directory and swaps file for symlink either way, but refuses to replace a directory with a non-directory, exiting 23 with "could not make way for". This matches that contract, including the refusal, so no directory is ever deleted and generated ebpf output cannot be destroyed. rm -f rather than rm -rf enforces it: a directory would fail instead of being removed. Paths absent from SOURCEDIR are untouched, keeping the non-delete behaviour that git ls-files --others --ignored relies on to collect artifacts. rsync was not reintroduced. apt-get update still exits 100 in these images on the expired bullseye-security Release file, so installing it remains impossible; a BCI-based prebuild image is the deeper fix and belongs in its own ticket. Validation, extracting the block from the committed script so the shipped code is what runs: 15 cases pass on the fix and 10 of them fail on 711b2ac5, covering all eight type transitions, symlink write-through into an outside directory, non-delete of generated artifacts, -u in both directions, and permissions, symlinks, hidden files and hard links on a fresh destination. A 21,646-file two-version reuse passes all ten assertions on the fix and fails five on 711b2ac5. xtrace is off for the loop. Under set -ex it emitted about 14 lines per path, 28,021 lines for 2,000 files, which would bury the build log; that is now 5. The walk costs roughly 1.2 to 1.7s per run at 21,646 files, against a multi-minute ebpf build. --- .../run-datadog-agent-prebuild.sh | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/prebuild-datadog-agent-scripts/run-datadog-agent-prebuild.sh b/prebuild-datadog-agent-scripts/run-datadog-agent-prebuild.sh index e1cca97e..292590a4 100755 --- a/prebuild-datadog-agent-scripts/run-datadog-agent-prebuild.sh +++ b/prebuild-datadog-agent-scripts/run-datadog-agent-prebuild.sh @@ -15,6 +15,53 @@ else # EOL, so apt-get update exits 100 on the expired bullseye-security Release # file and installing rsync is no longer possible. cp -au needs no network. mkdir -p $WORKDIR + + # WORKDIR is reused across dependency versions while SOURCEDIR is + # version-scoped, so a path can change type between runs. cp then either fails + # outright (it cannot overwrite a non-directory with a directory) or, when -u + # finds the destination no older, silently keeps the wrong type. rsync + # replaced the destination in both cases, so unlink it first. + # + # Only non-directories are ever unlinked, using rm -f so that a directory + # would fail rather than be removed: paths absent from SOURCEDIR must survive + # to keep cp -au's non-delete behaviour, which the generated ebpf artifacts + # rely on. + # + # xtrace off for the loop: it runs once per path in the dependency, which is + # about 14 trace lines each and would bury the rest of the build log. + set +x + ( cd "$SOURCEDIR" && find . -mindepth 1 -print0 ) | + while IFS= read -r -d '' rel; do + rel=${rel#./} + src="$SOURCEDIR/$rel" + dst="$WORKDIR/$rel" + + if [ -L "$dst" ]; then dst_type=l + elif [ -d "$dst" ]; then dst_type=d + elif [ -e "$dst" ]; then dst_type=f + else continue + fi + + if [ -L "$src" ]; then src_type=l + elif [ -d "$src" ]; then src_type=d + else src_type=f + fi + + if [ "$src_type" = "$dst_type" ]; then + continue + fi + + if [ "$dst_type" = d ]; then + # rsync refused this too, rather than deleting a directory that may hold + # generated output. Stop instead of building against a stale tree. + echo "$rel is a directory in $WORKDIR but not in $SOURCEDIR: delete $WORKDIR and re-run" >&2 + exit 1 + fi + + rm -f "$dst" + done + set -x + cp -au "$SOURCEDIR"/. $WORKDIR chown -R root:root $WORKDIR fi