Validate the release checksum contract for independent pinning (#1099)

* Add independently pinned threat-detect installation

* Replace standalone installer with release checksum contract validation

Author: davidslater

.github/workflows/publish-main.yml

@@ -38,6 +38,9 @@ jobs:
             echo "version=${VERSION}"
           } >> "$GITHUB_OUTPUT"
 
+      - name: Validate release checksum contract
+        run: bash scripts/validate-release-checksums.sh dist
+
       - name: Publish rolling main pre-release
         env:
           GH_TOKEN: ${{ github.token }}

.github/workflows/release.yml

@@ -38,6 +38,9 @@ jobs:
           cd dist
           sha256sum threat-detect-* > checksums.txt
 
+      - name: Validate release checksum contract
+        run: bash scripts/validate-release-checksums.sh dist
+
       - name: Upload release artifacts
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         with:
@@ -59,6 +62,9 @@ jobs:
           name: release-artifacts
           path: dist
 
+      - name: Validate downloaded release artifacts
+        run: bash scripts/validate-release-checksums.sh dist
+
       - name: Create GitHub release (prerelease)
         env:
           GH_TOKEN: ${{ github.token }}

README.md

@@ -675,6 +675,38 @@ with `contents: write`. `release-targets.txt` is the canonical build matrix for
 both tagged and rolling releases. The scheduled Release Platform Parity workflow
 compares its asset names with the platforms supported by `gh-aw`'s installer.
 
+### Checksum contract and independent pinning
+
+Each release supplies `checksums.txt` with exactly one entry for every binary
+in `release-targets.txt`, and no other entries. Each line consists of a
+64-character lowercase hexadecimal SHA-256 digest, two ASCII spaces, the asset
+basename, and a newline. Entries have no required ordering. The digest covers
+the exact published binary bytes, without packaging or transformation.
+
+Tagged and rolling release workflows validate the manifest against the matrix
+and hash every binary before publication. Tagged releases also validate after
+transferring artifacts into the publishing job. To run the same check locally:
+
+```bash
+bash scripts/validate-release-checksums.sh dist
+```
+
+This release contract supplies the inputs for
+[gh-aw#57792](https://github.com/github/gh-aw/issues/57792). The follow-up in
+gh-aw must review and commit the approved release tag and all platform digests
+together, emit those pins into compiled `*.lock.yml` detection jobs, and update
+its existing installer to verify against the emitted pin. A configured artifact
+mirror may change where bytes are downloaded, but must not change the expected
+digest. Missing pins or verification failures must prevent execution and
+downstream safe outputs, without falling back to an existing binary.
+
+Fetching `checksums.txt` alongside the binary at runtime is corruption detection,
+not an independent trust root. Trust comes from separately reviewing and pinning
+the release digests in gh-aw before compilation. The mutable rolling `main`
+release is for development, not production version pinning. This repository
+does not implement the compiler or installer changes; existing compiled
+workflows are unaffected by the release validation added here.
+
 Maintainers need to configure the following before the binary is consumed by `gh-aw`:
 
 1. Keep Actions enabled for this private repository.

scripts/test/validate-release-checksums-test.sh

@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# Offline contract tests use real hashes and no engine or network access.
+set -euo pipefail
+repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+fixture=$(mktemp -d)
+trap 'rm -rf "$fixture"' EXIT
+mkdir "$fixture/dist"
+cp "$repo_root/release-targets.txt" "$fixture/targets"
+if command -v sha256sum >/dev/null 2>&1; then
+  hash=(sha256sum)
+else
+  hash=(shasum -a 256)
+fi
+while read -r os arch asset; do
+  [[ -z "$os" || "$os" == \#* ]] && continue
+  printf 'fixture for %s/%s\n' "$os" "$arch" > "$fixture/dist/$asset"
+done < "$fixture/targets"
+(cd "$fixture/dist" && "${hash[@]}" threat-detect-*) > "$fixture/valid"
+cp "$fixture/valid" "$fixture/dist/checksums.txt"
+
+validate() {
+  bash "$repo_root/scripts/validate-release-checksums.sh" "$fixture/dist" "$fixture/targets" > "$fixture/log" 2>&1
+}
+reject() {
+  if validate; then
+    printf 'FAIL: accepted %s\n' "$1" >&2
+    exit 1
+  fi
+  if ! grep -q "$2" "$fixture/log"; then
+    cat "$fixture/log" >&2
+    printf 'FAIL: wrong diagnostic for %s\n' "$1" >&2
+    exit 1
+  fi
+}
+validate
+# The default matrix path works independently of the caller's working directory.
+(cd "$fixture" && bash "$repo_root/scripts/validate-release-checksums.sh" dist) > "$fixture/log"
+sort -r "$fixture/valid" > "$fixture/dist/checksums.txt"
+validate
+
+: > "$fixture/dist/checksums.txt"
+reject 'empty manifest' 'Missing checksum asset'
+sed '1d' "$fixture/valid" > "$fixture/dist/checksums.txt"
+reject 'missing platform' 'Missing checksum asset'
+cat "$fixture/valid" "$fixture/valid" > "$fixture/dist/checksums.txt"
+reject 'duplicate platform' 'Duplicate checksum asset'
+sed '1s/threat-detect-/unexpected-/' "$fixture/valid" > "$fixture/dist/checksums.txt"
+reject 'unexpected asset' 'Unexpected checksum asset'
+sed '1s/^./g/' "$fixture/valid" > "$fixture/dist/checksums.txt"
+reject 'nonhex digest' 'Invalid checksum line'
+sed '1s/^.//' "$fixture/valid" > "$fixture/dist/checksums.txt"
+reject 'short digest' 'Invalid checksum line'
+tr 'abcdef' 'ABCDEF' < "$fixture/valid" > "$fixture/dist/checksums.txt"
+reject 'uppercase digest' 'Invalid checksum line'
+sed '1s/  / /' "$fixture/valid" > "$fixture/dist/checksums.txt"
+reject 'wrong separator' 'Invalid checksum line'
+sed '1s/$/ extra/' "$fixture/valid" > "$fixture/dist/checksums.txt"
+reject 'extra field' 'Invalid checksum line'
+sed '1s|  |  ../|' "$fixture/valid" > "$fixture/dist/checksums.txt"
+reject 'path traversal' 'Unexpected checksum asset'
+printf '%s' "$(cat "$fixture/valid")" > "$fixture/dist/checksums.txt"
+reject 'missing final newline' 'must end with a newline'
+
+cp "$fixture/valid" "$fixture/dist/checksums.txt"
+: > "$fixture/targets"
+reject 'empty matrix' 'Release target matrix is empty'
+cat "$repo_root/release-targets.txt" "$repo_root/release-targets.txt" > "$fixture/targets"
+reject 'duplicate matrix' 'Duplicate release'
+printf 'linux amd64 ../binary\n' > "$fixture/targets"
+reject 'malformed matrix' 'Invalid release target'
+cp "$repo_root/release-targets.txt" "$fixture/targets"
+printf 'tampered\n' > "$fixture/dist/threat-detect-linux-amd64"
+reject 'digest mismatch' 'FAILED'
+printf 'fixture for linux/amd64\n' > "$fixture/dist/threat-detect-linux-amd64"
+mv "$fixture/dist/threat-detect-linux-arm64" "$fixture/absent"
+reject 'missing binary' 'threat-detect-linux-arm64'
+mv "$fixture/dist/checksums.txt" "$fixture/absent-checksums"
+reject 'missing manifest' 'checksums.txt'
+printf 'PASS: release checksum contract\n'

scripts/validate-release-checksums.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# Validate the release manifest against the platform matrix and actual bytes.
+set -euo pipefail
+export LC_ALL=C
+
+repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+if [[ $# -lt 1 || $# -gt 2 ]]; then
+  echo 'ERROR: Expected an artifact directory and optional target matrix. Example: bash scripts/validate-release-checksums.sh dist release-targets.txt' >&2
+  exit 1
+fi
+artifact_dir=$1
+targets_file=${2:-$repo_root/release-targets.txt}
+
+# Parse the matrix first, including when it is empty (NR == FNR is unsafe then).
+awk '
+  function fail(message) {
+    print "ERROR: " message > "/dev/stderr"
+    bad = 1
+  }
+  FILENAME == ARGV[1] {
+    if ($0 ~ /^[[:space:]]*(#|$)/) next
+    if (NF != 3 || $3 !~ /^threat-detect-[a-z0-9-]+$/) {
+      fail("Invalid release target. Expected <goos> <goarch> <asset>, for example linux amd64 threat-detect-linux-amd64.")
+      next
+    }
+    if (platforms[$1 "/" $2]++) fail("Duplicate release platform: " $1 "/" $2)
+    if (assets[$3]++) fail("Duplicate release asset: " $3)
+    count++
+    next
+  }
+  {
+    # Match the exact sha256sum text-mode format used by release generation.
+    if (length($1) != 64 || $1 ~ /[^0-9a-f]/ || NF != 2 || $0 != $1 "  " $2) {
+      fail("Invalid checksum line " FNR ". Expected 64 lowercase hexadecimal characters, two spaces, and a release asset name.")
+      next
+    }
+    if (!($2 in assets)) fail("Unexpected checksum asset: " $2)
+    if (seen[$2]++) fail("Duplicate checksum asset: " $2)
+  }
+  END {
+    if (!count) fail("Release target matrix is empty. Expected at least one platform asset.")
+    for (asset in assets) if (!(asset in seen)) fail("Missing checksum asset: " asset)
+    exit bad ? 1 : 0
+  }
+' "$targets_file" "$artifact_dir/checksums.txt"
+
+if [[ "$(tail -c 1 "$artifact_dir/checksums.txt" | od -An -tu1 | tr -d '[:space:]')" != 10 ]]; then
+  echo 'ERROR: Checksum manifest must end with a newline. Regenerate it with sha256sum.' >&2
+  exit 1
+fi
+
+# Only hash after validating the manifest filenames against the trusted matrix.
+if command -v sha256sum >/dev/null 2>&1; then
+  (cd "$artifact_dir" && sha256sum --check checksums.txt)
+else
+  (cd "$artifact_dir" && shasum -a 256 --check checksums.txt)
+fi

specs/usage-spec.md

@@ -51,8 +51,9 @@ here as `TD-XX`.
 ## 2. Acquisition
 
 **U-01**: A conforming host MUST acquire the detector as a published GitHub
-Release asset from `github/gh-aw-threat-detection`. The host MUST NOT build the
-detector from source as part of a production detection job.
+Release asset from `github/gh-aw-threat-detection`, either directly or through
+an organization-approved mirror or preinstallation of that asset. The host MUST
+NOT build the detector from source as part of a production detection job.
 
 **U-02**: The host MUST select the release asset matching the runner operating
 system and architecture: `threat-detect-linux-amd64`,
@@ -63,6 +64,33 @@ system and architecture: `threat-detect-linux-amd64`,
 recorded for that asset (via `checksums.txt` published alongside the assets, or
 the sha256 recorded in the release notes) before executing it.
 
+**U-03a**: A host requiring independently pinned artifacts MUST obtain the
+expected SHA-256 from trusted configuration reviewed separately from runtime
+acquisition (for example, compiler-embedded digests or a checksum file committed
+with the consuming workflow). Downloading both the binary and its checksum from
+the release or mirror during installation does not satisfy this requirement.
+Pins MUST be associated with an explicit release tag and platform asset name.
+Missing, malformed, duplicate, or mismatched pins MUST fail installation before
+execution, with no fallback to an unverified or existing binary. An approved
+mirror changes the source of the bytes, not the expected digest. A preinstalled
+asset MUST also be verified against the independent pin before execution.
+
+**U-03b**: A detector release MUST publish `checksums.txt` with exactly one
+entry for each binary asset in the release source revision's `release-targets.txt`
+and no additional entries. Each line MUST contain a 64-character lowercase
+hexadecimal SHA-256 digest, two ASCII spaces, the asset basename, and a newline.
+Entry order is not significant. Each digest MUST match the exact published
+binary bytes. Release publication MUST fail if the manifest is malformed,
+contains missing, duplicate, or unexpected entries, or does not match the
+binary assets. This applies to both tagged and rolling releases; a rolling
+release still MUST NOT be used as a production version pin (U-04).
+
+The release manifest supplies inputs for the host's independent pinning process;
+it is not itself an independent trust root. A host such as gh-aw must separately
+review and pin the release tag and platform digests, then embed them into its
+compiled workflows. Neither release validation nor publication implements that
+host integration.
+
 **U-04**: The host MUST pin acquisition to an explicit release tag (per TD-25,
 U-24, and U-26). A host that resolves the latest promoted (stable) release MUST
 first resolve it to a concrete release tag and then download that pinned tag; it