From 5583e8f6ca2e1cc6ebd07f357774a0c6749cf0c0 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Mon, 16 Nov 2026 21:32:03 -0500 Subject: [PATCH 1/5] Run the hosted Go flow on real Bazel in CI socket-patch has no Bazel-specific code, and until now no test ran it in a Bazel workspace. This adds a test target that does, and a workflow that runs it on pinned Bazel releases. No file under crates/*/src changes. The first tests hold hosted mode for a Go module that comes through gazelle's go_deps.from_file under Bzlmod, on Bazel 9.3.2, 8.8.2 and 7.7.2. Each one, in a fresh output base: - builds the fixture workspace and reads "OUT: PRISTINE" from `bazel run //:consumer`; - runs `scan --mode hosted --yes`, which rewrites go.mod and go.sum and nothing else; - runs the same two Bazel commands and reads the patched marker, which holds an id made for the run: the Socket module came from the stand-in patch host, Bazel changed no file of the checkout, and a `--lockfile_mode=error` build passes; - runs `rollback --yes`: every file of the checkout is the one of before, byte for byte, and the build prints "OUT: PRISTINE" again; - builds the rewritten checkout in a second output base against a patch host that serves other bytes: the build fails on the sum. The Socket API, the Go proxy, the patch host and the checksum-database lookup are one loopback wiremock server, and `go` is given no public proxy. The Bazel registry, the ruleset archives, the Go release list and the Go SDK come from the network. Bazel 9's repository contents cache stays on, as a user has it, with one cache per workspace copy. - crates/socket-patch-cli/tests/e2e_bazel_build/: harness.rs keeps every Bazel write inside a per-test scratch root (startup options, environment, repository cache, contents cache, server shutdown) and runs the socket-patch child; go.rs builds the two modules, the stand-in and the flow. The tests are #[ignore]d and print SKIP without a Bazel, or fail when SOCKET_PATCH_BAZEL_E2E_REQUIRED is set. Two hermetic tests run on every Unix runner. - crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/{e9,e7}: the two workspaces (gazelle 0.54.0 with rules_go 0.65.0, and gazelle 0.45.0 with rules_go 0.53.0 for Bazel 7). - scripts/install-bazel.sh and scripts/bazel-pins.txt: one Bazel release binary, checked against a pinned sha256 before it is installed. - .github/workflows/bazel-compatibility.yml: one Linux leg per Bazel release on a pull request that touches these files or a Go file the tests run through, on a main push that also touches the shared engine code and test helpers, nightly and on dispatch; one macOS leg nightly and on dispatch. Not required for merge. A later test of the same flow is one fixture directory and one test function. A new Bazel major adds its pin lines and one matrix line; a new ecosystem adds its module and one matrix line per release. Co-Authored-By: Claude Fable 5.2 --- .github/workflows/bazel-compatibility.yml | 205 +++++++ .../tests/e2e_bazel_build/go.rs | 409 ++++++++++++++ .../tests/e2e_bazel_build/harness.rs | 520 ++++++++++++++++++ .../tests/e2e_bazel_build/main.rs | 44 ++ .../bazel/gazelle/bzlmod/e7/BUILD.bazel | 8 + .../bazel/gazelle/bzlmod/e7/MODULE.bazel | 14 + .../fixtures/bazel/gazelle/bzlmod/e7/go.mod | 5 + .../fixtures/bazel/gazelle/bzlmod/e7/go.sum | 2 + .../fixtures/bazel/gazelle/bzlmod/e7/main.go | 9 + .../bazel/gazelle/bzlmod/e9/BUILD.bazel | 8 + .../bazel/gazelle/bzlmod/e9/MODULE.bazel | 15 + .../fixtures/bazel/gazelle/bzlmod/e9/go.mod | 5 + .../fixtures/bazel/gazelle/bzlmod/e9/go.sum | 2 + .../fixtures/bazel/gazelle/bzlmod/e9/main.go | 9 + scripts/bazel-pins.txt | 20 + scripts/install-bazel.sh | 113 ++++ 16 files changed, 1388 insertions(+) create mode 100644 .github/workflows/bazel-compatibility.yml create mode 100644 crates/socket-patch-cli/tests/e2e_bazel_build/go.rs create mode 100644 crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs create mode 100644 crates/socket-patch-cli/tests/e2e_bazel_build/main.rs create mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/BUILD.bazel create mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/MODULE.bazel create mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.mod create mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.sum create mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/main.go create mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/BUILD.bazel create mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/MODULE.bazel create mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.mod create mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.sum create mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/main.go create mode 100644 scripts/bazel-pins.txt create mode 100755 scripts/install-bazel.sh diff --git a/.github/workflows/bazel-compatibility.yml b/.github/workflows/bazel-compatibility.yml new file mode 100644 index 000000000..b983504f5 --- /dev/null +++ b/.github/workflows/bazel-compatibility.yml @@ -0,0 +1,205 @@ +name: Bazel patch compatibility + +# socket-patch in a Bazel workspace, on a real Bazel: one leg per Bazel +# release and ecosystem runs that ecosystem's `#[ignore]` tests of +# `e2e_bazel_build`, by name prefix (`go::b9_` on Bazel 9.3.2). A test builds +# a fixture workspace, runs the built binary in it and reads the result from +# what Bazel builds next. +# +# The package under test and every Socket host are loopback stand-ins. A leg +# still needs the network for the ruleset and the toolchain: github.com and +# release-assets.githubusercontent.com (the Bazel release, cached per +# release, and the ruleset archives), dl.google.com (the Go SDK) and, on +# every run, bcr.bazel.build (the registry) and go.dev (the Go release +# list). A repository cache saved from main holds the archives and the SDK. +# +# Not required for merge: no `merge_group` trigger, and no job here is a +# `needs` of ci.yml's `ci-ok`. Setting the repository variable +# BAZEL_COMPAT_DISABLED to `true` turns every leg off without a merge. +# +# A leg selects its tests by the major of its release, so a major has one +# release here. To add a Bazel major: one line per platform in +# scripts/bazel-pins.txt and one matrix line here. To add an ecosystem: its +# module in `e2e_bazel_build` and one matrix line per release. + +on: + pull_request: + types: [opened, synchronize, reopened, ready_for_review] + # The suite's own files and the Go files its tests run through, as + # go-compatibility.yml lists them. Shared engine code + # (patch/redirect/mod.rs, hosted/, commands/) gets its run on push to + # main (when it is in the push filter below) or nightly, or on demand + # via workflow_dispatch on the PR branch. + paths: + - '.github/workflows/bazel-compatibility.yml' + - 'scripts/install-bazel.sh' + - 'scripts/bazel-pins.txt' + - 'crates/socket-patch-cli/tests/e2e_bazel_build/**' + - 'crates/socket-patch-cli/tests/fixtures/bazel/**' + - 'crates/*/src/**/*golang*' + - 'crates/*/src/**/*golang*/**' + - 'crates/socket-patch-core/src/vendor/go*.rs' + push: + branches: [main] + # The PR paths, the shared engine code and test helpers the tests run + # through, and the toolchain files. A main push outside this set waits + # for the nightly run below. + paths: + - '.github/workflows/bazel-compatibility.yml' + - 'scripts/install-bazel.sh' + - 'scripts/bazel-pins.txt' + - 'crates/socket-patch-cli/tests/e2e_bazel_build/**' + - 'crates/socket-patch-cli/tests/fixtures/bazel/**' + - 'crates/*/src/**/*golang*' + - 'crates/*/src/**/*golang*/**' + - 'crates/socket-patch-core/src/vendor/go*.rs' + - 'crates/socket-patch-core/src/patch/redirect/**' + - 'crates/socket-patch-core/src/hosted/**' + - 'crates/socket-patch-cli/src/commands/**' + - 'crates/socket-patch-cli/tests/common/**' + - 'crates/socket-patch-cli/tests/hosted_memory_common/**' + - 'Cargo.lock' + - 'Cargo.toml' + - 'crates/*/Cargo.toml' + - 'rust-toolchain.toml' + - '.cargo/**' + schedule: + # Nightly, so a main change outside the push filter still gets a run + # within a day. + - cron: '17 4 * * *' + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: bazel-compat-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +env: + SOCKET_NO_CONFIG: '1' + SOCKET_NO_UPDATE_CHECK: '1' + SOCKET_TELEMETRY_DISABLED: '1' + +jobs: + bazel: + name: ${{ matrix.eco }} / Bazel ${{ matrix.bazel }} (${{ matrix.os }}) + if: github.event.pull_request.draft != true && vars.BAZEL_COMPAT_DISABLED != 'true' + strategy: + fail-fast: false + matrix: + include: + # `fixtures`: the ecosystem's directory under + # tests/fixtures/bazel, which keys its repository cache. + - {os: ubuntu-latest, bazel: '9.3.2', eco: go, fixtures: gazelle} + - {os: ubuntu-latest, bazel: '8.8.2', eco: go, fixtures: gazelle} + - {os: ubuntu-latest, bazel: '7.7.2', eco: go, fixtures: gazelle} + runs-on: ${{ vars.DISABLE_DEPOT_RUNNERS != 'true' && (matrix.os == 'ubuntu-latest' && 'depot-ubuntu-24.04-4' || matrix.os == 'ubuntu-22.04' && 'depot-ubuntu-22.04-4') || matrix.os }} + timeout-minutes: 45 + steps: &bazel-steps + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + + - name: Shellcheck the Bazel installer + # shellcheck is pre-installed on the Linux runners only. + if: runner.os == 'Linux' + run: shellcheck scripts/install-bazel.sh + + - run: rustup show + + - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1 + with: + shared-key: dev-${{ matrix.os }} + save-if: ${{ github.ref == 'refs/heads/main' }} + + - name: Restore the Bazel ${{ matrix.bazel }} binary + # The installer checks a restored file against its pin too, so a + # miss or a stale entry only means a download. Saved from main. + id: bazel-dist + uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0 + timeout-minutes: 5 + continue-on-error: true + with: + path: ${{ runner.temp }}/bazel-dist + key: bazel-dist-${{ matrix.bazel }}-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('scripts/bazel-pins.txt', 'scripts/install-bazel.sh') }} + + - name: Install Bazel ${{ matrix.bazel }} + env: + BAZEL_VERSION: ${{ matrix.bazel }} + # Prints SOCKET_PATCH_BAZEL_E2E_BAZEL, the pinned and verified binary. + run: scripts/install-bazel.sh "$BAZEL_VERSION" "$RUNNER_TEMP/bazel-dist" >> "$GITHUB_ENV" + + - name: Save the Bazel ${{ matrix.bazel }} binary + if: >- + github.ref == 'refs/heads/main' + && steps.bazel-dist.outputs.cache-hit != 'true' + && steps.bazel-dist.outcome == 'success' + uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0 + with: + path: ${{ runner.temp }}/bazel-dist + key: ${{ steps.bazel-dist.outputs.cache-primary-key }} + + - name: Restore the repository cache + # What Bazel downloads for the fixtures: ruleset archives and the + # language toolchain (about 100 MB for Go). Keyed by the + # ecosystem's fixtures, which pin all of it, so another ecosystem's + # fixture does not make this leg save a new copy; a run whose + # fixtures changed starts from the newest cache of its leg. The + # cache is content-addressed, so an older one is still correct, + # only incomplete. The package under test never comes from it: `go` + # fetches a module into the output base, which every test makes anew. + id: bazel-repos + uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0 + timeout-minutes: 5 + continue-on-error: true + with: + path: ${{ runner.temp }}/bazel-repos + key: bazel-repos-${{ matrix.eco }}-${{ matrix.bazel }}-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles(format('crates/socket-patch-cli/tests/fixtures/bazel/{0}/**', matrix.fixtures)) }} + restore-keys: | + bazel-repos-${{ matrix.eco }}-${{ matrix.bazel }}-${{ runner.os }}-${{ runner.arch }}- + + - name: Run the ${{ matrix.eco }} tests on Bazel ${{ matrix.bazel }} + env: + # A missing Bazel or another release fails the leg instead of + # printing SKIP. + SOCKET_PATCH_BAZEL_E2E_REQUIRED: '1' + SOCKET_PATCH_BAZEL_E2E_REPO_CACHE: ${{ runner.temp }}/bazel-repos + BAZEL_VERSION: ${{ matrix.bazel }} + ECO: ${{ matrix.eco }} + # Two tests at a time: each runs a Bazel server of its own. + run: | + set -euo pipefail + filter="${ECO}::b${BAZEL_VERSION%%.*}_" + cargo test --locked -p socket-patch-cli --test e2e_bazel_build -- \ + --ignored "$filter" --test-threads=2 --nocapture 2>&1 | tee "$RUNNER_TEMP/bazel-tests.log" + # A filter that selects no test passes for cargo. + passed=$(sed -n 's/^test result: .* \([0-9][0-9]*\) passed;.*/\1/p' "$RUNNER_TEMP/bazel-tests.log" | head -n 1) + if [ "${passed:-0}" = 0 ]; then + echo "::error::no test of e2e_bazel_build matches $filter" + exit 1 + fi + + - name: Save the repository cache + if: >- + github.ref == 'refs/heads/main' + && steps.bazel-repos.outputs.cache-hit != 'true' + && steps.bazel-repos.outcome == 'success' + uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0 + with: + path: ${{ runner.temp }}/bazel-repos + key: ${{ steps.bazel-repos.outputs.cache-primary-key }} + + # The macOS rows of `bazel`: nightly and on demand, never on a pull + # request or a push, so they do not queue on the small macOS runner pool. + bazel-macos: + name: ${{ matrix.eco }} / Bazel ${{ matrix.bazel }} (${{ matrix.os }}) + if: (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && vars.BAZEL_COMPAT_DISABLED != 'true' + strategy: + fail-fast: false + matrix: + include: + - {os: macos-latest, bazel: '9.3.2', eco: go, fixtures: gazelle} + runs-on: ${{ vars.DISABLE_DEPOT_RUNNERS != 'true' && (matrix.os == 'ubuntu-latest' && 'depot-ubuntu-24.04-4' || matrix.os == 'ubuntu-22.04' && 'depot-ubuntu-22.04-4') || matrix.os }} + timeout-minutes: 45 + steps: *bazel-steps diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs new file mode 100644 index 000000000..909aa2446 --- /dev/null +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs @@ -0,0 +1,409 @@ +//! Hosted mode for a Go module that comes through gazelle's +//! `go_deps.from_file(go_mod = "//:go.mod")` under Bzlmod. +//! +//! One flow per Bazel release, in a fresh output base: +//! +//! - (a) `bazel build` and `bazel run //:consumer` print the unpatched +//! marker. +//! - (b) `socket-patch scan --mode hosted --yes` rewrites `go.mod` and +//! `go.sum`, and nothing else. +//! - (c) The same two Bazel commands, with no step in between, print the +//! patched marker; the zip of the Socket module came from the stand-in +//! patch host, Bazel changed no file of the checkout, and a +//! `--lockfile_mode=error` build passes. +//! - (d) `socket-patch rollback --yes`: every file of the checkout is the +//! one of before, byte for byte, and the build prints the unpatched +//! marker again. +//! - (e) On a second machine (a new output base and, on Bazel 9, a new +//! contents cache, so no module fetched before is there) with the +//! rewritten `go.mod` and `go.sum`, a patch host that serves other bytes +//! fails the build on the sum, and the upstream module is not fetched in +//! its place. +//! +//! The marker is read from what Bazel built, and the patched one holds an +//! id made for the run, so no cache of an earlier run can supply it. +//! +//! One loopback server stands in for four hosts, by route prefix: the +//! Socket API, `proxy.golang.org` (the upstream module), `patch.socket.dev` +//! (the Socket module, in the GOPROXY layout) and `sum.golang.org` (the +//! lookup `rollback` reads through `SOCKET_GOSUMDB_URL`). gazelle fetches a +//! module with `go mod download`, which reads `GOPROXY`, so every Bazel +//! command gets +//! `--repo_env=GOPROXY=,` from the +//! first build on and step (c) changes no flag. `go` asks the first entry +//! and goes on to the second after a 404, as it goes from the public proxy +//! to the real patch host in a build outside these tests. No public proxy +//! is in the list: on these gazelle releases the two modules of the test +//! are all that `go` asks a proxy for. + +use std::io::Write as _; +use std::time::Instant; + +use base64::Engine as _; +use serde_json::json; +use sha2::{Digest, Sha256}; +use wiremock::matchers::{method, path}; +use wiremock::{Mock, MockServer, ResponseTemplate}; + +use crate::envelope::event_for; +use crate::harness::{changed, fixture_path, Cell, CONSUMER}; +use crate::hosted_memory_common::{mount_api, Patch}; + +/// The module both fixtures require, and the version `go.sum` pins. +const MODULE: &str = "example.com/upstream"; +const VERSION: &str = "v1.0.0"; +const PURL: &str = "pkg:golang/example.com/upstream@v1.0.0"; +/// Its two files. The fixtures' `go.sum` holds the `h1:` of these bytes. +const GO_MOD: &[u8] = b"module example.com/upstream\n\ngo 1.21\n"; +const UPSTREAM_LIB: &[u8] = b"package upstream\n\nfunc Greeting() string { return \"PRISTINE\" }\n"; +const UNPATCHED: &str = "OUT: PRISTINE"; +/// The version of the Socket module that replaces it. +const SOCKET_VERSION: &str = "v1.0.0-socketpatch.1"; +/// The route prefixes of the stand-in, one per host it plays. +const PROXY: &str = "proxy.golang.org"; +const PATCH_HOST: &str = "patch.socket.dev"; +const SUMDB: &str = "sum.golang.org"; +/// What [`changed`] gives for two equal trees. +const NOTHING: [&str; 0] = []; + +/// Go's `h1:` hash of a set of files: the sha256 of one +/// ` \n` line per file, in the order of the names. +fn h1(files: &[(String, &[u8])]) -> String { + let mut files = files.to_vec(); + files.sort(); + let mut summary = String::new(); + for (name, bytes) in files { + summary.push_str(&format!("{} {name}\n", hex::encode(Sha256::digest(bytes)))); + } + let digest = Sha256::digest(summary.as_bytes()); + let encoded = base64::engine::general_purpose::STANDARD.encode(digest); + format!("h1:{encoded}") +} + +/// One version of a module holding [`GO_MOD`] and a `lib.go`, as a Go +/// module proxy serves it. An `h1:` covers the names and the contents of +/// the files, not the bytes of the zip, so the zip is built here and no +/// binary is committed. +struct Module { + path: String, + version: String, + zip: Vec, + /// The ` ` line of `go.sum`. + zip_h1: String, + /// The ` /go.mod` line of `go.sum`. + mod_h1: String, +} + +impl Module { + fn new(path: &str, version: &str, lib: &[u8]) -> Module { + let name = |file: &str| format!("{path}@{version}/{file}"); + let files = [(name("go.mod"), GO_MOD), (name("lib.go"), lib)]; + let mut archive = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + for (name, bytes) in &files { + let options = zip::write::SimpleFileOptions::default(); + archive.start_file(name.as_str(), options).expect("zip"); + archive.write_all(bytes).expect("zip"); + } + Module { + path: path.to_string(), + version: version.to_string(), + zip: archive.finish().expect("zip").into_inner(), + zip_h1: h1(&files), + mod_h1: h1(&[("go.mod".to_string(), GO_MOD)]), + } + } + + /// The two lines a `go.sum` holds for this module version. + fn go_sum(&self) -> [String; 2] { + let (path, version) = (&self.path, &self.version); + [ + format!("{path} {version} {}", self.zip_h1), + format!("{path} {version}/go.mod {}", self.mod_h1), + ] + } + + /// The route of this version's `.info`, `.mod` or `.zip` on the proxy + /// the stand-in plays under `/`. + fn route(&self, host: &str, extension: &str) -> String { + format!("/{host}/{}/@v/{}.{extension}", self.path, self.version) + } +} + +/// The upstream module, as the public proxy would serve it. +fn upstream() -> Module { + Module::new(MODULE, VERSION, UPSTREAM_LIB) +} + +/// The loopback server behind every host of the test. +struct StandIn { + // Dropped before the runtime it runs on. + server: MockServer, + runtime: tokio::runtime::Runtime, +} + +impl StandIn { + fn start() -> StandIn { + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(2) + .enable_all() + .build() + .expect("a runtime for the stand-in"); + let server = runtime.block_on(MockServer::start()); + StandIn { server, runtime } + } + + /// `http://127.0.0.1:`. + fn origin(&self) -> String { + self.server.uri() + } + + /// Answer `GET ` with `body`. Any other route is a 404, which + /// sends `go` to the next entry of its `GOPROXY` list. wiremock asks + /// the mocks of the lowest `priority` number first. + fn serve(&self, priority: u8, route: &str, body: &[u8]) { + let mock = Mock::given(method("GET")) + .and(path(route)) + .respond_with(ResponseTemplate::new(200).set_body_bytes(body)) + .with_priority(priority); + self.runtime.block_on(mock.mount(&self.server)); + } + + /// Serve `module` in the GOPROXY layout under `/`. + fn serve_module(&self, host: &str, module: &Module) { + let version = &module.version; + let info = format!(r#"{{"Version": "{version}", "Time": "2026-01-01T00:00:00Z"}}"#); + self.serve(5, &module.route(host, "info"), info.as_bytes()); + self.serve(5, &module.route(host, "mod"), GO_MOD); + self.serve(5, &module.route(host, "zip"), &module.zip); + } + + /// The Socket API: one free patch of the upstream module, granted as + /// the module `socket` on the stand-in patch host. `go` fetches the + /// module, not Bazel's downloader, so the grant may name the loopback + /// origin. + fn grant(&self, uuid: &str, socket: &Module) { + let origin = self.origin(); + let url = format!("{origin}{}", socket.route(PATCH_HOST, "zip")); + let reference = json!({ + "status": "granted", + "url": url, + "purl": PURL, + "artifacts": [{ "kind": "tarball", "url": url, "integrity": {} }], + "registryOverride": { + "kind": "goproxy", + "indexUrl": format!("{origin}/{PATCH_HOST}"), + "identifiers": { + "name": MODULE, + "version": VERSION, + "goModulePath": socket.path, + "goModuleVersion": socket.version, + "goZipDirhashH1": socket.zip_h1, + "goModH1": socket.mod_h1, + }, + }, + }); + let patch = Patch { + purl: PURL.to_string(), + uuid: uuid.to_string(), + reference, + }; + self.runtime.block_on(mount_api(&self.server, &[patch])); + } + + /// How many requests for `route` the server has received. + fn asked(&self, route: &str) -> usize { + let received = self.runtime.block_on(self.server.received_requests()); + let received = received.expect("wiremock records requests"); + received.iter().filter(|r| r.url.path() == route).count() + } +} + +/// The flow of the module docs for `test`, on Bazel `release` and the +/// fixture workspace `fixture`. +fn hosted(test: &str, release: &str, fixture: &str) { + let started = Instant::now(); + let Some(mut cell) = Cell::open(test, release, fixture) else { + return; + }; + + // The Socket module: the upstream files with `lib.go` patched, under a + // path that is new for every run. + let uuid = uuid::Uuid::new_v4().to_string(); + let lib = + format!("package upstream\n\nfunc Greeting() string {{ return \"PATCHED {uuid}\" }}\n"); + let patched = format!("OUT: PATCHED {uuid}"); + let socket_path = format!("{PATCH_HOST}/gopatch/{uuid}"); + let socket = Module::new(&socket_path, SOCKET_VERSION, lib.as_bytes()); + let upstream = upstream(); + + let standin = StandIn::start(); + let origin = standin.origin(); + standin.serve_module(PROXY, &upstream); + standin.serve_module(PATCH_HOST, &socket); + // What `rollback` reads to restore the upstream `go.sum` lines, in the + // form of the checksum database: a record number, the two lines, a + // tree head. + let lookup = format!( + "1\n{}\n\ngo.sum database tree\n1\nAAAA\n", + upstream.go_sum().join("\n") + ); + let lookup_route = format!("/{SUMDB}/lookup/{MODULE}@{VERSION}"); + standin.serve(5, &lookup_route, lookup.as_bytes()); + standin.grant(&uuid, &socket); + let sumdb = format!("{origin}/{SUMDB}"); + let socket_patch = |cell: &Cell, args: &[&str]| { + cell.socket_patch(&origin, args, &[("SOCKET_GOSUMDB_URL", sumdb.as_str())]) + }; + let proxies = format!("{origin}/{PROXY},{origin}/{PATCH_HOST}"); + cell.repo_env.push(("GOPROXY".to_string(), proxies)); + let (upstream_zip, socket_zip) = ( + upstream.route(PROXY, "zip"), + socket.route(PATCH_HOST, "zip"), + ); + + // (a) The unpatched build, with the upstream module from the stand-in. + cell.builds(UNPATCHED); + let upstream_fetches = standin.asked(&upstream_zip); + assert!( + upstream_fetches > 0, + "{test}: the build did not fetch the upstream module from the stand-in" + ); + let before = cell.tree(); + assert!( + before.contains_key("MODULE.bazel.lock"), + "{test}: Bazel wrote no lock" + ); + + // (b) The one command of hosted mode. + let scan = socket_patch(&cell, &["scan", "--mode", "hosted", "--yes"]); + let said = &scan.dump; + assert_eq!(scan.code, 0, "{test}: {said}"); + assert_eq!(scan.envelope["status"], "success", "{test}: {said}"); + assert_eq!(scan.envelope["summary"]["applied"], 1, "{test}: {said}"); + let event = event_for(&scan.envelope, PURL); + assert_eq!(event["action"], "applied", "{test}: {said}"); + assert_eq!(event["uuid"], uuid.as_str(), "{test}: {said}"); + let rewritten = &scan.envelope["redirect"]["rewrittenFiles"]; + assert_eq!(*rewritten, json!(["go.mod", "go.sum"]), "{test}: {said}"); + // Those two files and no other: the one `replace` line, and the two + // sums of the Socket module in place of the upstream module's. + let pinned = cell.tree(); + assert_eq!(changed(&before, &pinned), ["go.mod", "go.sum"], "{test}"); + let text = |rel: &str| String::from_utf8(pinned[rel].clone()).expect("a text file"); + let (go_mod, go_sum) = (text("go.mod"), text("go.sum")); + let replace = format!("replace {MODULE} {VERSION} => {socket_path} {SOCKET_VERSION}"); + let replaces = go_mod.lines().filter(|line| line.trim() == replace).count(); + assert_eq!(replaces, 1, "{test}: go.mod:\n{go_mod}"); + let mut sums: Vec<&str> = go_sum.lines().collect(); + sums.sort_unstable(); + assert_eq!(sums, socket.go_sum(), "{test}: go.sum:\n{go_sum}"); + assert_eq!(standin.asked(&socket_zip), 0, "{test}: {said}"); + + // (c) The same build, no step in between: patched, from the patch host. + cell.builds(&patched); + assert!( + standin.asked(&socket_zip) > 0, + "{test}: the patched build did not fetch the Socket module from the stand-in" + ); + assert_eq!( + standin.asked(&upstream_zip), + upstream_fetches, + "{test}: the patched build fetched the upstream module" + ); + // Bazel changed no file, its own lock included, and that lock still + // stands when Bazel may not update it. + assert_eq!(changed(&pinned, &cell.tree()), NOTHING, "{test}"); + let frozen = cell.frozen_build(&[CONSUMER]); + assert!(frozen.ok(), "{test}: {}", frozen.dump()); + + // (d) The undo: every file is the one of before, and so is the build. + let undo = socket_patch(&cell, &["rollback", "--yes"]); + let said = &undo.dump; + assert_eq!(undo.code, 0, "{test}: {said}"); + assert_eq!(undo.envelope["summary"]["rolledBack"], 1, "{test}: {said}"); + assert_eq!(changed(&before, &cell.tree()), NOTHING, "{test}: {said}"); + cell.builds(UNPATCHED); + assert_eq!(changed(&before, &cell.tree()), NOTHING, "{test}"); + + // (e) A second machine with the rewritten files, and a patch host that + // serves other bytes at the pinned URL: still a module zip `go` + // unpacks, with another `h1:`. The build must fail on the sum, and + // must not go to the upstream module instead. + let twin = cell.twin(); + twin.write("go.mod", &go_mod); + twin.write("go.sum", &go_sum); + let other = b"package upstream\n\nfunc Greeting() string { return \"TAMPERED\" }\n"; + let tampered = Module::new(&socket_path, SOCKET_VERSION, other); + standin.serve(1, &socket_zip, &tampered.zip); + // Counted here, so that only the twin's own requests are judged. + let (socket_fetches, upstream_fetches) = + (standin.asked(&socket_zip), standin.asked(&upstream_zip)); + let build = twin.bazel("build", &[CONSUMER]); + assert!( + !build.ok(), + "{test}: the tampered module built:\n{}", + build.dump() + ); + // gazelle 0.54 fails with go's `checksum mismatch`, 0.45 with + // `expected sum `. + let stderr = build.stderr(); + let pinned_sum = format!("expected sum {}", socket.zip_h1); + let refusal = ["checksum mismatch", pinned_sum.as_str()] + .into_iter() + .find(|sign| stderr.contains(sign)); + let Some(refusal) = refusal else { + panic!( + "{test}: the build did not fail on the sum of the Socket module:\n{}", + build.dump() + ); + }; + assert!( + standin.asked(&socket_zip) > socket_fetches, + "{test}: the patch host was not asked, so the failed build proves nothing" + ); + assert_eq!( + standin.asked(&upstream_zip), + upstream_fetches, + "{test}: the build went to the upstream module" + ); + + println!( + "{test}: Bazel {release} built {patched:?} from {socket_path}@{SOCKET_VERSION} after \ + `scan --mode hosted`; `rollback` restored all {} files and the {UNPATCHED:?} build; other \ + bytes at the patch host failed a second machine's build ({refusal}); {:?}", + before.len(), + started.elapsed() + ); +} + +/// The fixtures pin the module these tests build: without a Bazel, this +/// holds the `h1:` arithmetic to sums that `go` computed. +#[test] +fn the_fixtures_pin_the_upstream_module_the_tests_serve() { + for era in ["e9", "e7"] { + let path = fixture_path(&format!("gazelle/bzlmod/{era}/go.sum")); + let go_sum = std::fs::read_to_string(&path).expect("the fixture's go.sum"); + let lines: Vec<&str> = go_sum.lines().collect(); + assert_eq!(lines, upstream().go_sum(), "{}", path.display()); + } +} + +#[test] +#[ignore = "real Bazel; run with --ignored"] +fn b9_hosted_go_deps() { + hosted("b9_hosted_go_deps", "9.3.2", "gazelle/bzlmod/e9"); +} + +#[test] +#[ignore = "real Bazel; run with --ignored"] +fn b8_hosted_go_deps() { + hosted("b8_hosted_go_deps", "8.8.2", "gazelle/bzlmod/e9"); +} + +// The fixture for Bazel 7 pins older rulesets: gazelle 0.45.0 and +// rules_go 0.53.0. +#[test] +#[ignore = "real Bazel; run with --ignored"] +fn b7_hosted_go_deps() { + hosted("b7_hosted_go_deps", "7.7.2", "gazelle/bzlmod/e7"); +} diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs new file mode 100644 index 000000000..c9ee60864 --- /dev/null +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs @@ -0,0 +1,520 @@ +//! What every real-Bazel test shares: the variables that configure a run, +//! the scratch root that holds everything one Bazel writes, the Bazel +//! command line, and the `socket-patch` child that runs in the workspace +//! copy. + +use std::collections::BTreeMap; +use std::ffi::OsString; +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +use serde_json::Value; + +use crate::fs_tree; +use crate::hosted_memory_common::{hermetic, read_tree, ORG}; + +/// The Bazel binary or launcher to run, by absolute path: every child runs +/// in the workspace copy. Unset: `bazel` on `PATH`. +pub const BAZEL: &str = "SOCKET_PATCH_BAZEL_E2E_BAZEL"; +/// Non-empty: a test that cannot run here (no Bazel, or another release) +/// fails. Unset: it prints `SKIP ` and passes. +pub const REQUIRED: &str = "SOCKET_PATCH_BAZEL_E2E_REQUIRED"; +/// A repository cache shared by the tests of a run, so that only the first +/// downloads the rulesets and the toolchain. Unset: an empty one per test. +pub const REPO_CACHE: &str = "SOCKET_PATCH_BAZEL_E2E_REPO_CACHE"; +/// Another `socket-patch` binary to hold the same tests against (a +/// release), by absolute path. Unset: the binary cargo built for this run. +pub const SOCKET_BIN: &str = "SOCKET_PATCH_BAZEL_E2E_SOCKET_BIN"; + +/// Removed from every Bazel child. `TEST_TMPDIR` puts a Bazel server in +/// test mode, where a failing module extension on 7.x is a crash (exit 37) +/// instead of an error. `USE_BAZEL_VERSION` is set again from the test's +/// release. `XDG_CACHE_HOME` would move a tool's cache out of the scratch +/// home. The CI markers change what some rulesets do, and a test must do +/// the same on a runner and on a developer's machine. +const REMOVED: &[&str] = &[ + "TEST_TMPDIR", + "USE_BAZEL_VERSION", + "XDG_CACHE_HOME", + "CI", + "GITHUB_ACTIONS", +]; + +/// The label every fixture builds: a binary that prints one `OUT: ...` line +/// from the package under test. +pub const CONSUMER: &str = "//:consumer"; + +/// Nothing listens there: a request the `socket-patch` child sends anywhere +/// but loopback fails at once. +const CLOSED_PORT: &str = "http://127.0.0.1:9"; + +/// The files of a workspace copy: `/`-separated relative path to bytes. +pub type Tree = BTreeMap>; + +/// `name`'s value when it is set and non-empty. +fn var(name: &str) -> Option { + std::env::var_os(name).filter(|value| !value.is_empty()) +} + +/// `9.3.2` as three numbers; `None` for anything else. +fn numbers(release: &str) -> Option<(u32, u32, u32)> { + let parts: Option> = release.split('.').map(|part| part.parse().ok()).collect(); + match parts?[..] { + [major, minor, patch] => Some((major, minor, patch)), + _ => None, + } +} + +/// The path `rel` under `tests/fixtures/bazel/`. +pub fn fixture_path(rel: &str) -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("tests/fixtures/bazel") + .join(rel) +} + +/// The names under which `before` and `after` differ. +pub fn changed(before: &Tree, after: &Tree) -> Vec { + let names = before.keys().chain(after.keys()); + let mut differing: Vec = names + .filter(|name| before.get(*name) != after.get(*name)) + .cloned() + .collect(); + differing.sort(); + differing.dedup(); + differing +} + +/// This machine cannot run `test`: a failure under [`REQUIRED`], else a +/// `SKIP` line. +fn unavailable(test: &str, why: &str) { + assert!( + var(REQUIRED).is_none(), + "{test}: {REQUIRED} is set and {why}" + ); + println!("SKIP {test}: {why}"); +} + +/// The temp root of one workspace copy, with a fixed place for everything a +/// Bazel run writes. Nothing lands in the developer's `~/.cache/bazel`, the +/// shared temp directory or the real home. +struct Scratch { + /// The workspace copy Bazel and `socket-patch` run in. + workspace: PathBuf, + /// `--output_user_root`: one output base, so one Bazel server. + output_user_root: PathBuf, + /// `java.io.tmpdir` of the server and `TMPDIR` of every child. + tmp: PathBuf, + /// `HOME` of every child. + home: PathBuf, + /// `--repository_cache`. + repository_cache: PathBuf, + /// `--repo_contents_cache` on Bazel 9, where that cache is on by + /// default. + contents_cache: PathBuf, + /// Removed on drop, after [`Scratch::drop`] made it removable. + dir: tempfile::TempDir, +} + +impl Scratch { + /// A new temp root. `repository_cache` is one that outlives it; `None` + /// gives it an empty one of its own. + fn new(repository_cache: Option) -> Scratch { + let dir = tempfile::Builder::new() + .prefix("sp-bazel-") + .tempdir() + .expect("create the scratch root"); + // Canonical: macOS temp directories sit behind a symlink, and a + // directory named by the environment may be relative, which Bazel + // would resolve against the workspace copy. + let made = |dir: PathBuf| { + std::fs::create_dir_all(&dir) + .unwrap_or_else(|e| panic!("create {}: {e}", dir.display())); + dir.canonicalize().expect("a canonical directory") + }; + let root = made(dir.path().to_path_buf()); + Scratch { + workspace: made(root.join("ws")), + output_user_root: made(root.join("root")), + tmp: made(root.join("tmp")), + home: made(root.join("home")), + repository_cache: made(repository_cache.unwrap_or_else(|| root.join("cache"))), + contents_cache: made(root.join("contents")), + dir, + } + } +} + +impl Drop for Scratch { + fn drop(&mut self) { + // Bazel marks its install base and the repositories it fetched + // read-only, which would leave the temp root behind. Silent: this + // also runs while a failed test unwinds. + fn make_writable(path: &Path) { + use std::os::unix::fs::PermissionsExt; + let Ok(meta) = std::fs::symlink_metadata(path) else { + return; + }; + if meta.file_type().is_symlink() { + return; + } + let mut perms = meta.permissions(); + perms.set_mode(perms.mode() | 0o700); + let _ = std::fs::set_permissions(path, perms); + if meta.is_dir() { + for entry in std::fs::read_dir(path).into_iter().flatten().flatten() { + make_writable(&entry.path()); + } + } + } + make_writable(self.dir.path()); + } +} + +/// The Bazel a test runs. +#[derive(Clone)] +struct Bazel { + /// A release binary, or a launcher that reads `USE_BAZEL_VERSION`. + program: OsString, + release: String, +} + +impl Bazel { + /// The Bazel of [`BAZEL`], if it is `release`. `bazel --version` is + /// answered by the client: no server starts. `Err` says why this + /// machine cannot run the test. + fn detect(scratch: &Scratch, release: &str) -> Result { + let program = var(BAZEL).unwrap_or_else(|| "bazel".into()); + let bazel = Bazel { + program, + release: release.to_string(), + }; + let shown = bazel.program.to_string_lossy().into_owned(); + let out = bazel + .child(scratch) + .arg("--version") + .output() + .map_err(|e| format!("`{shown}` did not run: {e}"))?; + let stdout = String::from_utf8_lossy(&out.stdout); + let found = stdout + .lines() + .find_map(|line| line.trim().strip_prefix("bazel ")); + match found { + Some(found) if found == release => Ok(bazel), + Some(found) => Err(format!("`{shown}` is Bazel {found}, not {release}")), + None => Err(format!( + "`{shown} --version` printed no `bazel ` line:\n{stdout}{}", + String::from_utf8_lossy(&out.stderr) + )), + } + } + + /// The program with the environment of every Bazel child and no + /// arguments. Built on `hermetic::command`, the one way this crate's + /// tests spawn a child; what is set here lands after its scrub. + fn child(&self, scratch: &Scratch) -> Command { + let mut cmd = hermetic::command(Path::new(&self.program)); + for name in REMOVED { + cmd.env_remove(name); + } + // A bazelisk's other variables change which Bazel it runs. + for (name, _) in std::env::vars_os() { + if name.to_string_lossy().starts_with("BAZELISK_") { + cmd.env_remove(name); + } + } + // The `go` a ruleset runs must not read the developer's module + // cache, flags or `go env -w` file. + hermetic::scrub_extra(&mut cmd, &[hermetic::Extra::Go]); + cmd.env("USE_BAZEL_VERSION", &self.release) + .env("HOME", &scratch.home) + .env("TMPDIR", &scratch.tmp); + cmd + } + + /// `bazel ` in the workspace copy. The + /// startup options never vary within a test: another set restarts the + /// server. They keep the rc files of the machine out and every write + /// inside the scratch root; without the two JVM options the server + /// writes `hsperfdata_` and its native libraries into the shared + /// temp directory. + fn command(&self, scratch: &Scratch, command: &str) -> Command { + let joined = |flag: &str, path: &Path| { + let mut arg = OsString::from(flag); + arg.push(path); + arg + }; + let mut cmd = self.child(scratch); + cmd.current_dir(&scratch.workspace) + .arg("--nohome_rc") + .arg("--nosystem_rc") + .arg(joined("--output_user_root=", &scratch.output_user_root)) + .arg("--max_idle_secs=15") + .arg(joined("--host_jvm_args=-Djava.io.tmpdir=", &scratch.tmp)) + .arg("--host_jvm_args=-XX:-UsePerfData") + .arg(command); + cmd + } +} + +/// One finished Bazel command. +pub struct Run { + /// The command line, for failure messages. + line: String, + output: Output, +} + +impl Run { + fn of(mut cmd: Command) -> Result { + let line = format!("{cmd:?}"); + let output = cmd.output().map_err(|e| format!("spawn {line}: {e}"))?; + Ok(Run { line, output }) + } + + pub fn ok(&self) -> bool { + self.output.status.success() + } + + pub fn stdout(&self) -> String { + String::from_utf8_lossy(&self.output.stdout).into_owned() + } + + pub fn stderr(&self) -> String { + String::from_utf8_lossy(&self.output.stderr).into_owned() + } + + pub fn dump(&self) -> String { + format!( + "{}\nexit {:?}\n--- stdout\n{}\n--- stderr\n{}", + self.line, + self.output.status.code(), + self.stdout(), + self.stderr() + ) + } +} + +/// One run of `socket-patch` in a workspace copy. +pub struct Ran { + pub code: i32, + /// stdout, parsed: the `--json` envelope. + pub envelope: Value, + /// The command line, the exit code and both streams, for failure + /// messages. + pub dump: String, +} + +/// One test's checkout of a fixture workspace, with a Bazel server and an +/// output base of its own. +pub struct Cell { + pub test: String, + /// `--repo_env=NAME=VALUE` pairs for every Bazel command (`GOPROXY`). + pub repo_env: Vec<(String, String)>, + bazel: Bazel, + // Dropped after `Cell::drop` has stopped the server. + scratch: Scratch, +} + +impl Cell { + /// The checkout of the fixture workspace `fixture` for `test`, which + /// runs on Bazel `release`. `None` after a `SKIP` line when this machine + /// has no such Bazel; a panic instead under [`REQUIRED`]. + pub fn open(test: &str, release: &str, fixture: &str) -> Option { + let (major, _, _) = + numbers(release).unwrap_or_else(|| panic!("{test}: {release} is no Bazel release")); + // A workflow leg selects the tests of its Bazel by this prefix. + assert!( + test.starts_with(&format!("b{major}_")), + "{test} runs on Bazel {release}: its name must start with b{major}_" + ); + let shared = var(REPO_CACHE).map(PathBuf::from); + let scratch = Scratch::new(shared); + let bazel = match Bazel::detect(&scratch, release) { + Ok(bazel) => bazel, + Err(why) => { + unavailable(test, &why); + return None; + } + }; + let source = fixture_path(fixture); + assert!(source.is_dir(), "{test}: no fixture {}", source.display()); + fs_tree::copy_tree(&source, &scratch.workspace); + // The checkout is its own repository: `socket-patch` looks no + // higher for a `socket.yml` or a project root. + std::fs::create_dir(scratch.workspace.join(".git")).expect("create .git"); + println!( + "{test}: Bazel {release} ({})", + bazel.program.to_string_lossy() + ); + Some(Cell { + test: test.to_string(), + repo_env: Vec::new(), + bazel, + scratch, + }) + } + + /// A second machine with the same checkout: the files of this one, in + /// a workspace copy with an output base and a contents cache of its + /// own, so nothing a build here fetched or built is there. Only the + /// repository cache is shared. This cell's server is stopped first, so + /// one runs at a time. + pub fn twin(&self) -> Cell { + if let Err(why) = self.shutdown() { + panic!("{}: {why}", self.test); + } + let twin = Cell { + test: self.test.clone(), + repo_env: self.repo_env.clone(), + bazel: self.bazel.clone(), + scratch: Scratch::new(Some(self.scratch.repository_cache.clone())), + }; + for (rel, bytes) in self.tree() { + twin.write(&rel, bytes); + } + std::fs::create_dir(twin.scratch.workspace.join(".git")).expect("create .git"); + twin + } + + /// Write a file of the workspace copy. + pub fn write(&self, rel: &str, bytes: impl AsRef<[u8]>) { + let path = self.scratch.workspace.join(rel); + std::fs::create_dir_all(path.parent().expect("a file path")).expect("create the parent"); + std::fs::write(&path, bytes).unwrap_or_else(|e| panic!("write {}: {e}", path.display())); + } + + /// Every regular file of the workspace copy. Bazel's `bazel-*` links + /// into the output base are links, so they are not followed. + pub fn tree(&self) -> Tree { + read_tree(&self.scratch.workspace) + } + + /// `bazel ` in the workspace copy. + pub fn bazel(&self, command: &str, args: &[&str]) -> Run { + self.bazel_with("update", command, args) + } + + /// `bazel build --lockfile_mode=error `: passes only when + /// `MODULE.bazel.lock` already holds what the build resolves. + pub fn frozen_build(&self, targets: &[&str]) -> Run { + self.bazel_with("error", "build", targets) + } + + fn bazel_with(&self, lockfile_mode: &str, command: &str, args: &[&str]) -> Run { + let scratch = &self.scratch; + let mut cmd = self.bazel.command(scratch, command); + let mut joined = |flag: &str, value: &std::ffi::OsStr| { + let mut arg = OsString::from(flag); + arg.push(value); + cmd.arg(arg); + }; + joined("--repository_cache=", scratch.repository_cache.as_os_str()); + let mut home = OsString::from("HOME="); + home.push(&scratch.home); + joined("--repo_env=", &home); + for (name, value) in &self.repo_env { + joined("--repo_env=", format!("{name}={value}").as_ref()); + } + // Stated, so no test depends on a release's default. + joined("--lockfile_mode=", lockfile_mode.as_ref()); + let release = numbers(&self.bazel.release).expect("checked by Cell::open"); + if release >= (9, 0, 0) { + // Bazel 9 keeps the repositories it fetched in a contents cache, + // by default under the repository cache, and hands them to the + // next workspace copy without a fetch. The cache stays on, as a + // user has it, in a place of this copy's own: a twin starts + // with an empty one and has to fetch. Bazel 8 has the cache + // off by default, Bazel 7 has none. + joined("--repo_contents_cache=", scratch.contents_cache.as_os_str()); + } + cmd.args(["--color=no", "--curses=no"]); + cmd.args(args); + Run::of(cmd).unwrap_or_else(|why| panic!("{}: {why}", self.test)) + } + + /// The build a user runs, and what it built: `bazel build //:consumer` + /// passes, and `bazel run //:consumer` prints the line `marker`. + pub fn builds(&self, marker: &str) { + let built = self.bazel("build", &[CONSUMER]); + assert!(built.ok(), "{}: {}", self.test, built.dump()); + let ran = self.bazel("run", &[CONSUMER]); + assert!( + ran.ok() && ran.stdout().lines().any(|line| line == marker), + "{}: {CONSUMER} did not print {marker:?}:\n{}", + self.test, + ran.dump() + ); + } + + /// `socket-patch --json` in the workspace copy, as a user in + /// that directory runs it, against the Socket API at `api`. Its home + /// and temp directory are the cell's, and every proxy is a closed port, + /// so it reaches no host but loopback. + pub fn socket_patch(&self, api: &str, args: &[&str], env: &[(&str, &str)]) -> Ran { + let mut cmd = match var(SOCKET_BIN) { + Some(bin) => hermetic::command(Path::new(&bin)), + None => hermetic::binary_command(), + }; + cmd.current_dir(&self.scratch.workspace) + .args(args) + .args(["--json", "--api-url", api, "--org", ORG]) + .args(["--api-token", "fake-token"]); + hermetic::scrub_crawler_inputs(&mut cmd); + cmd.env("HOME", &self.scratch.home) + .env("TMPDIR", &self.scratch.tmp); + for proxy in ["HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY"] { + cmd.env(proxy, CLOSED_PORT); + cmd.env(proxy.to_ascii_lowercase(), CLOSED_PORT); + } + cmd.env("NO_PROXY", "127.0.0.1,localhost"); + cmd.env("no_proxy", "127.0.0.1,localhost"); + cmd.envs(env.iter().copied()); + let line = format!("{cmd:?}"); + let out = cmd.output().unwrap_or_else(|e| panic!("spawn {line}: {e}")); + let stdout = String::from_utf8_lossy(&out.stdout); + let stderr = String::from_utf8_lossy(&out.stderr); + let code = out.status.code().unwrap_or(-1); + let dump = format!("{line}\nexit {code}\n--- stdout\n{stdout}\n--- stderr\n{stderr}"); + let envelope = serde_json::from_str(&stdout) + .unwrap_or_else(|e| panic!("{}: stdout is no JSON envelope ({e}):\n{dump}", self.test)); + Ran { + code, + envelope, + dump, + } + } + + /// `bazel shutdown`, when the pid file of a server is there. No command + /// options: the startup options are what address the server. + fn shutdown(&self) -> Result<(), String> { + let bases = std::fs::read_dir(&self.scratch.output_user_root); + let mut bases = bases.into_iter().flatten().flatten(); + if !bases.any(|base| base.path().join("server/server.pid.txt").exists()) { + return Ok(()); + } + let run = Run::of(self.bazel.command(&self.scratch, "shutdown"))?; + if run.ok() { + Ok(()) + } else { + Err(format!("`bazel shutdown` failed:\n{}", run.dump())) + } + } +} + +impl Drop for Cell { + fn drop(&mut self) { + // Never panic here: this also runs while a failed test unwinds. If + // the shutdown fails, the idle timeout ends the server. + if let Err(why) = self.shutdown() { + eprintln!("{}: leaving a Bazel server behind: {why}", self.test); + } + } +} + +#[test] +fn a_release_is_read_as_three_numbers() { + assert_eq!(numbers("9.3.2"), Some((9, 3, 2))); + assert_eq!(numbers("7.7.2"), Some((7, 7, 2))); + for unreadable in ["9.3", "9.3.2.1", "10.0.0rc3", ""] { + assert_eq!(numbers(unreadable), None, "{unreadable:?}"); + } +} diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs new file mode 100644 index 000000000..ffb26783d --- /dev/null +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs @@ -0,0 +1,44 @@ +#![cfg(unix)] +//! `socket-patch` in a Bazel workspace, on a real Bazel. +//! +//! Each test copies a fixture workspace from `tests/fixtures/bazel/`, builds +//! it with a pinned Bazel release, runs the built `socket-patch` binary in +//! it the way a user does, and builds again. What a test asserts, it reads +//! from what Bazel built: the line `bazel run //:consumer` prints. +//! +//! The tests are `#[ignore]`d: they need a Bazel release and the network +//! (the Bazel registry, the ruleset archives and the language toolchain), +//! and a host that does not answer fails them. The package under test and +//! every Socket host are loopback stand-ins. +//! `.github/workflows/bazel-compatibility.yml` runs them, one leg per Bazel +//! release and ecosystem. Locally: +//! +//! ```sh +//! eval "$(scripts/install-bazel.sh 9.3.2 "$HOME/.cache/socket-patch-bazel")" +//! export SOCKET_PATCH_BAZEL_E2E_BAZEL +//! cargo test -p socket-patch-cli --test e2e_bazel_build -- --ignored go::b9_ --nocapture +//! ``` +//! +//! With `SOCKET_PATCH_BAZEL_E2E_BAZEL` unset the tests run `bazel` from +//! `PATH` with `USE_BAZEL_VERSION` set to the test's release, so a bazelisk +//! picks it. A test whose Bazel is missing or is another release prints +//! `SKIP ` and passes; with `SOCKET_PATCH_BAZEL_E2E_REQUIRED=1` (CI) it +//! fails instead. `harness.rs` lists the other variables. +//! +//! To add a test of a flow that is here: one fixture directory and one +//! test function in the ecosystem's module. A test's name starts with its +//! Bazel major (`b9_`), which is how a workflow leg selects its own, so a +//! major has one release here. A new Bazel major adds its lines in +//! `scripts/bazel-pins.txt` and one line in the workflow's matrix; a new +//! ecosystem adds its module and one matrix line per release. + +#[path = "../common/envelope.rs"] +mod envelope; +#[path = "../common/fs_tree.rs"] +mod fs_tree; +// Also the home of `hermetic`, the environment every child is spawned with. +#[path = "../hosted_memory_common/mod.rs"] +mod hosted_memory_common; + +mod go; +mod harness; diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/BUILD.bazel b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/BUILD.bazel new file mode 100644 index 000000000..29489e934 --- /dev/null +++ b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/BUILD.bazel @@ -0,0 +1,8 @@ +load("@rules_go//go:def.bzl", "go_binary") + +# Prints OUT: PRISTINE until the module is patched. +go_binary( + name = "consumer", + srcs = ["main.go"], + deps = ["@com_example_upstream//:upstream"], +) diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/MODULE.bazel b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/MODULE.bazel new file mode 100644 index 000000000..6af9d9418 --- /dev/null +++ b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/MODULE.bazel @@ -0,0 +1,14 @@ +module(name = "socket_fixture", version = "0.0.0") + +bazel_dep(name = "rules_go", version = "0.53.0") +bazel_dep(name = "gazelle", version = "0.45.0") +single_version_override(module_name = "rules_go", version = "0.53.0") +single_version_override(module_name = "gazelle", version = "0.45.0") + +go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk") +go_sdk.download(version = "1.23.4") + +go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps") +go_deps.config(check_direct_dependencies = "error") +go_deps.from_file(go_mod = "//:go.mod") +use_repo(go_deps, "com_example_upstream") diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.mod b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.mod new file mode 100644 index 000000000..61cbb9690 --- /dev/null +++ b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.mod @@ -0,0 +1,5 @@ +module example.com/consumer + +go 1.21 + +require example.com/upstream v1.0.0 diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.sum b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.sum new file mode 100644 index 000000000..ad31912aa --- /dev/null +++ b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.sum @@ -0,0 +1,2 @@ +example.com/upstream v1.0.0 h1:SLqDtcPyVkkOYblt1rSVexJHaLWU/teyZ8o5mHyiUbA= +example.com/upstream v1.0.0/go.mod h1:WGm+Dt0Rps9QKvOZJwtpSRFNm3I9B5DlvBXyx4SHodQ= diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/main.go b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/main.go new file mode 100644 index 000000000..aa396ab81 --- /dev/null +++ b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/main.go @@ -0,0 +1,9 @@ +package main + +import ( + "fmt" + + "example.com/upstream" +) + +func main() { fmt.Println("OUT:", upstream.Greeting()) } diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/BUILD.bazel b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/BUILD.bazel new file mode 100644 index 000000000..29489e934 --- /dev/null +++ b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/BUILD.bazel @@ -0,0 +1,8 @@ +load("@rules_go//go:def.bzl", "go_binary") + +# Prints OUT: PRISTINE until the module is patched. +go_binary( + name = "consumer", + srcs = ["main.go"], + deps = ["@com_example_upstream//:upstream"], +) diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/MODULE.bazel b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/MODULE.bazel new file mode 100644 index 000000000..31957109a --- /dev/null +++ b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/MODULE.bazel @@ -0,0 +1,15 @@ +module(name = "socket_fixture", version = "0.0.0") + +bazel_dep(name = "rules_go", version = "0.65.0") +bazel_dep(name = "gazelle", version = "0.54.0") +single_version_override(module_name = "rules_go", version = "0.65.0") +single_version_override(module_name = "gazelle", version = "0.54.0") + +# rules_go 0.65.0 rejects an older SDK: its own go.mod says go 1.26.0. +go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk") +go_sdk.download(version = "1.26.0") + +go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps") +go_deps.config(check_direct_dependencies = "error") +go_deps.from_file(go_mod = "//:go.mod") +use_repo(go_deps, "com_example_upstream") diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.mod b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.mod new file mode 100644 index 000000000..61cbb9690 --- /dev/null +++ b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.mod @@ -0,0 +1,5 @@ +module example.com/consumer + +go 1.21 + +require example.com/upstream v1.0.0 diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.sum b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.sum new file mode 100644 index 000000000..ad31912aa --- /dev/null +++ b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.sum @@ -0,0 +1,2 @@ +example.com/upstream v1.0.0 h1:SLqDtcPyVkkOYblt1rSVexJHaLWU/teyZ8o5mHyiUbA= +example.com/upstream v1.0.0/go.mod h1:WGm+Dt0Rps9QKvOZJwtpSRFNm3I9B5DlvBXyx4SHodQ= diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/main.go b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/main.go new file mode 100644 index 000000000..aa396ab81 --- /dev/null +++ b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/main.go @@ -0,0 +1,9 @@ +package main + +import ( + "fmt" + + "example.com/upstream" +) + +func main() { fmt.Println("OUT:", upstream.Greeting()) } diff --git a/scripts/bazel-pins.txt b/scripts/bazel-pins.txt new file mode 100644 index 000000000..5a770a74b --- /dev/null +++ b/scripts/bazel-pins.txt @@ -0,0 +1,20 @@ +# Pinned Bazel release binaries for the real-Bazel tests. One pin per line: +# +# bazel +# +# Read by scripts/install-bazel.sh, which installs nothing that is not listed +# here. Platforms use the spelling of Bazel's release files. A release that +# .github/workflows/bazel-compatibility.yml runs needs a line for each +# platform it runs on. +# +# Each digest is the published `.sha256` sibling of the release file: +# https://releases.bazel.build//release/bazel--.sha256 +# To add a release, add one line per platform from those files; never copy a +# digest from a download you made yourself. + +bazel 7.7.2 linux-x86_64 f7b8c4e72f8776f6a352fbe699cc1de381342b5e7e0ad1f34d72c5a9b163288c +bazel 7.7.2 darwin-arm64 7f50ea270dce9f9315afb2a58371dd866e7b23a14f0533de470bb235aff2856f +bazel 8.8.2 linux-x86_64 bf463ba0f5f782e8fe31071bd2b821c2f6a27f03e5bb62f0879e88cbf3656e04 +bazel 8.8.2 darwin-arm64 ae640432a3fdb8a08c637562895f91c7ff1f63f04b1a9d34de9e9c46162bc10b +bazel 9.3.2 linux-x86_64 2e14c55c298752d71e8a0bbd762665ffce5966c9ad00ec5d1a8b82a58e7bfa06 +bazel 9.3.2 darwin-arm64 33fc3b489d159a86be0b465e9d1a58e5d03d2cc55faa87d8a19cd4de0a65f569 diff --git a/scripts/install-bazel.sh b/scripts/install-bazel.sh new file mode 100755 index 000000000..d7632db06 --- /dev/null +++ b/scripts/install-bazel.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# Install one pinned Bazel release binary for the real-Bazel tests +# (`e2e_bazel_build`), then print the variable a CI step appends to +# $GITHUB_ENV. +# +# scripts/install-bazel.sh +# +# The release binary is one file from the GitHub release +# (releases.bazel.build is tried second). It must match the sha256 pinned for +# this platform in scripts/bazel-pins.txt: a version or platform with no pin +# fails before anything is downloaded, and a download that does not match is +# deleted, never installed. A file already in that matches its pin (CI +# restores from the actions cache) is used without any download. +# +# The tests run the verified binary directly, so no bazelisk, `.bazelversion` +# or network version lookup is involved. +# +# Prints (stdout): +# SOCKET_PATCH_BAZEL_E2E_BAZEL= +set -euo pipefail + +die() { + echo "install-bazel.sh: $*" >&2 + exit 1 +} + +if [ $# -ne 2 ]; then + sed -n '2,19p' "$0" | sed 's/^# \{0,1\}//' >&2 + exit 2 +fi +VERSION="$1" +DIR="$2" +PINS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/bazel-pins.txt" + +# The version becomes part of two URLs and an awk variable. +if ! [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + die "'$VERSION' is not a Bazel release (expected e.g. 9.3.2)" +fi +[ -f "$PINS" ] || die "the pins file $PINS is missing" + +case "$(uname -s)" in + Linux) os=linux ;; + Darwin) os=darwin ;; + *) os="$(uname -s)" ;; +esac +case "$(uname -m)" in + x86_64 | amd64) arch=x86_64 ;; + arm64 | aarch64) arch=arm64 ;; + *) arch="$(uname -m)" ;; +esac +platform="$os-$arch" + +# The macOS runners have only shasum. +if command -v sha256sum >/dev/null 2>&1; then + sha256_of() { sha256sum <"$1" | cut -d' ' -f1; } +elif command -v shasum >/dev/null 2>&1; then + sha256_of() { shasum -a 256 <"$1" | cut -d' ' -f1; } +else + die "need sha256sum or shasum to verify the download" +fi + +# The one sha256 pinned for this version and platform. Anything else (no +# line, two lines, a malformed digest) is no pin. Resolved before the +# download, so an unpinned version or platform costs no network and leaves +# untouched. +sha="$(awk -v v="$VERSION" -v p="$platform" \ + '{ sub(/\r$/, "") } $1 == "bazel" && $2 == v && $3 == p { print $4 }' "$PINS")" +if ! [[ "$sha" =~ ^[0-9a-f]{64}$ ]]; then + die "no pinned sha256 for bazel $VERSION on $platform; add it to scripts/bazel-pins.txt from the release's .sha256 file" +fi + +mkdir -p "$DIR" +DIR="$(cd "$DIR" && pwd)" +file="bazel-$VERSION-$platform" +bazel="$DIR/$file" + +# Leave a verified file at $bazel. +fetch() { + local url got + if [ -f "$bazel" ]; then + if [ "$(sha256_of "$bazel")" = "$sha" ]; then + echo "install-bazel.sh: using the cached $file" >&2 + return 0 + fi + rm -f "$bazel" + fi + # Downloaded beside the final name and renamed only once verified, so an + # interrupted or tampered download never sits where a later run (or a + # cache save) would pick it up. + local tmp="$bazel.download" + for url in "$@"; do + rm -f "$tmp" + if curl -fsSL --retry 5 --retry-all-errors "$url" -o "$tmp"; then + got="$(sha256_of "$tmp")" + if [ "$got" = "$sha" ]; then + mv -f "$tmp" "$bazel" + return 0 + fi + echo "install-bazel.sh: sha256 mismatch for $url: got $got, pinned $sha" >&2 + else + echo "install-bazel.sh: download from $url failed" >&2 + fi + rm -f "$tmp" + done + die "no origin served a $file that matches its pin" +} + +fetch \ + "https://github.com/bazelbuild/bazel/releases/download/$VERSION/$file" \ + "https://releases.bazel.build/$VERSION/release/$file" +chmod +x "$bazel" + +echo "SOCKET_PATCH_BAZEL_E2E_BAZEL=$bazel" From 39893d319b322bd089148e259149459eb8cbab7d Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Mon, 16 Nov 2026 23:34:17 -0500 Subject: [PATCH 2/5] Tighten the Bazel end-to-end harness after review The real-Bazel test of the hosted Go flow asserts what it asserted. This changes how a leg selects it, what it takes from the machine it runs on, and how its Bazel is installed. - A flow is one `#[ignore]` test, `go::hosted_go_deps`. It takes its release from the Bazel it is given (`bazel --version`) and its fixture era from one table in the test code, `go::FIXTURE`; a release the table does not hold fails with a message that names the table. The workflow passes the module filter `go::`: no name prefix derived from the release, no assert on test names, no scrape of the log. `SOCKET_PATCH_BAZEL_E2E_VERSION` fails a leg whose Bazel is not its release, as `SOCKET_PATCH_GO_E2E_VERSION` does for Go. - The fixture is one `common` directory and one `MODULE.bazel` per era, in place of two full copies. A hermetic test holds the directories to the table and the one `go.sum` to the module the test serves. - The Go fetch settings that gazelle hands to `go mod download` (`GOPRIVATE`, `GONOPROXY`, `GONOSUMDB`, `GOSUMDB`, `GOAUTH`), and `GOFLAGS` and `GOINSECURE`, are pinned beside `GOPROXY`, as variables of the Bazel client and as `--repo_env`. An exported `GOPRIVATE=patch.socket.dev` used to send the fetch past the loopback stand-ins to the real host. - `XDG_CONFIG_HOME`, `XDG_DATA_HOME` and `XDG_STATE_HOME` are removed from every Bazel child, so on Linux the telemetry counters of the Go SDK land under the scratch home. `GOTELEMETRY` is not set: `go env` reports it, and Go does not read it from the environment. - A bazelisk launcher gets one `BAZELISK_HOME`, the caller's own cache, so it downloads a release at most once, not once per workspace copy. - A relative `SOCKET_PATCH_BAZEL_E2E_BAZEL` or `SOCKET_PATCH_BAZEL_E2E_SOCKET_BIN` fails at once with a message that names the variable, with or without `SOCKET_PATCH_BAZEL_E2E_REQUIRED`. - `scripts/install-bazel.sh` gives curl a connect timeout, a low-speed limit and a total time, and retries only transient failures: a 404 goes to the second origin at once and a stalled transfer ends. It prints one line on stdout, and the documented recipe is `export "$(scripts/install-bazel.sh )"`, which keeps a space in the path. - The workflow's jobs have no `name:`. GitHub showed the matrix expressions of the name as text for a job that its `if` skipped. Co-Authored-By: Claude Fable 5.2 --- .github/workflows/bazel-compatibility.yml | 53 +-- .../tests/e2e_bazel_build/go.rs | 111 +++--- .../tests/e2e_bazel_build/harness.rs | 337 +++++++++++++----- .../tests/e2e_bazel_build/main.rs | 36 +- .../gazelle/bzlmod/{e7 => common}/BUILD.bazel | 0 .../gazelle/bzlmod/{e7 => common}/go.mod | 0 .../gazelle/bzlmod/{e7 => common}/go.sum | 0 .../gazelle/bzlmod/{e7 => common}/main.go | 0 .../bazel/gazelle/bzlmod/e9/BUILD.bazel | 8 - .../fixtures/bazel/gazelle/bzlmod/e9/go.mod | 5 - .../fixtures/bazel/gazelle/bzlmod/e9/go.sum | 2 - .../fixtures/bazel/gazelle/bzlmod/e9/main.go | 9 - scripts/install-bazel.sh | 22 +- 13 files changed, 385 insertions(+), 198 deletions(-) rename crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/{e7 => common}/BUILD.bazel (100%) rename crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/{e7 => common}/go.mod (100%) rename crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/{e7 => common}/go.sum (100%) rename crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/{e7 => common}/main.go (100%) delete mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/BUILD.bazel delete mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.mod delete mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.sum delete mode 100644 crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/main.go diff --git a/.github/workflows/bazel-compatibility.yml b/.github/workflows/bazel-compatibility.yml index b983504f5..f1d212366 100644 --- a/.github/workflows/bazel-compatibility.yml +++ b/.github/workflows/bazel-compatibility.yml @@ -1,10 +1,11 @@ name: Bazel patch compatibility # socket-patch in a Bazel workspace, on a real Bazel: one leg per Bazel -# release and ecosystem runs that ecosystem's `#[ignore]` tests of -# `e2e_bazel_build`, by name prefix (`go::b9_` on Bazel 9.3.2). A test builds -# a fixture workspace, runs the built binary in it and reads the result from -# what Bazel builds next. +# release and ecosystem runs the `#[ignore]` tests of that ecosystem's module +# of `e2e_bazel_build` (`go::`) on the release it installed. A test builds a +# fixture workspace, runs the built binary in it and reads the result from +# what Bazel builds next; the test code says which fixture files go with +# which release. # # The package under test and every Socket host are loopback stand-ins. A leg # still needs the network for the ruleset and the toolchain: github.com and @@ -17,10 +18,14 @@ name: Bazel patch compatibility # `needs` of ci.yml's `ci-ok`. Setting the repository variable # BAZEL_COMPAT_DISABLED to `true` turns every leg off without a merge. # -# A leg selects its tests by the major of its release, so a major has one -# release here. To add a Bazel major: one line per platform in -# scripts/bazel-pins.txt and one matrix line here. To add an ecosystem: its -# module in `e2e_bazel_build` and one matrix line per release. +# To add a Bazel release: one line per platform in scripts/bazel-pins.txt +# and one matrix line here. To add an ecosystem: its module in +# `e2e_bazel_build` and one matrix line per release. +# +# No job has a `name:`. GitHub does not expand a matrix expression in the +# name of a job that its `if` skips, so such a name shows up as its source +# text in a pull request's checks. The default title of a leg is the job's +# id and its matrix values: `bazel (ubuntu-latest, 9.3.2, go, gazelle)`. on: pull_request: @@ -83,14 +88,14 @@ env: jobs: bazel: - name: ${{ matrix.eco }} / Bazel ${{ matrix.bazel }} (${{ matrix.os }}) if: github.event.pull_request.draft != true && vars.BAZEL_COMPAT_DISABLED != 'true' strategy: fail-fast: false matrix: include: - # `fixtures`: the ecosystem's directory under - # tests/fixtures/bazel, which keys its repository cache. + # `eco`: the ecosystem's module of the test target. `fixtures`: + # its directory under tests/fixtures/bazel, which keys its + # repository cache. - {os: ubuntu-latest, bazel: '9.3.2', eco: go, fixtures: gazelle} - {os: ubuntu-latest, bazel: '8.8.2', eco: go, fixtures: gazelle} - {os: ubuntu-latest, bazel: '7.7.2', eco: go, fixtures: gazelle} @@ -127,7 +132,9 @@ jobs: - name: Install Bazel ${{ matrix.bazel }} env: BAZEL_VERSION: ${{ matrix.bazel }} - # Prints SOCKET_PATCH_BAZEL_E2E_BAZEL, the pinned and verified binary. + # Prints one line, SOCKET_PATCH_BAZEL_E2E_BAZEL=. A line of $GITHUB_ENV is a name, `=` and the + # value up to the end of the line, so a space in the path is kept. run: scripts/install-bazel.sh "$BAZEL_VERSION" "$RUNNER_TEMP/bazel-dist" >> "$GITHUB_ENV" - name: Save the Bazel ${{ matrix.bazel }} binary @@ -161,24 +168,19 @@ jobs: - name: Run the ${{ matrix.eco }} tests on Bazel ${{ matrix.bazel }} env: - # A missing Bazel or another release fails the leg instead of - # printing SKIP. + # A missing Bazel fails the leg instead of printing SKIP, and so + # does a Bazel that is not the release of this leg. SOCKET_PATCH_BAZEL_E2E_REQUIRED: '1' + SOCKET_PATCH_BAZEL_E2E_VERSION: ${{ matrix.bazel }} SOCKET_PATCH_BAZEL_E2E_REPO_CACHE: ${{ runner.temp }}/bazel-repos - BAZEL_VERSION: ${{ matrix.bazel }} ECO: ${{ matrix.eco }} - # Two tests at a time: each runs a Bazel server of its own. + # The ecosystem's module holds one `#[ignore]` test per flow, and a + # test takes its release from the Bazel installed above, so the + # filter is the module's name and nothing is derived from the + # release. Two tests at a time: each runs a Bazel server of its own. run: | - set -euo pipefail - filter="${ECO}::b${BAZEL_VERSION%%.*}_" cargo test --locked -p socket-patch-cli --test e2e_bazel_build -- \ - --ignored "$filter" --test-threads=2 --nocapture 2>&1 | tee "$RUNNER_TEMP/bazel-tests.log" - # A filter that selects no test passes for cargo. - passed=$(sed -n 's/^test result: .* \([0-9][0-9]*\) passed;.*/\1/p' "$RUNNER_TEMP/bazel-tests.log" | head -n 1) - if [ "${passed:-0}" = 0 ]; then - echo "::error::no test of e2e_bazel_build matches $filter" - exit 1 - fi + --ignored "${ECO}::" --test-threads=2 --nocapture - name: Save the repository cache if: >- @@ -193,7 +195,6 @@ jobs: # The macOS rows of `bazel`: nightly and on demand, never on a pull # request or a push, so they do not queue on the small macOS runner pool. bazel-macos: - name: ${{ matrix.eco }} / Bazel ${{ matrix.bazel }} (${{ matrix.os }}) if: (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && vars.BAZEL_COMPAT_DISABLED != 'true' strategy: fail-fast: false diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs index 909aa2446..4494637e7 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs @@ -1,7 +1,8 @@ //! Hosted mode for a Go module that comes through gazelle's //! `go_deps.from_file(go_mod = "//:go.mod")` under Bzlmod. //! -//! One flow per Bazel release, in a fresh output base: +//! One flow, on the Bazel the run is given ([`FIXTURE`] says which rulesets +//! go with its release), in a fresh output base: //! //! - (a) `bazel build` and `bazel run //:consumer` print the unpatched //! marker. @@ -34,7 +35,9 @@ //! and goes on to the second after a 404, as it goes from the public proxy //! to the real patch host in a build outside these tests. No public proxy //! is in the list: on these gazelle releases the two modules of the test -//! are all that `go` asks a proxy for. +//! are all that `go` asks a proxy for. The caller's other fetch settings +//! are pinned beside it ([`go_env`]), so a `GOPRIVATE` of the machine +//! cannot send `go` past the stand-ins. use std::io::Write as _; use std::time::Instant; @@ -46,14 +49,27 @@ use wiremock::matchers::{method, path}; use wiremock::{Mock, MockServer, ResponseTemplate}; use crate::envelope::event_for; -use crate::harness::{changed, fixture_path, Cell, CONSUMER}; +use crate::harness::{changed, fixture_path, Cell, Fixture, CONSUMER}; use crate::hosted_memory_common::{mount_api, Patch}; -/// The module both fixtures require, and the version `go.sum` pins. +/// The workspace the test builds, and the rulesets each Bazel release +/// builds it with. +const FIXTURE: Fixture = Fixture { + name: "go::FIXTURE", + dir: "gazelle/bzlmod", + eras: &[ + // gazelle 0.45.0 and rules_go 0.53.0, with Go 1.23.4. + ((7, 0, 0)..(8, 0, 0), "e7"), + // gazelle 0.54.0 and rules_go 0.65.0, with Go 1.26.0. + ((8, 0, 0)..(10, 0, 0), "e9"), + ], +}; + +/// The module the fixture requires, and the version `go.sum` pins. const MODULE: &str = "example.com/upstream"; const VERSION: &str = "v1.0.0"; const PURL: &str = "pkg:golang/example.com/upstream@v1.0.0"; -/// Its two files. The fixtures' `go.sum` holds the `h1:` of these bytes. +/// Its two files. The fixture's `go.sum` holds the `h1:` of these bytes. const GO_MOD: &[u8] = b"module example.com/upstream\n\ngo 1.21\n"; const UPSTREAM_LIB: &[u8] = b"package upstream\n\nfunc Greeting() string { return \"PRISTINE\" }\n"; const UNPATCHED: &str = "OUT: PRISTINE"; @@ -218,13 +234,45 @@ impl StandIn { } } -/// The flow of the module docs for `test`, on Bazel `release` and the -/// fixture workspace `fixture`. -fn hosted(test: &str, release: &str, fixture: &str) { +/// What a `go` that Bazel starts reads in place of the caller's settings, +/// for [`Cell::repo_env`]: the proxy list, and every other setting that +/// decides where `go` fetches a module from and what it trusts. gazelle +/// hands `GOPROXY`, `GOPRIVATE`, `GONOPROXY`, `GONOSUMDB`, `GOSUMDB` and +/// (0.54) `GOAUTH` on to `go mod download` and clears the rest of the +/// environment, so a `GOPRIVATE=patch.socket.dev` of the machine would +/// send that fetch past the stand-ins to the real host. +fn go_env(proxies: &str) -> Vec<(String, String)> { + let pins = [ + ("GOPROXY", proxies), + ("GOPRIVATE", ""), + ("GONOPROXY", ""), + ("GONOSUMDB", ""), + // No checksum database. gazelle turns it off for the fetch, which + // it holds to `go.sum`; any other `go` that asked one would fail + // on this name instead of asking sum.golang.org about a module of + // the test. + ("GOSUMDB", "sum.invalid.example"), + // No credential helper of the machine. + ("GOAUTH", "off"), + // Not handed on by gazelle, but read by a `go` that a rule starts + // with the environment it inherits. + ("GOFLAGS", ""), + ("GOINSECURE", ""), + ]; + let owned = pins.map(|(name, value)| (name.to_string(), value.to_string())); + owned.to_vec() +} + +/// The flow of the module docs. +#[test] +#[ignore = "real Bazel; run with --ignored"] +fn hosted_go_deps() { + let test = "go::hosted_go_deps"; let started = Instant::now(); - let Some(mut cell) = Cell::open(test, release, fixture) else { + let Some(mut cell) = Cell::open(test, &FIXTURE) else { return; }; + let release = cell.release().to_string(); // The Socket module: the upstream files with `lib.go` patched, under a // path that is new for every run. @@ -255,7 +303,7 @@ fn hosted(test: &str, release: &str, fixture: &str) { cell.socket_patch(&origin, args, &[("SOCKET_GOSUMDB_URL", sumdb.as_str())]) }; let proxies = format!("{origin}/{PROXY},{origin}/{PATCH_HOST}"); - cell.repo_env.push(("GOPROXY".to_string(), proxies)); + cell.repo_env = go_env(&proxies); let (upstream_zip, socket_zip) = ( upstream.route(PROXY, "zip"), socket.route(PATCH_HOST, "zip"), @@ -370,40 +418,23 @@ fn hosted(test: &str, release: &str, fixture: &str) { println!( "{test}: Bazel {release} built {patched:?} from {socket_path}@{SOCKET_VERSION} after \ `scan --mode hosted`; `rollback` restored all {} files and the {UNPATCHED:?} build; other \ - bytes at the patch host failed a second machine's build ({refusal}); {:?}", + bytes at the patch host failed a second machine's build ({refusal}); {origin} answered \ + {} requests for the upstream zip and {} for the Socket zip; {:?}", before.len(), + standin.asked(&upstream_zip), + standin.asked(&socket_zip), started.elapsed() ); } -/// The fixtures pin the module these tests build: without a Bazel, this -/// holds the `h1:` arithmetic to sums that `go` computed. -#[test] -fn the_fixtures_pin_the_upstream_module_the_tests_serve() { - for era in ["e9", "e7"] { - let path = fixture_path(&format!("gazelle/bzlmod/{era}/go.sum")); - let go_sum = std::fs::read_to_string(&path).expect("the fixture's go.sum"); - let lines: Vec<&str> = go_sum.lines().collect(); - assert_eq!(lines, upstream().go_sum(), "{}", path.display()); - } -} - +/// Without a Bazel: the fixture's directories are those of its table, and +/// its `go.sum` pins the module the test serves, which holds the `h1:` +/// arithmetic here to sums that `go` computed. #[test] -#[ignore = "real Bazel; run with --ignored"] -fn b9_hosted_go_deps() { - hosted("b9_hosted_go_deps", "9.3.2", "gazelle/bzlmod/e9"); -} - -#[test] -#[ignore = "real Bazel; run with --ignored"] -fn b8_hosted_go_deps() { - hosted("b8_hosted_go_deps", "8.8.2", "gazelle/bzlmod/e9"); -} - -// The fixture for Bazel 7 pins older rulesets: gazelle 0.45.0 and -// rules_go 0.53.0. -#[test] -#[ignore = "real Bazel; run with --ignored"] -fn b7_hosted_go_deps() { - hosted("b7_hosted_go_deps", "7.7.2", "gazelle/bzlmod/e7"); +fn the_fixture_pins_the_upstream_module_the_test_serves() { + FIXTURE.check(); + let path = fixture_path(FIXTURE.dir).join("common/go.sum"); + let go_sum = std::fs::read_to_string(&path).expect("the fixture's go.sum"); + let lines: Vec<&str> = go_sum.lines().collect(); + assert_eq!(lines, upstream().go_sum(), "{}", path.display()); } diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs index c9ee60864..eca06895f 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs @@ -1,10 +1,12 @@ //! What every real-Bazel test shares: the variables that configure a run, +//! the Bazel a run is given and the fixture files that go with its release, //! the scratch root that holds everything one Bazel writes, the Bazel //! command line, and the `socket-patch` child that runs in the workspace //! copy. use std::collections::BTreeMap; -use std::ffi::OsString; +use std::ffi::{OsStr, OsString}; +use std::ops::Range; use std::path::{Path, PathBuf}; use std::process::{Command, Output}; @@ -14,10 +16,15 @@ use crate::fs_tree; use crate::hosted_memory_common::{hermetic, read_tree, ORG}; /// The Bazel binary or launcher to run, by absolute path: every child runs -/// in the workspace copy. Unset: `bazel` on `PATH`. +/// in the workspace copy. Unset: `bazel` on `PATH`. A test runs on the +/// release this Bazel says it is. pub const BAZEL: &str = "SOCKET_PATCH_BAZEL_E2E_BAZEL"; -/// Non-empty: a test that cannot run here (no Bazel, or another release) -/// fails. Unset: it prints `SKIP ` and passes. +/// The release that Bazel must be (`9.3.2`), so that a workflow leg cannot +/// pass on another one: a test fails when `bazel --version` says otherwise. +/// Unset: any release. +pub const VERSION: &str = "SOCKET_PATCH_BAZEL_E2E_VERSION"; +/// Non-empty: a test fails when no Bazel runs here. Unset: it prints +/// `SKIP ` and passes. pub const REQUIRED: &str = "SOCKET_PATCH_BAZEL_E2E_REQUIRED"; /// A repository cache shared by the tests of a run, so that only the first /// downloads the rulesets and the toolchain. Unset: an empty one per test. @@ -28,14 +35,19 @@ pub const SOCKET_BIN: &str = "SOCKET_PATCH_BAZEL_E2E_SOCKET_BIN"; /// Removed from every Bazel child. `TEST_TMPDIR` puts a Bazel server in /// test mode, where a failing module extension on 7.x is a crash (exit 37) -/// instead of an error. `USE_BAZEL_VERSION` is set again from the test's -/// release. `XDG_CACHE_HOME` would move a tool's cache out of the scratch -/// home. The CI markers change what some rulesets do, and a test must do -/// the same on a runner and on a developer's machine. +/// instead of an error. An `XDG_*_HOME` directory would move what a tool +/// keeps out of the scratch home: a cache, or on Linux the telemetry +/// counters of a Go SDK command that a ruleset starts, which Go puts under +/// `XDG_CONFIG_HOME` and no variable turns off. Without them a tool falls +/// back to a directory under `HOME`. The CI markers change what some +/// rulesets do, and a test must do the same on a runner and on a +/// developer's machine. const REMOVED: &[&str] = &[ "TEST_TMPDIR", - "USE_BAZEL_VERSION", "XDG_CACHE_HOME", + "XDG_CONFIG_HOME", + "XDG_DATA_HOME", + "XDG_STATE_HOME", "CI", "GITHUB_ACTIONS", ]; @@ -56,12 +68,86 @@ fn var(name: &str) -> Option { std::env::var_os(name).filter(|value| !value.is_empty()) } -/// `9.3.2` as three numbers; `None` for anything else. -fn numbers(release: &str) -> Option<(u32, u32, u32)> { - let parts: Option> = release.split('.').map(|part| part.parse().ok()).collect(); - match parts?[..] { - [major, minor, patch] => Some((major, minor, patch)), - _ => None, +/// The program `name` names, which must be an absolute path: a relative +/// one names nothing in the workspace copy a child runs in. A wrong +/// setting is a mistake, not a missing tool, so it fails and never skips. +fn program(name: &str) -> Option { + let path = PathBuf::from(var(name)?); + assert!( + path.is_absolute(), + "{name}={} must be an absolute path: the tests run it from a workspace copy", + path.display() + ); + Some(path) +} + +/// A Bazel release as three numbers: `9.3.2` is `(9, 3, 2)`. +pub type Release = (u32, u32, u32); + +/// The three numbers `release` starts with, whatever a build adds after +/// them: `9.3.2` and `9.3.2-homebrew` are `(9, 3, 2)`. `None` for anything +/// else. +fn numbers(release: &str) -> Option { + let mut parts = release.splitn(3, '.'); + let major = parts.next()?.parse().ok()?; + let minor = parts.next()?.parse().ok()?; + let patch = parts.next()?; + let digits = patch + .find(|c: char| !c.is_ascii_digit()) + .unwrap_or(patch.len()); + Some((major, minor, patch[..digits].parse().ok()?)) +} + +/// One line of a fixture's table: the Bazel releases of the range (its end +/// is the first release after it) build the workspace with the files of +/// the directory of that name. +pub type Era = (Range, &'static str); + +/// A fixture workspace. Its directory `common` holds the files every Bazel +/// release builds, and one directory per era holds the rest: the +/// `MODULE.bazel` that pins the rulesets of that era. A test copies both +/// into one workspace, so Bazel never meets two `MODULE.bazel` in a tree. +/// A new era is one such directory and one line of `eras`. +pub struct Fixture { + /// Where the constant is written (`go::FIXTURE`), for the failure of a + /// release its table does not hold. + pub name: &'static str, + /// The directory under `tests/fixtures/bazel/`. + pub dir: &'static str, + /// The one place that says which release builds which era. + pub eras: &'static [Era], +} + +impl Fixture { + /// The era directory for `release`. + fn era(&self, release: &str) -> Option<&'static str> { + let numbers = numbers(release)?; + let line = self.eras.iter().find(|(range, _)| range.contains(&numbers)); + line.map(|(_, era)| *era) + } + + /// Without a Bazel: the directories of the fixture are `common` and the + /// eras of the table, and each era holds a `MODULE.bazel`, which + /// `common` does not. + pub fn check(&self) { + let root = fixture_path(self.dir); + let mut found: Vec = std::fs::read_dir(&root) + .unwrap_or_else(|e| panic!("read {}: {e}", root.display())) + .map(|entry| entry.expect("a directory entry").path()) + .filter(|path| path.is_dir()) + .filter_map(|path| Some(path.file_name()?.to_string_lossy().into_owned())) + .collect(); + found.sort(); + let mut named: Vec<&str> = self.eras.iter().map(|(_, era)| *era).collect(); + named.push("common"); + named.sort_unstable(); + named.dedup(); + assert_eq!(found, named, "{}: {}", self.name, root.display()); + assert!(!root.join("common/MODULE.bazel").exists(), "{}", self.name); + for era in named.iter().filter(|era| **era != "common") { + let module = root.join(era).join("MODULE.bazel"); + assert!(module.is_file(), "{}: no {}", self.name, module.display()); + } } } @@ -84,8 +170,8 @@ pub fn changed(before: &Tree, after: &Tree) -> Vec { differing } -/// This machine cannot run `test`: a failure under [`REQUIRED`], else a -/// `SKIP` line. +/// No Bazel runs on this machine for `test`: a failure under [`REQUIRED`], +/// else a `SKIP` line. fn unavailable(test: &str, why: &str) { assert!( var(REQUIRED).is_none(), @@ -96,7 +182,8 @@ fn unavailable(test: &str, why: &str) { /// The temp root of one workspace copy, with a fixed place for everything a /// Bazel run writes. Nothing lands in the developer's `~/.cache/bazel`, the -/// shared temp directory or the real home. +/// shared temp directory or the real home, but for the one cache of a +/// launcher ([`launcher_cache`]). struct Scratch { /// The workspace copy Bazel and `socket-patch` run in. workspace: PathBuf, @@ -170,37 +257,83 @@ impl Drop for Scratch { } } +/// Where a launcher (bazelisk) keeps the Bazel releases it downloads: the +/// caller's own `BAZELISK_HOME`, or the directory bazelisk picks for the +/// caller without one (Go's `os.UserCacheDir()`, then `bazelisk`). Every +/// child gets it as `BAZELISK_HOME`: with the scratch `HOME` alone, a +/// launcher would download a whole release for every cell and every twin. +/// A release there is the caller's tool, like the `bazel` on `PATH`, and +/// nothing a workspace under test or a package manager wrote; a temp +/// directory of the test process would cost a download per run instead. +fn launcher_cache() -> Option { + if let Some(home) = var("BAZELISK_HOME") { + return Some(PathBuf::from(home)); + } + let macos = cfg!(target_os = "macos"); + let caches = match var("XDG_CACHE_HOME") { + Some(dir) if !macos => PathBuf::from(dir), + _ => PathBuf::from(var("HOME")?).join(if macos { "Library/Caches" } else { ".cache" }), + }; + Some(caches.join("bazelisk")) +} + +/// `program` in the workspace copy, with the environment of every Bazel +/// child and no arguments. Built on `hermetic::command`, the one way this +/// crate's tests spawn a child; what is set here lands after its scrub. +fn child(program: &OsStr, scratch: &Scratch) -> Command { + let mut cmd = hermetic::command(Path::new(program)); + for name in REMOVED { + cmd.env_remove(name); + } + // A bazelisk's other variables change what it does around a command + // (`BAZELISK_CLEAN`) and where it gets a release from. + for (name, _) in std::env::vars_os() { + if name.to_string_lossy().starts_with("BAZELISK_") { + cmd.env_remove(name); + } + } + if let Some(cache) = launcher_cache() { + cmd.env("BAZELISK_HOME", cache); + } + // The `go` a ruleset runs must not read the developer's module cache, + // flags or `go env -w` file. + hermetic::scrub_extra(&mut cmd, &[hermetic::Extra::Go]); + cmd.env("HOME", &scratch.home) + .env("TMPDIR", &scratch.tmp) + .current_dir(&scratch.workspace); + cmd +} + /// The Bazel a test runs. #[derive(Clone)] struct Bazel { /// A release binary, or a launcher that reads `USE_BAZEL_VERSION`. program: OsString, + /// What `--version` printed after `bazel `. release: String, } impl Bazel { - /// The Bazel of [`BAZEL`], if it is `release`. `bazel --version` is - /// answered by the client: no server starts. `Err` says why this - /// machine cannot run the test. - fn detect(scratch: &Scratch, release: &str) -> Result { - let program = var(BAZEL).unwrap_or_else(|| "bazel".into()); - let bazel = Bazel { - program, - release: release.to_string(), - }; - let shown = bazel.program.to_string_lossy().into_owned(); - let out = bazel - .child(scratch) + /// `program` (`bazel` on `PATH` without one) and the release it says + /// it is. `bazel --version` is answered by the client: no server + /// starts. A launcher answers for the release the caller's own + /// `USE_BAZEL_VERSION` picks. `Err` says why no Bazel runs here. + fn detect(scratch: &Scratch, program: Option) -> Result { + let program = program.map_or_else(|| "bazel".into(), PathBuf::into_os_string); + let shown = program.to_string_lossy().into_owned(); + let out = child(&program, scratch) .arg("--version") .output() .map_err(|e| format!("`{shown}` did not run: {e}"))?; let stdout = String::from_utf8_lossy(&out.stdout); - let found = stdout + let release = stdout .lines() .find_map(|line| line.trim().strip_prefix("bazel ")); - match found { - Some(found) if found == release => Ok(bazel), - Some(found) => Err(format!("`{shown}` is Bazel {found}, not {release}")), + match release { + Some(release) => Ok(Bazel { + program, + release: release.to_string(), + }), None => Err(format!( "`{shown} --version` printed no `bazel ` line:\n{stdout}{}", String::from_utf8_lossy(&out.stderr) @@ -208,43 +341,20 @@ impl Bazel { } } - /// The program with the environment of every Bazel child and no - /// arguments. Built on `hermetic::command`, the one way this crate's - /// tests spawn a child; what is set here lands after its scrub. - fn child(&self, scratch: &Scratch) -> Command { - let mut cmd = hermetic::command(Path::new(&self.program)); - for name in REMOVED { - cmd.env_remove(name); - } - // A bazelisk's other variables change which Bazel it runs. - for (name, _) in std::env::vars_os() { - if name.to_string_lossy().starts_with("BAZELISK_") { - cmd.env_remove(name); - } - } - // The `go` a ruleset runs must not read the developer's module - // cache, flags or `go env -w` file. - hermetic::scrub_extra(&mut cmd, &[hermetic::Extra::Go]); - cmd.env("USE_BAZEL_VERSION", &self.release) - .env("HOME", &scratch.home) - .env("TMPDIR", &scratch.tmp); - cmd - } - /// `bazel ` in the workspace copy. The /// startup options never vary within a test: another set restarts the /// server. They keep the rc files of the machine out and every write /// inside the scratch root; without the two JVM options the server /// writes `hsperfdata_` and its native libraries into the shared - /// temp directory. + /// temp directory. A launcher is held to the release it answered for. fn command(&self, scratch: &Scratch, command: &str) -> Command { let joined = |flag: &str, path: &Path| { let mut arg = OsString::from(flag); arg.push(path); arg }; - let mut cmd = self.child(scratch); - cmd.current_dir(&scratch.workspace) + let mut cmd = child(&self.program, scratch); + cmd.env("USE_BAZEL_VERSION", &self.release) .arg("--nohome_rc") .arg("--nosystem_rc") .arg(joined("--output_user_root=", &scratch.output_user_root)) @@ -307,52 +417,80 @@ pub struct Ran { /// output base of its own. pub struct Cell { pub test: String, - /// `--repo_env=NAME=VALUE` pairs for every Bazel command (`GOPROXY`). + /// What every repository rule must read in place of the caller's own + /// setting (`GOPROXY`): each pair is a variable of every Bazel command + /// and its `--repo_env=NAME=VALUE`. On Bazel 7, 8 and 9 a rule reads the + /// client's environment with `--repo_env` on top, and under + /// `--experimental_strict_repo_env` the flag alone. pub repo_env: Vec<(String, String)>, bazel: Bazel, + numbers: Release, + /// [`SOCKET_BIN`]. + socket_bin: Option, // Dropped after `Cell::drop` has stopped the server. scratch: Scratch, } impl Cell { - /// The checkout of the fixture workspace `fixture` for `test`, which - /// runs on Bazel `release`. `None` after a `SKIP` line when this machine - /// has no such Bazel; a panic instead under [`REQUIRED`]. - pub fn open(test: &str, release: &str, fixture: &str) -> Option { - let (major, _, _) = - numbers(release).unwrap_or_else(|| panic!("{test}: {release} is no Bazel release")); - // A workflow leg selects the tests of its Bazel by this prefix. - assert!( - test.starts_with(&format!("b{major}_")), - "{test} runs on Bazel {release}: its name must start with b{major}_" - ); + /// The checkout of `fixture` for `test`, on the Bazel of [`BAZEL`]: + /// the files of the fixture's `common` and of the era its table names + /// for that Bazel's release. `None` after a `SKIP` line when no Bazel + /// runs here; a panic instead under [`REQUIRED`]. A panic always for a + /// relative [`BAZEL`] or [`SOCKET_BIN`], a release that is not + /// [`VERSION`], and a release the table does not hold. + pub fn open(test: &str, fixture: &Fixture) -> Option { + let (program, socket_bin) = (program(BAZEL), program(SOCKET_BIN)); let shared = var(REPO_CACHE).map(PathBuf::from); let scratch = Scratch::new(shared); - let bazel = match Bazel::detect(&scratch, release) { + let bazel = match Bazel::detect(&scratch, program) { Ok(bazel) => bazel, Err(why) => { unavailable(test, &why); return None; } }; - let source = fixture_path(fixture); - assert!(source.is_dir(), "{test}: no fixture {}", source.display()); - fs_tree::copy_tree(&source, &scratch.workspace); + let release = &bazel.release; + let shown = bazel.program.to_string_lossy().into_owned(); + if let Some(wanted) = var(VERSION) { + assert!( + wanted.to_string_lossy() == *release, + "{test}: {VERSION}={} but `{shown}` is Bazel {release}", + wanted.to_string_lossy() + ); + } + let (Some(numbers), Some(era)) = (numbers(release), fixture.era(release)) else { + panic!( + "{test}: `{shown}` is Bazel {release}, which no line of the era table of {} \ + holds ({:?}): add its line, and its directory under {}", + fixture.name, + fixture.eras, + fixture_path(fixture.dir).display() + ); + }; + for part in ["common", era] { + let source = fixture_path(fixture.dir).join(part); + assert!(source.is_dir(), "{test}: no fixture {}", source.display()); + fs_tree::copy_tree(&source, &scratch.workspace); + } // The checkout is its own repository: `socket-patch` looks no // higher for a `socket.yml` or a project root. std::fs::create_dir(scratch.workspace.join(".git")).expect("create .git"); - println!( - "{test}: Bazel {release} ({})", - bazel.program.to_string_lossy() - ); + println!("{test}: Bazel {release} ({shown}), {}/{era}", fixture.dir); Some(Cell { test: test.to_string(), repo_env: Vec::new(), bazel, + numbers, + socket_bin, scratch, }) } + /// The release of this cell's Bazel, as its `--version` prints it. + pub fn release(&self) -> &str { + &self.bazel.release + } + /// A second machine with the same checkout: the files of this one, in /// a workspace copy with an output base and a contents cache of its /// own, so nothing a build here fetched or built is there. Only the @@ -366,6 +504,8 @@ impl Cell { test: self.test.clone(), repo_env: self.repo_env.clone(), bazel: self.bazel.clone(), + numbers: self.numbers, + socket_bin: self.socket_bin.clone(), scratch: Scratch::new(Some(self.scratch.repository_cache.clone())), }; for (rel, bytes) in self.tree() { @@ -402,7 +542,8 @@ impl Cell { fn bazel_with(&self, lockfile_mode: &str, command: &str, args: &[&str]) -> Run { let scratch = &self.scratch; let mut cmd = self.bazel.command(scratch, command); - let mut joined = |flag: &str, value: &std::ffi::OsStr| { + cmd.envs(self.repo_env.iter().map(|(name, value)| (name, value))); + let mut joined = |flag: &str, value: &OsStr| { let mut arg = OsString::from(flag); arg.push(value); cmd.arg(arg); @@ -416,8 +557,7 @@ impl Cell { } // Stated, so no test depends on a release's default. joined("--lockfile_mode=", lockfile_mode.as_ref()); - let release = numbers(&self.bazel.release).expect("checked by Cell::open"); - if release >= (9, 0, 0) { + if self.numbers >= (9, 0, 0) { // Bazel 9 keeps the repositories it fetched in a contents cache, // by default under the repository cache, and hands them to the // next workspace copy without a fetch. The cache stays on, as a @@ -450,8 +590,8 @@ impl Cell { /// and temp directory are the cell's, and every proxy is a closed port, /// so it reaches no host but loopback. pub fn socket_patch(&self, api: &str, args: &[&str], env: &[(&str, &str)]) -> Ran { - let mut cmd = match var(SOCKET_BIN) { - Some(bin) => hermetic::command(Path::new(&bin)), + let mut cmd = match &self.socket_bin { + Some(bin) => hermetic::command(bin), None => hermetic::binary_command(), }; cmd.current_dir(&self.scratch.workspace) @@ -514,7 +654,32 @@ impl Drop for Cell { fn a_release_is_read_as_three_numbers() { assert_eq!(numbers("9.3.2"), Some((9, 3, 2))); assert_eq!(numbers("7.7.2"), Some((7, 7, 2))); - for unreadable in ["9.3", "9.3.2.1", "10.0.0rc3", ""] { + assert_eq!(numbers("9.3.2-homebrew"), Some((9, 3, 2))); + assert_eq!(numbers("10.0.0rc3"), Some((10, 0, 0))); + for unreadable in ["9.3", "9.x.2", "no_version", ""] { assert_eq!(numbers(unreadable), None, "{unreadable:?}"); } } + +#[test] +fn a_release_builds_the_era_whose_range_holds_it() { + let fixture = Fixture { + name: "a table", + dir: "none", + eras: &[ + ((7, 0, 0)..(8, 0, 0), "old"), + ((8, 0, 0)..(10, 0, 0), "new"), + ], + }; + for (release, era) in [ + ("7.0.0", Some("old")), + ("7.7.2", Some("old")), + ("8.0.0", Some("new")), + ("9.3.2-homebrew", Some("new")), + ("6.5.0", None), + ("10.0.0", None), + ("no_version", None), + ] { + assert_eq!(fixture.era(release), era, "{release}"); + } +} diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs index ffb26783d..9b9b318b7 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs @@ -2,9 +2,9 @@ //! `socket-patch` in a Bazel workspace, on a real Bazel. //! //! Each test copies a fixture workspace from `tests/fixtures/bazel/`, builds -//! it with a pinned Bazel release, runs the built `socket-patch` binary in -//! it the way a user does, and builds again. What a test asserts, it reads -//! from what Bazel built: the line `bazel run //:consumer` prints. +//! it with the Bazel release it is given, runs the built `socket-patch` +//! binary in it the way a user does, and builds again. What a test asserts, +//! it reads from what Bazel built: the line `bazel run //:consumer` prints. //! //! The tests are `#[ignore]`d: they need a Bazel release and the network //! (the Bazel registry, the ruleset archives and the language toolchain), @@ -14,23 +14,25 @@ //! release and ecosystem. Locally: //! //! ```sh -//! eval "$(scripts/install-bazel.sh 9.3.2 "$HOME/.cache/socket-patch-bazel")" -//! export SOCKET_PATCH_BAZEL_E2E_BAZEL -//! cargo test -p socket-patch-cli --test e2e_bazel_build -- --ignored go::b9_ --nocapture +//! export "$(scripts/install-bazel.sh 9.3.2 "$HOME/.cache/socket-patch-bazel")" +//! cargo test -p socket-patch-cli --test e2e_bazel_build -- --ignored go:: --nocapture //! ``` //! -//! With `SOCKET_PATCH_BAZEL_E2E_BAZEL` unset the tests run `bazel` from -//! `PATH` with `USE_BAZEL_VERSION` set to the test's release, so a bazelisk -//! picks it. A test whose Bazel is missing or is another release prints -//! `SKIP ` and passes; with `SOCKET_PATCH_BAZEL_E2E_REQUIRED=1` (CI) it -//! fails instead. `harness.rs` lists the other variables. +//! A test takes its release from the Bazel it runs: the one +//! `SOCKET_PATCH_BAZEL_E2E_BAZEL` names, else `bazel` on `PATH` (a bazelisk +//! there picks its release as it does anywhere, from `USE_BAZEL_VERSION`). +//! With no Bazel a test prints `SKIP ` and passes; with +//! `SOCKET_PATCH_BAZEL_E2E_REQUIRED=1` (CI) it fails instead. `harness.rs` +//! lists the other variables. //! -//! To add a test of a flow that is here: one fixture directory and one -//! test function in the ecosystem's module. A test's name starts with its -//! Bazel major (`b9_`), which is how a workflow leg selects its own, so a -//! major has one release here. A new Bazel major adds its lines in -//! `scripts/bazel-pins.txt` and one line in the workflow's matrix; a new -//! ecosystem adds its module and one matrix line per release. +//! A flow is one `#[ignore]`d test in its ecosystem's module, and its +//! fixture is one directory of files that every release builds beside one +//! `MODULE.bazel` per era of rulesets; the fixture's table in the test code +//! says which releases build which era (`harness::Fixture`). So another +//! Bazel release is one line per platform in `scripts/bazel-pins.txt` and +//! one line of the workflow's matrix; another era is one `MODULE.bazel` +//! and one line of the table; another ecosystem is its module, its fixture +//! and one line of the workflow's matrix per release. #[path = "../common/envelope.rs"] mod envelope; diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/BUILD.bazel b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/common/BUILD.bazel similarity index 100% rename from crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/BUILD.bazel rename to crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/common/BUILD.bazel diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.mod b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/common/go.mod similarity index 100% rename from crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.mod rename to crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/common/go.mod diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.sum b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/common/go.sum similarity index 100% rename from crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/go.sum rename to crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/common/go.sum diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/main.go b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/common/main.go similarity index 100% rename from crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e7/main.go rename to crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/common/main.go diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/BUILD.bazel b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/BUILD.bazel deleted file mode 100644 index 29489e934..000000000 --- a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/BUILD.bazel +++ /dev/null @@ -1,8 +0,0 @@ -load("@rules_go//go:def.bzl", "go_binary") - -# Prints OUT: PRISTINE until the module is patched. -go_binary( - name = "consumer", - srcs = ["main.go"], - deps = ["@com_example_upstream//:upstream"], -) diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.mod b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.mod deleted file mode 100644 index 61cbb9690..000000000 --- a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.mod +++ /dev/null @@ -1,5 +0,0 @@ -module example.com/consumer - -go 1.21 - -require example.com/upstream v1.0.0 diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.sum b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.sum deleted file mode 100644 index ad31912aa..000000000 --- a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/go.sum +++ /dev/null @@ -1,2 +0,0 @@ -example.com/upstream v1.0.0 h1:SLqDtcPyVkkOYblt1rSVexJHaLWU/teyZ8o5mHyiUbA= -example.com/upstream v1.0.0/go.mod h1:WGm+Dt0Rps9QKvOZJwtpSRFNm3I9B5DlvBXyx4SHodQ= diff --git a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/main.go b/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/main.go deleted file mode 100644 index aa396ab81..000000000 --- a/crates/socket-patch-cli/tests/fixtures/bazel/gazelle/bzlmod/e9/main.go +++ /dev/null @@ -1,9 +0,0 @@ -package main - -import ( - "fmt" - - "example.com/upstream" -) - -func main() { fmt.Println("OUT:", upstream.Greeting()) } diff --git a/scripts/install-bazel.sh b/scripts/install-bazel.sh index d7632db06..6b4a688cb 100755 --- a/scripts/install-bazel.sh +++ b/scripts/install-bazel.sh @@ -1,7 +1,6 @@ #!/usr/bin/env bash # Install one pinned Bazel release binary for the real-Bazel tests -# (`e2e_bazel_build`), then print the variable a CI step appends to -# $GITHUB_ENV. +# (`e2e_bazel_build`) and print where it is. # # scripts/install-bazel.sh # @@ -15,8 +14,11 @@ # The tests run the verified binary directly, so no bazelisk, `.bazelversion` # or network version lookup is involved. # -# Prints (stdout): +# Prints one line on stdout, and everything else on stderr: # SOCKET_PATCH_BAZEL_E2E_BAZEL= +# A CI step appends that line to $GITHUB_ENV, and a shell takes it with +# export "$(scripts/install-bazel.sh )" +# Both keep a space in . set -euo pipefail die() { @@ -25,11 +27,13 @@ die() { } if [ $# -ne 2 ]; then - sed -n '2,19p' "$0" | sed 's/^# \{0,1\}//' >&2 + sed -n '2,21p' "$0" | sed 's/^# \{0,1\}//' >&2 exit 2 fi VERSION="$1" DIR="$2" +# What is printed is one line. +case "$DIR" in *$'\n'*) die " must not hold a newline" ;; esac PINS="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/bazel-pins.txt" # The version becomes part of two URLs and an awk variable. @@ -88,9 +92,17 @@ fetch() { # interrupted or tampered download never sits where a later run (or a # cache save) would pick it up. local tmp="$bazel.download" + # A stalled transfer must not sit until the job times out, and a file an + # origin does not have must not be asked for again. Per attempt: 20 s to + # connect, an end after 30 s below 30 kB/s, and 10 minutes in all (the + # binary is about 60 MB, so that is a link of 1 Mbit/s). curl retries only + # what it calls transient (a timeout, HTTP 408, 429, 500, 502, 503 and + # 504) and starts no retry after 5 minutes. Any other failure, a 404 + # included, goes to the next origin at once. for url in "$@"; do rm -f "$tmp" - if curl -fsSL --retry 5 --retry-all-errors "$url" -o "$tmp"; then + if curl -fsSL --connect-timeout 20 --speed-limit 30000 --speed-time 30 \ + --max-time 600 --retry 3 --retry-max-time 300 "$url" -o "$tmp"; then got="$(sha256_of "$tmp")" if [ "$got" = "$sha" ]; then mv -f "$tmp" "$bazel" From b6564c0f2f0d218938f82932c89eb64912b46a66 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Tue, 17 Nov 2026 02:36:16 -0500 Subject: [PATCH 3/5] Share the Bazel test harness across ecosystems, one workflow each The Go flow kept to itself what any flow on a real Bazel needs, and the workflow had one path list and one matrix for every ecosystem. A second ecosystem would copy the first, and each would start its legs on every change to the other. harness.rs now holds what the flows share: - StandIn, the loopback server behind the hosts of a flow (start, origin, serve, api, requests, asked), and the steps a flow is made of: the unpatched build with Bazel's lock, the envelope and the files of a hosted scan, the patched build that changes no file and passes with a frozen lock, and the undo. go.rs uses them and asserts what it asserted before. - Cell::route_downloads and rewrite, for a ruleset that fetches with Bazel's own downloader: a downloader config beside the workspace copy, named on every later command of the cell and of its twin by the option of the release (--experimental_downloader_config before Bazel 8.2.0, --downloader_config from there on; 8.2.0 and later still take the old name, with a warning). A rule quotes its URL prefix, so a `.` or a `+` in a file name is no pattern. Cell::numbers gives a flow the parsed release. - The environment of every Bazel child. The tool settings of a developer's shell no longer reach the repository rules and `bazel run`: GEM_HOME, GEM_PATH, NODE_OPTIONS, NODE_PATH, PYTHONHOME, PYTHONPATH, PYTHONSTARTUP, PIP_* and UV_*, and through the scrubs of tests/common/hermetic.rs BUNDLE_*, RUBYOPT, RUBYLIB, npm_config_*, PNPM_* and VIRTUAL_ENV, beside the Go settings removed before. The proxy of the machine stays, because a build may need it to reach the Bazel registry, and loopback is added to no_proxy and NO_PROXY: without it Bazel's downloader asks the proxy for a stand-in's URL. Two hermetic tests hold this. One checks the option name, and the config a cell and its twin name. The other starts the test binary again with every removed variable and a proxy set, and reads the environment of a child spawned there. .github/workflows/bazel-compatibility.yml becomes a reusable workflow that holds the legs: the Linux matrix, the macOS job, the installer, the caches, the REQUIRED and VERSION settings and the BAZEL_COMPAT_DISABLED switch. .github/workflows/bazel-go.yml calls it with the Go module, fixture and releases, under its own triggers, path lists and concurrency group. Another ecosystem adds a caller and touches neither file. A pull request run covers the ecosystem's own flow and fixture, the harness, the installer with its pins and the two workflows; shared engine code is covered on the push to main and nightly, as before. The repository cache is one per ecosystem and fixture era now, not one per Bazel release: the legs of one era share it. A caller names the era of each release it runs, and Fixture::check fails when that is not the era the test code's table gives the release. The run step fails when its module filter matches no test, which would pass with nothing run. Co-Authored-By: Claude Fable 5.2 --- .github/workflows/bazel-compatibility.yml | 190 +++---- .github/workflows/bazel-go.yml | 104 ++++ .../tests/e2e_bazel_build/go.rs | 176 ++----- .../tests/e2e_bazel_build/harness.rs | 490 ++++++++++++++++-- .../tests/e2e_bazel_build/main.rs | 11 +- 5 files changed, 722 insertions(+), 249 deletions(-) create mode 100644 .github/workflows/bazel-go.yml diff --git a/.github/workflows/bazel-compatibility.yml b/.github/workflows/bazel-compatibility.yml index f1d212366..9a1a7079b 100644 --- a/.github/workflows/bazel-compatibility.yml +++ b/.github/workflows/bazel-compatibility.yml @@ -1,86 +1,79 @@ -name: Bazel patch compatibility - -# socket-patch in a Bazel workspace, on a real Bazel: one leg per Bazel -# release and ecosystem runs the `#[ignore]` tests of that ecosystem's module -# of `e2e_bazel_build` (`go::`) on the release it installed. A test builds a -# fixture workspace, runs the built binary in it and reads the result from -# what Bazel builds next; the test code says which fixture files go with -# which release. +name: Bazel patch compatibility (legs) + +# socket-patch in a Bazel workspace, on a real Bazel: the legs of one +# ecosystem. One leg per Bazel release runs the `#[ignore]` tests of the +# ecosystem's module of `e2e_bazel_build` (`go::`) on the release it +# installed. A test builds a fixture workspace, runs the built binary in it +# and reads the result from what Bazel builds next; the test code says which +# fixture files go with which release. +# +# This is a reusable workflow, and nothing starts it but a caller. Each +# ecosystem has one (bazel-go.yml), which says when its legs run (its +# triggers and path lists, its concurrency group) and on which Bazel +# releases (the inputs below). Another ecosystem is one more caller, and no +# change here or in another ecosystem's caller. A called workflow runs with +# the event, the ref and the token of its caller: every `github.*` below is +# the caller's, the token is the `contents: read` the caller grants, and the +# file is the one of the caller's commit, so a pull request that changes it +# runs its own version. # # The package under test and every Socket host are loopback stand-ins. A leg # still needs the network for the ruleset and the toolchain: github.com and # release-assets.githubusercontent.com (the Bazel release, cached per -# release, and the ruleset archives), dl.google.com (the Go SDK) and, on -# every run, bcr.bazel.build (the registry) and go.dev (the Go release -# list). A repository cache saved from main holds the archives and the SDK. +# release, and the ruleset archives), bcr.bazel.build (the registry) on +# every run, and the hosts of the ecosystem's toolchain, which its caller +# names. A repository cache saved from main holds the archives and the +# toolchain. # -# Not required for merge: no `merge_group` trigger, and no job here is a -# `needs` of ci.yml's `ci-ok`. Setting the repository variable -# BAZEL_COMPAT_DISABLED to `true` turns every leg off without a merge. +# Not required for merge: no caller has a `merge_group` trigger, and no job +# here is a `needs` of ci.yml's `ci-ok`. Setting the repository variable +# BAZEL_COMPAT_DISABLED to `true` turns every leg of every ecosystem off +# without a merge. # # To add a Bazel release: one line per platform in scripts/bazel-pins.txt -# and one matrix line here. To add an ecosystem: its module in -# `e2e_bazel_build` and one matrix line per release. +# and one line of `releases` in each caller that runs it. # # No job has a `name:`. GitHub does not expand a matrix expression in the # name of a job that its `if` skips, so such a name shows up as its source -# text in a pull request's checks. The default title of a leg is the job's -# id and its matrix values: `bazel (ubuntu-latest, 9.3.2, go, gazelle)`. +# text in a pull request's checks. The default title of a leg is the +# caller's job, the job's id here and its matrix values: +# `go / bazel (9.3.2, e9)`. on: - pull_request: - types: [opened, synchronize, reopened, ready_for_review] - # The suite's own files and the Go files its tests run through, as - # go-compatibility.yml lists them. Shared engine code - # (patch/redirect/mod.rs, hosted/, commands/) gets its run on push to - # main (when it is in the push filter below) or nightly, or on demand - # via workflow_dispatch on the PR branch. - paths: - - '.github/workflows/bazel-compatibility.yml' - - 'scripts/install-bazel.sh' - - 'scripts/bazel-pins.txt' - - 'crates/socket-patch-cli/tests/e2e_bazel_build/**' - - 'crates/socket-patch-cli/tests/fixtures/bazel/**' - - 'crates/*/src/**/*golang*' - - 'crates/*/src/**/*golang*/**' - - 'crates/socket-patch-core/src/vendor/go*.rs' - push: - branches: [main] - # The PR paths, the shared engine code and test helpers the tests run - # through, and the toolchain files. A main push outside this set waits - # for the nightly run below. - paths: - - '.github/workflows/bazel-compatibility.yml' - - 'scripts/install-bazel.sh' - - 'scripts/bazel-pins.txt' - - 'crates/socket-patch-cli/tests/e2e_bazel_build/**' - - 'crates/socket-patch-cli/tests/fixtures/bazel/**' - - 'crates/*/src/**/*golang*' - - 'crates/*/src/**/*golang*/**' - - 'crates/socket-patch-core/src/vendor/go*.rs' - - 'crates/socket-patch-core/src/patch/redirect/**' - - 'crates/socket-patch-core/src/hosted/**' - - 'crates/socket-patch-cli/src/commands/**' - - 'crates/socket-patch-cli/tests/common/**' - - 'crates/socket-patch-cli/tests/hosted_memory_common/**' - - 'Cargo.lock' - - 'Cargo.toml' - - 'crates/*/Cargo.toml' - - 'rust-toolchain.toml' - - '.cargo/**' - schedule: - # Nightly, so a main change outside the push filter still gets a run - # within a day. - - cron: '17 4 * * *' - workflow_dispatch: + workflow_call: + inputs: + eco: + description: >- + The ecosystem's module of the `e2e_bazel_build` test target + (`go`). A leg runs the `#[ignore]` tests of that module. + required: true + type: string + fixtures: + description: >- + The ecosystem's fixture, a directory under + crates/socket-patch-cli/tests/fixtures/bazel (`gazelle/bzlmod`). + required: true + type: string + releases: + description: >- + The Linux legs, as a JSON list with one + {"bazel": "", "era": ""} per leg: a Bazel + release that scripts/bazel-pins.txt pins, and the directory of the + fixture that holds the `MODULE.bazel` this release builds. The + test code decides the era of a release, and the hermetic test of + the ecosystem's module fails when a leg names another. + required: true + type: string + macos: + description: >- + The macOS legs, in the same form. They run nightly and on demand + only. + required: true + type: string permissions: contents: read -concurrency: - group: bazel-compat-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - env: SOCKET_NO_CONFIG: '1' SOCKET_NO_UPDATE_CHECK: '1' @@ -92,14 +85,8 @@ jobs: strategy: fail-fast: false matrix: - include: - # `eco`: the ecosystem's module of the test target. `fixtures`: - # its directory under tests/fixtures/bazel, which keys its - # repository cache. - - {os: ubuntu-latest, bazel: '9.3.2', eco: go, fixtures: gazelle} - - {os: ubuntu-latest, bazel: '8.8.2', eco: go, fixtures: gazelle} - - {os: ubuntu-latest, bazel: '7.7.2', eco: go, fixtures: gazelle} - runs-on: ${{ vars.DISABLE_DEPOT_RUNNERS != 'true' && (matrix.os == 'ubuntu-latest' && 'depot-ubuntu-24.04-4' || matrix.os == 'ubuntu-22.04' && 'depot-ubuntu-22.04-4') || matrix.os }} + include: ${{ fromJSON(inputs.releases) }} + runs-on: ${{ vars.DISABLE_DEPOT_RUNNERS == 'true' && 'ubuntu-latest' || 'depot-ubuntu-24.04-4' }} timeout-minutes: 45 steps: &bazel-steps - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 @@ -115,12 +102,14 @@ jobs: - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1 with: - shared-key: dev-${{ matrix.os }} + # The key the other workflows give the runner image of this job. + shared-key: dev-${{ runner.os == 'macOS' && 'macos-latest' || 'ubuntu-latest' }} save-if: ${{ github.ref == 'refs/heads/main' }} - name: Restore the Bazel ${{ matrix.bazel }} binary # The installer checks a restored file against its pin too, so a - # miss or a stale entry only means a download. Saved from main. + # miss or a stale entry only means a download. Saved from main, and + # shared by every ecosystem that runs the release. id: bazel-dist uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0 timeout-minutes: 5 @@ -148,41 +137,53 @@ jobs: key: ${{ steps.bazel-dist.outputs.cache-primary-key }} - name: Restore the repository cache - # What Bazel downloads for the fixtures: ruleset archives and the - # language toolchain (about 100 MB for Go). Keyed by the - # ecosystem's fixtures, which pin all of it, so another ecosystem's - # fixture does not make this leg save a new copy; a run whose - # fixtures changed starts from the newest cache of its leg. The - # cache is content-addressed, so an older one is still correct, - # only incomplete. The package under test never comes from it: `go` - # fetches a module into the output base, which every test makes anew. + # What Bazel downloads for the fixture: the ruleset archives and the + # language toolchain. One cache per ecosystem and fixture era, keyed + # by the files that pin all of it: `common` and the era's + # directory. The legs of one era share it, whatever their Bazel + # release, and another era or another ecosystem never makes a leg + # save a new copy. A run whose fixture changed starts from the + # newest cache of its era. The cache is content-addressed, so an + # older one, or one that another release of the era saved, is still + # correct, only incomplete: a leg downloads what it lacks, which for + # a second release of an era is what that Bazel resolves for its own + # built-in modules. Whether the package under test can come from it + # is the ecosystem's matter, and its test says how it rules that out. id: bazel-repos uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0 timeout-minutes: 5 continue-on-error: true with: path: ${{ runner.temp }}/bazel-repos - key: bazel-repos-${{ matrix.eco }}-${{ matrix.bazel }}-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles(format('crates/socket-patch-cli/tests/fixtures/bazel/{0}/**', matrix.fixtures)) }} + key: bazel-repos-${{ inputs.eco }}-${{ matrix.era }}-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles(format('crates/socket-patch-cli/tests/fixtures/bazel/{0}/common/**', inputs.fixtures), format('crates/socket-patch-cli/tests/fixtures/bazel/{0}/{1}/**', inputs.fixtures, matrix.era)) }} restore-keys: | - bazel-repos-${{ matrix.eco }}-${{ matrix.bazel }}-${{ runner.os }}-${{ runner.arch }}- + bazel-repos-${{ inputs.eco }}-${{ matrix.era }}-${{ runner.os }}-${{ runner.arch }}- - - name: Run the ${{ matrix.eco }} tests on Bazel ${{ matrix.bazel }} + - name: Run the ${{ inputs.eco }} tests on Bazel ${{ matrix.bazel }} env: # A missing Bazel fails the leg instead of printing SKIP, and so # does a Bazel that is not the release of this leg. SOCKET_PATCH_BAZEL_E2E_REQUIRED: '1' SOCKET_PATCH_BAZEL_E2E_VERSION: ${{ matrix.bazel }} SOCKET_PATCH_BAZEL_E2E_REPO_CACHE: ${{ runner.temp }}/bazel-repos - ECO: ${{ matrix.eco }} + ECO: ${{ inputs.eco }} # The ecosystem's module holds one `#[ignore]` test per flow, and a # test takes its release from the Bazel installed above, so the # filter is the module's name and nothing is derived from the - # release. Two tests at a time: each runs a Bazel server of its own. + # release. The first command lists the tests of the filter and fails + # when there is none: a module name that matches nothing would run + # no test and pass. Two tests at a time: each runs a Bazel server of + # its own. run: | + cargo test --locked -p socket-patch-cli --test e2e_bazel_build -- \ + --ignored "${ECO}::" --list | grep ': test$' cargo test --locked -p socket-patch-cli --test e2e_bazel_build -- \ --ignored "${ECO}::" --test-threads=2 --nocapture - name: Save the repository cache + # The first leg of an era to finish on main saves it. Another leg + # of the era that also missed finds the key taken, which the action + # logs without failing the step. if: >- github.ref == 'refs/heads/main' && steps.bazel-repos.outputs.cache-hit != 'true' @@ -192,15 +193,14 @@ jobs: path: ${{ runner.temp }}/bazel-repos key: ${{ steps.bazel-repos.outputs.cache-primary-key }} - # The macOS rows of `bazel`: nightly and on demand, never on a pull - # request or a push, so they do not queue on the small macOS runner pool. + # The macOS legs: nightly and on demand, never on a pull request or a + # push, so they do not queue on the small macOS runner pool. bazel-macos: if: (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && vars.BAZEL_COMPAT_DISABLED != 'true' strategy: fail-fast: false matrix: - include: - - {os: macos-latest, bazel: '9.3.2', eco: go, fixtures: gazelle} - runs-on: ${{ vars.DISABLE_DEPOT_RUNNERS != 'true' && (matrix.os == 'ubuntu-latest' && 'depot-ubuntu-24.04-4' || matrix.os == 'ubuntu-22.04' && 'depot-ubuntu-22.04-4') || matrix.os }} + include: ${{ fromJSON(inputs.macos) }} + runs-on: macos-latest timeout-minutes: 45 steps: *bazel-steps diff --git a/.github/workflows/bazel-go.yml b/.github/workflows/bazel-go.yml new file mode 100644 index 000000000..d809f6510 --- /dev/null +++ b/.github/workflows/bazel-go.yml @@ -0,0 +1,104 @@ +name: Bazel patch compatibility (Go) + +# The Go flow of the real-Bazel tests (`go::` in `e2e_bazel_build`): hosted +# mode for a module that gazelle's `go_deps` resolves from `go.mod`, on +# each Bazel release below. bazel-compatibility.yml holds the legs and what +# every ecosystem shares; this file says when the Go legs run and on which +# releases. +# +# Beyond what every leg needs, a Go leg downloads the Go SDK from +# dl.google.com (about 100 MB, held by its repository cache) and asks go.dev +# for the Go release list on every run. A Go module under test never comes +# from the repository cache: `go` fetches a module into the output base, +# which every test makes anew. +# +# Not required for merge: no `merge_group` trigger, and no job here is a +# `needs` of ci.yml's `ci-ok`. + +on: + pull_request: + types: [opened, synchronize, reopened, ready_for_review] + # What a pull request run covers: the Go flow and its fixture, the + # harness and the workflows that run it, the Bazel installer and its + # pins, and the Go files the flow runs through, as go-compatibility.yml + # lists them. Nothing another ecosystem's flow owns is here, so a change + # to one ecosystem starts that ecosystem's legs only. + # + # What it does not cover: shared engine code (patch/redirect/mod.rs, + # hosted/, commands/), the shared test helpers and the toolchain files. + # A change there gets its run on the push to main (when the file is in + # the list below) or in the nightly run, not on the pull request: that + # is the price of not running every ecosystem's legs on every shared + # change. On demand, workflow_dispatch runs the legs on any branch. + paths: + - '.github/workflows/bazel-go.yml' + - '.github/workflows/bazel-compatibility.yml' + - 'scripts/install-bazel.sh' + - 'scripts/bazel-pins.txt' + - 'crates/socket-patch-cli/tests/e2e_bazel_build/go.rs' + - 'crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs' + - 'crates/socket-patch-cli/tests/fixtures/bazel/gazelle/**' + - 'crates/*/src/**/*golang*' + - 'crates/*/src/**/*golang*/**' + - 'crates/socket-patch-core/src/vendor/go*.rs' + push: + branches: [main] + # The pull request paths, and what only a push to main covers: the + # root of the test target (it changes when an ecosystem's module is + # added; ci.yml builds the target and runs its hermetic tests on every + # pull request), the shared engine code and test helpers the flow runs + # through, and the toolchain files. A main push outside this list waits + # for the nightly run below. + paths: + - '.github/workflows/bazel-go.yml' + - '.github/workflows/bazel-compatibility.yml' + - 'scripts/install-bazel.sh' + - 'scripts/bazel-pins.txt' + - 'crates/socket-patch-cli/tests/e2e_bazel_build/go.rs' + - 'crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs' + - 'crates/socket-patch-cli/tests/fixtures/bazel/gazelle/**' + - 'crates/*/src/**/*golang*' + - 'crates/*/src/**/*golang*/**' + - 'crates/socket-patch-core/src/vendor/go*.rs' + - 'crates/socket-patch-cli/tests/e2e_bazel_build/main.rs' + - 'crates/socket-patch-core/src/patch/redirect/**' + - 'crates/socket-patch-core/src/hosted/**' + - 'crates/socket-patch-cli/src/commands/**' + - 'crates/socket-patch-cli/tests/common/**' + - 'crates/socket-patch-cli/tests/hosted_memory_common/**' + - 'Cargo.lock' + - 'Cargo.toml' + - 'crates/*/Cargo.toml' + - 'rust-toolchain.toml' + - '.cargo/**' + schedule: + # Nightly, so a main change outside the push list still gets a run + # within a day. The macOS leg runs here and on demand only. + - cron: '17 4 * * *' + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: bazel-go-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + go: + uses: ./.github/workflows/bazel-compatibility.yml + with: + eco: go + fixtures: gazelle/bzlmod + # One leg per line: a Bazel release, and the era of the fixture that + # `go::FIXTURE` gives it. + releases: >- + [ + {"bazel": "9.3.2", "era": "e9"}, + {"bazel": "8.8.2", "era": "e9"}, + {"bazel": "7.7.2", "era": "e7"} + ] + macos: >- + [ + {"bazel": "9.3.2", "era": "e9"} + ] diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs index 4494637e7..3060b0a77 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/go.rs @@ -45,18 +45,16 @@ use std::time::Instant; use base64::Engine as _; use serde_json::json; use sha2::{Digest, Sha256}; -use wiremock::matchers::{method, path}; -use wiremock::{Mock, MockServer, ResponseTemplate}; -use crate::envelope::event_for; -use crate::harness::{changed, fixture_path, Cell, Fixture, CONSUMER}; -use crate::hosted_memory_common::{mount_api, Patch}; +use crate::harness::{fixture_path, Cell, Fixture, StandIn, CONSUMER}; +use crate::hosted_memory_common::Patch; /// The workspace the test builds, and the rulesets each Bazel release /// builds it with. const FIXTURE: Fixture = Fixture { name: "go::FIXTURE", dir: "gazelle/bzlmod", + workflow: "bazel-go.yml", eras: &[ // gazelle 0.45.0 and rules_go 0.53.0, with Go 1.23.4. ((7, 0, 0)..(8, 0, 0), "e7"), @@ -79,8 +77,6 @@ const SOCKET_VERSION: &str = "v1.0.0-socketpatch.1"; const PROXY: &str = "proxy.golang.org"; const PATCH_HOST: &str = "patch.socket.dev"; const SUMDB: &str = "sum.golang.org"; -/// What [`changed`] gives for two equal trees. -const NOTHING: [&str; 0] = []; /// Go's `h1:` hash of a set of files: the sha256 of one /// ` \n` line per file, in the order of the names. @@ -150,88 +146,48 @@ fn upstream() -> Module { Module::new(MODULE, VERSION, UPSTREAM_LIB) } -/// The loopback server behind every host of the test. -struct StandIn { - // Dropped before the runtime it runs on. - server: MockServer, - runtime: tokio::runtime::Runtime, +/// Serve `module` on `standin` in the GOPROXY layout under `/`. A +/// route the stand-in does not serve is a 404, which sends `go` to the next +/// entry of its `GOPROXY` list. +fn serve_module(standin: &StandIn, host: &str, module: &Module) { + let version = &module.version; + let info = format!(r#"{{"Version": "{version}", "Time": "2026-01-01T00:00:00Z"}}"#); + standin.serve(5, &module.route(host, "info"), info.as_bytes()); + standin.serve(5, &module.route(host, "mod"), GO_MOD); + standin.serve(5, &module.route(host, "zip"), &module.zip); } -impl StandIn { - fn start() -> StandIn { - let runtime = tokio::runtime::Builder::new_multi_thread() - .worker_threads(2) - .enable_all() - .build() - .expect("a runtime for the stand-in"); - let server = runtime.block_on(MockServer::start()); - StandIn { server, runtime } - } - - /// `http://127.0.0.1:`. - fn origin(&self) -> String { - self.server.uri() - } - - /// Answer `GET ` with `body`. Any other route is a 404, which - /// sends `go` to the next entry of its `GOPROXY` list. wiremock asks - /// the mocks of the lowest `priority` number first. - fn serve(&self, priority: u8, route: &str, body: &[u8]) { - let mock = Mock::given(method("GET")) - .and(path(route)) - .respond_with(ResponseTemplate::new(200).set_body_bytes(body)) - .with_priority(priority); - self.runtime.block_on(mock.mount(&self.server)); - } - - /// Serve `module` in the GOPROXY layout under `/`. - fn serve_module(&self, host: &str, module: &Module) { - let version = &module.version; - let info = format!(r#"{{"Version": "{version}", "Time": "2026-01-01T00:00:00Z"}}"#); - self.serve(5, &module.route(host, "info"), info.as_bytes()); - self.serve(5, &module.route(host, "mod"), GO_MOD); - self.serve(5, &module.route(host, "zip"), &module.zip); - } - - /// The Socket API: one free patch of the upstream module, granted as - /// the module `socket` on the stand-in patch host. `go` fetches the - /// module, not Bazel's downloader, so the grant may name the loopback - /// origin. - fn grant(&self, uuid: &str, socket: &Module) { - let origin = self.origin(); - let url = format!("{origin}{}", socket.route(PATCH_HOST, "zip")); - let reference = json!({ - "status": "granted", - "url": url, - "purl": PURL, - "artifacts": [{ "kind": "tarball", "url": url, "integrity": {} }], - "registryOverride": { - "kind": "goproxy", - "indexUrl": format!("{origin}/{PATCH_HOST}"), - "identifiers": { - "name": MODULE, - "version": VERSION, - "goModulePath": socket.path, - "goModuleVersion": socket.version, - "goZipDirhashH1": socket.zip_h1, - "goModH1": socket.mod_h1, - }, +/// The Socket API on `standin`: one free patch of the upstream module, +/// granted as the module `socket` on the stand-in patch host. `go` fetches +/// the module, not Bazel's downloader, so the grant may name the loopback +/// origin. +fn grant(standin: &StandIn, uuid: &str, socket: &Module) { + let origin = standin.origin(); + let url = format!("{origin}{}", socket.route(PATCH_HOST, "zip")); + let reference = json!({ + "status": "granted", + "url": url, + "purl": PURL, + "artifacts": [{ "kind": "tarball", "url": url, "integrity": {} }], + "registryOverride": { + "kind": "goproxy", + "indexUrl": format!("{origin}/{PATCH_HOST}"), + "identifiers": { + "name": MODULE, + "version": VERSION, + "goModulePath": socket.path, + "goModuleVersion": socket.version, + "goZipDirhashH1": socket.zip_h1, + "goModH1": socket.mod_h1, }, - }); - let patch = Patch { - purl: PURL.to_string(), - uuid: uuid.to_string(), - reference, - }; - self.runtime.block_on(mount_api(&self.server, &[patch])); - } - - /// How many requests for `route` the server has received. - fn asked(&self, route: &str) -> usize { - let received = self.runtime.block_on(self.server.received_requests()); - let received = received.expect("wiremock records requests"); - received.iter().filter(|r| r.url.path() == route).count() - } + }, + }); + let patch = Patch { + purl: PURL.to_string(), + uuid: uuid.to_string(), + reference, + }; + standin.api(&[patch]); } /// What a `go` that Bazel starts reads in place of the caller's settings, @@ -286,8 +242,8 @@ fn hosted_go_deps() { let standin = StandIn::start(); let origin = standin.origin(); - standin.serve_module(PROXY, &upstream); - standin.serve_module(PATCH_HOST, &socket); + serve_module(&standin, PROXY, &upstream); + serve_module(&standin, PATCH_HOST, &socket); // What `rollback` reads to restore the upstream `go.sum` lines, in the // form of the checksum database: a record number, the two lines, a // tree head. @@ -297,7 +253,7 @@ fn hosted_go_deps() { ); let lookup_route = format!("/{SUMDB}/lookup/{MODULE}@{VERSION}"); standin.serve(5, &lookup_route, lookup.as_bytes()); - standin.grant(&uuid, &socket); + grant(&standin, &uuid, &socket); let sumdb = format!("{origin}/{SUMDB}"); let socket_patch = |cell: &Cell, args: &[&str]| { cell.socket_patch(&origin, args, &[("SOCKET_GOSUMDB_URL", sumdb.as_str())]) @@ -310,33 +266,19 @@ fn hosted_go_deps() { ); // (a) The unpatched build, with the upstream module from the stand-in. - cell.builds(UNPATCHED); + let before = cell.builds_unpatched(UNPATCHED); let upstream_fetches = standin.asked(&upstream_zip); assert!( upstream_fetches > 0, "{test}: the build did not fetch the upstream module from the stand-in" ); - let before = cell.tree(); - assert!( - before.contains_key("MODULE.bazel.lock"), - "{test}: Bazel wrote no lock" - ); - // (b) The one command of hosted mode. + // (b) The one command of hosted mode. It rewrites those two files and + // no other: the one `replace` line, and the two sums of the Socket + // module in place of the upstream module's. let scan = socket_patch(&cell, &["scan", "--mode", "hosted", "--yes"]); let said = &scan.dump; - assert_eq!(scan.code, 0, "{test}: {said}"); - assert_eq!(scan.envelope["status"], "success", "{test}: {said}"); - assert_eq!(scan.envelope["summary"]["applied"], 1, "{test}: {said}"); - let event = event_for(&scan.envelope, PURL); - assert_eq!(event["action"], "applied", "{test}: {said}"); - assert_eq!(event["uuid"], uuid.as_str(), "{test}: {said}"); - let rewritten = &scan.envelope["redirect"]["rewrittenFiles"]; - assert_eq!(*rewritten, json!(["go.mod", "go.sum"]), "{test}: {said}"); - // Those two files and no other: the one `replace` line, and the two - // sums of the Socket module in place of the upstream module's. - let pinned = cell.tree(); - assert_eq!(changed(&before, &pinned), ["go.mod", "go.sum"], "{test}"); + let pinned = cell.pinned_by(&scan, PURL, &uuid, &before, &["go.mod", "go.sum"]); let text = |rel: &str| String::from_utf8(pinned[rel].clone()).expect("a text file"); let (go_mod, go_sum) = (text("go.mod"), text("go.sum")); let replace = format!("replace {MODULE} {VERSION} => {socket_path} {SOCKET_VERSION}"); @@ -347,8 +289,10 @@ fn hosted_go_deps() { assert_eq!(sums, socket.go_sum(), "{test}: go.sum:\n{go_sum}"); assert_eq!(standin.asked(&socket_zip), 0, "{test}: {said}"); - // (c) The same build, no step in between: patched, from the patch host. - cell.builds(&patched); + // (c) The same build, no step in between: patched, from the patch + // host. Bazel changed no file, its own lock included, and that lock + // still stands when Bazel may not update it. + cell.builds_patched(&pinned, &patched); assert!( standin.asked(&socket_zip) > 0, "{test}: the patched build did not fetch the Socket module from the stand-in" @@ -358,20 +302,10 @@ fn hosted_go_deps() { upstream_fetches, "{test}: the patched build fetched the upstream module" ); - // Bazel changed no file, its own lock included, and that lock still - // stands when Bazel may not update it. - assert_eq!(changed(&pinned, &cell.tree()), NOTHING, "{test}"); - let frozen = cell.frozen_build(&[CONSUMER]); - assert!(frozen.ok(), "{test}: {}", frozen.dump()); // (d) The undo: every file is the one of before, and so is the build. let undo = socket_patch(&cell, &["rollback", "--yes"]); - let said = &undo.dump; - assert_eq!(undo.code, 0, "{test}: {said}"); - assert_eq!(undo.envelope["summary"]["rolledBack"], 1, "{test}: {said}"); - assert_eq!(changed(&before, &cell.tree()), NOTHING, "{test}: {said}"); - cell.builds(UNPATCHED); - assert_eq!(changed(&before, &cell.tree()), NOTHING, "{test}"); + cell.undone(&undo, "rolledBack", &before, UNPATCHED); // (e) A second machine with the rewritten files, and a patch host that // serves other bytes at the pinned URL: still a module zip `go` diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs index eca06895f..8736f4aa6 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs @@ -1,8 +1,9 @@ //! What every real-Bazel test shares: the variables that configure a run, //! the Bazel a run is given and the fixture files that go with its release, //! the scratch root that holds everything one Bazel writes, the Bazel -//! command line, and the `socket-patch` child that runs in the workspace -//! copy. +//! command line and the environment of its children, the `socket-patch` +//! child that runs in the workspace copy, the loopback server that stands +//! in for the hosts of a flow, and the steps every flow is made of. use std::collections::BTreeMap; use std::ffi::{OsStr, OsString}; @@ -10,10 +11,14 @@ use std::ops::Range; use std::path::{Path, PathBuf}; use std::process::{Command, Output}; -use serde_json::Value; +use serde_json::{json, Value}; +use wiremock::matchers::{method, path}; +use wiremock::{Mock, MockServer, ResponseTemplate}; +use crate::envelope::event_for; use crate::fs_tree; -use crate::hosted_memory_common::{hermetic, read_tree, ORG}; +use crate::hosted_memory_common::hermetic::{self, Extra}; +use crate::hosted_memory_common::{mount_api, read_tree, Patch, ORG}; /// The Bazel binary or launcher to run, by absolute path: every child runs /// in the workspace copy. Unset: `bazel` on `PATH`. A test runs on the @@ -33,25 +38,65 @@ pub const REPO_CACHE: &str = "SOCKET_PATCH_BAZEL_E2E_REPO_CACHE"; /// release), by absolute path. Unset: the binary cargo built for this run. pub const SOCKET_BIN: &str = "SOCKET_PATCH_BAZEL_E2E_SOCKET_BIN"; -/// Removed from every Bazel child. `TEST_TMPDIR` puts a Bazel server in -/// test mode, where a failing module extension on 7.x is a crash (exit 37) -/// instead of an error. An `XDG_*_HOME` directory would move what a tool -/// keeps out of the scratch home: a cache, or on Linux the telemetry -/// counters of a Go SDK command that a ruleset starts, which Go puts under -/// `XDG_CONFIG_HOME` and no variable turns off. Without them a tool falls -/// back to a directory under `HOME`. The CI markers change what some -/// rulesets do, and a test must do the same on a runner and on a -/// developer's machine. +/// Removed from every Bazel child, so that a test does the same in every +/// shell. What a child keeps reaches the repository rules, the tools they +/// start and the binary `bazel run` starts. const REMOVED: &[&str] = &[ + // Bazel: `TEST_TMPDIR` puts a server in test mode, where a failing + // module extension on 7.x is a crash (exit 37) instead of an error. "TEST_TMPDIR", + // Any tool: an `XDG_*_HOME` directory would move what it keeps out of + // the scratch home: a cache, or on Linux the telemetry counters of a Go + // SDK command that a ruleset starts, which Go puts under + // `XDG_CONFIG_HOME` and no variable turns off. Without them a tool + // falls back to a directory under `HOME`. "XDG_CACHE_HOME", "XDG_CONFIG_HOME", "XDG_DATA_HOME", "XDG_STATE_HOME", + // Rulesets: the CI markers change what some of them do, and a test + // must do the same on a runner and on a developer's machine. "CI", "GITHUB_ACTIONS", + // Ruby and RubyGems: where gems are installed and looked up. rvm, + // chruby and direnv export them for the Ruby of the shell, which is + // not the Ruby a ruleset downloads. + "GEM_HOME", + "GEM_PATH", + // Node: options and a module search path for every `node`, the + // consumer that `bazel run` starts included. + "NODE_OPTIONS", + "NODE_PATH", + // Python: another standard library, more module directories, and a + // file to run at start, for every interpreter. + "PYTHONHOME", + "PYTHONPATH", + "PYTHONSTARTUP", ]; +/// Removed from every Bazel child under every name that starts so. +const REMOVED_PREFIXES: &[&str] = &[ + // A bazelisk: what it does around a command (`BAZELISK_CLEAN`) and + // where it gets a release from. `BAZELISK_HOME` is set again, to + // [`launcher_cache`]. + "BAZELISK_", + // pip: any option of its command line (an index, a proxy, a cache). + "PIP_", + // uv: the same. + "UV_", +]; + +/// The scrubs of `hermetic` that every Bazel child gets, for the tools a +/// ruleset starts: `go` (its module cache, flags and `go env -w` file), +/// Bundler and Ruby (`BUNDLE_*`, and the `RUBYOPT` and `RUBYLIB` of a shell +/// under `bundle exec`), npm and pnpm (`npm_config_*`, which outranks an +/// rc file, and `PNPM_*`) and Python (the `VIRTUAL_ENV` of an activated +/// environment). +const EXTRAS: &[Extra] = &[Extra::Go, Extra::Bundle, Extra::Pnpm, Extra::Venv]; + +/// The hosts a proxy is never asked for: a stand-in listens on loopback. +const LOOPBACK: [&str; 2] = ["127.0.0.1", "localhost"]; + /// The label every fixture builds: a binary that prints one `OUT: ...` line /// from the package under test. pub const CONSUMER: &str = "//:consumer"; @@ -63,6 +108,9 @@ const CLOSED_PORT: &str = "http://127.0.0.1:9"; /// The files of a workspace copy: `/`-separated relative path to bytes. pub type Tree = BTreeMap>; +/// What [`changed`] gives for two equal trees. +pub const NOTHING: [&str; 0] = []; + /// `name`'s value when it is set and non-empty. fn var(name: &str) -> Option { std::env::var_os(name).filter(|value| !value.is_empty()) @@ -114,6 +162,9 @@ pub struct Fixture { pub name: &'static str, /// The directory under `tests/fixtures/bazel/`. pub dir: &'static str, + /// The file under `.github/workflows/` that says on which Bazel + /// releases the ecosystem's tests run. + pub workflow: &'static str, /// The one place that says which release builds which era. pub eras: &'static [Era], } @@ -128,7 +179,9 @@ impl Fixture { /// Without a Bazel: the directories of the fixture are `common` and the /// eras of the table, and each era holds a `MODULE.bazel`, which - /// `common` does not. + /// `common` does not. The workflow names this fixture, and for every + /// Bazel release it runs the era the table gives that release: a leg's + /// repository cache is the one of its era. pub fn check(&self) { let root = fixture_path(self.dir); let mut found: Vec = std::fs::read_dir(&root) @@ -148,6 +201,25 @@ impl Fixture { let module = root.join(era).join("MODULE.bazel"); assert!(module.is_file(), "{}: no {}", self.name, module.display()); } + + let workflow = Path::new(env!("CARGO_MANIFEST_DIR")) + .join("../../.github/workflows") + .join(self.workflow); + let said = format!("{}: {}", self.name, workflow.display()); + let text = std::fs::read_to_string(&workflow).unwrap_or_else(|e| panic!("{said}: {e}")); + let fixtures = format!("fixtures: {}", self.dir); + let named = text.lines().any(|line| line.trim() == fixtures); + assert!(named, "{said}: no `{fixtures}`"); + let leg = regex::Regex::new(r#"\{"bazel": "([^"]+)", "era": "([^"]+)"\}"#).expect("a leg"); + let legs: Vec<[&str; 2]> = leg.captures_iter(&text).map(|c| c.extract().1).collect(); + // Every leg is written the one way the pattern reads. + assert!( + !legs.is_empty() && legs.len() == text.matches("\"bazel\":").count(), + "{said}: a leg is not written as {{\"bazel\": \"9.3.2\", \"era\": \"e9\"}}" + ); + for [release, era] in legs { + assert_eq!(self.era(release), Some(era), "{said}: Bazel {release}"); + } } } @@ -198,6 +270,9 @@ struct Scratch { /// `--repo_contents_cache` on Bazel 9, where that cache is on by /// default. contents_cache: PathBuf, + /// The downloader config, which only [`Cell::route_downloads`] writes. + /// Beside the workspace copy, so it is no file of the checkout. + downloads: PathBuf, /// Removed on drop, after [`Scratch::drop`] made it removable. dir: tempfile::TempDir, } @@ -226,6 +301,7 @@ impl Scratch { home: made(root.join("home")), repository_cache: made(repository_cache.unwrap_or_else(|| root.join("cache"))), contents_cache: made(root.join("contents")), + downloads: root.join("downloader.cfg"), dir, } } @@ -277,6 +353,29 @@ fn launcher_cache() -> Option { Some(caches.join("bazelisk")) } +/// What every Bazel child gets as `no_proxy` and as `NO_PROXY`: the entries +/// the caller has under either name, and [`LOOPBACK`]. +/// +/// The proxy settings of the machine stay as they are: where a proxy is the +/// way to the Bazel registry and the ruleset archives, a build without it +/// fetches nothing. Loopback must not go through it, because a flow sends +/// Bazel's downloader and the tools of a ruleset to a stand-in there, which +/// a proxy on another machine cannot reach. A tool may read either name +/// first (Bazel's downloader reads `no_proxy`, and `NO_PROXY` only when +/// that one is empty), so both get the one list. +fn no_proxy() -> String { + let own = ["no_proxy", "NO_PROXY"].map(|name| var(name).unwrap_or_default()); + let own = own.each_ref().map(|list| list.to_string_lossy()); + let named = own.iter().flat_map(|list| list.split(',')).map(str::trim); + let mut hosts: Vec<&str> = Vec::new(); + for host in named.chain(LOOPBACK) { + if !host.is_empty() && !hosts.contains(&host) { + hosts.push(host); + } + } + hosts.join(",") +} + /// `program` in the workspace copy, with the environment of every Bazel /// child and no arguments. Built on `hermetic::command`, the one way this /// crate's tests spawn a child; what is set here lands after its scrub. @@ -285,20 +384,23 @@ fn child(program: &OsStr, scratch: &Scratch) -> Command { for name in REMOVED { cmd.env_remove(name); } - // A bazelisk's other variables change what it does around a command - // (`BAZELISK_CLEAN`) and where it gets a release from. for (name, _) in std::env::vars_os() { - if name.to_string_lossy().starts_with("BAZELISK_") { - cmd.env_remove(name); + let text = name.to_string_lossy(); + if REMOVED_PREFIXES + .iter() + .any(|prefix| text.starts_with(prefix)) + { + cmd.env_remove(&name); } } if let Some(cache) = launcher_cache() { cmd.env("BAZELISK_HOME", cache); } - // The `go` a ruleset runs must not read the developer's module cache, - // flags or `go env -w` file. - hermetic::scrub_extra(&mut cmd, &[hermetic::Extra::Go]); - cmd.env("HOME", &scratch.home) + hermetic::scrub_extra(&mut cmd, EXTRAS); + let no_proxy = no_proxy(); + cmd.env("no_proxy", &no_proxy) + .env("NO_PROXY", &no_proxy) + .env("HOME", &scratch.home) .env("TMPDIR", &scratch.tmp) .current_dir(&scratch.workspace); cmd @@ -413,6 +515,91 @@ pub struct Ran { pub dump: String, } +/// The loopback server that stands in for the hosts of a flow: the Socket +/// API at its own paths ([`StandIn::api`]), and whatever else a flow serves +/// under a route prefix per host (the registry of the package under test, +/// the patch host). +pub struct StandIn { + // Dropped before the runtime it runs on. + server: MockServer, + runtime: tokio::runtime::Runtime, +} + +impl StandIn { + pub fn start() -> StandIn { + let runtime = tokio::runtime::Builder::new_multi_thread() + .worker_threads(2) + .enable_all() + .build() + .expect("a runtime for the stand-in"); + let server = runtime.block_on(MockServer::start()); + StandIn { server, runtime } + } + + /// `http://127.0.0.1:`. + pub fn origin(&self) -> String { + self.server.uri() + } + + /// Answer `GET ` with `body`. A route nobody serves is a 404. + /// wiremock asks the mocks of the lowest `priority` number first, so a + /// number below the one of an earlier answer replaces that answer. The + /// routes of [`StandIn::api`] have 5. + pub fn serve(&self, priority: u8, route: &str, body: &[u8]) { + let mock = Mock::given(method("GET")) + .and(path(route)) + .respond_with(ResponseTemplate::new(200).set_body_bytes(body)) + .with_priority(priority); + self.runtime.block_on(mock.mount(&self.server)); + } + + /// The Socket API for `patches`: what a `scan` asks about the packages + /// of a checkout, with each patch free for every caller and granted as + /// its `reference` says. + pub fn api(&self, patches: &[Patch]) { + self.runtime.block_on(mount_api(&self.server, patches)); + } + + /// The path of every request the server has received, in order. + pub fn requests(&self) -> Vec { + let received = self.runtime.block_on(self.server.received_requests()); + let received = received.expect("wiremock records requests"); + received.iter().map(|r| r.url.path().to_string()).collect() + } + + /// How many requests for `route` the server has received. + pub fn asked(&self, route: &str) -> usize { + self.requests().iter().filter(|path| *path == route).count() + } +} + +/// The line of a downloader config ([`Cell::route_downloads`]) that sends +/// every URL starting with `prefix`, a host and the start of a path, to the +/// stand-in at `origin`, which plays a host under the route prefix of its +/// name: `https://` is fetched from `/`. +/// Bazel reads the pattern as a regular expression over the whole URL after +/// its scheme, so the prefix is quoted: a `.` or a `+` in a file name or a +/// version stands for itself. +pub fn rewrite(prefix: &str, origin: &str) -> String { + // `\E` would end the quote, and a config line is split at white space. + assert!( + !prefix.contains(|c: char| c == '\\' || c.is_whitespace()), + "{prefix:?} is no URL" + ); + format!("rewrite (\\Q{prefix}\\E.*) {origin}/$1") +} + +/// The option that names a downloader config on the release `numbers`. +/// Bazel 8.2.0 gave it this name; before, it is the experimental option of +/// the same meaning, which 8.2.0 and later still take, with a warning. +fn downloader_option(numbers: Release) -> &'static str { + if numbers < (8, 2, 0) { + "--experimental_downloader_config=" + } else { + "--downloader_config=" + } +} + /// One test's checkout of a fixture workspace, with a Bazel server and an /// output base of its own. pub struct Cell { @@ -427,6 +614,8 @@ pub struct Cell { numbers: Release, /// [`SOCKET_BIN`]. socket_bin: Option, + /// The rules of the downloader config, once a flow has asked for one. + downloads: Option>, // Dropped after `Cell::drop` has stopped the server. scratch: Scratch, } @@ -482,6 +671,7 @@ impl Cell { bazel, numbers, socket_bin, + downloads: None, scratch, }) } @@ -491,30 +681,54 @@ impl Cell { &self.bazel.release } + /// That release as three numbers, for what one release does and + /// another does not: `cell.numbers() >= (8, 5, 0)`. + pub fn numbers(&self) -> Release { + self.numbers + } + /// A second machine with the same checkout: the files of this one, in /// a workspace copy with an output base and a contents cache of its /// own, so nothing a build here fetched or built is there. Only the - /// repository cache is shared. This cell's server is stopped first, so - /// one runs at a time. + /// repository cache is shared, and the settings of this cell go along: + /// its `repo_env` and its downloader config. This cell's server is + /// stopped first, so one runs at a time. pub fn twin(&self) -> Cell { if let Err(why) = self.shutdown() { panic!("{}: {why}", self.test); } - let twin = Cell { + let mut twin = Cell { test: self.test.clone(), repo_env: self.repo_env.clone(), bazel: self.bazel.clone(), numbers: self.numbers, socket_bin: self.socket_bin.clone(), + downloads: None, scratch: Scratch::new(Some(self.scratch.repository_cache.clone())), }; for (rel, bytes) in self.tree() { twin.write(&rel, bytes); } std::fs::create_dir(twin.scratch.workspace.join(".git")).expect("create .git"); + if let Some(rules) = &self.downloads { + twin.route_downloads(rules); + } twin } + /// Send the downloads of Bazel's own downloader where `rules` say: the + /// lines of a downloader config ([`rewrite`] makes the usual one), which + /// every Bazel command of this cell names from now on, by the option of + /// its release. A flow asks for it before its first build, as a machine + /// behind a mirror has the option in an rc file, so no step of the flow + /// changes a flag. A second call replaces the rules of the first. + pub fn route_downloads(&mut self, rules: &[String]) { + let file = &self.scratch.downloads; + std::fs::write(file, rules.join("\n") + "\n") + .unwrap_or_else(|e| panic!("write {}: {e}", file.display())); + self.downloads = Some(rules.to_vec()); + } + /// Write a file of the workspace copy. pub fn write(&self, rel: &str, bytes: impl AsRef<[u8]>) { let path = self.scratch.workspace.join(rel); @@ -535,11 +749,17 @@ impl Cell { /// `bazel build --lockfile_mode=error `: passes only when /// `MODULE.bazel.lock` already holds what the build resolves. - pub fn frozen_build(&self, targets: &[&str]) -> Run { + fn frozen_build(&self, targets: &[&str]) -> Run { self.bazel_with("error", "build", targets) } fn bazel_with(&self, lockfile_mode: &str, command: &str, args: &[&str]) -> Run { + let cmd = self.command(lockfile_mode, command, args); + Run::of(cmd).unwrap_or_else(|why| panic!("{}: {why}", self.test)) + } + + /// `bazel `, not started yet. + fn command(&self, lockfile_mode: &str, command: &str, args: &[&str]) -> Command { let scratch = &self.scratch; let mut cmd = self.bazel.command(scratch, command); cmd.envs(self.repo_env.iter().map(|(name, value)| (name, value))); @@ -557,7 +777,7 @@ impl Cell { } // Stated, so no test depends on a release's default. joined("--lockfile_mode=", lockfile_mode.as_ref()); - if self.numbers >= (9, 0, 0) { + if self.numbers() >= (9, 0, 0) { // Bazel 9 keeps the repositories it fetched in a contents cache, // by default under the repository cache, and hands them to the // next workspace copy without a fetch. The cache stays on, as a @@ -566,9 +786,13 @@ impl Cell { // off by default, Bazel 7 has none. joined("--repo_contents_cache=", scratch.contents_cache.as_os_str()); } + if self.downloads.is_some() { + let option = downloader_option(self.numbers()); + joined(option, scratch.downloads.as_os_str()); + } cmd.args(["--color=no", "--curses=no"]); cmd.args(args); - Run::of(cmd).unwrap_or_else(|why| panic!("{}: {why}", self.test)) + cmd } /// The build a user runs, and what it built: `bazel build //:consumer` @@ -585,6 +809,67 @@ impl Cell { ); } + /// Step (a) of a flow: the build prints `marker`, and Bazel wrote its + /// lock. Gives the files of the checkout, that lock among them. + pub fn builds_unpatched(&self, marker: &str) -> Tree { + self.builds(marker); + let before = self.tree(); + let locked = before.contains_key("MODULE.bazel.lock"); + assert!(locked, "{}: Bazel wrote no lock", self.test); + before + } + + /// Step (b) of a hosted flow, for its finished `scan --mode hosted + /// --yes`: the command applied the one patch `uuid` of `purl`, it names + /// `files` as what it rewrote, and those are the files of the checkout + /// that differ from `before`, and no other. Gives the files of the + /// checkout as the command left them. + pub fn pinned_by( + &self, + scan: &Ran, + purl: &str, + uuid: &str, + before: &Tree, + files: &[&str], + ) -> Tree { + let (test, said) = (&self.test, &scan.dump); + assert_eq!(scan.code, 0, "{test}: {said}"); + assert_eq!(scan.envelope["status"], "success", "{test}: {said}"); + assert_eq!(scan.envelope["summary"]["applied"], 1, "{test}: {said}"); + let event = event_for(&scan.envelope, purl); + assert_eq!(event["action"], "applied", "{test}: {said}"); + assert_eq!(event["uuid"], uuid, "{test}: {said}"); + let rewritten = &scan.envelope["redirect"]["rewrittenFiles"]; + assert_eq!(*rewritten, json!(files), "{test}: {said}"); + let pinned = self.tree(); + assert_eq!(changed(before, &pinned), files, "{test}: {said}"); + pinned + } + + /// Step (c) of a flow whose build writes no file of the checkout: the + /// build prints `marker`, Bazel changed no file of `pinned`, its own + /// lock included, and that lock still stands when Bazel may not update + /// it. + pub fn builds_patched(&self, pinned: &Tree, marker: &str) { + self.builds(marker); + assert_eq!(changed(pinned, &self.tree()), NOTHING, "{}", self.test); + let frozen = self.frozen_build(&[CONSUMER]); + assert!(frozen.ok(), "{}: {}", self.test, frozen.dump()); + } + + /// Step (d) of such a flow, for its finished undo (`rollback --yes`, or + /// `vendor --revert`): the command counts one package under `counted` + /// of its summary, every file of the checkout is the one of `before`, + /// and so is the build, which prints `marker` and changes no file. + pub fn undone(&self, undo: &Ran, counted: &str, before: &Tree, marker: &str) { + let (test, said) = (&self.test, &undo.dump); + assert_eq!(undo.code, 0, "{test}: {said}"); + assert_eq!(undo.envelope["summary"][counted], 1, "{test}: {said}"); + assert_eq!(changed(before, &self.tree()), NOTHING, "{test}: {said}"); + self.builds(marker); + assert_eq!(changed(before, &self.tree()), NOTHING, "{test}"); + } + /// `socket-patch --json` in the workspace copy, as a user in /// that directory runs it, against the Socket API at `api`. Its home /// and temp directory are the cell's, and every proxy is a closed port, @@ -605,8 +890,8 @@ impl Cell { cmd.env(proxy, CLOSED_PORT); cmd.env(proxy.to_ascii_lowercase(), CLOSED_PORT); } - cmd.env("NO_PROXY", "127.0.0.1,localhost"); - cmd.env("no_proxy", "127.0.0.1,localhost"); + cmd.env("NO_PROXY", LOOPBACK.join(",")); + cmd.env("no_proxy", LOOPBACK.join(",")); cmd.envs(env.iter().copied()); let line = format!("{cmd:?}"); let out = cmd.output().unwrap_or_else(|e| panic!("spawn {line}: {e}")); @@ -666,6 +951,7 @@ fn a_release_builds_the_era_whose_range_holds_it() { let fixture = Fixture { name: "a table", dir: "none", + workflow: "none", eras: &[ ((7, 0, 0)..(8, 0, 0), "old"), ((8, 0, 0)..(10, 0, 0), "new"), @@ -683,3 +969,147 @@ fn a_release_builds_the_era_whose_range_holds_it() { assert_eq!(fixture.era(release), era, "{release}"); } } + +/// A cell no Bazel stands behind, for what is decided without one: the +/// command line of the release `release`. +fn cell_without_bazel(release: &str) -> Cell { + Cell { + test: "a test".to_string(), + repo_env: Vec::new(), + bazel: Bazel { + program: "bazel".into(), + release: release.to_string(), + }, + numbers: numbers(release).expect("a release"), + socket_bin: None, + downloads: None, + scratch: Scratch::new(None), + } +} + +/// Bazel 8.1.1 has the experimental option alone and 8.2.0 has both names +/// (`RepositoryOptions.java` at Bazel's two tags), so 8.2.0 is where the +/// name changes. A cell names its config on every command once a flow has +/// asked for one, never before, and its twin names a config of its own. +#[test] +fn a_downloader_config_is_named_by_the_option_of_the_release() { + let rule = rewrite("registry.example/a+b/-/a+b-1.0.0.tgz", "http://127.0.0.1:1"); + assert_eq!( + rule, + r"rewrite (\Qregistry.example/a+b/-/a+b-1.0.0.tgz\E.*) http://127.0.0.1:1/$1" + ); + let rules = [rule, "block patch.example".to_string()]; + // The arguments of a build that name a downloader config. + let named = |cell: &Cell| -> Vec { + let build = cell.command("update", "build", &[CONSUMER]); + let args = build + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()); + args.filter(|arg| arg.contains("downloader_config")) + .collect() + }; + for (release, option) in [ + ("7.7.2", "--experimental_downloader_config="), + ("8.1.1", "--experimental_downloader_config="), + ("8.2.0", "--downloader_config="), + ("8.8.2", "--downloader_config="), + ("9.3.2-homebrew", "--downloader_config="), + ] { + let mut cell = cell_without_bazel(release); + assert_eq!(named(&cell), NOTHING, "{release}"); + cell.route_downloads(&rules); + let twin = cell.twin(); + for cell in [&cell, &twin] { + let file = &cell.scratch.downloads; + assert_eq!(named(cell), [format!("{option}{}", file.display())]); + let config = std::fs::read_to_string(file).expect("the downloader config"); + assert_eq!(config, format!("{}\nblock patch.example\n", rules[0])); + assert!(!file.starts_with(&cell.scratch.workspace), "{release}"); + } + assert_ne!(cell.scratch.downloads, twin.scratch.downloads, "{release}"); + } +} + +/// Set in the copy of this test binary that the next test starts, which +/// then plays the shell of a developer. +const PLAYS_THE_SHELL: &str = "SOCKET_PATCH_BAZEL_E2E_PLAYS_THE_SHELL"; + +/// What such a shell has set that must not reach a Bazel child: every name +/// of [`REMOVED`], one name under each of [`REMOVED_PREFIXES`], and what +/// each scrub of [`EXTRAS`] removes. +fn settings_of_a_shell() -> Vec { + let scrubbed = [ + // `Extra::Go`. + "GOFLAGS", + "GOWORK", + "GOPATH", + "GOMODCACHE", + // `Extra::Bundle`. + "BUNDLE_GEMFILE", + "RUBYOPT", + "RUBYLIB", + // `Extra::Pnpm`. + "npm_config_registry", + "NPM_CONFIG_USERCONFIG", + "PNPM_HOME", + // `Extra::Venv`. + "VIRTUAL_ENV", + ]; + let named = REMOVED.iter().chain(&scrubbed).map(|name| name.to_string()); + let prefixed = REMOVED_PREFIXES + .iter() + .map(|prefix| format!("{prefix}SETTING")); + named.chain(prefixed).collect() +} + +/// A child of [`child`] sees none of the tool settings of the process that +/// spawns it. It keeps that process's proxy, and gets loopback beside the +/// hosts the process already reaches without one, under both names of the +/// list. The environment is one per process and the tests of this binary +/// share theirs, so the test starts the binary again with the settings, +/// and that copy looks at a child of its own. +#[test] +fn a_bazel_child_sees_no_tool_setting_of_the_shell() { + const PROXY: &str = "http://proxy.example:3128"; + if var(PLAYS_THE_SHELL).is_none() { + let mut copy = Command::new(std::env::current_exe().expect("the test binary")); + let this = "harness::a_bazel_child_sees_no_tool_setting_of_the_shell"; + copy.args(["--exact", this, "--nocapture"]) + .env(PLAYS_THE_SHELL, "1") + .envs( + settings_of_a_shell() + .iter() + .map(|name| (name, "of-the-shell")), + ) + .env("http_proxy", PROXY) + .env("HTTPS_PROXY", PROXY) + .env("no_proxy", "corp.example, 127.0.0.1") + .env("NO_PROXY", "other.example"); + let out = copy.output().expect("start the test binary again"); + let said = String::from_utf8_lossy(&out.stdout); + assert!( + out.status.success() && said.contains("test result: ok. 1 passed"), + "{said}{}", + String::from_utf8_lossy(&out.stderr) + ); + return; + } + let scratch = Scratch::new(None); + let out = child(OsStr::new("env"), &scratch).output().expect("env"); + let printed = String::from_utf8_lossy(&out.stdout); + let seen: BTreeMap<&str, &str> = printed.lines().filter_map(|l| l.split_once('=')).collect(); + for name in settings_of_a_shell() { + assert!(std::env::var_os(&name).is_some(), "{name} is not set here"); + assert!(!seen.contains_key(name.as_str()), "a child sees {name}"); + } + assert_eq!(seen.get("GOENV"), Some(&"off")); + assert_eq!( + seen.get("HOME").map(Path::new), + Some(scratch.home.as_path()) + ); + assert_eq!(seen.get("http_proxy"), Some(&PROXY)); + assert_eq!(seen.get("HTTPS_PROXY"), Some(&PROXY)); + let direct = "corp.example,127.0.0.1,other.example,localhost"; + assert_eq!(seen.get("no_proxy"), Some(&direct)); + assert_eq!(seen.get("NO_PROXY"), Some(&direct)); +} diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs index 9b9b318b7..23dd4ae28 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/main.rs @@ -11,7 +11,8 @@ //! and a host that does not answer fails them. The package under test and //! every Socket host are loopback stand-ins. //! `.github/workflows/bazel-compatibility.yml` runs them, one leg per Bazel -//! release and ecosystem. Locally: +//! release, for the ecosystem of the workflow that calls it +//! (`bazel-go.yml`). Locally: //! //! ```sh //! export "$(scripts/install-bazel.sh 9.3.2 "$HOME/.cache/socket-patch-bazel")" @@ -30,9 +31,13 @@ //! `MODULE.bazel` per era of rulesets; the fixture's table in the test code //! says which releases build which era (`harness::Fixture`). So another //! Bazel release is one line per platform in `scripts/bazel-pins.txt` and -//! one line of the workflow's matrix; another era is one `MODULE.bazel` +//! one line of the ecosystem's workflow; another era is one `MODULE.bazel` //! and one line of the table; another ecosystem is its module, its fixture -//! and one line of the workflow's matrix per release. +//! and a workflow of its own, which calls `bazel-compatibility.yml` as +//! `bazel-go.yml` does. `harness.rs` holds what the flows share: the +//! stand-in server, the steps of a flow, the downloader config for a +//! ruleset that fetches with Bazel's own downloader, and the environment +//! of every Bazel child. #[path = "../common/envelope.rs"] mod envelope; From 442bb9fef8811ab23e56681fd9a88d29252c5ea4 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Tue, 17 Nov 2026 03:56:25 -0500 Subject: [PATCH 4/5] Hold the scrubbed environment list in its test, and pick Bazel tests by exact name The test that a Bazel child sees none of a shell's tool settings read its names from the two lists it was meant to guard, so a name dropped from either still passed. The names are now written out in the test, and the test also fails when a list gains a name it does not cover. Dropping GEM_HOME, the PIP_ prefix or the virtualenv scrub each fails it. The run step chose an ecosystem's tests with a libtest filter, which is a substring: `go::` is also inside `cargo::`. It now lists the ignored tests, keeps the names that start with the module, and runs those by exact name; it still fails when the module has no test. The header of bazel-pins.txt names the callers as the place that says which releases run. Co-Authored-By: Claude Fable 5.2 --- .github/workflows/bazel-compatibility.yml | 18 ++++---- .../tests/e2e_bazel_build/harness.rs | 41 +++++++++++++++---- scripts/bazel-pins.txt | 4 +- 3 files changed, 45 insertions(+), 18 deletions(-) diff --git a/.github/workflows/bazel-compatibility.yml b/.github/workflows/bazel-compatibility.yml index 9a1a7079b..61cc9b960 100644 --- a/.github/workflows/bazel-compatibility.yml +++ b/.github/workflows/bazel-compatibility.yml @@ -170,15 +170,19 @@ jobs: # The ecosystem's module holds one `#[ignore]` test per flow, and a # test takes its release from the Bazel installed above, so the # filter is the module's name and nothing is derived from the - # release. The first command lists the tests of the filter and fails - # when there is none: a module name that matches nothing would run - # no test and pass. Two tests at a time: each runs a Bazel server of - # its own. + # release. The tests are chosen by the module's name as a prefix and + # run by their exact names: a test filter is a substring, and `go::` + # is also inside `cargo::`. The step fails when the module has no + # test: a name that matches nothing would run no test and pass. Two + # tests at a time: each runs a Bazel server of its own. run: | + names=$(cargo test --locked -p socket-patch-cli --test e2e_bazel_build -- \ + --ignored --list | sed -n "s/^\(${ECO}::.*\): test\$/\1/p") + test -n "$names" + echo "$names" + # shellcheck disable=SC2086 # one test name per word cargo test --locked -p socket-patch-cli --test e2e_bazel_build -- \ - --ignored "${ECO}::" --list | grep ': test$' - cargo test --locked -p socket-patch-cli --test e2e_bazel_build -- \ - --ignored "${ECO}::" --test-threads=2 --nocapture + --ignored --exact $names --test-threads=2 --nocapture - name: Save the repository cache # The first leg of an era to finish on main saves it. Another leg diff --git a/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs b/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs index 8736f4aa6..3d23f1b6d 100644 --- a/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs +++ b/crates/socket-patch-cli/tests/e2e_bazel_build/harness.rs @@ -1034,11 +1034,30 @@ fn a_downloader_config_is_named_by_the_option_of_the_release() { /// then plays the shell of a developer. const PLAYS_THE_SHELL: &str = "SOCKET_PATCH_BAZEL_E2E_PLAYS_THE_SHELL"; -/// What such a shell has set that must not reach a Bazel child: every name -/// of [`REMOVED`], one name under each of [`REMOVED_PREFIXES`], and what -/// each scrub of [`EXTRAS`] removes. +/// What such a shell has set that must not reach a Bazel child. Written +/// out, not read from [`REMOVED`] and [`REMOVED_PREFIXES`]: a name dropped +/// from either list, or from a scrub of [`EXTRAS`], fails the test. fn settings_of_a_shell() -> Vec { - let scrubbed = [ + let names = [ + // `REMOVED`. + "TEST_TMPDIR", + "XDG_CACHE_HOME", + "XDG_CONFIG_HOME", + "XDG_DATA_HOME", + "XDG_STATE_HOME", + "CI", + "GITHUB_ACTIONS", + "GEM_HOME", + "GEM_PATH", + "NODE_OPTIONS", + "NODE_PATH", + "PYTHONHOME", + "PYTHONPATH", + "PYTHONSTARTUP", + // One name under each of `REMOVED_PREFIXES`. + "BAZELISK_SETTING", + "PIP_SETTING", + "UV_SETTING", // `Extra::Go`. "GOFLAGS", "GOWORK", @@ -1055,11 +1074,15 @@ fn settings_of_a_shell() -> Vec { // `Extra::Venv`. "VIRTUAL_ENV", ]; - let named = REMOVED.iter().chain(&scrubbed).map(|name| name.to_string()); - let prefixed = REMOVED_PREFIXES - .iter() - .map(|prefix| format!("{prefix}SETTING")); - named.chain(prefixed).collect() + // The other direction: a name added to a list is added here too. + for name in REMOVED { + assert!(names.contains(name), "{name} is removed and not tested"); + } + for prefix in REMOVED_PREFIXES { + let tested = names.iter().any(|name| name.starts_with(prefix)); + assert!(tested, "{prefix}* is removed and not tested"); + } + names.iter().map(|name| name.to_string()).collect() } /// A child of [`child`] sees none of the tool settings of the process that diff --git a/scripts/bazel-pins.txt b/scripts/bazel-pins.txt index 5a770a74b..d36044084 100644 --- a/scripts/bazel-pins.txt +++ b/scripts/bazel-pins.txt @@ -4,8 +4,8 @@ # # Read by scripts/install-bazel.sh, which installs nothing that is not listed # here. Platforms use the spelling of Bazel's release files. A release that -# .github/workflows/bazel-compatibility.yml runs needs a line for each -# platform it runs on. +# a caller of .github/workflows/bazel-compatibility.yml runs (bazel-go.yml +# and its siblings) needs a line for each platform it runs on. # # Each digest is the published `.sha256` sibling of the release file: # https://releases.bazel.build//release/bazel--.sha256 From f1aa02b798186384b66fbab71570bc9f41c6f730 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Tue, 17 Nov 2026 04:19:54 -0500 Subject: [PATCH 5/5] Give the nightly Bazel run of the Go caller a concurrency group of its own A concurrency group holds one pending run. In the group of main's pushes, a second push replaces a nightly run that waits behind the first, and the nightly is the only run with the macOS leg and the only run for a main change outside the push path list. Co-Authored-By: Claude Fable 5.2 --- .github/workflows/bazel-go.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/bazel-go.yml b/.github/workflows/bazel-go.yml index d809f6510..f42bc6af0 100644 --- a/.github/workflows/bazel-go.yml +++ b/.github/workflows/bazel-go.yml @@ -80,8 +80,11 @@ on: permissions: contents: read +# The nightly run has a group of its own. A group holds one pending run, so +# in the group of main's pushes a second push would replace a nightly that +# waits behind the first, and the nightly is the only run with the macOS leg. concurrency: - group: bazel-go-${{ github.event.pull_request.number || github.ref }} + group: bazel-go-${{ github.event.pull_request.number || github.ref }}${{ github.event_name == 'schedule' && '-nightly' || '' }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} jobs: