e2e: add an option to test a branch with our own tooling.

A run hands over to the revision it tests: it re-executes that
revision's e2e-runner and builds e2e-report out of its worktree. That
is right when the revision is what we are testing, and backwards when
the tooling is. Tooling changes have nowhere to be exercised -- on a
branch they are tested against whatever else is on it, and against
main they are thrown away at the handover.

--own-tooling keeps both tools ours. The worktree is still the
branch's, so the tests and the plugins still are too. Both tools or
neither: it is the runner which records what a report reads.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>

Author: klihub

scripts/testing/nightly/README.md

@@ -49,6 +49,7 @@ environment:
 | `PACK_RESULTS` | `--pack-results` | off, see below |
 | `K8SCRI` | `--runtime` | containerd and cri-o on alternating days |
 | `FULL_BUILD` | `--full-build`, `--minimal-build` | everything once a week |
+| `OWN_TOOLING` | `--own-tooling` | off, the tested revision's tooling runs |
 | `RUN_IF_CHANGED` | `--run-if-changed` | off, test whenever asked |
 | `FORCE_AFTER` | `--force-after` | `24h`, with `--run-if-changed` |
 
@@ -132,6 +133,40 @@ read -r verdict _ < "$RESULT_ROOT/latest/status.txt"
 [ "$verdict" = PASS ] || echo "e2e run $verdict, see $RESULT_ROOT/latest/"
 ```
 
+## Testing a branch with your own tooling
+
+A run hands over to the tested revision: it adds the worktree, then re-executes
+the `e2e-runner` it finds there and builds `e2e-report` out of it, so a run is
+driven and reported on by the revision under test. That is what you want when the
+revision is what you are testing.
+
+It is the wrong way round when the *tooling* is what you are testing. Changes to
+the runner or to `e2e-report` have nowhere to be exercised: putting them on a
+branch and testing that branch tests them against whatever else is on it, and
+testing `main` throws them away at the handover. `--own-tooling` keeps them:
+
+```shell
+scripts/testing/nightly/e2e-runner --branch main --own-tooling \
+    --results /opt/e2e-test/nri-plugins/results
+```
+
+The worktree is still added at the branch, and the tests and the plugins are
+still the branch's -- only the tooling is ours. So the line above runs `main`'s
+tests against `main`'s plugins, with the runner and the reporter of the tree it
+was started from. `E2E_OWN_TOOLING=1` in the `e2e-cron-job` settings does the same
+from cron.
+
+Both tools or neither, deliberately: it is the runner which records what a report
+reads, so a run driven by one revision's runner and reported on by another's
+loses whatever the two do not agree about. Asking for it from a tree which has no
+`test/e2e/cmd/e2e-report` in it is refused before the run rather than after the
+tests, when there would be nothing left to report them with.
+
+The alternative is to keep a tooling branch rebased on `main` and test that,
+which works as long as the branch touches nothing outside the tooling --
+`git diff --name-only main..<branch>` says whether it does. `--own-tooling` is
+what saves the rebasing.
+
 ## Serving the results
 
 `e2e-report serve` is what serves the results, and the only thing which can: the

scripts/testing/nightly/e2e-cron-job

@@ -68,6 +68,11 @@ E2E_PACK_RESULTS="${E2E_PACK_RESULTS-1}"
 # E2E_OPTIONS: anything else to pass the runner, as it would be on its command
 # line, for what this script has no setting of its own
 E2E_OPTIONS="${E2E_OPTIONS:-}"
+# E2E_OWN_TOOLING: test the branch with the tooling in the tree this script is
+# installed from, rather than the branch's. The tests and the plugins are still
+# the branch's, so this is how to exercise runner or e2e-report changes against a
+# branch which knows nothing of them.
+E2E_OWN_TOOLING="${E2E_OWN_TOOLING-}"
 # E2E_UPDATE_CLONE: bring the clone up to the branch before running, so that the
 # copy of the runner which parses our options is the branch's too. Off by
 # default: it is a reset --hard of a directory which may be someone's tree.
@@ -151,6 +156,12 @@ case "$E2E_PACK_RESULTS" in
         ;;
 esac
 
+case "$E2E_OWN_TOOLING" in
+    1|true|yes|y)
+        opts+=(--own-tooling)
+        ;;
+esac
+
 if [ -n "$E2E_PROXIES" ]; then
     opts+=(--source-proxies "$E2E_PROXIES")
 fi

scripts/testing/nightly/e2e-cron-job.env

@@ -43,6 +43,13 @@
 # flags in one value are fine.
 #E2E_OPTIONS=--runtime crio --retention-days 30
 
+# Test the branch with the tooling in the tree this script is installed from,
+# rather than the branch's own: the tests and the plugins stay the branch's, the
+# runner and e2e-report are ours. This is how to exercise tooling changes against
+# a branch which knows nothing about them -- E2E_BRANCH=main with this on tests
+# main's plugins and main's tests with your runner.
+#E2E_OWN_TOOLING=1
+
 # Bring the clone up to the branch before running. The runner already re-execs
 # itself out of the worktree it makes, so the tests, the framework and the report
 # tool are the branch's whatever is checked out here -- but the copy which parses

scripts/testing/nightly/e2e-runner

@@ -45,6 +45,9 @@ K8SCRI="${K8SCRI:-}"
 # FULL_BUILD: 1 to build all binaries and images, 0 for the minimal set of
 # them, empty to build everything once a week and the minimal set otherwise
 FULL_BUILD="${FULL_BUILD:-}"
+# OWN_TOOLING: run the tests of the branch with the tooling in the tree we were
+# started from rather than the branch's own.
+OWN_TOOLING="${OWN_TOOLING:-}"
 # RUN_IF_CHANGED: only test if the remote branch has moved since we last tried,
 # which is what makes it sane to trigger us from cron every few minutes
 RUN_IF_CHANGED="${RUN_IF_CHANGED:-}"
@@ -116,6 +119,7 @@ RUN_LOCK_FD="E2E_RUN_LOCK_FD"
 # E2E_REPORT_DIR:    where it was built, ours to remove
 # VMS_LEFT_RUNNING:  test VMs we failed to destroy, if any
 # RE_EXECED:         whether we are the copy the bootstrap re-execed into
+# OWN_TREE:          the tree we were started from, for --own-tooling
 
 RESULT_DIR=""
 LOG=""
@@ -133,6 +137,11 @@ if [ -n "${!RUN_LOCK_FD}" ]; then
     RE_EXECED=1
 fi
 
+# The tree we were started from, which --own-tooling takes the tooling from:
+# three levels up from scripts/testing/nightly, resolved so that being invoked
+# through a relative path or a symlink makes no difference.
+OWN_TREE="$(realpath -m "$(dirname "$(realpath "$SELF")")/../../..")"
+
 usage() {
     local exit_code="${exit_code:-0}"
 
@@ -159,6 +168,11 @@ usage() {
     echo "  --no-pack-results         publish the results as plain files"
     echo "  --runtime <runtime>       containerd or crio, by default the two of"
     echo "                            them on alternating days"
+    echo "  --own-tooling             test the branch with the tooling in the"
+    echo "                            tree we were started from, not the"
+    echo "                            branch's: the tests and the plugins are"
+    echo "                            the branch's, the runner and e2e-report"
+    echo "                            stay ours"
     echo "  --run-if-changed          do nothing unless the remote branch has"
     echo "                            moved since we last tried to test it"
     echo "  --force-after <time>      with --run-if-changed, test anyway once this"
@@ -469,6 +483,14 @@ prepare_worktree() {
         fatal "failed to add git worktree $WORKTREE"
     fi
 
+    # Asked to be the runner of this run ourselves, so there is nothing to hand
+    # over to: the worktree is there to supply the tests and the plugins only.
+    if [ -n "$OWN_TOOLING" ]; then
+        info "testing $TEST_BRANCH with our own tooling..."
+        must-cd "$WORKTREE"
+        return 0
+    fi
+
     if [ -x "$WORKTREE/scripts/testing/nightly/e2e-runner" ]; then
         # The options below are passed on, the rest default from the
         # environment, so export those to keep them across the re-exec.
@@ -823,10 +845,18 @@ policies_tested() {
 }
 
 build_e2e_report() {
-    local dir
+    local dir tree="$WORKTREE"
+
+    # Reported on by the revision which was tested, so that a report says what
+    # that revision knew to say -- unless the tooling of this run is ours, in
+    # which case the reporting is ours too. Half of the tooling would be worse
+    # than neither: it is the runner which records what the report reads.
+    if [ -n "$OWN_TOOLING" ]; then
+        tree="$OWN_TREE"
+    fi
 
-    if [ ! -d "$WORKTREE/$E2E_REPORT" ]; then
-        info "no $E2E_REPORT in $WORKTREE, leaving the results unreported..."
+    if [ ! -d "$tree/$E2E_REPORT" ]; then
+        info "no $E2E_REPORT in $tree, leaving the results unreported..."
         return 0
     fi
 
@@ -836,8 +866,8 @@ build_e2e_report() {
     fi
     E2E_REPORT_DIR="$dir"
 
-    info "building $E2E_REPORT..."
-    must-cd "$WORKTREE"
+    info "building $E2E_REPORT out of $tree..."
+    must-cd "$tree"
     if ! go build -o "$dir/e2e-report" "./$E2E_REPORT"; then
         error "failed to build $E2E_REPORT"
         return 1
@@ -1220,6 +1250,10 @@ while [  "$#" -gt 0 ] && [ "${1#-}" != "$1" ]; do
             K8SCRI="$2"
             shift 2
             ;;
+        --own-tooling)
+            OWN_TOOLING=1
+            shift
+            ;;
         --run-if-changed)
             RUN_IF_CHANGED=1
             shift
@@ -1263,6 +1297,13 @@ case "$K8SCRI" in
         ;;
 esac
 
+# Said up front rather than when we get to it: asking for our tooling from a
+# tree which has none of it is a mistake worth hearing about before a run, not
+# after the tests have been run and there is nothing to report them with.
+if [ -n "$OWN_TOOLING" ] && [ ! -d "$OWN_TREE/$E2E_REPORT" ]; then
+    fatal "no $E2E_REPORT in $OWN_TREE, so there is no tooling of ours to use"
+fi
+
 # Packing keeps everything in one archive, and everything is what makes it
 # worth serving: keep the artifacts of every test and the coverage data of each
 # as well, unless we were told what to keep with them.