Skip to content

Commit f4c18f3

Browse files
committed
e2e: put the whole cron command behind a script.
Add e2e-cron-job, which is all a crontab line needs: it reads what a host wants from /etc/sysconfig/nri-plugins-e2e-cron-job and runs the runner with it. Anything on its own command line is passed through, for a one-off. E2E_UPDATE_CLONE fetches and resets the clone to the branch first, so that the copy of the runner which parses the options is the branch's too. Off by default: it is a reset --hard of a directory which may be someone's working tree. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
1 parent c8281c2 commit f4c18f3

3 files changed

Lines changed: 260 additions & 22 deletions

File tree

scripts/testing/nightly/README.md

Lines changed: 42 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -70,21 +70,49 @@ something else.
7070

7171
## From cron
7272

73-
Nothing here wants root, so this is the crontab of the user which owns the result
74-
root and can run docker, vagrant and qemu, `crontab -e` as that user:
73+
`e2e-cron-job` is a simple bash wrapper to help set up a configurable cron job.
74+
You can use it like this:
7575

76-
```crontab
77-
PATH=/usr/local/bin:/usr/bin:/bin
78-
79-
0 2 * * * /opt/e2e-test/nri-plugins/nri-plugins/scripts/testing/nightly/e2e-runner \
80-
--results /opt/e2e-test/nri-plugins/results \
81-
>>$HOME/e2e-runner.log 2>&1
76+
```shell
77+
env=$HOME/.config/nri-plugins-e2e-cron-job
78+
install -m 644 -D scripts/testing/nightly/e2e-cron-job.env "$env"
79+
"${EDITOR:-vi}" "$env"
80+
81+
job=$PWD/scripts/testing/nightly/e2e-cron-job
82+
{
83+
crontab -l 2>/dev/null
84+
echo 'PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin'
85+
echo "E2E_CRON_ENV=$env"
86+
echo "*/5 * * * * $job >>\$HOME/e2e-runner.log 2>&1"
87+
echo "2 2 * * * $job --force-after 0 >>\$HOME/e2e-runner.log 2>&1"
88+
echo "00 00 * * 0 docker image prune -a -f --filter 'until=168h'"
89+
} | crontab -
8290
```
8391

92+
That sets up cron jobs to
93+
- poll the repo and kick off a test run if something was merged
94+
- kick off unconditional nightly test runs
95+
- clean up old docker images
96+
8497
A drop-in in `/etc/cron.d` works as well, and takes the user to run as in a field
8598
of its own after the five of the schedule. Name the user there: every example of
8699
the format says `root`, and this needs none of it.
87100

101+
### The one thing to watch
102+
103+
The copy of `e2e-runner` which parses the options is whichever one you invoke --
104+
the clone's, when cron runs it out of the clone. The runner re-executes itself
105+
out of the worktree it makes, so the tests, the framework and the report tool are
106+
always the branch's, but it cannot re-execute its way out of not understanding an
107+
option the checked-out copy has never heard of. A clone left on a revision older
108+
than `--run-if-changed` fails with `unknown command line option`.
109+
110+
Either keep the tree you install `e2e-cron-job` from separate from the clone the
111+
runner tests in, or set `E2E_UPDATE_CLONE=1`, which fetches and resets the clone
112+
to the branch before running. That is a `reset --hard` of that directory, which
113+
is why it is off by default: turn it on only where the clone is nobody's working
114+
tree.
115+
88116
Once a run has a directory to publish into, everything it prints goes to
89117
`e2e-runner.log.txt` there, and only what happens before that lands in the log
90118
above. Run by hand from a terminal it prints to both.
@@ -97,16 +125,6 @@ On a host behind a proxy, put the proxy variables in a file and point
97125
`--source-proxies` at it: the runner exports them and passes them into the test
98126
VMs.
99127

100-
Note that the copy of `e2e-runner` which parses the options is whichever one you
101-
invoke -- the clone's, when cron runs it out of the clone. The runner
102-
re-executes itself out of the worktree it makes, so the tests, the framework and
103-
the report tool are always the branch's, but it cannot re-execute its way out of
104-
not understanding an option the checked-out copy has never heard of. A clone left
105-
on a revision older than `--run-if-changed` fails with `unknown command line
106-
option`.
107-
108-
## Being told how a run went
109-
110128
To be told when a run fails, look at the verdict rather than at the exit status:
111129

112130
```shell
@@ -188,10 +206,12 @@ pruned and packed the rest away. `--pack-results` publishes all of it in a
188206
single `results.tar.zst` of some 2.5M instead, keeping the artifacts of every
189207
test and the coverage data of each:
190208

191-
```crontab
192-
0 2 * * * /opt/e2e-test/nri-plugins/nri-plugins/scripts/testing/nightly/e2e-runner \
193-
--results /opt/e2e-test/nri-plugins/results --pack-results \
194-
>>$HOME/e2e-runner.log 2>&1
209+
`E2E_PACK_RESULTS=1` in the settings file, which is the default there, or
210+
`--pack-results` on the runner:
211+
212+
```shell
213+
scripts/testing/nightly/e2e-runner --results /opt/e2e-test/nri-plugins/results \
214+
--pack-results
195215
```
196216

197217
`results.json`, `status.txt` and `summary.txt` stay where they are, so how a run
Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
#!/bin/bash
2+
3+
# The whole of what a cron job needs to run the e2e tests, so that a crontab
4+
# line is this script and nothing else:
5+
#
6+
# PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin
7+
# */5 * * * * /path/to/nri-plugins/scripts/testing/nightly/e2e-cron-job
8+
#
9+
# Every setting comes from the environment, so a host is configured without
10+
# editing this file. Install the settings file next to it:
11+
#
12+
# install -m 644 scripts/testing/nightly/e2e-cron-job.env \
13+
# /etc/sysconfig/nri-plugins-e2e-cron-job
14+
#
15+
# and edit that. What it leaves out defaults below. Anything given on our own
16+
# command line is passed on to the runner, so a one-off is
17+
#
18+
# e2e-cron-job --no-pack-results --runtime crio
19+
#
20+
# Every five minutes is not a typo: with E2E_RUN_IF_CHANGED the runner does
21+
# nothing unless the branch it tests has moved since it last tried, so polling
22+
# often is how a merged PR gets tested soon after it lands rather than the
23+
# following night. A run already going is left alone, and once E2E_FORCE_AFTER
24+
# has passed with nothing new, the tests run anyway.
25+
26+
set -o pipefail
27+
28+
SELF="$0"
29+
HERE="$(dirname "$(realpath "$SELF")")"
30+
31+
# ENV_FILE: where a host says what it wants, and the only thing to edit
32+
ENV_FILE="${E2E_CRON_ENV:-/etc/sysconfig/nri-plugins-e2e-cron-job}"
33+
34+
if [ -r "$ENV_FILE" ]; then
35+
# shellcheck disable=SC1090
36+
if ! source "$ENV_FILE"; then
37+
echo "$SELF: failed to read $ENV_FILE" >&2
38+
exit 1
39+
fi
40+
fi
41+
42+
# E2E_REMOTE: repository to test a branch of
43+
E2E_REMOTE="${E2E_REMOTE:-https://github.com/containers/nri-plugins}"
44+
# E2E_BRANCH: which branch of it
45+
E2E_BRANCH="${E2E_BRANCH:-main}"
46+
# E2E_CLONE: where to keep the clone of it. A clone belongs to one remote: the
47+
# runner refuses one whose origin is another, so a second remote wants a second
48+
# directory here.
49+
E2E_CLONE="${E2E_CLONE:-/opt/e2e-test/nri-plugins/nri-plugins}"
50+
# E2E_RESULTS: where to publish the results of each run
51+
E2E_RESULTS="${E2E_RESULTS:-/opt/e2e-test/nri-plugins/results}"
52+
# E2E_PROXIES: a file setting up proxies in the environment, if this host needs
53+
# one to reach the remote and pull images
54+
E2E_PROXIES="${E2E_PROXIES:-}"
55+
# E2E_RUN_IF_CHANGED: only test when the branch has moved, which is what makes
56+
# it sane to be triggered every few minutes. Empty, no or 0 to test on every
57+
# trigger. Defaulted with ${VAR-...} rather than ${VAR:-...}, so that a setting
58+
# emptied on purpose stays empty instead of coming back as the default.
59+
E2E_RUN_IF_CHANGED="${E2E_RUN_IF_CHANGED-1}"
60+
# E2E_FORCE_AFTER: test anyway once this long has passed with nothing new.
61+
# Defaulted with ${VAR-...} like the switches above, so that an empty value stays
62+
# empty: the runner reads that as never, and coming back as a day instead would be
63+
# the opposite of what emptying it asks for.
64+
E2E_FORCE_AFTER="${E2E_FORCE_AFTER-24h}"
65+
# E2E_PACK_RESULTS: publish a run as a single archive, keeping everything it
66+
# collected. Needs e2e-report serve to browse.
67+
E2E_PACK_RESULTS="${E2E_PACK_RESULTS-1}"
68+
# E2E_OPTIONS: anything else to pass the runner, as it would be on its command
69+
# line, for what this script has no setting of its own
70+
E2E_OPTIONS="${E2E_OPTIONS:-}"
71+
# E2E_UPDATE_CLONE: bring the clone up to the branch before running, so that the
72+
# copy of the runner which parses our options is the branch's too. Off by
73+
# default: it is a reset --hard of a directory which may be someone's tree.
74+
E2E_UPDATE_CLONE="${E2E_UPDATE_CLONE-}"
75+
76+
update_clone() {
77+
# Bring the clone up to the branch we are to test. The runner re-execs itself
78+
# out of the worktree it makes, so everything it does after parsing its
79+
# options is the branch's code already -- but it cannot re-exec its way out
80+
# of not understanding an option the checked-out copy has never heard of.
81+
# This is what closes that gap, for whoever wants it closed.
82+
#
83+
# Safe to do although we may well be running out of this very clone: git
84+
# replaces a file rather than writing over it, and the copy of us the shell
85+
# is reading stays open and therefore intact. Checked, not assumed.
86+
local origin
87+
88+
# Nothing to update yet. The runner clones it, at the branch, by itself.
89+
if [ ! -e "$E2E_CLONE/.git" ]; then
90+
return 0
91+
fi
92+
93+
# Fetching needs whatever proxies reach the remote. The runner sets those up
94+
# for itself, but we are ahead of it here.
95+
if [ -n "$E2E_PROXIES" ] && [ -r "$E2E_PROXIES" ]; then
96+
# shellcheck disable=SC1090
97+
source "$E2E_PROXIES" || true
98+
export http_proxy https_proxy no_proxy HTTP_PROXY HTTPS_PROXY NO_PROXY
99+
fi
100+
101+
# A clone belongs to one remote. Resetting it to some other remote's branch
102+
# would be worse than leaving it alone, and the runner says so properly.
103+
origin=$(git -C "$E2E_CLONE" remote get-url origin 2>/dev/null)
104+
if [ "$origin" != "$E2E_REMOTE" ]; then
105+
echo "$SELF: $E2E_CLONE is a clone of ${origin:-nothing} and not of" \
106+
"$E2E_REMOTE, leaving it as it is" >&2
107+
return 0
108+
fi
109+
110+
# A failure here is not ours to be fatal about: the checkout we have still
111+
# runs, and the runner fetches for itself and fails properly if it cannot.
112+
if ! git -C "$E2E_CLONE" fetch -q origin; then
113+
echo "$SELF: failed to fetch $E2E_REMOTE in $E2E_CLONE," \
114+
"running with the clone as it is" >&2
115+
return 0
116+
fi
117+
if ! git -C "$E2E_CLONE" reset -q --hard "origin/$E2E_BRANCH"; then
118+
echo "$SELF: failed to put $E2E_CLONE on $E2E_BRANCH," \
119+
"running with the clone as it is" >&2
120+
return 0
121+
fi
122+
}
123+
124+
case "$E2E_UPDATE_CLONE" in
125+
1|true|yes|y)
126+
update_clone
127+
;;
128+
esac
129+
130+
RUNNER="${E2E_RUNNER:-$HERE/e2e-runner}"
131+
132+
if [ ! -x "$RUNNER" ]; then
133+
echo "$SELF: no e2e-runner to run at $RUNNER" >&2
134+
exit 1
135+
fi
136+
137+
opts=(--remote "$E2E_REMOTE"
138+
--branch "$E2E_BRANCH"
139+
--local "$E2E_CLONE"
140+
--results "$E2E_RESULTS")
141+
142+
case "$E2E_RUN_IF_CHANGED" in
143+
1|true|yes|y)
144+
opts+=(--run-if-changed --force-after "$E2E_FORCE_AFTER")
145+
;;
146+
esac
147+
148+
case "$E2E_PACK_RESULTS" in
149+
1|true|yes|y)
150+
opts+=(--pack-results)
151+
;;
152+
esac
153+
154+
if [ -n "$E2E_PROXIES" ]; then
155+
opts+=(--source-proxies "$E2E_PROXIES")
156+
fi
157+
158+
# Unquoted on purpose: a setting of several flags in one value is the point of
159+
# it, and an empty one has to come to nothing rather than to an empty argument.
160+
# shellcheck disable=SC2206,SC2086
161+
opts+=($E2E_OPTIONS)
162+
163+
exec "$RUNNER" "${opts[@]}" "$@"
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Configuration for e2e-cron-job, to be installed as
2+
# /etc/sysconfig/nri-plugins-e2e-cron-job. Every setting left out here keeps the
3+
# default the script itself has, so uncomment only what this host changes.
4+
5+
# The repository to test a branch of, and which branch.
6+
#E2E_REMOTE=https://github.com/containers/nri-plugins
7+
#E2E_BRANCH=main
8+
9+
# Where to keep the clone. A clone belongs to one remote -- the runner refuses
10+
# one whose origin is another -- so testing a fork wants its own directory here,
11+
# and the URL has to be spelled the same way every time (an https clone is not
12+
# an ssh one as far as that check is concerned).
13+
#E2E_CLONE=/opt/e2e-test/nri-plugins/nri-plugins
14+
15+
# Where each run publishes its results. This is what e2e-report serve serves,
16+
# and the runs of one result root are kept apart by a lock in it: two runners
17+
# pointed at the same root would share a worktree and fight over the test VMs.
18+
#E2E_RESULTS=/opt/e2e-test/nri-plugins/results
19+
20+
# A file setting up proxies in the environment, if this host needs one to reach
21+
# the remote and pull images.
22+
#E2E_PROXIES=/etc/sysconfig/proxy
23+
24+
# Only test when the branch has moved since we last tried, which is what makes
25+
# it sane to be triggered every few minutes: a merged PR gets tested soon after
26+
# it lands instead of the following night. Set this to nothing to test on every
27+
# trigger, which is what a once-a-night crontab line wants.
28+
#E2E_RUN_IF_CHANGED=1
29+
30+
# Test anyway once this long has passed with nothing new, so that a quiet week
31+
# still gets a run a day. Takes 30m, 12h, 3d or a bare number of hours; never
32+
# (or nothing at all) to leave a quiet branch alone, and 0 to test on every
33+
# trigger. The clock runs from the last run of any kind, not from the last forced
34+
# one, so a branch which moves daily is never forced at all.
35+
#E2E_FORCE_AFTER=24h
36+
37+
# Publish a run as a single archive, keeping everything it collected: a tenth of
38+
# what it takes as plain files, and nothing thrown away. Browsing it needs
39+
# e2e-report serve, so set this to nothing if a file server is all there is.
40+
#E2E_PACK_RESULTS=1
41+
42+
# Anything else to pass the runner, as it would be on its command line. Several
43+
# flags in one value are fine.
44+
#E2E_OPTIONS=--runtime crio --retention-days 30
45+
46+
# Bring the clone up to the branch before running. The runner already re-execs
47+
# itself out of the worktree it makes, so the tests, the framework and the report
48+
# tool are the branch's whatever is checked out here -- but the copy which parses
49+
# the runner's options is whatever the clone has, so an option newer than that
50+
# checkout is an error it never gets past. Turn this on to close that gap.
51+
#
52+
# Off by default, because it is a reset --hard of E2E_CLONE: turn it on only
53+
# where that directory is nobody's working tree. Keeping the tree this script is
54+
# installed from separate from E2E_CLONE avoids the whole question.
55+
#E2E_UPDATE_CLONE=1

0 commit comments

Comments
 (0)