Skip to content

e2e: improve nightly runner. - #793

Merged
askervin merged 13 commits into
e2e-coveragefrom
e2e-nightly-runner
Sep 18, 2026
Merged

e2e: improve nightly runner.#793
askervin merged 13 commits into
e2e-coveragefrom
e2e-nightly-runner

Conversation

@klihub

@klihub klihub commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Improve the nightly e2e runner.

  • Publish a report per run: the failures first, each with the reason it failed
    and links to the logs, the commands and the test source at the tested
    revision, then the coverage of each plugin, then every test case. Plus an
    index of all runs, and a status.txt and a latest symlink for the cronjob.
  • Keep results for RETENTION_DAYS (100 by default) and pack up the artifacts of
    the tests which passed: a full run publishes 17M instead of 93M.
  • New options: --runtime, --full-build|--minimal-build, --retention-days,
    --retention-keep, --keep-coverage-data, --keep-artifacts.
  • Fixes: the results of a topology not matching n[0-9]-c were left out of the
    summary, so a run whose only failures were there was published as passing;
    the first run on a host died of a git internal error.
  • A caddy configuration for serving the results.

Exit status semantics are unchanged: non-zero only if the runner itself failed,
never because the tested system failed.

@klihub
klihub added this pull request to stack #796 September 16, 2026 11:44
@klihub klihub changed the title e2e nightly runner e2e: improve nightly runner. Sep 16, 2026
@klihub
klihub force-pushed the e2e-nightly-runner branch 2 times, most recently from bb66c6a to fdae1be Compare September 17, 2026 12:09
klihub added 13 commits September 17, 2026 15:46
@klihub
e2e-runner: collect and summarize results reliably.
3897f67 
Collect the results of all VMs, not just the ones matched by the
incorrect 'n[0-9]*-c*' glob pattern. Rename the runtime logs to
.txt like the other logs, so that a browser renders them. Clone
the repository into CLONED_REPO, not to an unset variable.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
e2e-runner: publish a coverage report, clean old results.
4fc57e3 
Generate a coverage report of every run and publish it with the
results. Remove the results of a run once they are older than
RETENTION_DAYS (100 by default). Also remove coverage data of a
run once its report exists.

Control with these extra options
  --retention-days <days>    0 keeps every result, whatever its age
  --retention-keep <count>   always keep this many latest results
  --keep-coverage-data       keep the data collected for each test

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
testing: report on e2e test runs, hyperlinked.
4f511cc 
Add e2e-report to generate report for a test run. List runs which never
got to a test distinctively from runs with some failing tests.

Collect numbers to results.json, use it when generating the index of
all runs. report-coverage.sh takes several input directories and one
to write the report to. The coverage of one plugin can thus be reported
across all the topologies it ran on, and write the summary.json for the
report of a run to pick up.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
e2e-runner: pack up the artifacts of passing tests.
63385e2 
The command transcripts and the plugin logs are the bulk of what
a run publishes, and nobody reads either for a test which passed.
Pack them into an artifacts.tar.xz per test, but leave artifacts
of failing tests as they are, browsable.

Packing beats compressing the files one by one, currently the
transcripts being 3090 tiny files: 824K packed against 15M plain
gzipped, most of which is the block each file takes. Controllable
with the new extra option

  --keep-artifacts all|failed|none  whose artifacts to keep as they are

A test keeps its own log either way, nothing is removed unless it made
it into the tarball first.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
testing: finish off the report of a nightly run.
f610b3a 
Remove the byte compiled python of the pyexec snippets from the
results. Record the verdict of a run in a status.txt and point a
latest symlink at the newest run.

Give the report dark mode, a badge for each verdict, and a section
per VM which folds away when everything in it passed.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
e2e-runner: let the runtime and the build be chosen.
431b0fa 
Take --runtime and --full-build|--minimal-build, and keep deciding
by the day of the year when neither is given. Refuse a runtime which
is neither containerd nor crio right away, instead of half an hour
later. Export the options which default from the environment across
the re-exec.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
testing: add a caddy file for serving the results.
b51cc0e 
The results of the runs are static files, so serving them takes
a file server with browsing for the directories which have no
index of their own. Take the directory to serve and the port from
the environment. Caddy also sniffs the type of a file which has
no extension, so the command transcripts of a test case show up
in a browser instead of being downloaded.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
e2e-runner: set up the worktree on a fresh host too.
d620271 
Resolve the path of the worktree with 'realpath -m' and create
the parent directory. Otherwise on a host with the first test
run ever we'd end up running "git worktree add ''" which crashes
git with an internal error like this:

    BUG: builtin/worktree.c:498: How come '' becomes empty after
    sanitization?

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
e2e-runner: prune results regardless of their --name.
ef1021a 
Prune anything with results in it. Leave links and whatever else
alone. Order the runs by when they ran.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
e2e-report: report on a run whatever it is called, and link to it.
4e906b0 
Index anything with results or a log of a run in it, give a run which
has no report one of its own, and order them by when they ran, told from
the name of a run or from the timestamp of its directory.

Link each policy of the coverage table to the source of its plugin at
the tested revision, and recognise the ssh spellings of a remote, which
is what the nightly is cloned from. Right-align the header of a column
whose cells are right-aligned, call the log of run.sh a test log, and
leave out a link to an empty file, such as the pyexec output of a test
whose last pyexec printed nothing.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
e2e: pack leftovers of a passing test, link failing ones.
a277dc1 
A test also leaves behind the python snippet of its last pyexec, the
state it ran against, what it printed and the cache of the plugin, some
160 kilobytes per test and of interest only when a test failed. Pack
those away as well for the tests which passed, which takes a run of the
whole suite from 25M to 17M, and link them from the report for the tests
which failed.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
e2e-runner: show the log of runs on a terminal.
6456c86 
Tee the output of a run when we have a terminal to write to. Keep
redirecting it to the log when we have not. From cron there is nobody
watching, and printing anything would have the whole log of the run
mailed to whoever gets our output.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
@klihub
testing: tell ongoing runs apart from stopped ones.
ad89cbc 
Say RUNNING in the status of a run at the start, take a log written to
within the hour as one still going and a quiet one as aborted. Report
on a run as soon as its directory is there, so the index has it from
the start.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
askervin
askervin approved these changes Sep 18, 2026
View reviewed changes

@askervin askervin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@askervin
askervin merged commit 2aa2c1f into main Sep 18, 2026
25 checks passed
@klihub
klihub deleted the e2e-nightly-runner branch September 21, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@askervin askervin askervin approved these changes
@fmuyassarov fmuyassarov Awaiting requested review from fmuyassarov

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@klihub @askervin