Skip to content

docs+ci: report the GNU test suite pass rate in the README - #260

Open
metif12 wants to merge 3 commits into
vlang:mainfrom
metif12:gnu-suite-report
Open

metif12 wants to merge 3 commits into
vlang:mainfrom
metif12:gnu-suite-report

Conversation

@metif12

@metif12 metif12 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

The suite job in ci.yml already measures us on every push. This makes the number
visible: a scheduled workflow runs the same measurement and writes the pass rate
into the README between two HTML comment markers, so the headline figure is a
measurement rather than a count someone typed by hand.

Both jobs call util/gnu_test_suite.vsh, so they cannot disagree about the
measurement itself - which is the point, since a README figure that drifts from
the CI figure is worse than no figure. The harness now emits a machine-readable
GNU_SUITE_RESULT line; the workflow parses that instead of the prose summary,
and commits only README.md so the commit it makes is reviewable as one line.

The rate's scope is stated next to it: tests for tools that are not built are
counted as
ot-run and excluded, because a test with nothing to invoke fails on
a missing command and says nothing about this implementation. The full
est-suite.log is uploaded as an artifact so any failure is readable rather
than inferred.

The Completed count is left alone. It is maintained by an existing workflow
from what has actually merged. I changed it to 109/109 while writing this and then
reverted it: 18 of the port PRs are still open, so that figure would have been
false.

Reads and writes terminal attributes, and is the last unimplemented tool.

Verified against GNU 9.4 in a pty: all 13 cases byte-identical, including the non-tty error path. -a matches on all ten lines, -g on all four flag words and 32 control characters, and size, sane, -icanon, icanon, echo, -echo and the -F/--file forms all match.

Four things were measured rather than recalled, and each was a bug:

V's termios.Termios has a different layout from glibc's struct termios. c_cc sits at offset 20 instead of 17 because the compiler pads after c_line. An offsetof probe showed glibc reporting c_lflag 0x105073 where V read 0x8a3b, so every control character and the top byte of every flag word was wrong. This port includes termios.h and uses C.termios directly.

The \ format does not apply to a C-typed field. A probe printed c_iflag=1280 and hex: iflag=1280, identical with the format silently ignored, which made -g wrong by a whole nibble. Each word is cast to u32 first.

The flag polarity in -a is the opposite of what the option syntax suggests: a bare name means the flag is set and a leading dash means it is clear.

The wrap width is 79 not 76, the input-mode order is ixon, ixoff, iuclc, ixany, and several bit values differ from recall, so every constant comes from a C probe printing the preprocessor values.

Not implemented, stated rather than hidden: the control settings rows, columns, cols, drain and flush, the evenp/oddp/parity aliases and the special-character assignment forms. The suite is stty_nix_test.v because termios.h does not exist on Windows tcc, and V cannot generate C for a test target, so the Linux CI job is what runs it.
The repo measures itself with the unit tests it writes for itself. This adds the
other measurement: the tests GNU writes for its own implementation, run against
ours. The pass rate of that suite is the number that tells us where we actually
stand, and it is not a number this repo has ever had.

The flow is the one uutils/coreutils uses, and is not original work: fetch the
release tarball, configure it, copy lib/stdckdint.in.h over lib/stdckdint.h,
build, point tests/local.mk at our bin/ instead of GNU's own src/, and run
make check.

Two things had to be worked out, and both cost a build cycle each:

**Building GNU at all.** The tree ships lib/stdckdint.in.h, gnulib's replacement
for the C23 header that gcc 13 on Ubuntu 24.04 does not provide, but the make
rule that copies it into place never fires, so lib/malloca.c dies on
`#include <stdckdint.h>`. Copying it by hand is the whole fix - the file has no
substitution markers. The release tarball is used rather than the git sources
because it ships configure and tests/ pre-generated, so no autoreconf, gperf,
bison or texinfo is needed.

**Pointing the tests at our binaries.** This is done in two places, because the
tests find their subject through two independent mechanisms: the PATH= line in
tests/local.mk, and `path_prepend_ ./src`, which most tests call to prepend the
build directory. Patching only the first leaves every test silently measuring
GNU - I measured 417 "passes" that were GNU's, and it took a control run against
GNU's own binaries to prove the harness was sound before trusting any number.

Results are filtered to the tools actually present in bin/, because a test for a
tool nobody built fails with a missing-command error that says nothing about this
implementation. Those are counted as not-run rather than dropped, so the report
never hides how much of the suite was measured.

The job does not gate the build. It is informational until the pass rate means
something, and the log is uploaded as an artifact so the failures are readable
rather than inferred.

Local run against 43 cross-compiled tools: 0 pass, 2 fail, 72 skip, 4 not-run. The
skips are legitimate - no gdb, no root, no ACLs - and the two failures are real
and actionable: our `env` rejects `---` and `-u` where GNU accepts them, and
mishandles `env --` and `env -u 'a=b'`.
The suite job in ci.yml already measures us on every push. This makes that
number visible: a scheduled workflow runs the same measurement and writes the
pass rate into the README between two HTML comment markers, so the headline
figure is a measurement rather than a count someone typed by hand.

Both jobs call util/gnu_test_suite.vsh, so they cannot disagree about the
measurement itself - which is the point, since a README figure that drifts from
the CI figure is worse than no figure.

The harness now emits a machine-readable GNU_SUITE_RESULT line. The workflow
parses that instead of the prose summary, and commits only README.md, so the
commit it makes is reviewable as a single line.

What the rate covers is stated next to it: tests for tools that are not built
are counted as not-run and excluded, because a test with nothing to invoke
fails on a missing command and says nothing about this implementation. The
full test-suite.log is uploaded as an artifact so any failure is readable
rather than inferred.

The Completed count is left alone. It is maintained by an existing workflow
from what has actually merged, and 109/109 would be false while the port PRs
are still open.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant