Repository navigation
Chart SQL benchmark file sizes on bench, and reorder the groups - #222
Merged
Merged
Conversation
Adds `file_sizes` (migration 014) as the sixth family. The new `SQL File Size` group has one chart per benchmark and scale factor, and each chart has one line per Vortex format, summed over the benchmark's files. A line per file would be thousands of lines; this is about 16 charts of at most 2 lines. - Discovery and the summary skip-scan `idx_file_sizes_baseline` for the scopes rather than reading every file row. - The summary reuses the compression-size card: each format's total and its min/geomean/max ratio to vortex-file-compressed, each scope at its newest commit with a baseline, so nightly-only scopes count too. - A size summary with its own baseline gets no headline. The Parquet headline read "same size as Parquet" for this group. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🔦 Lighthouse audit
Run |
The group list and group bundles now include the `SQL File Size` group and its summary, so payloads cached under `v12` by the deployed code are not what this code returns for the same data. Without a new key, a deployment of this change serves the old cached group list, without the group, until the next ingest or the one-day backstop, and shares the key with the old deployment. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
enabled auto-merge (squash)
November 13, 2026 14:45
joseph-isaacs
added a commit
to vortex-data/vortex
that referenced
this pull request
Nov 13, 2026
## Summary #11443 started storing SQL benchmark file sizes in the benchmarks database, but only for `vortex-file-compressed` and `vortex-compact`. This PR stores every on-disk format, so bench can chart Parquet, DuckDB, Lance and the rest beside Vortex (vortex-data/vortex.dev#222). PR comments still compare only the Vortex formats. ## Changes - `scripts/capture-file-sizes.py` records every format directory that `Format::name()` in `vortex-bench/src/lib.rs` names: `arrow-ipc`, `csv`, `duckdb`, `lance`, `litdata`, `parquet`, `vortex-compact`, `vortex-file-compressed` and `vortex-spatial-native`. Each develop commit adds roughly one row per Parquet file; ClickBench has about 100. - **DuckDB catalogs:** DuckDB keeps a catalog of views (`duckdb.db`, and `duckdb.db.wal` while open) in every format directory it queries (`duckdb-bench/src/lib.rs`). That isn't the format's data, so it's now skipped everywhere except the `duckdb` directory, where the database is the data. This also takes the catalog out of the Vortex totals. Rows recorded today include it, so PR comments based on today's develop commits list `duckdb.db` as a removed file, once. - `scripts/compare-benchmark-jsons.py`: the "File Size Changes" section compares only formats starting with `vortex`. The other formats' files come from the data generators and don't change with a Vortex PR. New tests check that capture records every format, skips the DuckDB catalogs outside `duckdb`, and ignores unknown directories. Another checks that the report leaves out non-Vortex formats. ### Checks - `uv run --no-project --with pytest --with pandas --with numpy --with orjson --with tabulate --with xxhash pytest scripts/tests/test_measurement_id.py scripts/tests/test_benchmark_reporting.py scripts/tests/test_post_ingest.py`: 49 passed. - `uvx ruff format --check` and `uvx ruff check` on the changed files pass, as does `git diff --check`. - Not run: a CI benchmark run. `action/bench-sql` on this PR would exercise capture and the comment. The new formats only reach the database from develop runs. --- _Generated by [Claude Code](https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s)_ Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
3 tasks done
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
bench, in two parts.
SQL File Size group
A new SQL File Size group charts the on-disk size of the files each SQL benchmark writes. The data comes from the
file_sizestable added in #221. Develop runs fill it since vortex-data/vortex#11443, and vortex-data/vortex#11447 adds every format, not only Vortex.Charts and lines. The table has one row per file, which would be far too many lines; ClickBench alone writes about 100 files per format. Instead:
vortexandvortex-compacttoday, plus Parquet, DuckDB and the rest once #11447 lands, so about 4 lines per chart.clickbench,tpch,tpcds, then the other benchmarks by name, each by numeric scale factor.tpch sf=1.0,tpch sf=10.0). Benchmarks without scale factors use the plain name (clickbench).Summary card. It reuses the compression-size card: each format's total, plus its min, geomean and max ratio to
vortex, each scope at its newest commit with avortexsize. The "vs Arrow" column shows "—", because file sizes record no Arrow size. A size summary with its own baseline gets no headline. Without that, this group's header read "same size as Parquet".Implementation:
file_sizesis the sixth family infamilies.ts,slug.ts,queries.ts(chart, batched group query, discovery) andsummary.ts.idx_file_sizes_baselineinstead of reading every file row.CLAUDE.md,CONTRACT.mdanddesign-decisions.mdnow say six families.Group order
The landing page now runs:
The string-bench groups now come after everything else, from a separate
GROUP_ORDER_LASTlist.Cache
GROUP_PAYLOAD_CACHE_VERSIONis nowv14, so a deployment of this change doesn't serve group lists cached by older code, which lack the new group and order. The preview hit exactly this before the bump.How it was checked
bun run test:benchagainst a local PostgreSQL 16: 1004 passed, 9 skipped. The new tests cover:chartPayloadbyte for byte at?n=1…4,all;bun run check:cipasses, andtsc --noEmitreports no errors in bench.bun run devagainst a seeded database: the group, card and charts render.Production has rows since develop's run for Store SQL benchmark file sizes in the benchmarks database vortex#11443.
Not run locally: build, knip, Playwright and Lighthouse; CI runs them.
Every commit is signed off (
git commit -s), for the DCO check.bun run check:cipasses.Each new image or other asset has an annotation in
REUSE.toml. (No new assets.)🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s