Skip to content

Chart SQL benchmark file sizes on bench, and reorder the groups - #222

Merged
joseph-isaacs merged 2 commits into
mainfrom
ji/file-size-charts
Nov 13, 2026
Merged

joseph-isaacs merged 2 commits into
mainfrom
ji/file-size-charts

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

What changes

bench, in two parts.

SQL File Size group

A new SQL File Size group charts the on-disk size of the files each SQL benchmark writes. The data comes from the file_sizes table added in #221. Develop runs fill it since vortex-data/vortex#11443, and vortex-data/vortex#11447 adds every format, not only Vortex.

Charts and lines. The table has one row per file, which would be far too many lines; ClickBench alone writes about 100 files per format. Instead:

  • One chart per benchmark and scale factor, about 16 today.
  • One line per format, summed over the benchmark's files. That's vortex and vortex-compact today, plus Parquet, DuckDB and the rest once #11447 lands, so about 4 lines per chart.
  • Chart order: clickbench, tpch, tpcds, then the other benchmarks by name, each by numeric scale factor.
  • Names: TPC-H, TPC-DS and nested TPC-H always name their scale factor (tpch sf=1.0, tpch sf=10.0). Benchmarks without scale factors use the plain name (clickbench).

Summary card. It reuses the compression-size card: each format's total, plus its min, geomean and max ratio to vortex, each scope at its newest commit with a vortex size. The "vs Arrow" column shows "—", because file sizes record no Arrow size. A size summary with its own baseline gets no headline. Without that, this group's header read "same size as Parquet".

Implementation:

  • file_sizes is the sixth family in families.ts, slug.ts, queries.ts (chart, batched group query, discovery) and summary.ts.
  • Discovery and the summary use a recursive skip scan over idx_file_sizes_baseline instead of reading every file row.
  • Docs: CLAUDE.md, CONTRACT.md and design-decisions.md now say six families.

Group order

The landing page now runs:

  1. Compression
  2. Compression Size
  3. SQL File Size
  4. Random Access (NVMe)
  5. Random Access (S3)
  6. The query groups, in their existing order
  7. Groups outside the fixed list, by name
  8. String Encoding
  9. String Encoding Size

The string-bench groups now come after everything else, from a separate GROUP_ORDER_LAST list.

Cache

GROUP_PAYLOAD_CACHE_VERSION is now v14, so a deployment of this change doesn't serve group lists cached by older code, which lack the new group and order. The preview hit exactly this before the bump.

How it was checked

  • bun run test:bench against a local PostgreSQL 16: 1004 passed, 9 skipped. The new tests cover:

    • per-format sums;
    • the group payload matching chartPayload byte for byte at ?n=1…4,all;
    • chart order and names;
    • the summary against real rows;
    • no headline for a summary with its own baseline;
    • the group order, with the string groups last.
  • bun run check:ci passes, and tsc --noEmit reports no errors in bench.

  • bun run dev against a seeded database: the group, card and charts render.

  • Production has rows since develop's run for Store SQL benchmark file sizes in the benchmarks database vortex#11443.

  • Not run locally: build, knip, Playwright and Lighthouse; CI runs them.

  • Every commit is signed off (git commit -s), for the DCO check.

  • bun run check:ci passes.

  • Each new image or other asset has an annotation in REUSE.toml. (No new assets.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s

Adds `file_sizes` (migration 014) as the sixth family. The new `SQL File Size`
group has one chart per benchmark and scale factor, and each chart has one line
per Vortex format, summed over the benchmark's files. A line per file would be
thousands of lines; this is about 16 charts of at most 2 lines.

- Discovery and the summary skip-scan `idx_file_sizes_baseline` for the
  scopes rather than reading every file row.
- The summary reuses the compression-size card: each format's total and its
  min/geomean/max ratio to vortex-file-compressed, each scope at its newest
  commit with a baseline, so nightly-only scopes count too.
- A size summary with its own baseline gets no headline. The Parquet headline
  read "same size as Parquet" for this group.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
vortex.dev Ready Ready Preview Nov 13, 2026 2:43pm UTC

Request Review

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🔦 Lighthouse audit

URL Perf A11y Best practices SEO
/ 🟢 100 🟢 100 🟢 100 🟢 100
/bench 🟢 100 🟢 100 🟢 100 🟢 100
/bench/chart/ct.eyJrIjoiQ29tcHJlc3Npb25UaW1lIiwiZGF0YXNldCI6ImFyYWRlIiwiZGF0YXNldF92YXJpYW50IjpudWxsfQ 🟢 100 🟢 100 🟢 100 🟢 100
/blog 🟢 100 🟢 100 🟢 100 🟢 100
/blog/btrblocks-compressor 🟢 100 🟢 100 🟢 100 🟢 100

Run 43445344911 · 2026-11-13 14:46 UTC

The group list and group bundles now include the `SQL File Size` group and
its summary, so payloads cached under `v12` by the deployed code are not what
this code returns for the same data. Without a new key, a deployment of this
change serves the old cached group list, without the group, until the next
ingest or the one-day backstop, and shares the key with the old deployment.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) November 13, 2026 14:45
@joseph-isaacs
joseph-isaacs merged commit 2a38ef3 into main Nov 13, 2026
20 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/file-size-charts branch November 13, 2026 14:48
joseph-isaacs added a commit to vortex-data/vortex that referenced this pull request Nov 13, 2026
## Summary

#11443 started storing SQL benchmark file sizes in the benchmarks
database, but only for `vortex-file-compressed` and `vortex-compact`.
This PR stores every on-disk format, so bench can chart Parquet, DuckDB,
Lance and the rest beside Vortex (vortex-data/vortex.dev#222). PR
comments still compare only the Vortex formats.

## Changes

- `scripts/capture-file-sizes.py` records every format directory that
`Format::name()` in `vortex-bench/src/lib.rs` names: `arrow-ipc`, `csv`,
`duckdb`, `lance`, `litdata`, `parquet`, `vortex-compact`,
`vortex-file-compressed` and `vortex-spatial-native`. Each develop
commit adds roughly one row per Parquet file; ClickBench has about 100.
- **DuckDB catalogs:** DuckDB keeps a catalog of views (`duckdb.db`, and
`duckdb.db.wal` while open) in every format directory it queries
(`duckdb-bench/src/lib.rs`). That isn't the format's data, so it's now
skipped everywhere except the `duckdb` directory, where the database is
the data. This also takes the catalog out of the Vortex totals. Rows
recorded today include it, so PR comments based on today's develop
commits list `duckdb.db` as a removed file, once.
- `scripts/compare-benchmark-jsons.py`: the "File Size Changes" section
compares only formats starting with `vortex`. The other formats' files
come from the data generators and don't change with a Vortex PR.

New tests check that capture records every format, skips the DuckDB
catalogs outside `duckdb`, and ignores unknown directories. Another
checks that the report leaves out non-Vortex formats.

### Checks

- `uv run --no-project --with pytest --with pandas --with numpy --with
orjson --with tabulate --with xxhash pytest
scripts/tests/test_measurement_id.py
scripts/tests/test_benchmark_reporting.py
scripts/tests/test_post_ingest.py`: 49 passed.
- `uvx ruff format --check` and `uvx ruff check` on the changed files
pass, as does `git diff --check`.
- Not run: a CI benchmark run. `action/bench-sql` on this PR would
exercise capture and the comment. The new formats only reach the
database from develop runs.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s)_

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@joseph-isaacs joseph-isaacs changed the title Chart SQL benchmark file sizes on bench Chart SQL benchmark file sizes on bench, and reorder the groups Nov 13, 2026

This branch was successfully deployed

1 active deployment
Preview — f61e347b Deployed Nov 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant