Repository navigation
Store every format's SQL benchmark file sizes - #11447
Conversation
`capture-file-sizes.py` recorded only the two Vortex formats. It now records every on-disk format directory `Format::name()` names (Parquet, DuckDB, Lance, Arrow IPC, CSV, LitData and the Vortex formats), so bench can chart them side by side. It is a few more rows per develop commit. DuckDB keeps a catalog of views in each format directory it queries (`duckdb.db` and its WAL). That is not the format's data, so it is skipped everywhere except the `duckdb` format's directory, where the database is the data. This also drops the catalog from the Vortex totals. The PR comment's "File Size Changes" table still compares only the `vortex*` formats: the other formats' files come from the data generators and do not change with a Vortex PR. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will improve performance by 12.21%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Summary
#11443 started storing SQL benchmark file sizes in the benchmarks database, but only for
vortex-file-compressedandvortex-compact. This PR stores every on-disk format, so bench can chart Parquet, DuckDB, Lance and the rest beside Vortex (vortex-data/vortex.dev#222).