Skip to content

Add the file_sizes table for SQL benchmark file sizes - #221

Merged
joseph-isaacs merged 3 commits into
mainfrom
ji/file-sizes-table
Nov 13, 2026
Merged

joseph-isaacs merged 3 commits into
mainfrom
ji/file-sizes-table

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

What changes

bench (database schema and docs only; the site's pages and charts don't change).

The PR benchmark comments in vortex-data/vortex compare against baselines read from this database. SQL benchmark file sizes were never stored here, so when vortex-data/vortex#10987 moved the baseline from S3 to Postgres, the comments lost their file-size section. This adds a table for those sizes.

  • db/migrations/014_file_sizes.sql creates file_sizes, keyed by (commit_sha, benchmark, scale_factor, format, file) with value_bytes.
    • Its index is (benchmark, scale_factor, format, commit_sha). CI chooses the baseline commit from the query measurements, then reads that commit's files for each scope, and this index serves that read as a range scan.
    • It needs no grants: the ALTER DEFAULT PRIVILEGES rules in 004, 005 and 011 give bench_ingest, bench_read and bench_ci_read their access when migrator creates the table.
    • It needs no requires-superuser marker either, so bench-schema-deploy.yml can apply it.
  • docs/bench/CONTRACT.md, docs/bench/README.md and docs/bench/architecture/data-pipeline.md add the file_size kind. CONTRACT.md also records where the new golden vectors came from, and why a new kind with its own table needs no SCHEMA_VERSION bump.
  • infra/RESTORE.md adds the table to the restore data check, and db/migrations/README.md lists 014.

The writer and the reader are in vortex-data/vortex#11443. Apply this migration in production before that PR merges. Develop's ingest is required and writes each file in one transaction, so file_size records sent before the table exists would fail every develop SQL benchmark job. The PR-side reader now skips a missing table, but develop's ingest does not.

How it was checked

  • Applied the migrations to a local PostgreSQL 16 and loaded 1,500 commits × 100 files. With the PR 100 commits behind develop, the vortex PR's baseline fetch took 5 ms, and the file read used idx_file_sizes_baseline (an index scan of 100 rows). Under the earlier index and commit search it took 5.3 s.

  • An independent review applied 001–013 as a bootstrap role and 014 as migrator through db/migrate-schema.py. It found file_sizes owned by migrator, with SELECT/INSERT/UPDATE for bench_ingest and SELECT for both read roles.

  • bun run check:ci and db/tests were not run locally. CI ran them on the first commit and passed.

  • Every commit is signed off (git commit -s), for the DCO check.

  • bun run check:ci passes.

  • Each new image or other asset has an annotation in REUSE.toml. (No new assets.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s

The monorepo's PR benchmark comments compare against baselines from this
database, and SQL benchmark file sizes were never stored, so the comments lost
their file-size section. Migration 014 adds `file_sizes`, keyed by
(commit_sha, benchmark, scale_factor, format, file), with an index on the dims
the CI baseline lookup filters on. The default privileges from 004, 005 and 011
grant the ingest, read and CI-read roles their access.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
vortex.dev Ready Ready Preview Nov 13, 2026 1:56pm UTC

Request Review

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🔦 Lighthouse audit

URL Perf A11y Best practices SEO
/ 🟢 100 🟢 100 🟢 100 🟢 100
/bench 🟢 100 🟢 100 🟢 100 🟢 100
/bench/chart/ct.eyJrIjoiQ29tcHJlc3Npb25UaW1lIiwiZGF0YXNldCI6ImFyYWRlIiwiZGF0YXNldF92YXJpYW50IjpudWxsfQ 🟢 100 🟢 100 🟢 100 🟢 100
/blog 🟢 100 🟢 100 🟢 100 🟢 100
/blog/btrblocks-compressor 🟢 100 🟢 100 🟢 100 🟢 100

Run 43438100699 · 2026-11-13 14:00 UTC

Review of the CI reader: the baseline commit is chosen from the query
measurements, and file sizes are then read for that commit alone, so the index
ends in `commit_sha` rather than `file`.

Also document the sixth kind in the bench README, the provenance of the
`file_sizes` golden vectors, why a new kind needs no `SCHEMA_VERSION` bump,
and the table in the restore check.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) November 13, 2026 13:56
@joseph-isaacs
joseph-isaacs merged commit adfee9b into main Nov 13, 2026
20 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/file-sizes-table branch November 13, 2026 14:02

This branch was successfully deployed

1 active deployment
Preview — 10b9b7f8 Deployed Nov 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant