Skip to content

Add a builtin schema migration step to finalize shards - #40898

Open
SangJunBak wants to merge 2 commits into
jun/type-relationsfrom
jun/builtin-view-migration
Open

SangJunBak wants to merge 2 commits into
jun/type-relationsfrom
jun/builtin-view-migration

Conversation

@SangJunBak

@SangJunBak SangJunBak commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

We add MigrationStep::Drop such that we can convert builtin materialized views and tables to views and clean up any stray shards. This will be used to migrate our builtin function and type relations.

See commit messages for details, but it essentially sees if these converted relations have shards then adds it to the unfinalized_shards inside the migration transaction.

A simpler solution could've been something like:

  • Like at what's a view in the catalog
  • See if it has a shard associated. If it does, it must've been from when the view was a storage backed collection.
  • Finalize the shard in a transaction

And this could've been done in migrate.rs. However, it seemed cleaner to add it to the existing framework.

Motivation

Because we're trying to reduce the number of materialized views, we provide a way to drop them and convert them into views with this PR.

Verification

  • We created unit tests in builtin_schema_migration_tests.rs such that a migration results in the correct results in unfinalized_shards, the queue of shards to finalize. I figured it wasn't worth refactoring our current random turmoil test for this because the functionality doesn't do any intermediate persist work.

@SangJunBak
SangJunBak requested a review from a team as a code owner November 18, 2026 07:49
@SangJunBak
SangJunBak added this pull request to stack #40899 November 18, 2026 07:50
@SangJunBak
SangJunBak requested review from a team as code owners November 18, 2026 21:28
@SangJunBak
SangJunBak force-pushed the jun/builtin-view-migration branch from de5d7b3 to dbcb891 Compare November 18, 2026 22:01
…e durable catalog

- In the durable catalog via migrate.rs, we change the catalog's system mapping of any builtin relation that wants to go from a table/mv to a view. A followup change is making sure that the shard is cleaned up via a builtin schema migration.
- In open.rs, we remove a stale comment because it predates our builtin schema migration and it's now possible to convert a table to a view in stable schemas. The assertion below where the comment panics if we try to remove an object in a stable schema. This doesn't trigger because we're mutating the type of the object in place rather than deleting/adding it.
The new MigrationStep::Drop allows us to drop the shard when converting a table/mv into a view. It does this by looking at the builtin items to drop, then adds it to the unfinalized shards queue. 

For forced migrations, although there isn't an explicit one for Drop, we still clean up shards that are associated with views.
@SangJunBak
SangJunBak force-pushed the jun/builtin-view-migration branch from dbcb891 to 1da0c53 Compare November 18, 2026 23:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant