Skip to content

adapter: hydrate new builtin collections before 0dt promotion - #40903

Open
ggevay wants to merge 5 commits into
MaterializeInc:mainfrom
ggevay:gabor/new-builtin-mv-ro-hydration
Open

ggevay wants to merge 5 commits into
MaterializeInc:mainfrom
ggevay:gabor/new-builtin-mv-ro-hydration

Conversation

@ggevay

@ggevay ggevay commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes: SQL-948

Motivation

During a 0dt upgrade, a builtin storage collection that is new in the target version gets its shard inside the read-only process's catalog savepoint, which is never committed, and the promoted leader allocates a different one. So dataflows that hydrated while read-only are rebuilt at cut-over, and anything reading a new collection cannot hydrate before promotion.

On main this hits the three query-policy materialized views from #38762, which are small leaves. If #40067's mz_catalog_items and #40843 on top of it ship in one upgrade, the 12 catalog-server indexes that depend on mz_types only hydrate after cut-over.

Description

Reviewable commit by commit:

  1. Shards from the migration shard. Builtin tables, sources and materialized views without a shard get theirs from the builtin migration shard, as Replacement-migrated collections do, and the read-only process writes them the same way.
    • An entry is only reused for the same GlobalId and builtin. A new builtin's GlobalId exists only in the uncommitted savepoint, and a later boot can give it to another builtin, for example after another deploy generation was promoted.
    • Versions without the new key field decode these entries and never use them. Once released, the field must not be reverted or left out of a backport, or the next version that has it panics on leftover entries.
  2. Leader clock. For a collection the leader reports no frontier for, such as a new builtin, the caught-up check compares against the leader's clock instead of the current time. The clock is the upper of mz_cluster_replica_frontiers, which the leader advances every second. With the old leader down, as when a region is enabled again after a soft delete, such a collection's inputs stop, so against the current time it would lag forever.
  3. New builtins gate promotion. Only a new storage collection without a shard from the migration shard still seeds the caught-up exclusion.
    • New views and indexes no longer seed it. Otherwise a new view would keep everything reading it out of the check, such as mz_types and its indexes, which read SQL-495: Convert the type system tables to catalog views #40843's new mz_builtin_types. So a new builtin index over existing collections now has to hydrate before promotion.
    • With the break-glass flag enable_0dt_hydrate_migrated_builtin_mvs off, or a leader older than v26.17, every new builtin seeds it, as before. New tables and sources are written either way.
  4. 0dt workflow new-builtins-survive-promotion. It and the two builtin-schema-migrations-* workflows now wait until the replica reports its reconciliation metrics, since a leader reports IsLeader before it reconnects to its replicas.
  5. Stricter checks in that workflow. It also compares the GlobalIds of the collections read for the new builtins across the promotion, because a replica only counts a dataflow as reused or replaced if it had one with the same ids before. The wait for the reconciliation metrics ends once two reads in a row agree.

The coverage of tables and sources, the matching by both GlobalId and builtin, and the workflow come from Aljoscha's #40904, which fixes the same bug.

Verification

  • New unit tests for the allocation and matching, the key encoding, and the leader-clock comparison. Each fails when the behavior it covers is reverted.
  • The new workflow upgrades from v26.17.1, and from v26.47.1 with the old leader stopped first, after it recorded a REFRESH materialized view's frontier in the future. It checks that the new builtin materialized views and sources were written on the read-only generation, promotes, and checks that the collections read for them kept their GlobalIds. With a running leader it also asserts that the catalog server replaced no dataflow. With the old leader killed, a promotion can replace dataflows that have nothing to do with new builtins, also on main, so that run does not assert it. On main the workflow fails, because the new builtin materialized views are not written on the read-only generation.
  • With SQL-932: Introduce shared parsed-items layer #40067 and the PRs stacked on it (through SQL-496: Convert the function system tables to catalog views #40900) merged in locally, a 0dt upgrade from v26.51.0-rc.3 replaces no dataflow. Without this PR the same upgrade replaces 23: the 12 catalog-server indexes that read mz_types, the dataflows of the new collections, and user dataflows that read the converted relations.
  • Nightly (subsets): https://buildkite.com/materialize/nightly/builds/18803 (upgrade and 0dt jobs, before the last test-only commit) and https://buildkite.com/materialize/nightly/builds/18805 (0dt jobs, final head)

🤖 Generated with Claude Code

@ggevay ggevay added the A-ADAPTER Topics related to the ADAPTER layer label Nov 18, 2026
@ggevay ggevay changed the title adapter: hydrate new builtin MVs before 0dt promotion adapter: hydrate new builtin collections before 0dt promotion Nov 18, 2026
@ggevay
ggevay force-pushed the gabor/new-builtin-mv-ro-hydration branch from 2f5ebfc to 713b85c Compare November 19, 2026 14:20
ggevay and others added 4 commits November 19, 2026 16:49
During a 0dt upgrade, a builtin table, source or materialized view that
is new in the target version got its shard from the storage controller
inside the read-only process's catalog savepoint, which is never
committed. The promoted leader allocated a different shard, so dataflows
that hydrated while read-only were rebuilt at cut-over, and the
read-only process could not write these collections.

Builtin storage collections without a shard now get their shard from the
builtin migration shard, as Replacement-migrated collections do. The
read-only process writes them the same way: it backfills the tables,
writes the introspection sources, and write-enables the materialized
views. The caught-up check still leaves them out.

Their migration-shard entries also record the builtin's schema, name and
type, and are only reused for the same global id and builtin. A new
builtin's global id exists only in the uncommitted savepoint, and a
later boot can give it to another builtin, for example when another
deploy generation was promoted in between. On a mismatch, a fresh shard
is allocated. Versions that don't know the new key field still decode
these entries and never use them, since the entries have a higher build
version than theirs. Once released, the field must not be reverted or
left out of a backport.

The coverage of tables and sources and the matching by both global id
and builtin come from Aljoscha's MaterializeInc#40904.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…lock

The caught-up check compares a collection's write frontier with the
frontier the leader reported for it. For a collection without one, such
as a builtin table converted to a materialized view, it compared against
the current time. With the old leader down, as when a region is enabled
again after a soft delete, inputs that only the leader advances, such as
the catalog shard, stop. Such a collection then lags behind the current
time for good, and the new generation never becomes ready.

Compare it against the leader's clock instead: the upper of
mz_cluster_replica_frontiers, which the leader advances to its wall
clock every second and a read-only process does not. While the leader
runs, the two are close. The newest frontier the leader reported is no
substitute, since a REFRESH materialized view holds its frontier at its
next refresh.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
New builtin storage collections are written while read-only, so they and
what reads them can hydrate before promotion. Seed the caught-up
exclusion only with a new builtin storage collection that did not get
its shard from the builtin migration shard.

New builtins without a shard, such as views and indexes, no longer seed
it either. Otherwise a new view would keep everything that reads it out
of the check. As a result, a new builtin index over existing collections
now has to hydrate before promotion.

With enable_0dt_hydrate_migrated_builtin_mvs off, or a leader older than
v26.17, every new builtin seeds the exclusion, as before. New tables and
sources are written either way.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
new-builtins-survive-promotion upgrades from v26.17.1, and from v26.47.1
with the old leader stopped first, after it recorded a REFRESH
materialized view's frontier in the future. It checks that the builtin
materialized views and sources that are new since then were written on
the read-only generation, and promotes. With a running leader it also
asserts that no dataflow was replaced. With the old leader killed, a
promotion can replace dataflows that have nothing to do with new
builtins, also on main, so that run does not assert it.

The new workflow and the two builtin-schema-migrations workflows wait
until the replica reports its reconciliation metrics, since a leader
reports IsLeader before it reconnects to its replicas.

Based on the workflow in Aljoscha's MaterializeInc#40904.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ggevay
ggevay force-pushed the gabor/new-builtin-mv-ro-hydration branch from 713b85c to b4a1ff9 Compare November 19, 2026 15:49
@ggevay
ggevay marked this pull request as ready for review November 19, 2026 16:00
@ggevay
ggevay requested review from a team as code owners November 19, 2026 16:00
@def-

def- commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- The promotion test can pass when a new builtin dataflow is rebuilt

test/0dt/mzcompose.py:2101

If a new builtin MV gets a different export ID at promotion, compute drops its hydrated dataflow and creates a fresh one without incrementing the replacement counter. Reusing one unrelated catalog index is enough for the workflow to pass even in the running-leader case, where it asserts that every dataflow survived.

Details

Compute looks up the old dataflow by its export IDs at src/compute/src/server.rs:1097. The unmatched branch queues creation without recording either metric at src/compute/src/server.rs:1168, and the old export is dropped at src/compute/src/server.rs:1190. Even for matching IDs, the polling helper at test/0dt/mzcompose.py:1859 returns on the first recorded dataflow, so a scrape can miss a later replacement in the same reconciliation. Compare the target builtins’ global IDs and shard IDs across promotion, and wait for reconciliation to finish before asserting the replacement count.

…lows

A replica counts a dataflow as reused or replaced only if it had one with
the same global ids before. A collection that came back under another id
would be rebuilt without being counted. So `new-builtins-survive-promotion`
now also checks that each written collection kept its global id across the
promotion.

`await_reconciliation_metrics` now waits until two reads in a row agree, so
that a read in the middle of a reconciliation is not taken for the result.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ggevay

ggevay commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Hardened the test workflow in b75eeba:

  1. It now also checks that the collections read for each new builtin keep their GlobalIds across the promotion. In this workflow the ids are deterministic, but as the comment says, a replica only counts a dataflow as reused or replaced if it matches an earlier one by its export ids.
  2. await_reconciliation_metrics now waits until two reads in a row agree.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ADAPTER Topics related to the ADAPTER layer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants