Repository navigation
Conversation
2f5ebfc to
713b85c
Compare
During a 0dt upgrade, a builtin table, source or materialized view that is new in the target version got its shard from the storage controller inside the read-only process's catalog savepoint, which is never committed. The promoted leader allocated a different shard, so dataflows that hydrated while read-only were rebuilt at cut-over, and the read-only process could not write these collections. Builtin storage collections without a shard now get their shard from the builtin migration shard, as Replacement-migrated collections do. The read-only process writes them the same way: it backfills the tables, writes the introspection sources, and write-enables the materialized views. The caught-up check still leaves them out. Their migration-shard entries also record the builtin's schema, name and type, and are only reused for the same global id and builtin. A new builtin's global id exists only in the uncommitted savepoint, and a later boot can give it to another builtin, for example when another deploy generation was promoted in between. On a mismatch, a fresh shard is allocated. Versions that don't know the new key field still decode these entries and never use them, since the entries have a higher build version than theirs. Once released, the field must not be reverted or left out of a backport. The coverage of tables and sources and the matching by both global id and builtin come from Aljoscha's MaterializeInc#40904. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…lock The caught-up check compares a collection's write frontier with the frontier the leader reported for it. For a collection without one, such as a builtin table converted to a materialized view, it compared against the current time. With the old leader down, as when a region is enabled again after a soft delete, inputs that only the leader advances, such as the catalog shard, stop. Such a collection then lags behind the current time for good, and the new generation never becomes ready. Compare it against the leader's clock instead: the upper of mz_cluster_replica_frontiers, which the leader advances to its wall clock every second and a read-only process does not. While the leader runs, the two are close. The newest frontier the leader reported is no substitute, since a REFRESH materialized view holds its frontier at its next refresh. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
New builtin storage collections are written while read-only, so they and what reads them can hydrate before promotion. Seed the caught-up exclusion only with a new builtin storage collection that did not get its shard from the builtin migration shard. New builtins without a shard, such as views and indexes, no longer seed it either. Otherwise a new view would keep everything that reads it out of the check. As a result, a new builtin index over existing collections now has to hydrate before promotion. With enable_0dt_hydrate_migrated_builtin_mvs off, or a leader older than v26.17, every new builtin seeds the exclusion, as before. New tables and sources are written either way. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
new-builtins-survive-promotion upgrades from v26.17.1, and from v26.47.1 with the old leader stopped first, after it recorded a REFRESH materialized view's frontier in the future. It checks that the builtin materialized views and sources that are new since then were written on the read-only generation, and promotes. With a running leader it also asserts that no dataflow was replaced. With the old leader killed, a promotion can replace dataflows that have nothing to do with new builtins, also on main, so that run does not assert it. The new workflow and the two builtin-schema-migrations workflows wait until the replica reports its reconciliation metrics, since a leader reports IsLeader before it reconnects to its replicas. Based on the workflow in Aljoscha's MaterializeInc#40904. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
713b85c to
b4a1ff9
Compare
QA LLM Review1. MEDIUM -- The promotion test can pass when a new builtin dataflow is rebuilt
If a new builtin MV gets a different export ID at promotion, compute drops its hydrated dataflow and creates a fresh one without incrementing the replacement counter. Reusing one unrelated catalog index is enough for the workflow to pass even in the running-leader case, where it asserts that every dataflow survived. DetailsCompute looks up the old dataflow by its export IDs at |
…lows A replica counts a dataflow as reused or replaced only if it had one with the same global ids before. A collection that came back under another id would be rebuilt without being counted. So `new-builtins-survive-promotion` now also checks that each written collection kept its global id across the promotion. `await_reconciliation_metrics` now waits until two reads in a row agree, so that a read in the middle of a reconciliation is not taken for the result. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Hardened the test workflow in b75eeba:
|
Closes: SQL-948
Motivation
During a 0dt upgrade, a builtin storage collection that is new in the target version gets its shard inside the read-only process's catalog savepoint, which is never committed, and the promoted leader allocates a different one. So dataflows that hydrated while read-only are rebuilt at cut-over, and anything reading a new collection cannot hydrate before promotion.
On main this hits the three query-policy materialized views from #38762, which are small leaves. If #40067's
mz_catalog_itemsand #40843 on top of it ship in one upgrade, the 12 catalog-server indexes that depend onmz_typesonly hydrate after cut-over.Description
Reviewable commit by commit:
Replacement-migrated collections do, and the read-only process writes them the same way.GlobalIdand builtin. A new builtin'sGlobalIdexists only in the uncommitted savepoint, and a later boot can give it to another builtin, for example after another deploy generation was promoted.mz_cluster_replica_frontiers, which the leader advances every second. With the old leader down, as when a region is enabled again after a soft delete, such a collection's inputs stop, so against the current time it would lag forever.mz_typesand its indexes, which read SQL-495: Convert the type system tables to catalog views #40843's newmz_builtin_types. So a new builtin index over existing collections now has to hydrate before promotion.enable_0dt_hydrate_migrated_builtin_mvsoff, or a leader older than v26.17, every new builtin seeds it, as before. New tables and sources are written either way.new-builtins-survive-promotion. It and the twobuiltin-schema-migrations-*workflows now wait until the replica reports its reconciliation metrics, since a leader reportsIsLeaderbefore it reconnects to its replicas.GlobalIds of the collections read for the new builtins across the promotion, because a replica only counts a dataflow as reused or replaced if it had one with the same ids before. The wait for the reconciliation metrics ends once two reads in a row agree.The coverage of tables and sources, the matching by both
GlobalIdand builtin, and the workflow come from Aljoscha's #40904, which fixes the same bug.Verification
REFRESHmaterialized view's frontier in the future. It checks that the new builtin materialized views and sources were written on the read-only generation, promotes, and checks that the collections read for them kept theirGlobalIds. With a running leader it also asserts that the catalog server replaced no dataflow. With the old leader killed, a promotion can replace dataflows that have nothing to do with new builtins, also on main, so that run does not assert it. On main the workflow fails, because the new builtin materialized views are not written on the read-only generation.mz_types, the dataflows of the new collections, and user dataflows that read the converted relations.🤖 Generated with Claude Code