Skip to content

SQL-932: Introduce shared parsed-items layer - #40067

Open
SangJunBak wants to merge 11 commits into
mainfrom
jun/type-relations
Open

SangJunBak wants to merge 11 commits into
mainfrom
jun/type-relations

Conversation

@SangJunBak

@SangJunBak SangJunBak commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor
  • Adds mz_catalog_state, a materialized view over mz_catalog_raw that filters out audit log entries, so that our other builtin relations don't have to hydrate from the audit log each time.
  • On top of it we add mz_catalog_items, a materialized view that decodes each item's DDL once, plus a narrow indexed view mz_catalog_item_metadata that excludes the DDL text so we don't store text-heavy fields in memory.
    • Since one parse failure could poison the entire relation, we create try_decode_catalog_create_sql, which exposes an ok and error field and lets downstream relations error via mz_unsafe.mz_error_if_null.
  • Adds mz_builtin_object_mappings, an indexed view over the builtin GidMapping records

For an example of how these are intended to be used, this is the definition of mz_types for a later PR in the stack:

WITH
    user_types AS (
        SELECT
            id,
            oid,
            schema_id,
            name,
            COALESCE(mz_unsafe.mz_error_if_null(details, decode_error)->>'category', '') AS category,
            owner_id,
            privileges,
            create_sql,
            redacted_create_sql
        FROM mz_internal.mz_catalog_items
        WHERE type = 'type'
    ),
    builtin_types AS (
        SELECT
            m.id,
            t.oid,
            s.id AS schema_id,
            t.name,
            t.category,
            '{MZ_SYSTEM_ROLE_ID}' AS owner_id,
            t.privileges,
            NULL::text AS create_sql,
            NULL::text AS redacted_create_sql
        FROM mz_internal.mz_builtin_types t
        JOIN mz_internal.mz_builtin_object_mappings m
            ON m.schema_name = t.schema_name AND m.name = t.name AND m.object_type = {object_type}
        JOIN mz_catalog.mz_schemas s ON s.name = t.schema_name AND s.database_id IS NULL
    )
SELECT * FROM user_types
UNION ALL
SELECT * FROM builtin_types",
            object_type = gid_mapping_object_type(CatalogItemType::Type),
        )

Motivation

We introduce these shared materialized views such that we can eventually replace our builtin materialized views with views built on top of them. Removing the other builtin mvs will come in a followup PR.

Part of SQL-932

Verification

In followup PRs, we'll start building our builtin relations on these shared mvs. It seems unnecessary to create individual SLTs for these relations given these will be tested in the SLTs of downstream builtin relations

@SangJunBak
SangJunBak force-pushed the jun/type-relations branch 2 times, most recently from 187e43c to b493cc1 Compare November 11, 2026 18:47
@SangJunBak
SangJunBak force-pushed the jun/type-relations branch 2 times, most recently from 48c6fac to 8560f45 Compare November 17, 2026 08:58
@linear-code

linear-code Bot commented Nov 17, 2026

Copy link
Copy Markdown

SQL-932

@SangJunBak SangJunBak changed the title draft: Implement type builtin relations SQL-932: Introduce shared parsed-items layer Nov 17, 2026
@SangJunBak
SangJunBak marked this pull request as ready for review November 17, 2026 10:56
@SangJunBak
SangJunBak requested review from a team as code owners November 17, 2026 10:56
@ggevay
ggevay self-requested a review November 17, 2026 11:10
@SangJunBak
SangJunBak added this pull request to stack #40844 November 17, 2026 12:16
@aljoscha

Copy link
Copy Markdown
Contributor

We introduce these shared materialized views such that we can eventually replace our builtin materialized views with views built on top of them. Removing the other builtin mvs will come in a followup PR. We do

What's up with the end there? 🤔

@ggevay ggevay left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments inline. The important ones are the SQL-948 blocker, the mz_error_if_null stuff, and the item fields that readers need.

SUPPORT_SELECT,
};

// TODO(SQL-932): every builtin relation but `mz_catalog_state` and the audit

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, there is SQL-948, which I think we should treat as a blocker for this PR: in a 0dt upgrade, a new builtin MV such as mz_catalog_items, and everything that reads it, only hydrates after promotion.

With #40843 on top, those readers include the 12 catalog-server indexes that depend on mz_types, such as mz_objects_ind, mz_show_columns_ind and pg_type_all_databases_ind. (Shipping the readers a release later would avoid this problem on Cloud, but self-managed upgrades can skip releases.)

@ggevay ggevay Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a draft PR for SQL-948:
#40903
I'll try to undraft it today.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, will undraft it tomorrow.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries and thanks for working on it to unblock me!

Comment thread src/catalog/src/builtin/mz_internal.rs Outdated
access: vec![],
ontology: None,
});
/// Every user-defined catalog item.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Temporary items and mz_internal.mz_analytics (s851) are Item rows too, so they show up here:

Suggested change
/// Every user-defined catalog item.
/// Every catalog item that has an `Item` record: user items, including temporary ones, and
/// runtime-alterable builtins such as `mz_internal.mz_analytics`.

Comment thread src/catalog/src/builtin/mz_internal.rs Outdated
Comment on lines +90 to +92
/// A relation reading this filters on `type` and if a row's DDL fails to parse,
/// it populates the column `details` as null and `decode_error` with an error message. Readers of this
/// should raise the error through `mz_unsafe.mz_error_if_null`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mz_error_if_null would be a nice thing to use here in theory, but unfortunately mz_error_if_null is fraught with evaluation order issues.

So I'd suggest not using it in this PR: mz_catalog_items could decode with parse_catalog_create_sql, as the relations that read mz_catalog_raw do today. A row that doesn't decode then fails every reader of the layer, much like it fails the relations that parse every item today, and try_decode_catalog_create_sql isn't needed.

For example, a specific issue here would be that mz_error_if_null only fires when a query evaluates the column it wraps. With a user view of the shape described here (WHERE type = 'source', mz_error_if_null(details, decode_error)->>'cluster_id') and one bad source row:

  • SELECT * fails, but SELECT id and count(*) return the bad row without an error.
  • With an index on the view, every peek fails, including lookups of good rows. SQL-495: Convert the type system tables to catalog views #40843's mz_types is such an indexed view (mz_types_ind).
  • WHERE mz_error_if_null(details, decode_error) IS NOT NULL is optimized away, since the function never returns NULL.

For reference, mz_error_if_null's known issues, open and closed:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah that's unfortunate, but makes sense!

SELECT
mz_internal.parse_catalog_id(data->'key'->'gid') AS id,
(data->'value'->>'oid')::oid AS oid,
mz_internal.parse_catalog_id(data->'value'->'schema_id') AS schema_id,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this also carry ephemeral_owner_session and global_id? Readers need them:

  • Temporary items are Item rows too, and here they get schema_id u0. mz_tables and mz_views report 0 for them, based on ephemeral_owner_session.
  • mz_kafka_sources reads global_id.

Adding columns later would need another replacement step for this MV.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call! Makes sense to me

Comment thread src/catalog/src/builtin/mz_internal.rs Outdated
COALESCE('s' || (data->'value'->>'catalog_id'), '') AS id,
COALESCE('s' || (data->'value'->>'global_id'), '') AS global_id
FROM mz_internal.mz_catalog_state
WHERE kind = 'GidMapping'",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this leave out runtime-alterable builtins, as mz_object_global_ids does with the <RUNTIME-ALTERABLE> fingerprint? mz_internal.mz_analytics (s851) has both a GidMapping and an Item row, so it is in this view and in mz_catalog_items, and readers that combine the two would list it twice.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

/// item kind of the parsed statement, and NULL when the statement does not
/// parse or defines no catalog item.
#[sqlfunc]
fn try_decode_catalog_create_sql<'a>(a: &'a str) -> Jsonb {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing in CI reads rows of the new relations or results of this function, since the goldens pin only plans and names. Could you add an SLT file along these lines?

statement ok
CREATE TABLE t (a INT DEFAULT 1, b TEXT)

statement ok
CREATE VIEW v AS SELECT 'secret' AS s, a FROM t

statement ok
CREATE DEFAULT INDEX ON v

# mz_catalog_items decodes every item the way the existing decoders do.

simple conn=mz_system,user=mz_system
SELECT
    count(*) FILTER (WHERE name IN ('t', 'v', 'v_primary_idx')),
    count(*) FILTER (
        WHERE type IS DISTINCT FROM details->>'type'
        OR details IS DISTINCT FROM mz_internal.parse_catalog_create_sql(create_sql)
        OR redacted_create_sql IS DISTINCT FROM mz_internal.redact_sql(create_sql)
        OR decode_error IS NOT NULL
    )
FROM mz_internal.mz_catalog_items;
----
3,0
COMPLETE 1

# Its identity columns match mz_objects, and every user object is in it.

simple conn=mz_system,user=mz_system
WITH items AS (SELECT id, oid, schema_id, name, owner_id, type FROM mz_internal.mz_catalog_items),
objects AS (SELECT id, oid, schema_id, name, owner_id, type FROM mz_catalog.mz_objects)
SELECT count(*) FROM (
    (SELECT * FROM items EXCEPT ALL SELECT * FROM objects)
    UNION ALL
    (SELECT * FROM objects WHERE id LIKE 'u%' EXCEPT ALL SELECT * FROM items)
);
----
0
COMPLETE 1

# mz_builtin_object_mappings names every builtin object it maps.

simple conn=mz_system,user=mz_system
SELECT count(*) > 800, count(*) FILTER (WHERE o.id IS NULL OR o.name <> m.name OR s.name <> m.schema_name)
FROM mz_internal.mz_builtin_object_mappings m
LEFT JOIN (SELECT DISTINCT id, name, schema_id FROM mz_catalog.mz_objects) o ON o.id = m.id
LEFT JOIN mz_catalog.mz_schemas s ON s.id = o.schema_id;
----
t,0
COMPLETE 1

# Decode failures become data instead of errors.

query T
SELECT mz_internal.try_decode_catalog_create_sql('CREATE MATERIALIZED VIEW "materialize"."public"."mv" AS SELECT 1')::text
----
{"error":"missing IN CLUSTER","type":"materialized-view"}

query T
SELECT mz_internal.try_decode_catalog_create_sql('SELECT 1')::text
----
{"error":"not a CREATE item statement","type":null}

@SangJunBak

Copy link
Copy Markdown
Contributor Author

What's up with the end there? 🤔

Oops! Removed

We add a materialized view `mz_catalog_state` that filters out audit log entries. We plan to use this for our other builtin relations such that we don't hydrate from the audit log each time. It would also ensure queries ran outside of mz_catalog_server don't query all of the audit log.
mz_catalog_items is a materialized view over the item records of the durable catalog, derived from mz_catalog_state. Although the additional materialized view creates an additional persist round trip, it saves us from reparsing every item row. 

Since many of our builtin relations join on this relation or don't need the item's DDL text, we create a narrow indexed view that excludes the DDL statement. This way we avoid having to store text-heavy fields in memory.

An issue with having one unified relation is when we call `parse_catalog_create_sql`, one parse failure could poison the entire relation.  Thus we create `try_decode_catalog_create_sql` that  exposes an `ok` and `error` field and allows downstream builtin relations to error via `mz_unsafe.mz_error_if_null`. 

We also need to return the item type even when parsing fails. Thus we extract the `item_type` match statement into a function and reuse it.
Creates mz_internal.mz_builtin_object_mappings, an indexed view over the
builtin GidMapping records of mz_catalog_state. With this and mz_catalog_items, many builtin relations can just be views over a join of these two. Currently all builtin relations define this as a CTE in their own materialized view definition. We ideally want to refactor each of these, but we'll wait until fingerprint changes don't require a builtin replacement.
mz_unsafe.mz_error_if_null raises only when a query evaluates the column
it wraps, and its evaluation order is not guaranteed (SQL-795).  Thus we revert the change of adding `try_decode_catalog_create_sql` and go back to using `parse_catalog_create_sql`.
This is a revert since we no longer need to extract `item_type` as a separate function
Readers of the items layer need both. mz_tables and mz_views report a
temporary item's schema as 0 based on ephemeral_owner_session, and
mz_kafka_sources reads global_id. mz_catalog_item_metadata carries them
too.
…ppings

A runtime-alterable builtin such as mz_internal.mz_analytics has both a
GidMapping record and an Item record, so it was in
mz_builtin_object_mappings and in mz_catalog_items, and a reader
combining the two would list it twice. Filter on the
<RUNTIME-ALTERABLE> fingerprint, as mz_object_global_ids does.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants