Repository navigation
SQL-932: Introduce shared parsed-items layer - #40067
SangJunBak wants to merge 11 commits into
Conversation
187e43c to
b493cc1
Compare
48c6fac to
8560f45
Compare
8560f45 to
ed2311a
Compare
What's up with the end there? 🤔 |
ggevay
left a comment
There was a problem hiding this comment.
Some comments inline. The important ones are the SQL-948 blocker, the mz_error_if_null stuff, and the item fields that readers need.
| SUPPORT_SELECT, | ||
| }; | ||
|
|
||
| // TODO(SQL-932): every builtin relation but `mz_catalog_state` and the audit |
There was a problem hiding this comment.
Unfortunately, there is SQL-948, which I think we should treat as a blocker for this PR: in a 0dt upgrade, a new builtin MV such as mz_catalog_items, and everything that reads it, only hydrates after promotion.
With #40843 on top, those readers include the 12 catalog-server indexes that depend on mz_types, such as mz_objects_ind, mz_show_columns_ind and pg_type_all_databases_ind. (Shipping the readers a release later would avoid this problem on Cloud, but self-managed upgrades can skip releases.)
There was a problem hiding this comment.
Sorry, will undraft it tomorrow.
There was a problem hiding this comment.
No worries and thanks for working on it to unblock me!
| access: vec![], | ||
| ontology: None, | ||
| }); | ||
| /// Every user-defined catalog item. |
There was a problem hiding this comment.
Temporary items and mz_internal.mz_analytics (s851) are Item rows too, so they show up here:
| /// Every user-defined catalog item. | |
| /// Every catalog item that has an `Item` record: user items, including temporary ones, and | |
| /// runtime-alterable builtins such as `mz_internal.mz_analytics`. |
| /// A relation reading this filters on `type` and if a row's DDL fails to parse, | ||
| /// it populates the column `details` as null and `decode_error` with an error message. Readers of this | ||
| /// should raise the error through `mz_unsafe.mz_error_if_null`. |
There was a problem hiding this comment.
mz_error_if_null would be a nice thing to use here in theory, but unfortunately mz_error_if_null is fraught with evaluation order issues.
So I'd suggest not using it in this PR: mz_catalog_items could decode with parse_catalog_create_sql, as the relations that read mz_catalog_raw do today. A row that doesn't decode then fails every reader of the layer, much like it fails the relations that parse every item today, and try_decode_catalog_create_sql isn't needed.
For example, a specific issue here would be that mz_error_if_null only fires when a query evaluates the column it wraps. With a user view of the shape described here (WHERE type = 'source', mz_error_if_null(details, decode_error)->>'cluster_id') and one bad source row:
SELECT *fails, butSELECT idandcount(*)return the bad row without an error.- With an index on the view, every peek fails, including lookups of good rows. SQL-495: Convert the type system tables to catalog views #40843's
mz_typesis such an indexed view (mz_types_ind). WHERE mz_error_if_null(details, decode_error) IS NOT NULLis optimized away, since the function never returns NULL.
For reference, mz_error_if_null's known issues, open and closed:
- Errors where a guard should prevent them: database-issues#6049 and database-issues#5826 (both open), database-issues#6004, database-issues#6012.
- Predicate pushdown moved it through a join: database-issues#6258, fixed by Don't push
error_if_nullthrough a Join #20784. - Panics on a NULL error message: database-issues#5817, fixed by sql: Prevent panic in mz_error_if_null #19575, and database-issues#5280.
- A guard written as
mz_error_if_null(...) IS NULLis folded away: Reworkhas_cluster_privilegeto error intentionally #28034.
There was a problem hiding this comment.
Ah that's unfortunate, but makes sense!
| SELECT | ||
| mz_internal.parse_catalog_id(data->'key'->'gid') AS id, | ||
| (data->'value'->>'oid')::oid AS oid, | ||
| mz_internal.parse_catalog_id(data->'value'->'schema_id') AS schema_id, |
There was a problem hiding this comment.
Should this also carry ephemeral_owner_session and global_id? Readers need them:
- Temporary items are
Itemrows too, and here they getschema_idu0.mz_tablesandmz_viewsreport0for them, based onephemeral_owner_session. mz_kafka_sourcesreadsglobal_id.
Adding columns later would need another replacement step for this MV.
There was a problem hiding this comment.
Good call! Makes sense to me
| COALESCE('s' || (data->'value'->>'catalog_id'), '') AS id, | ||
| COALESCE('s' || (data->'value'->>'global_id'), '') AS global_id | ||
| FROM mz_internal.mz_catalog_state | ||
| WHERE kind = 'GidMapping'", |
There was a problem hiding this comment.
Should this leave out runtime-alterable builtins, as mz_object_global_ids does with the <RUNTIME-ALTERABLE> fingerprint? mz_internal.mz_analytics (s851) has both a GidMapping and an Item row, so it is in this view and in mz_catalog_items, and readers that combine the two would list it twice.
| /// item kind of the parsed statement, and NULL when the statement does not | ||
| /// parse or defines no catalog item. | ||
| #[sqlfunc] | ||
| fn try_decode_catalog_create_sql<'a>(a: &'a str) -> Jsonb { |
There was a problem hiding this comment.
Nothing in CI reads rows of the new relations or results of this function, since the goldens pin only plans and names. Could you add an SLT file along these lines?
statement ok
CREATE TABLE t (a INT DEFAULT 1, b TEXT)
statement ok
CREATE VIEW v AS SELECT 'secret' AS s, a FROM t
statement ok
CREATE DEFAULT INDEX ON v
# mz_catalog_items decodes every item the way the existing decoders do.
simple conn=mz_system,user=mz_system
SELECT
count(*) FILTER (WHERE name IN ('t', 'v', 'v_primary_idx')),
count(*) FILTER (
WHERE type IS DISTINCT FROM details->>'type'
OR details IS DISTINCT FROM mz_internal.parse_catalog_create_sql(create_sql)
OR redacted_create_sql IS DISTINCT FROM mz_internal.redact_sql(create_sql)
OR decode_error IS NOT NULL
)
FROM mz_internal.mz_catalog_items;
----
3,0
COMPLETE 1
# Its identity columns match mz_objects, and every user object is in it.
simple conn=mz_system,user=mz_system
WITH items AS (SELECT id, oid, schema_id, name, owner_id, type FROM mz_internal.mz_catalog_items),
objects AS (SELECT id, oid, schema_id, name, owner_id, type FROM mz_catalog.mz_objects)
SELECT count(*) FROM (
(SELECT * FROM items EXCEPT ALL SELECT * FROM objects)
UNION ALL
(SELECT * FROM objects WHERE id LIKE 'u%' EXCEPT ALL SELECT * FROM items)
);
----
0
COMPLETE 1
# mz_builtin_object_mappings names every builtin object it maps.
simple conn=mz_system,user=mz_system
SELECT count(*) > 800, count(*) FILTER (WHERE o.id IS NULL OR o.name <> m.name OR s.name <> m.schema_name)
FROM mz_internal.mz_builtin_object_mappings m
LEFT JOIN (SELECT DISTINCT id, name, schema_id FROM mz_catalog.mz_objects) o ON o.id = m.id
LEFT JOIN mz_catalog.mz_schemas s ON s.id = o.schema_id;
----
t,0
COMPLETE 1
# Decode failures become data instead of errors.
query T
SELECT mz_internal.try_decode_catalog_create_sql('CREATE MATERIALIZED VIEW "materialize"."public"."mv" AS SELECT 1')::text
----
{"error":"missing IN CLUSTER","type":"materialized-view"}
query T
SELECT mz_internal.try_decode_catalog_create_sql('SELECT 1')::text
----
{"error":"not a CREATE item statement","type":null}
Oops! Removed |
We add a materialized view `mz_catalog_state` that filters out audit log entries. We plan to use this for our other builtin relations such that we don't hydrate from the audit log each time. It would also ensure queries ran outside of mz_catalog_server don't query all of the audit log.
mz_catalog_items is a materialized view over the item records of the durable catalog, derived from mz_catalog_state. Although the additional materialized view creates an additional persist round trip, it saves us from reparsing every item row. Since many of our builtin relations join on this relation or don't need the item's DDL text, we create a narrow indexed view that excludes the DDL statement. This way we avoid having to store text-heavy fields in memory. An issue with having one unified relation is when we call `parse_catalog_create_sql`, one parse failure could poison the entire relation. Thus we create `try_decode_catalog_create_sql` that exposes an `ok` and `error` field and allows downstream builtin relations to error via `mz_unsafe.mz_error_if_null`. We also need to return the item type even when parsing fails. Thus we extract the `item_type` match statement into a function and reuse it.
Creates mz_internal.mz_builtin_object_mappings, an indexed view over the builtin GidMapping records of mz_catalog_state. With this and mz_catalog_items, many builtin relations can just be views over a join of these two. Currently all builtin relations define this as a CTE in their own materialized view definition. We ideally want to refactor each of these, but we'll wait until fingerprint changes don't require a builtin replacement.
ed2311a to
081441a
Compare
mz_unsafe.mz_error_if_null raises only when a query evaluates the column it wraps, and its evaluation order is not guaranteed (SQL-795). Thus we revert the change of adding `try_decode_catalog_create_sql` and go back to using `parse_catalog_create_sql`.
This is a revert since we no longer need to extract `item_type` as a separate function
Readers of the items layer need both. mz_tables and mz_views report a temporary item's schema as 0 based on ephemeral_owner_session, and mz_kafka_sources reads global_id. mz_catalog_item_metadata carries them too.
…ppings A runtime-alterable builtin such as mz_internal.mz_analytics has both a GidMapping record and an Item record, so it was in mz_builtin_object_mappings and in mz_catalog_items, and a reader combining the two would list it twice. Filter on the <RUNTIME-ALTERABLE> fingerprint, as mz_object_global_ids does.
081441a to
4d7de5c
Compare
For an example of how these are intended to be used, this is the definition of
mz_typesfor a later PR in the stack:WITH user_types AS ( SELECT id, oid, schema_id, name, COALESCE(mz_unsafe.mz_error_if_null(details, decode_error)->>'category', '') AS category, owner_id, privileges, create_sql, redacted_create_sql FROM mz_internal.mz_catalog_items WHERE type = 'type' ), builtin_types AS ( SELECT m.id, t.oid, s.id AS schema_id, t.name, t.category, '{MZ_SYSTEM_ROLE_ID}' AS owner_id, t.privileges, NULL::text AS create_sql, NULL::text AS redacted_create_sql FROM mz_internal.mz_builtin_types t JOIN mz_internal.mz_builtin_object_mappings m ON m.schema_name = t.schema_name AND m.name = t.name AND m.object_type = {object_type} JOIN mz_catalog.mz_schemas s ON s.name = t.schema_name AND s.database_id IS NULL ) SELECT * FROM user_types UNION ALL SELECT * FROM builtin_types", object_type = gid_mapping_object_type(CatalogItemType::Type), )Motivation
We introduce these shared materialized views such that we can eventually replace our builtin materialized views with views built on top of them. Removing the other builtin mvs will come in a followup PR.
Part of SQL-932
Verification
In followup PRs, we'll start building our builtin relations on these shared mvs. It seems unnecessary to create individual SLTs for these relations given these will be tested in the SLTs of downstream builtin relations