Skip to content

storage: support AWS IAM authentication for PostgreSQL connections - #40911

Open
val-materialize wants to merge 6 commits into
mainfrom
valeriechiang/ss-593-postgres-iam-auth
Open

val-materialize wants to merge 6 commits into
mainfrom
valeriechiang/ss-593-postgres-iam-auth

Conversation

@val-materialize

@val-materialize val-materialize commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Part of SS-593. IAM authentication works today for RDS MySQL connections only. Users with PostgreSQL on Amazon RDS or Aurora have asked for the same, so they can stop managing a database password.

Builds on #40909 (merged), which moved the token signer into mz-aws-util.

Description

CREATE CONNECTION ... TO POSTGRES now accepts AWS CONNECTION:

CREATE CONNECTION pg TO POSTGRES (
    HOST 'instance.foo000.us-west-1.rds.amazonaws.com',
    DATABASE 'postgres',
    USER 'materialize',
    AWS CONNECTION aws_rds_postgres,
    SSL MODE 'verify_full'
);
  • Planner. Accepts the option for PostgreSQL. Rejects it together with PASSWORD, and requires SSL, because the token is sent as the password. Same rules and error messages as MySQL.
  • PostgresConnection. Gains an optional aws_connection, and loads the AWS SDK config when it builds the connection config.
  • mz_postgres_util::Config. Takes an optional SdkConfig. When set, every call to connect signs a fresh token and uses it as the password. A token is valid for 15 minutes and is only checked when a connection opens.
  • Tunnels. The token is signed for the database's own host and port, also when the connection goes through an SSH tunnel or AWS PrivateLink.
  • Docs. Split out to doc: document AWS IAM authentication for PostgreSQL connections #40923, because merging to main publishes docs immediately. That PR merges when the release containing this change is live.

Things for reviewers to weigh in on:

  • There is no feature flag, matching MySQL. Say so if this should be gated.
  • PostgresConnection gains a field. I believe this is safe in the same way the MySQL field was, but please check version skew between environmentd and clusterd.

Verification

  • New test/pg-cdc/create-alter-iam-connection.td covers planning: create, the two ALTER CONNECTION rejections, and the create-time errors.
  • CI is green on the current head, including the builds, rust-cargo-test, and the postgres-cdc and mysql-cdc suites. Cargo.lock was edited by hand, and CI accepted it.
  • Tested end to end against RDS for PostgreSQL 17.11 and Aurora PostgreSQL 17.11, using the CI build of 2900dbc14. Passed, including replication past the 15-minute token lifetime and reconnecting with a fresh token. Receipt: storage: support AWS IAM authentication for PostgreSQL connections #40911 (comment) . The only change since that commit removes the docs from this PR. SSH tunnel and PrivateLink were not tested.

🤖 Generated with Claude Code

val-materialize and others added 2 commits November 18, 2026 15:21
The RDS IAM auth token generator lived in mz-mysql-util, but nothing in
it is specific to MySQL: it signs a request for a host, port, and
database user, and RDS and Aurora accept the same token for MySQL and
PostgreSQL.

Move it to mz-aws-util behind a new `rds` feature and make it public,
so that other crates (notably mz-postgres-util) can use it for IAM
authentication. mz-mysql-util now calls the shared function. No
behavior change.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Allow `AWS CONNECTION` on `CREATE CONNECTION ... TO POSTGRES`, so that
a connection can authenticate to Amazon RDS or Aurora PostgreSQL with an
IAM authentication token in place of a password. This matches what
MySQL connections already support.

* Planner: accept the `AWS CONNECTION` option for PostgreSQL. Reject it
  together with `PASSWORD`, and require SSL, because the token is sent
  as the password.
* `PostgresConnection` gains an optional `aws_connection` and loads the
  AWS SDK config when it builds the connection config.
* `mz_postgres_util::Config` takes an optional `SdkConfig`. When set,
  every new connection signs a fresh token for the database's own
  host and port, including when it connects through an SSH tunnel or
  AWS PrivateLink.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Nov 18, 2026

Copy link
Copy Markdown

SS-593

Base automatically changed from valeriechiang/rds-iam-auth-shared to main November 18, 2026 21:03
@val-materialize
val-materialize marked this pull request as ready for review November 18, 2026 21:03
@val-materialize
val-materialize requested review from a team as code owners November 18, 2026 21:03
@val-materialize
val-materialize removed request for a team November 18, 2026 21:03
Materialize ingests over a replication connection, and Amazon RDS only
accepts IAM authentication on replication connections when
rds.iam_auth_for_replication is set. Also list the rds_replication
role.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@val-materialize
val-materialize requested review from a team and martykulma November 18, 2026 21:39

@martykulma martykulma left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ends up causing mz cli and mz_debug to pull in the AWS crate.

Not sure if that's a problem, but it can be avoided:

  • set default-features=false for mz-postgres-util in those crates
  • make mz-ore a non-optional dependency of mz-postgres-util and drop it from the tunnel list (every feature pulls in tunnel, so they all required mz-ore)

It needs to be tested against an actual RDS PG and Aurora to make sure things are working correctly before merge. I'll hold off on approving until we can get an E2E test.

mz-postgres-util's tunnel feature now pulls in the AWS crates for IAM
authentication. mz and mz-debug only use the query helpers, so depend
on mz-postgres-util without default features there.

To make that build, mz-ore becomes a non-optional dependency of
mz-postgres-util: the crate re-exports mz_ore::sql unconditionally, so
it never compiled without it. The tunnel feature now only adds mz-ore's
async feature.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@val-materialize
val-materialize requested a review from a team as a code owner November 18, 2026 21:46
@val-materialize

Copy link
Copy Markdown
Contributor Author

Thanks, both make sense.

AWS crates in mz / mz-debug: fixed in 2900dbc, following your two suggestions. Both crates now use mz-postgres-util with default-features = false, and mz-ore is a non-optional dependency of mz-postgres-util, with the tunnel feature only adding mz-ore/async. I left mz-deploy alone since it already depends on mz-aws-util. CI is still running on that commit.

E2E test: agreed, this should not merge without it. I will try to test the CI build against RDS PG and Aurora myself, but I might need some help getting it deployed. The steps are on SS-593; the main thing I want to confirm is that a source stays healthy past the 15-minute token lifetime.

@def-

def- commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- Validation succeeds when IAM replication is disabled

src/storage-types/src/connections.rs:2686

With IAM database authentication and logical replication enabled, leaving rds.iam_auth_for_replication at its default of 0 lets VALIDATE CONNECTION succeed even though the source cannot ingest. The same false positive reaches CREATE SOURCE, which uses this validator during planning.

Details

The validator opens an ordinary SQL session and checks the WAL level, WAL senders, and available slots. All three checks can pass. The replication worker then calls connect_replication at src/storage/src/source/postgres/replication.rs:296, where the upstream rejects IAM authentication for that session. For IAM connections, check the replication-specific setting during validation or verify that a logical replication session can authenticate. Upstream replication authentication requirements

2. MEDIUM -- Role credential fetching runs on the Timely worker

src/postgres-util/src/tunnel.rs:215

With assume-role authentication, opening a PostgreSQL connection now polls a network credential request directly inside the source operator. On a busy Timely worker, delays between polls can exhaust the STS request timeout and restart a source whose credentials and database are healthy.

Details

The replication worker requests InTask::Yes at src/storage/src/source/postgres/replication.rs:290. Loading the SDK configuration honors that setting, but the actual credential fetch happens later in rds_auth_token at src/aws-util/src/rds.rs:70, before the PostgreSQL connect future enters its Tokio task. Run token generation through run_in_task_if(self.in_task, ...) as well so credential I/O can progress independently of the worker.

3. MEDIUM -- The verified TLS example omits the database CA

doc/user/content/sql/create-connection.md:916

The example selects verify_full without supplying the CA that signs the database certificate. On a stock installation, the documented CREATE CONNECTION fails certificate verification before IAM authentication starts, even with the correct endpoint, role, and database settings.

Details

The production image ships the distribution CA bundle (misc/images/distroless-prod-base/Dockerfile:58). The regional database roots are outside that bundle, and the PostgreSQL connector loads additional roots from SSL CERTIFICATE AUTHORITY (src/tls-util/src/lib.rs:79). Show how to supply the database CA bundle through that option while keeping hostname verification enabled. Upstream TLS setup

Copy link
Copy Markdown
Contributor Author

Is medium… good?

Copy link
Copy Markdown
Contributor Author

Running e2e tests. Will keep you posted

@val-materialize

Copy link
Copy Markdown
Contributor Author

E2E test against RDS PostgreSQL and Aurora PostgreSQL: passed

Tested the CI-built image for this PR's head (v26.52.0-dev.0 (2900dbc14)), running on an EC2 machine in the scratch account, against a real RDS for PostgreSQL 17.11 instance and a real Aurora PostgreSQL 17.11 cluster (serverless v2). Both were private, in the same VPC as the machine, with IAM database authentication enabled, rds.logical_replication = 1, and rds.iam_auth_for_replication = 1.

The database user (mz_iam) has no password and holds only rds_iam and rds_replication. Materialize authenticated through an AWS connection using ASSUME ROLE ARN, to a role allowed only rds-db:connect for that user. No access keys were used.

Check RDS PostgreSQL Aurora PostgreSQL
CREATE CONNECTION ... AWS CONNECTION, SSL MODE 'require', validated pass pass
SSL MODE 'verify_full' with the Amazon RDS global CA bundle, validated pass pass
CREATE SOURCE over a replication connection reaches running pass pass
Snapshot of existing rows pass (3/3) pass (3/3)
Live replication, one upstream row every 30s pass pass
Token expiry: same replication connection still streaming after the 15-minute token lifetime pass (same backend pid at 23 min, rows matching upstream) pass (same backend pid at 18 min, rows matching upstream)
Reconnect: restart Materialize after the original tokens expired pass (new upstream connection, replication resumes) pass (new upstream connection, replication resumes)
Recovery after rds.iam_auth_for_replication goes 0 and back to 1, with no intervention pass pass

Planner rejections behave as intended: AWS CONNECTION without SSL and AWS CONNECTION with PASSWORD are both rejected with the same messages as MySQL.

What this answers

  • Token lifetime. The token is only checked when a connection opens. A replication connection that outlives its token keeps streaming, on both RDS and Aurora. No refresh mechanism is needed for a healthy connection.
  • Fresh token on reconnect. After a restart, long after the first tokens expired, both sources reconnected. So each new connection signs a new token rather than reusing one.
  • Aurora. Aurora PostgreSQL has the rds.iam_auth_for_replication cluster parameter and behaves the same as RDS.

Things worth knowing

  • With rds.iam_auth_for_replication = 0, VALIDATE CONNECTION still passes but the source stalls. An ordinary connection authenticates with IAM; the replication connection does not, and the source reports FATAL: password authentication failed for user "mz_iam". So a user who misses that parameter gets a connection that validates and a source that fails with a message about passwords. The docs in this PR call out the parameter; a clearer error or a validation check could be a follow-up.
  • SHOW rds.iam_auth_for_replication is not supported inside Postgres. The value is only visible in the parameter group.
  • bad_user in the log is the master user, which does not have rds_iam, so RDS falls back to password auth and rejects the token.

Not covered

  • SSH tunnel and AWS PrivateLink. The token is signed for the database's own host and port in both cases, but neither path was exercised.
  • PostgreSQL versions other than 17.
  • Runs longer than about 30 minutes.

The test resources (EC2 machine, RDS instance, Aurora cluster, IAM roles) were torn down right after the run.

Full log: every statement, its output, and UTC timestamps (AWS account ID redacted)
=== RDS PostgreSQL: verify parameters; create IAM user, table, publication (as master user)
=== started 2026-11-18T22:07:29Z
SELECT version();
                                                    version                                                    
---------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.11 on aarch64-unknown-linux-gnu, compiled by aarch64-unknown-linux-gnu-gcc (GCC) 12.4.0, 64-bit
(1 row)

SHOW wal_level;
 wal_level 
-----------
 logical
(1 row)

SHOW rds.logical_replication;
 rds.logical_replication 
-------------------------
 on
(1 row)

SHOW rds.iam_auth_for_replication;
ERROR:  unrecognized configuration parameter "rds.iam_auth_for_replication"
[stderr] failed to run commands: exit status 3
[status=Failed rc=3]

=== RDS PostgreSQL (retry; SHOW of rds.iam_auth_for_replication is not supported, read from the parameter group instead)
=== started 2026-11-18T22:07:39Z
rds.iam_auth_for_replication	1	dynamic
rds.logical_replication	1	static
SELECT version();
                                                    version                                                    
---------------------------------------------------------------------------------------------------------------
 PostgreSQL 17.11 on aarch64-unknown-linux-gnu, compiled by aarch64-unknown-linux-gnu-gcc (GCC) 12.4.0, 64-bit
(1 row)

SHOW wal_level;
 wal_level 
-----------
 logical
(1 row)

SHOW rds.logical_replication;
 rds.logical_replication 
-------------------------
 on
(1 row)

CREATE USER mz_iam;
CREATE ROLE
GRANT rds_iam TO mz_iam;
GRANT ROLE
GRANT rds_replication TO mz_iam;
GRANT ROLE
CREATE TABLE public.e2e (id int PRIMARY KEY, note text, ts timestamptz DEFAULT now());
CREATE TABLE
ALTER TABLE public.e2e REPLICA IDENTITY FULL;
ALTER TABLE
INSERT INTO public.e2e (id, note) VALUES (1, 'snapshot-1'), (2, 'snapshot-2'), (3, 'snapshot-3');
INSERT 0 3
GRANT USAGE ON SCHEMA public TO mz_iam;
GRANT
GRANT SELECT ON public.e2e TO mz_iam;
GRANT
CREATE PUBLICATION mz_source FOR TABLE public.e2e;
CREATE PUBLICATION
SELECT r.rolname AS mz_iam_member_of FROM pg_auth_members m JOIN pg_roles r ON r.oid = m.roleid JOIN pg_roles u ON u.oid = m.member WHERE u.rolname = 'mz_iam' ORDER BY 1;
 mz_iam_member_of 
------------------
 rds_iam
 rds_replication
(2 rows)


[status=Success rc=0]

=== Materialize: CREATE CONNECTION to RDS PostgreSQL with AWS CONNECTION (IAM auth, validated), CREATE SOURCE, snapshot
=== started 2026-11-18T22:07:55Z
SELECT mz_version();
         mz_version         
----------------------------
 v26.52.0-dev.0 (2900dbc14)
(1 row)

SHOW CREATE CONNECTION aws_rds;
            name            |                                             create_sql                                              
----------------------------+-----------------------------------------------------------------------------------------------------
 materialize.public.aws_rds | CREATE CONNECTION materialize.public.aws_rds                                                       +
                            | TO AWS                                                                                             +
                            |     (ASSUME ROLE ARN = 'arn:aws:iam::<acct>:role/ss-593-e2e-Customer', REGION = 'us-east-1');
(1 row)

CREATE CONNECTION pg_rds TO POSTGRES (
    HOST 'ss-593-iam-e2e.ctthmav6dsti.us-east-1.rds.amazonaws.com',
    PORT 5432,
    DATABASE 'postgres',
    USER 'mz_iam',
    AWS CONNECTION aws_rds,
    SSL MODE 'require'
);
CREATE CONNECTION
VALIDATE CONNECTION pg_rds;
VALIDATE CONNECTION
CREATE SOURCE pg_rds_src FROM POSTGRES CONNECTION pg_rds (PUBLICATION 'mz_source');
CREATE SOURCE
CREATE TABLE e2e_rds FROM SOURCE pg_rds_src (REFERENCE public.e2e);
CREATE TABLE
SELECT name, type, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_rds_src', 'e2e_rds') ORDER BY name;
    name    |   type   | status  | error 
------------+----------+---------+-------
 e2e_rds    | table    | running | 
 pg_rds_src | postgres | running | 
(2 rows)

SELECT id, note FROM e2e_rds ORDER BY id;
 id |    note    
----+------------
  1 | snapshot-1
  2 | snapshot-2
  3 | snapshot-3
(3 rows)


[status=Success rc=0]

=== RDS PostgreSQL: start upstream writer (1 row / 30s); record the replication connection (user, backend_start)
=== started 2026-11-18T22:08:45Z
writer rds started
SELECT now() AT TIME ZONE 'utc' AS utc_now, pid, usename, application_name, backend_start AT TIME ZONE 'utc' AS backend_start_utc, state FROM pg_stat_replication
          utc_now           | pid  | usename | application_name |     backend_start_utc      |   state   
----------------------------+------+---------+------------------+----------------------------+-----------
 2026-11-18 22:08:56.897675 | 1626 | mz_iam  | materialize      | 2026-11-18 22:08:03.608796 | streaming
(1 row)

SELECT slot_name, active, active_pid FROM pg_replication_slots
                  slot_name                   | active | active_pid 
----------------------------------------------+--------+------------
 materialize_75be7e2780ef4e04a51eb96901df9d36 | t      |       1626
(1 row)


[status=Success rc=0]

=== Aurora PostgreSQL: verify parameters; create IAM user, table, publication (as master user)
=== started 2026-11-18T22:12:48Z
rds.iam_auth_for_replication	1	dynamic
rds.logical_replication	1	static
SELECT version();
                              version                              
-------------------------------------------------------------------
 PostgreSQL 17.11 on aarch64-linux, compiled by gcc-10.5.0, 64-bit
(1 row)

SHOW wal_level;
 wal_level 
-----------
 logical
(1 row)

SHOW rds.logical_replication;
 rds.logical_replication 
-------------------------
 on
(1 row)

CREATE USER mz_iam;
CREATE ROLE
GRANT rds_iam TO mz_iam;
GRANT ROLE
GRANT rds_replication TO mz_iam;
GRANT ROLE
CREATE TABLE public.e2e (id int PRIMARY KEY, note text, ts timestamptz DEFAULT now());
CREATE TABLE
ALTER TABLE public.e2e REPLICA IDENTITY FULL;
ALTER TABLE
INSERT INTO public.e2e (id, note) VALUES (1, 'snapshot-1'), (2, 'snapshot-2'), (3, 'snapshot-3');
INSERT 0 3
GRANT USAGE ON SCHEMA public TO mz_iam;
GRANT
GRANT SELECT ON public.e2e TO mz_iam;
GRANT
CREATE PUBLICATION mz_source FOR TABLE public.e2e;
CREATE PUBLICATION
SELECT r.rolname AS mz_iam_member_of FROM pg_auth_members m JOIN pg_roles r ON r.oid = m.roleid JOIN pg_roles u ON u.oid = m.member WHERE u.rolname = 'mz_iam' ORDER BY 1;
 mz_iam_member_of 
------------------
 rds_iam
 rds_replication
(2 rows)


[status=Success rc=0]

=== Materialize: CREATE CONNECTION to Aurora PostgreSQL with AWS CONNECTION (IAM auth, validated), CREATE SOURCE, snapshot
=== started 2026-11-18T22:13:03Z
CREATE CONNECTION pg_aurora TO POSTGRES (
    HOST 'ss-593-iam-aurora.cluster-ctthmav6dsti.us-east-1.rds.amazonaws.com',
    PORT 5432,
    DATABASE 'postgres',
    USER 'mz_iam',
    AWS CONNECTION aws_rds,
    SSL MODE 'require'
);
CREATE CONNECTION
VALIDATE CONNECTION pg_aurora;
VALIDATE CONNECTION
CREATE SOURCE pg_aurora_src FROM POSTGRES CONNECTION pg_aurora (PUBLICATION 'mz_source');
CREATE SOURCE
CREATE TABLE e2e_aurora FROM SOURCE pg_aurora_src (REFERENCE public.e2e);
CREATE TABLE
SELECT name, type, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_aurora_src', 'e2e_aurora') ORDER BY name;
     name      |   type   | status  | error 
---------------+----------+---------+-------
 e2e_aurora    | table    | running | 
 pg_aurora_src | postgres | running | 
(2 rows)

SELECT id, note FROM e2e_aurora ORDER BY id;
 id |    note    
----+------------
  1 | snapshot-1
  2 | snapshot-2
  3 | snapshot-3
(3 rows)


[status=Success rc=0]

=== Aurora PostgreSQL: start upstream writer (1 row / 30s); record the replication connection (user, backend_start)
=== started 2026-11-18T22:13:40Z
writer aurora started
SELECT now() AT TIME ZONE 'utc' AS utc_now, pid, usename, application_name, backend_start AT TIME ZONE 'utc' AS backend_start_utc, state FROM pg_stat_replication
          utc_now           | pid | usename | application_name |     backend_start_utc      |   state   
----------------------------+-----+---------+------------------+----------------------------+-----------
 2026-11-18 22:13:51.255839 | 919 | mz_iam  | materialize      | 2026-11-18 22:13:06.996922 | streaming
(1 row)

SELECT slot_name, active, active_pid FROM pg_replication_slots
                  slot_name                   | active | active_pid 
----------------------------------------------+--------+------------
 materialize_c1fd5149cafe424ca3d084887cb78030 | t      |        919
(1 row)


[status=Success rc=0]

=== Materialize: SSL MODE verify_full with the Amazon RDS global CA bundle; planner rejections; auth failure for a database user the IAM policy does not allow
=== started 2026-11-18T22:14:29Z
CA bundle from https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem: 114 certificates (stored as secret rds_ca; contents not echoed)
CREATE CONNECTION pg_rds_verify_full TO POSTGRES (
    HOST 'ss-593-iam-e2e.ctthmav6dsti.us-east-1.rds.amazonaws.com', DATABASE 'postgres', USER 'mz_iam',
    AWS CONNECTION aws_rds, SSL MODE 'verify_full', SSL CERTIFICATE AUTHORITY SECRET rds_ca
);
CREATE CONNECTION
VALIDATE CONNECTION pg_rds_verify_full;
VALIDATE CONNECTION
CREATE CONNECTION pg_aurora_verify_full TO POSTGRES (
    HOST 'ss-593-iam-aurora.cluster-ctthmav6dsti.us-east-1.rds.amazonaws.com', DATABASE 'postgres', USER 'mz_iam',
    AWS CONNECTION aws_rds, SSL MODE 'verify_full', SSL CERTIFICATE AUTHORITY SECRET rds_ca
);
CREATE CONNECTION
VALIDATE CONNECTION pg_aurora_verify_full;
VALIDATE CONNECTION
CREATE CONNECTION bad_no_ssl TO POSTGRES (
    HOST 'ss-593-iam-e2e.ctthmav6dsti.us-east-1.rds.amazonaws.com', DATABASE 'postgres', USER 'mz_iam', AWS CONNECTION aws_rds
);
ERROR:  invalid CONNECTION: AWS IAM authentication requires SSL to be enabled
CREATE SECRET dummy_pw AS 'not-a-real-password';
CREATE SECRET
CREATE CONNECTION bad_password TO POSTGRES (
    HOST 'ss-593-iam-e2e.ctthmav6dsti.us-east-1.rds.amazonaws.com', DATABASE 'postgres', USER 'mz_iam', PASSWORD SECRET dummy_pw,
    AWS CONNECTION aws_rds, SSL MODE 'require'
);
ERROR:  invalid CONNECTION: AWS IAM authentication is not supported with password
CREATE CONNECTION bad_user TO POSTGRES (
    HOST 'ss-593-iam-e2e.ctthmav6dsti.us-east-1.rds.amazonaws.com', DATABASE 'postgres', USER 'postgres',
    AWS CONNECTION aws_rds, SSL MODE 'require'
) WITH (VALIDATE = false);
CREATE CONNECTION
VALIDATE CONNECTION bad_user;
ERROR:  db error: FATAL: password authentication failed for user "postgres"

[status=Success rc=0]

=== Token-expiry check #1 (RDS connection > 15 min old; Aurora just past 15 min): same replication connections still streaming, rows still arriving
=== started 2026-11-18T22:31:23Z
--- upstream (rds): replication connection and latest row
SELECT now() AT TIME ZONE 'utc' AS utc_now, pid, usename, backend_start AT TIME ZONE 'utc' AS backend_start_utc, date_trunc('second', now() - backend_start) AS connection_age, state FROM pg_stat_replication
          utc_now           | pid  | usename |     backend_start_utc      | connection_age |   state   
----------------------------+------+---------+----------------------------+----------------+-----------
 2026-11-18 22:31:27.026153 | 1626 | mz_iam  | 2026-11-18 22:08:03.608796 | 00:23:23       | streaming
(1 row)

SELECT count(*) AS upstream_rows, (SELECT note FROM public.e2e ORDER BY id DESC LIMIT 1) AS latest_note FROM public.e2e
 upstream_rows |  latest_note  
---------------+---------------
            49 | live-22:31:25
(1 row)

--- materialize (rds): source status and latest row
SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_rds_src', 'e2e_rds') ORDER BY name
    name    | status  | error 
------------+---------+-------
 e2e_rds    | running | 
 pg_rds_src | running | 
(2 rows)

SELECT count(*) AS mz_rows, (SELECT note FROM e2e_rds ORDER BY id DESC LIMIT 1) AS latest_note, now() AT TIME ZONE 'utc' AS utc_now FROM e2e_rds
 mz_rows |  latest_note  |         utc_now         
---------+---------------+-------------------------
      49 | live-22:31:25 | 2026-11-18 22:31:27.919
(1 row)

--- upstream (aurora): replication connection and latest row
SELECT now() AT TIME ZONE 'utc' AS utc_now, pid, usename, backend_start AT TIME ZONE 'utc' AS backend_start_utc, date_trunc('second', now() - backend_start) AS connection_age, state FROM pg_stat_replication
          utc_now           | pid | usename |     backend_start_utc      | connection_age |   state   
----------------------------+-----+---------+----------------------------+----------------+-----------
 2026-11-18 22:31:30.625735 | 919 | mz_iam  | 2026-11-18 22:13:06.996922 | 00:18:23       | streaming
(1 row)

SELECT count(*) AS upstream_rows, (SELECT note FROM public.e2e ORDER BY id DESC LIMIT 1) AS latest_note FROM public.e2e
 upstream_rows |  latest_note  
---------------+---------------
            39 | live-22:31:16
(1 row)

--- materialize (aurora): source status and latest row
SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_aurora_src', 'e2e_aurora') ORDER BY name
     name      | status  | error 
---------------+---------+-------
 e2e_aurora    | running | 
 pg_aurora_src | running | 
(2 rows)

SELECT count(*) AS mz_rows, (SELECT note FROM e2e_aurora ORDER BY id DESC LIMIT 1) AS latest_note, now() AT TIME ZONE 'utc' AS utc_now FROM e2e_aurora
 mz_rows |  latest_note  |         utc_now         
---------+---------------+-------------------------
      39 | live-22:31:16 | 2026-11-18 22:31:30.915
(1 row)

--- materialize: status history (any stall/restart would show here)
SELECT s.name, h.occurred_at, h.status, h.error FROM mz_internal.mz_source_status_history h JOIN mz_sources s ON s.id = h.source_id WHERE s.name IN ('pg_rds_src','pg_aurora_src') ORDER BY h.occurred_at
     name      |        occurred_at         |  status  | error 
---------------+----------------------------+----------+-------
 pg_rds_src    | 2026-11-18 22:08:03.453+00 | running  | 
 pg_rds_src    | 2026-11-18 22:08:03.453+00 | starting | 
 pg_aurora_src | 2026-11-18 22:13:06.905+00 | running  | 
 pg_aurora_src | 2026-11-18 22:13:06.905+00 | starting | 
(4 rows)


[status=Success rc=0]

=== Reconnect check: restart Materialize after the original tokens expired; sources must reconnect with fresh tokens (new upstream pids) and keep replicating
=== started 2026-11-18T22:31:45Z
--- restarting Materialize at 22:31:46Z (both original IAM tokens, minted 22:08 and 22:13, are expired by now)
--- Materialize accepting connections again at 22:31:53Z; waiting 60s for sources to resume
--- upstream (rds): replication connection and latest row
SELECT now() AT TIME ZONE 'utc' AS utc_now, pid, usename, backend_start AT TIME ZONE 'utc' AS backend_start_utc, date_trunc('second', now() - backend_start) AS connection_age, state FROM pg_stat_replication
          utc_now           | pid  | usename |     backend_start_utc      | connection_age |   state   
----------------------------+------+---------+----------------------------+----------------+-----------
 2026-11-18 22:32:56.012038 | 2709 | mz_iam  | 2026-11-18 22:31:53.653265 | 00:01:02       | streaming
(1 row)

SELECT count(*) AS upstream_rows, (SELECT note FROM public.e2e ORDER BY id DESC LIMIT 1) AS latest_note FROM public.e2e
 upstream_rows |  latest_note  
---------------+---------------
            52 | live-22:32:55
(1 row)

--- materialize (rds): source status and latest row
SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_rds_src', 'e2e_rds') ORDER BY name
    name    | status  | error 
------------+---------+-------
 e2e_rds    | running | 
 pg_rds_src | running | 
(2 rows)

SELECT count(*) AS mz_rows, (SELECT note FROM e2e_rds ORDER BY id DESC LIMIT 1) AS latest_note, now() AT TIME ZONE 'utc' AS utc_now FROM e2e_rds
 mz_rows |  latest_note  |         utc_now         
---------+---------------+-------------------------
      51 | live-22:32:25 | 2026-11-18 22:32:56.064
(1 row)

--- upstream (aurora): replication connection and latest row
SELECT now() AT TIME ZONE 'utc' AS utc_now, pid, usename, backend_start AT TIME ZONE 'utc' AS backend_start_utc, date_trunc('second', now() - backend_start) AS connection_age, state FROM pg_stat_replication
          utc_now           | pid  | usename |     backend_start_utc      | connection_age |   state   
----------------------------+------+---------+----------------------------+----------------+-----------
 2026-11-18 22:32:58.625741 | 1739 | mz_iam  | 2026-11-18 22:31:53.676765 | 00:01:04       | streaming
(1 row)

SELECT count(*) AS upstream_rows, (SELECT note FROM public.e2e ORDER BY id DESC LIMIT 1) AS latest_note FROM public.e2e
 upstream_rows |  latest_note  
---------------+---------------
            42 | live-22:32:47
(1 row)

--- materialize (aurora): source status and latest row
SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_aurora_src', 'e2e_aurora') ORDER BY name
     name      | status  | error 
---------------+---------+-------
 e2e_aurora    | running | 
 pg_aurora_src | running | 
(2 rows)

SELECT count(*) AS mz_rows, (SELECT note FROM e2e_aurora ORDER BY id DESC LIMIT 1) AS latest_note, now() AT TIME ZONE 'utc' AS utc_now FROM e2e_aurora
 mz_rows |  latest_note  |         utc_now         
---------+---------------+-------------------------
      42 | live-22:32:47 | 2026-11-18 22:32:58.688
(1 row)

--- materialize: status history (any stall/restart would show here)
SELECT s.name, h.occurred_at, h.status, h.error FROM mz_internal.mz_source_status_history h JOIN mz_sources s ON s.id = h.source_id WHERE s.name IN ('pg_rds_src','pg_aurora_src') ORDER BY h.occurred_at
     name      |        occurred_at         |  status  | error 
---------------+----------------------------+----------+-------
 pg_rds_src    | 2026-11-18 22:08:03.453+00 | running  | 
 pg_rds_src    | 2026-11-18 22:08:03.453+00 | starting | 
 pg_aurora_src | 2026-11-18 22:13:06.905+00 | running  | 
 pg_aurora_src | 2026-11-18 22:13:06.905+00 | starting | 
 pg_rds_src    | 2026-11-18 22:31:53.466+00 | running  | 
 pg_rds_src    | 2026-11-18 22:31:53.466+00 | starting | 
 pg_aurora_src | 2026-11-18 22:31:53.604+00 | running  | 
 pg_aurora_src | 2026-11-18 22:31:53.604+00 | starting | 
(8 rows)


[status=Success rc=0]

=== Informational: rds.iam_auth_for_replication = 0 on both, then force a reconnect
=== started 2026-11-18T22:34:36Z
--- parameter groups now:
rds.iam_auth_for_replication	0
rds.iam_auth_for_replication	0
--- existing connections before forcing a reconnect:
--- upstream (rds): replication connection and latest row
SELECT now() AT TIME ZONE 'utc' AS utc_now, pid, usename, backend_start AT TIME ZONE 'utc' AS backend_start_utc, date_trunc('second', now() - backend_start) AS connection_age, state FROM pg_stat_replication
          utc_now          | pid  | usename |     backend_start_utc      | connection_age |   state   
---------------------------+------+---------+----------------------------+----------------+-----------
 2026-11-18 22:34:42.41853 | 2709 | mz_iam  | 2026-11-18 22:31:53.653265 | 00:02:48       | streaming
(1 row)

SELECT count(*) AS upstream_rows, (SELECT note FROM public.e2e ORDER BY id DESC LIMIT 1) AS latest_note FROM public.e2e
 upstream_rows |  latest_note  
---------------+---------------
            55 | live-22:34:26
(1 row)

--- materialize (rds): source status and latest row
SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_rds_src', 'e2e_rds') ORDER BY name
    name    | status  | error 
------------+---------+-------
 e2e_rds    | running | 
 pg_rds_src | running | 
(2 rows)

SELECT count(*) AS mz_rows, (SELECT note FROM e2e_rds ORDER BY id DESC LIMIT 1) AS latest_note, now() AT TIME ZONE 'utc' AS utc_now FROM e2e_rds
 mz_rows |  latest_note  |         utc_now         
---------+---------------+-------------------------
      55 | live-22:34:26 | 2026-11-18 22:34:42.528
(1 row)

--- upstream (aurora): replication connection and latest row
SELECT now() AT TIME ZONE 'utc' AS utc_now, pid, usename, backend_start AT TIME ZONE 'utc' AS backend_start_utc, date_trunc('second', now() - backend_start) AS connection_age, state FROM pg_stat_replication
          utc_now           | pid  | usename |     backend_start_utc      | connection_age |   state   
----------------------------+------+---------+----------------------------+----------------+-----------
 2026-11-18 22:34:45.558811 | 1739 | mz_iam  | 2026-11-18 22:31:53.676765 | 00:02:51       | streaming
(1 row)

SELECT count(*) AS upstream_rows, (SELECT note FROM public.e2e ORDER BY id DESC LIMIT 1) AS latest_note FROM public.e2e
 upstream_rows |  latest_note  
---------------+---------------
            45 | live-22:34:17
(1 row)

--- materialize (aurora): source status and latest row
SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_aurora_src', 'e2e_aurora') ORDER BY name
     name      | status  | error 
---------------+---------+-------
 e2e_aurora    | running | 
 pg_aurora_src | running | 
(2 rows)

SELECT count(*) AS mz_rows, (SELECT note FROM e2e_aurora ORDER BY id DESC LIMIT 1) AS latest_note, now() AT TIME ZONE 'utc' AS utc_now FROM e2e_aurora
 mz_rows |  latest_note  |        utc_now         
---------+---------------+------------------------
      45 | live-22:34:17 | 2026-11-18 22:34:45.62
(1 row)

--- restarting Materialize at 22:34:46Z to force new replication connections
SELECT name, type, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_rds_src','e2e_rds','pg_aurora_src','e2e_aurora') ORDER BY name
     name      |   type   | status  |                                    error                                    
---------------+----------+---------+-----------------------------------------------------------------------------
 e2e_aurora    | table    | stalled | postgres: db error: FATAL: password authentication failed for user "mz_iam"
 e2e_rds       | table    | stalled | postgres: db error: FATAL: password authentication failed for user "mz_iam"
 pg_aurora_src | postgres | stalled | postgres: db error: FATAL: password authentication failed for user "mz_iam"
 pg_rds_src    | postgres | stalled | postgres: db error: FATAL: password authentication failed for user "mz_iam"
(4 rows)

VALIDATE CONNECTION pg_rds
VALIDATE CONNECTION
VALIDATE CONNECTION pg_aurora
VALIDATE CONNECTION
SELECT s.name, h.occurred_at, h.status, h.error FROM mz_internal.mz_source_status_history h JOIN mz_sources s ON s.id = h.source_id WHERE s.name IN ('pg_rds_src','pg_aurora_src') AND h.occurred_at > now() - interval '3 minutes' ORDER BY h.occurred_at
     name      |        occurred_at         |  status  |                                    error                                    
---------------+----------------------------+----------+-----------------------------------------------------------------------------
 pg_rds_src    | 2026-11-18 22:34:52.469+00 | running  | 
 pg_rds_src    | 2026-11-18 22:34:52.469+00 | starting | 
 pg_aurora_src | 2026-11-18 22:34:52.751+00 | starting | 
 pg_aurora_src | 2026-11-18 22:34:52.757+00 | running  | 
 pg_rds_src    | 2026-11-18 22:34:52.904+00 | stalled  | postgres: db error: FATAL: password authentication failed for user "mz_iam"
 pg_aurora_src | 2026-11-18 22:34:52.974+00 | stalled  | postgres: db error: FATAL: password authentication failed for user "mz_iam"
 pg_rds_src    | 2026-11-18 22:34:57.924+00 | running  | 
 pg_rds_src    | 2026-11-18 22:34:57.924+00 | starting | 
 pg_aurora_src | 2026-11-18 22:34:58.004+00 | running  | 
 ... (same starting -> running -> stalled cycle repeats every ~5s for both sources until the parameter is restored; rows trimmed) ...
(72 rows)


[status=Success rc=0]

=== Recovery: rds.iam_auth_for_replication back to 1; sources must recover on their own and catch up
=== started 2026-11-18T22:37:52Z
--- rds.iam_auth_for_replication set back to 1 at 22:36:04Z; no Materialize restart or other intervention
--- upstream (rds): replication connection and latest row
SELECT now() AT TIME ZONE 'utc' AS utc_now, pid, usename, backend_start AT TIME ZONE 'utc' AS backend_start_utc, date_trunc('second', now() - backend_start) AS connection_age, state FROM pg_stat_replication
          utc_now          | pid  | usename |     backend_start_utc      | connection_age |   state   
---------------------------+------+---------+----------------------------+----------------+-----------
 2026-11-18 22:37:55.90112 | 2975 | mz_iam  | 2026-11-18 22:36:10.109365 | 00:01:45       | streaming
(1 row)

SELECT count(*) AS upstream_rows, (SELECT note FROM public.e2e ORDER BY id DESC LIMIT 1) AS latest_note FROM public.e2e
 upstream_rows |  latest_note  
---------------+---------------
            61 | live-22:37:27
(1 row)

--- materialize (rds): source status and latest row
SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_rds_src', 'e2e_rds') ORDER BY name
    name    | status  | error 
------------+---------+-------
 e2e_rds    | running | 
 pg_rds_src | running | 
(2 rows)

SELECT count(*) AS mz_rows, (SELECT note FROM e2e_rds ORDER BY id DESC LIMIT 1) AS latest_note, now() AT TIME ZONE 'utc' AS utc_now FROM e2e_rds
 mz_rows |  latest_note  |         utc_now         
---------+---------------+-------------------------
      61 | live-22:37:27 | 2026-11-18 22:37:56.068
(1 row)

--- upstream (aurora): replication connection and latest row
SELECT now() AT TIME ZONE 'utc' AS utc_now, pid, usename, backend_start AT TIME ZONE 'utc' AS backend_start_utc, date_trunc('second', now() - backend_start) AS connection_age, state FROM pg_stat_replication
          utc_now           | pid  | usename |     backend_start_utc      | connection_age |   state   
----------------------------+------+---------+----------------------------+----------------+-----------
 2026-11-18 22:37:58.777685 | 2037 | mz_iam  | 2026-11-18 22:36:10.438243 | 00:01:48       | streaming
(1 row)

SELECT count(*) AS upstream_rows, (SELECT note FROM public.e2e ORDER BY id DESC LIMIT 1) AS latest_note FROM public.e2e
 upstream_rows |  latest_note  
---------------+---------------
            52 | live-22:37:48
(1 row)

--- materialize (aurora): source status and latest row
SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE name IN ('pg_aurora_src', 'e2e_aurora') ORDER BY name
     name      | status  | error 
---------------+---------+-------
 e2e_aurora    | running | 
 pg_aurora_src | running | 
(2 rows)

SELECT count(*) AS mz_rows, (SELECT note FROM e2e_aurora ORDER BY id DESC LIMIT 1) AS latest_note, now() AT TIME ZONE 'utc' AS utc_now FROM e2e_aurora
 mz_rows |  latest_note  |         utc_now         
---------+---------------+-------------------------
      52 | live-22:37:48 | 2026-11-18 22:37:59.068
(1 row)

SELECT s.name, h.occurred_at, h.status, h.error FROM mz_internal.mz_source_status_history h JOIN mz_sources s ON s.id = h.source_id WHERE s.name IN ('pg_rds_src','pg_aurora_src') AND h.status <> 'stalled' ORDER BY h.occurred_at DESC LIMIT 4
     name      |        occurred_at         |  status  | error 
---------------+----------------------------+----------+-------
 pg_aurora_src | 2026-11-18 22:36:10.381+00 | running  | 
 pg_aurora_src | 2026-11-18 22:36:10.381+00 | starting | 
 pg_rds_src    | 2026-11-18 22:36:10.054+00 | running  | 
 pg_rds_src    | 2026-11-18 22:36:10.054+00 | starting | 
(4 rows)

SELECT s.name, count(*) AS stalled_events, min(h.occurred_at) AS first_stall, max(h.occurred_at) AS last_stall FROM mz_internal.mz_source_status_history h JOIN mz_sources s ON s.id = h.source_id WHERE h.status = 'stalled' GROUP BY 1 ORDER BY 1
     name      | stalled_events |        first_stall         |         last_stall         
---------------+----------------+----------------------------+----------------------------
 pg_aurora_src |             15 | 2026-11-18 22:34:52.974+00 | 2026-11-18 22:36:05.359+00
 pg_rds_src    |             15 | 2026-11-18 22:34:52.904+00 | 2026-11-18 22:36:05.031+00
(2 rows)


[status=Success rc=0]

@martykulma martykulma left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice - thanks for running those tests!

Are those doc changes going to go live after the merge? You might have to split them out so they go out with the next release.

The rest looks good 🐟

Merging to main publishes the docs immediately, before the release
that contains this change is live. The docs now live in a separate PR
that merges when that release is out.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@val-materialize

val-materialize commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

@martykulma tysm for the review! Splitting out docs to another PR then merging this

@val-materialize

Copy link
Copy Markdown
Contributor Author

docs in #40923 to be merged during v26.52.0 (next week)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants