Skip to content

SQL-522: serve HTTP/2 on environmentd and balancerd HTTP endpoints - #37913

Draft
jubrad wants to merge 8 commits into
MaterializeInc:mainfrom
jubrad:justin/sql-522-add-adapter-support-for-http2
Draft

jubrad wants to merge 8 commits into
MaterializeInc:mainfrom
jubrad:justin/sql-522-add-adapter-support-for-http2

Conversation

@jubrad

@jubrad jubrad commented Jul 28, 2026 •

Copy link
Copy Markdown
Member

Problem

environmentd and balancerd only speak HTTP/1.1. HTTP/2 is a step toward HTTP/3 and gives multiplexing.

Solution

  • environmentd: serve with hyper-util's auto builder. TLS advertises h2, http/1.1 via ALPN; cleartext listeners accept h2c by preface sniffing. WebSockets work over HTTP/2 via RFC 8441 extended CONNECT, and the HTTP/1.1 upgrade path is kept.
  • balancerd: the internal HTTP server uses the auto builder. The HTTPS proxy is a byte proxy that answers ALPN before it connects upstream, so advertising h2 in front of a pre-HTTP/2 environmentd breaks clients (found HTTP2 preface). It is gated on the dyncfg balancerd_https_enable_http2_alpn, default off, read once at startup.
  • No outbound HTTP client changes.

Rollout

  1. Deploy. environmentd serves h2 immediately; balancerd unchanged.
  2. Once every environmentd behind a balancer fleet supports HTTP/2, set balancerd_https_enable_http2_alpn=true in LaunchDarkly and rolling-restart balancerd.
  3. Rollback: flag off, restart balancerd. Disable the flag before downgrading any environmentd to a pre-HTTP/2 version.

Tests

  • test_http2_tls (ALPN selection), test_http2_cleartext (h2c), test_http2_websocket_auth (WS over extended CONNECT and HTTP/1.1 upgrade).
  • balancerd test_balancer: ALPN selection with the flag on, h2 end to end, HTTP/1.1 clients still served, h2c on the internal server.
  • test/balancerd mzcompose turns the flag on for its TLS balancer.

Reviewer notes

  • The TLS context is shared with pgwire. Clients that send no ALPN get no ALPN in the handshake (AlpnError::NOACK), so pgwire is unaffected.
  • HTTP/2 requests have no Host header (:authority lives in the URI). resolve_host (OAuth metadata) and the console CSRF check is_same_origin now fall back to the URI authority. The console proxy pins upstream requests to HTTP/1.1, since hyper-util's legacy client rejects an HTTP/2 request on an HTTP/1 connection.
  • When balancerd can't reach upstream, an h2 client gets a closed connection instead of the hand-written HTTP/1.1 502, which it couldn't parse anyway.

Closes SQL-522. Supersedes #37541.

🤖 Generated with Claude Code

@jubrad
jubrad force-pushed the justin/sql-522-add-adapter-support-for-http2 branch 2 times, most recently from ed28527 to f876fb1 Compare August 26, 2026 03:16
@jubrad
jubrad force-pushed the justin/sql-522-add-adapter-support-for-http2 branch 3 times, most recently from 8af397f to e22f1ea Compare September 8, 2026 14:13
@jubrad
jubrad force-pushed the justin/sql-522-add-adapter-support-for-http2 branch 3 times, most recently from dfcde15 to 325aec7 Compare September 22, 2026 19:07
@jubrad
jubrad force-pushed the justin/sql-522-add-adapter-support-for-http2 branch 6 times, most recently from dc4c443 to a9df141 Compare October 1, 2026 12:31
jubrad and others added 7 commits November 18, 2026 15:36
Negotiate HTTP/2 or HTTP/1.1 via TLS ALPN and serve both protocols with
hyper's auto builder (h2c via preface sniffing on plaintext listeners).
HTTP/1.1-only clients are unaffected. pgwire listeners share the TLS
context, but pgwire clients do not send ALPN, so the negotiation
callback no-ops for them.

balancerd remains a byte proxy for HTTPS: HTTP/2 frames pass through to
environmentd unchanged. Its raw HTTP/1.1 502 fallback is now skipped
for clients that negotiated h2, and its internal HTTP server serves
h2c.

Also accept WebSockets over HTTP/2 (RFC 8441 extended CONNECT):
environmentd advertises SETTINGS_ENABLE_CONNECT_PROTOCOL and the
/api/experimental/sql route accepts CONNECT in addition to GET. This
requires axum >= 0.8.9, as 0.8.8 has a method-routing bug that
registers CONNECT handlers under OPTIONS. The axum bump pulls
tungstenite 0.29, which duplicates the 0.28 pinned by kube-client, so
deny.toml gains skip entries for the old versions.

reqwest gains the native-tls-alpn feature: without it the native-tls
backend never offers ALPN and clients silently stay on HTTP/1.1, which
the new tests would catch in CI.

Tests: HTTP/2 over TLS with HTTP/1.1 downgrade, h2c prior knowledge,
WebSocket authentication (valid and invalid credentials) over both
HTTP/2 and HTTP/1.1, and h2/h1.1/h2c assertions through balancerd.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
balancerd is a byte proxy: it terminates TLS and forwards the decrypted
stream to environmentd. If balancerd advertises HTTP/2 via ALPN before
environmentd supports it, clients negotiate h2 but environmentd receives
frames it cannot parse ("invalid HTTP version parsed (found HTTP2
preface)").

Add `balancerd_https_enable_http2_alpn` dyncfg (default false) to
control whether balancerd advertises h2. During upgrade:
1. Upgrade all environmentd instances (they now support h2)
2. Enable the dyncfg via LaunchDarkly
3. Restart balancerd instances (the dyncfg is read at startup)

Also add `enable_http2_alpn` parameter to `TlsCertConfig::load_context`
so callers can control ALPN advertisement. environmentd passes `true`
since it handles HTTP/2 directly via hyper's auto builder.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
bin/lint-test-flags requires every mz_dyncfg::Config to be known to
parallel-workload's FlipFlagsAction and to mzcompose's system parameter
lists. balancerd dyncfgs are not SQL-settable system parameters, so add
the new flag alongside the other balancerd_* entries in the
uninteresting lists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The new balancerd_https_enable_http2_alpn dyncfg defaults off, which
broke the balancerd server test's assertion that a request through
balancerd comes back as HTTP/2.

Follow the convention that a new flag defaults off in production but on
in the test configuration: teach set_defaults to accept the flag so it
can be set via --default-config, and have the test enable it.

Also document that the flag is read once when the TLS context is built,
so a change only takes effect after balancerd restarts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
reqwest's `native-tls-alpn` feature applies workspace-wide, so enabling
it made every outbound HTTPS client in Materialize offer ALPN and
potentially switch to HTTP/2: schema registry, OIDC, webhooks, and the
rest. This PR only adds HTTP/2 on the serving side and does not need
that.

Drop the feature and assert ALPN selection directly with openssl, which
covers the callback in `TlsCertConfig::load_context` rather than
reqwest's protocol choice. End-to-end HTTP/2 request handling stays
covered by the h2c prior-knowledge tests and the WebSocket test's manual
h2 handshake.

This is also the only client-visible change in the PR, and so the
suspected cause of the kafka-auth-2 regression.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…efault

reqwest 0.13's `native-tls-vendored` feature implies ALPN support, so
the test clients negotiate h2 with servers that advertise it. The TLS
tests asserted HTTP/1.1 for a default client, which only held with
reqwest 0.12. Assert HTTP/2 for the default client and keep a dedicated
`http1_only` client for the HTTP/1.1 path.

The cert-reloading tests inspect the peer certificate per request and
rely on a fresh connection each time, so pin those clients to HTTP/1.1
rather than let HTTP/2 multiplex requests over one connection.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ments

Turn on balancerd_https_enable_http2_alpn for the TLS balancerd in
test/balancerd so its curl-based HTTPS workflows negotiate h2 end to end.
The rationale for the flag lives on the dyncfg; drop the restatements at
call sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jubrad
jubrad force-pushed the justin/sql-522-add-adapter-support-for-http2 branch from 0f6bb07 to f7de4db Compare November 18, 2026 21:39
@DAlperin
DAlperin self-requested a review November 18, 2026 21:40
…ole proxy

HTTP/2 requests carry `:authority` in the URI and no `Host` header. The
OAuth protected-resource metadata endpoint returned 400 for them, and the
console preview CSRF check treated any cross-origin POST as same-origin.
Both now fall back to the URI authority.

The console proxy forwarded the inbound version, and hyper-util's legacy
client rejects an HTTP/2 request on an HTTP/1 upstream connection, so
/internal-console failed for h2 clients. Pin it to HTTP/1.1.

Also correct comments on how the auto builder picks a protocol and on
pgwire ALPN (libpq 17+ offers `postgresql`, now probed in
test_http2_tls), and enable the balancerd h2 flag for every TLS balancerd
in test/balancerd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant