[improve][pip] PIP-496: Pulsar Functions and IO support for the V5 client and scalable topics

[lhotari]

PIP: PIP-496, pip/pip-496.md

Motivation

Pulsar Functions and Pulsar IO use the v4 Java client throughout. As a result, a function, source or sink cannot
read or write a scalable topic (topic://, PIP-460), which only the V5 client (PIP-466) supports. Every function
or connector attached to a persistent:// topic also blocks the PIP-475 migration of that topic, because
migration refuses while a v4 producer or consumer is attached.

Modifications

This PR adds the proposal document pip/pip-496.md. The proposal:

• Picks the client per component from its topics, with the rule of the pulsar-client and pulsar-perf CLIs:
  • topic:// topics use the V5 client and other topics the v4 client, with no configuration.
  • A new optional clientApi setting (V4/V5, also --client-api on pulsar-admin functions|sources|sinks)
    makes the V5 client drive persistent:// topics. That lets PIP-475 migrate the topics of a running component
    with a Shared subscription.
• Maps the subscription types onto the V5 consumer models:
  • Shared uses a QueueConsumer, with the same ack, negative-ack, timeout and dead letter behavior as with v4.
  • Failover, Key_Shared and EFFECTIVELY_ONCE use a StreamConsumer, which gives per-key order and acknowledges
    the completed prefix of the records.
  • The differences from v4 are stated explicitly, including how failures are handled.
• Rejects at submission every combination that the V5 runtime cannot honor. The rules may only be relaxed in
  later versions, because instances check them again at start.
• Adds one user API method, BaseContext.getPulsarClientV5(). Context.newOutputMessage(...) publishes to
  topic:// topics with the V5 client in every component.
• Specifies compatibility: stored components are unchanged (the default is AUTO). The upgrade and rollback
  sections cover the new proto field, older workers and older runtime images.

Reference implementation. #26697 has unit tests and end-to-end tests
against a TLS-authenticated broker with the thread runtime. Those tests cover:

• a source and a Shared function on topic:// topics;
• a Key_Shared function with parallelism 2;
• a clientApi=V5 Shared function whose persistent:// input is migrated with PIP-475 while it runs.

Review. The document went through three rounds of independent review with a subtractive, minimalism-focused
lens. The Resolved design decisions section records what was cut, each with its additive way back, and what was
kept and why. The review cut three API methods (newOutputMessageV5, getPulsarClientBuilderV5,
Record.getMessageV5) and led to fixes in the reference implementation.

Open question for the discussion. Should one component be allowed to mix topic:// topics with other
topics? The proposal keeps the CLIs' stricter one-domain rule, which can be relaxed later without breaking
anything (see Alternatives).

Verifying this change

• Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

[Dream95]

This PIP shows that while Pulsar Functions and IO can now support scalable topics through the V5 client, V5 still has some gaps compared with V4. As more components move to V5, I think we should focus on closing those gaps.