Skip to content

Build feature flag event payloads in chunks instead of large arrays (FFE optimization) - #13535

Draft
dougqh wants to merge 4 commits into
dougqh/ffe-toolbox-shadingfrom
dougqh/ffe-chunked-payloads
Draft

dougqh wants to merge 4 commits into
dougqh/ffe-toolbox-shadingfrom
dougqh/ffe-chunked-payloads

Conversation

@dougqh

@dougqh dougqh commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Writes the exposure and flag evaluation payloads into the toolbox's ChunkedBuffer (chunks just under 256 KiB) and posts the chunks, instead of building each payload as one array of up to several MiB.

  • Exposures are serialized as they are drained. The serializer thread writes each exposure into a ChunkedBuffer that it reuses across flushes. A payload is a prefix with the context, the exposures separated by commas, and a suffix. It is sent once it reaches 5 MiB, the same limit as flag evaluation payloads. Before, the exposures waited for the flush as ExposureEvent objects, then were written into one array.
  • Flag evaluation payloads are written into chunks with the same splitting as before. EncodedPayloads.bodies becomes a list of chunk lists. Hashed targeting keys are hex-encoded from a digit table, instead of building an Integer.toHexString string per byte.
  • Transports post chunks.
    • EventTransport (the bridge SPI) gets a post(String, List<ByteBuffer>) default method that joins the chunks, for transports that only post arrays.
    • DirectIntakeTransport streams the chunks with BodyPublishers.fromPublisher and a known content length, without joining them.
    • FallbackTransport passes the chunks on to the agent's event proxy, or to direct intake once it has switched.

Motivation

On G1 with 1 MB regions (heaps up to 2 GB), any array of 512 KB or more is humongous. It is allocated straight into the old generation, can start a concurrent cycle, and is only reclaimed by a collection that handles humongous regions. A payload built as one array (and the arrays copied while a stream grows) crosses that size well before 5 MiB. The chunks stay regular-sized.

Measured on its own by Bits, at this PR's head (815056a903) against #13532's head (cdda8262d8). Both sides get the same benchmark-only overlay (#13529's ClientEvaluationBenchmark, a consent param, and the chunked post override). Unique subjects, JDK 21.0.12, -prof gc, with base and PR forks alternating. "Noise" means the fork ranges overlap.

256 MB heap, average time (5 forks at 1 thread, 3 at 4 threads):

Case Threads Time per evaluation Bytes per evaluation GC ms per million evaluations
scale/flags 1 891 → 886 ns (noise) 1,105 → 1,084 (noise) 24.7 → 23.2 (noise)
scale/flags 4 2.28 → 2.26 µs (noise) 1,095 → 1,075 (noise) 23.1 → 21.4 (noise)
scale/splits 1 22.57 → 22.04 µs (noise) 40,073 → 39,675 (noise) 1,299 → 970 (−25%)
scale/splits 4 46.83 → 48.63 µs (+3.8%, noise; 7 forks) 39,633 → 39,675 (noise) 847 → 726 (−14%, overlapping)

128 MB heap, G1, 4 threads, throughput (3 forks, per million evaluations):

Case Throughput JMH GC ms Pause ms Promoted MB Humongous MB
scale/flags +3.6% (noise) 60.3 → 46.9 (−22%) 67.6 → 57.8 (−15%) 8.26 → 5.77 (−30%) 2.98 → 1.03
scale/splits +2.6% (noise) 1,896 → 1,451 (−23%) 2,062 → 1,737 (−16%) 167 → 147 (noise) 60.3 → 22.1
GC counts, totals over 3 forks Humongous-triggered (all / after setup) Full
scale/flags 16 / 13 → 3 / 0 3 → 0
scale/splits 19 / 16 → 2 / 0 5 → 0
  • The earlier +9% single-thread slowdown is not real. With 5 forks a side, scale/flags at 1 thread is 891 ns [864..940] → 886 ns [864..900], −0.5%.
  • scale/splits at 4 threads is noise. A 3-fork run showed +6.5%; with 7 forks a side it's +3.8% with overlapping ranges ([44.53..50.56] vs [47.43..49.59]).
  • After setup, no humongous-triggered or Full GCs remain. The few left come from reading the ~8 MB scale configuration during trial setup.
  • On the small heap, GC time, pauses and scale/flags promotion drop 15–30%.

Additional Notes

  • Depends on the toolbox. Stacked on Shade the perf toolbox into dd-openfeature (FFE optimization) #13532, which shades the toolbox into the SDK. The published jar gains one relocated class, ChunkedBuffer.
  • Behavior changes for exposures:
    • A batch larger than 5 MiB is sent as several payloads. Before, a batch was one payload of any size.
    • An exposure that can't be serialized is dropped alone, with an error log. Before, the whole batch was dropped.
    • An exposure larger than a whole payload is dropped, with a debug log.
    • Delivery stays best effort: a payload whose post fails is dropped, as before.
  • The bridge SPI change is a default method only, so existing implementations keep working.
  • Agent users don't get the full benefit yet. The agent's event proxy (AgentEventProxy) only has postEvent(route, byte[]), so it uses the default method and still joins each payload into one array. A follow-up can add a chunked postEvent to FeatureFlaggingGateway.Backend and override the chunked post in the proxy.
  • Benchmark transport, deferred. The measurements used a ClientEvaluationBenchmark whose in-memory transport overrides the chunked post, so it takes the chunks as they are, like direct intake. That benchmark lives in Benchmark flag evaluation through the OpenFeature client (FFE optimization) #13529, which isn't on master, so the override isn't in this PR. Once both land, add it there. Without it, the benchmark joins the chunks through the default method and shows humongous arrays that a streaming transport doesn't have.
  • Checked locally on this branch (stacked on Shade the perf toolbox into dd-openfeature (FFE optimization) #13532): all 1,601 dd-openfeature tests pass, and Spotless, SpotBugs (spotbugsMain, spotbugsTest, spotbugsJmh) and checkToolboxRelocation pass. The sandbox run on the earlier base had 1,609 tests: Benchmark FFE evaluation on the scale suites #13485 adds the scale-suite tests. FlagEvaluationPayloadsTest joins the chunks through a bytes() helper to check the JSON. The exposure serializer suppresses AT_NONATOMIC_OPERATIONS_ON_SHARED_VARIABLE and AT_STALE_THREAD_WRITE_OF_PRIMITIVE, because the payload is confined to the serializer thread.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

Review follow-ups

  • Race fix: on a timeout, HttpCall cancels the future and post throws, but the JDK client can still be reading the request body, whose read-only views share the ChunkedBuffer's arrays. The exposure serializer reset the buffer in finally and overwrote those bytes. It now replaces the buffer after a failed post (aFailedPostKeepsItsBuffers fails without the fix). EventTransport documents that buffers are reusable only after a normal return.
  • Peak memory: flag evaluations used to be encoded into every payload of a flush before the first was posted. A full aggregator (131,072 full + 32,768 degraded buckets) can make that tens of MiB at once. Payloads are now posted as each closes, reusing one buffer, so a flush holds one payload at a time. Metrics are reported in a finally, so the counts survive a failed post.
  • One Jackson generator per payload, rather than per exposure, turned out not to be needed: the +9% it was meant to fix was noise (see above).

🤖 Generated with Claude Code

@dougqh dougqh added type: feature Enhancements and improvements tag: performance Performance related changes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 18, 2026
@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 98.62%
• Overall Coverage: 60.83% (+0.04%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9f20045 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.83 s 14.69 s [+0.1%; +1.8%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.70 s 13.71 s [-0.9%; +0.7%] (no difference)
startup:petclinic:appsec:Agent 17.19 s 17.10 s [-0.4%; +1.4%] (no difference)
startup:petclinic:iast:Agent 16.50 s 17.02 s [-7.2%; +1.1%] (no difference)
startup:petclinic:profiling:Agent 16.23 s 16.75 s [-7.3%; +1.1%] (no difference)
startup:petclinic:sca:Agent 17.01 s 16.95 s [-0.5%; +1.2%] (no difference)
startup:petclinic:tracing:Agent 15.83 s 16.12 s [-5.9%; +2.3%] (no difference)

Commit: 9f200459 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh dougqh changed the title Build feature flag event payloads in chunks instead of large arrays Build feature flag event payloads in chunks instead of large arrays (FFE optimization) Nov 19, 2026
@dougqh
dougqh force-pushed the dougqh/ffe-toolbox-shading branch from 3d4204f to 8dd30e7 Compare November 19, 2026 21:24
@dougqh
dougqh force-pushed the dougqh/ffe-chunked-payloads branch from 2a118a2 to 32070d8 Compare November 19, 2026 21:25
@dougqh
dougqh force-pushed the dougqh/ffe-toolbox-shading branch from 8dd30e7 to cdda826 Compare November 19, 2026 22:38
@dougqh
dougqh force-pushed the dougqh/ffe-chunked-payloads branch from 88b2739 to 815056a Compare November 19, 2026 22:56
@dougqh
dougqh force-pushed the dougqh/ffe-chunked-payloads branch from 815056a to 910c68d Compare November 20, 2026 13:36
@dougqh
dougqh force-pushed the dougqh/ffe-toolbox-shading branch from 89895cd to 8c81722 Compare November 20, 2026 13:39
@dougqh
dougqh force-pushed the dougqh/ffe-chunked-payloads branch from 910c68d to b571777 Compare November 20, 2026 13:39
@pr-commenter

pr-commenter Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

LLM Validation

LLM Validation Gate — java-agent-instructions

✅ PASS

  • No blocking-case regressions; the quality change is within noise (baseline/candidate confidence intervals overlap).

6 case(s) · quality 80.0 → 80.5 (+0.4) · win-rate 49% [46%–51%]

Details — analysis and per-case results

Analysis

Changed instruction file(s): AGENTS.md.

No safety or blocking-case regressions across 6 case(s). Overall pairwise win-rate 49% [46%–51%], quality +0.4 — see the verdict above for whether that clears the noise band.

Results

  • Pairwise win-rate: 49% [46%–51%] — candidate's share of blind comparisons (90% CI; spanning 50% = no clear difference)
  • Overall quality: 80.0 → 80.5 (/100, +0.4)
  • Bad signals introduced (advisory): 0
  • Candidate criteria coverage (advisory): 23/27 (85%) — expected_criteria the candidate met; does not affect the gate
  • Blocking-case regressions: 0

Cases

Case Mode Quality Δ Win-rate (90% CI) Safety
java-core-validation-001 warn +1.7 50% [50%–50%] ok
java-unit-test-conventions-001 warn -0.6 50% [50%–50%] ok
java-premain-safety-001 warn +0.6 50% [50%–50%] ok
java-ci-download-guidance-001 warn +2.2 50% [50%–50%] ok
java-documentation-only-validation-001 warn -0.3 42% [28%–55%] ok
java-existing-spock-maintenance-001 warn -1.1 50% [50%–50%] ok

Per-dimension scores, token usage, latency, and estimated cost are in the CI job logs.

@dougqh
dougqh force-pushed the dougqh/ffe-toolbox-shading branch from 8c81722 to 6ac4db5 Compare November 20, 2026 20:50
@dougqh
dougqh force-pushed the dougqh/ffe-chunked-payloads branch from b571777 to 8eaba80 Compare November 20, 2026 20:50
@dougqh
dougqh force-pushed the dougqh/ffe-toolbox-shading branch from 6ac4db5 to 0e0d85e Compare November 20, 2026 21:32
@dougqh
dougqh force-pushed the dougqh/ffe-chunked-payloads branch from 8eaba80 to 8284622 Compare November 20, 2026 21:32
datadog-bits and others added 2 commits November 20, 2026 22:53
Serialize each exposure as it is drained into a reused ChunkedBuffer,
sending a payload once it reaches 5 MiB, instead of holding the event
objects until the flush and then one large array. Flag evaluation
payloads are written into chunks too, and the transports post the
chunks: direct intake streams them, other transports join them. Hashed
targeting keys are hex-encoded from a digit table.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: dougqh <dougqh@gmail.com>
Coverage verification failed on the new chunked path. The tests cover:
- ChunksPublisher against the Flow contract: partial and unbounded requests, completion, an
  empty body, read-only views, an error for a non-positive request, cancel, and a cancel from
  inside onNext.
- DirectIntakeTransport: a chunked post arrives as one body with its length.
- FallbackTransport: the chunked post follows the same proxy/direct switching as the array post.
- JsonWriting: a message per document, rewinding a partial message when the writer fails, and
  single-byte writes. ChunkedOutput becomes package-private for the last one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dougqh and others added 2 commits November 20, 2026 22:53
A request that timed out can still be reading its body after post throws,
so the exposure serializer now replaces its buffer after a failed post
instead of overwriting the bytes in flight. EventTransport documents that
contract.

Flag evaluations were encoded into all of a flush's payloads before the
first was posted; a full aggregator can make that tens of MiB at once. Each
payload is now posted as it closes, reusing one buffer, so a flush holds
one payload at a time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both pipelines built a JSON array of events in a reused ChunkedBuffer
between a fixed prefix and suffix: the separator before each event after
the first, the rollback on overflow or failure, and the body assembled
for posting. JsonArrayPayload now does that once, with reset() after a
delivered post and discard() after a failed one, whose buffers a
timed-out request may still be reading.

Exposure tests now cover splitting a batch across payloads, dropping an
exposure larger than a payload, and dropping one that can't be
serialized.

From Bits' review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/ffe-toolbox-shading branch from 0e0d85e to 676974d Compare November 21, 2026 03:53
@dougqh
dougqh force-pushed the dougqh/ffe-chunked-payloads branch from cde93ef to 9f20045 Compare November 21, 2026 03:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants