Skip to content

Benchmark flag evaluation through the OpenFeature client (FFE optimization) - #13529

Draft
dougqh wants to merge 3 commits into
masterfrom
dougqh/ffe-client-benchmark
Draft

dougqh wants to merge 3 commits into
masterfrom
dougqh/ffe-client-benchmark

Conversation

@dougqh

@dougqh dougqh commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Adds ClientEvaluationBenchmark, which evaluates the scale suite cases the way an application does: through an OpenFeature client, with the provider's hooks and a started runtime. That covers evaluation metrics, evaluation counts and exposure logging. ScaleSuiteBenchmark calls the provider directly and measures none of those.

  • The runtime gets the suite's configuration from a fake Remote Configuration source and posts its events to a transport that drops them. Its background threads aggregate and serialize as in production, without network I/O. The OpenTelemetry API is the no-op one.
  • subjects=repeated evaluates each case with its own targeting key, so exposures are deduplicated after the first one.
  • subjects=unique gives every evaluation a new subject, like a service with many users: every exposure is new, and evaluation counts open a bucket per subject up to the per-flag cap. The 2^18 subject keys are built in the setup, and the context wrapper is reused per thread, so the benchmark allocates nothing per operation.
  • ScaleCase gains a client-path evaluate.

This PR has no change to shipped code.

Motivation

Applications evaluate through the client. For a simple flag, about two thirds of the per-evaluation allocation happens outside the provider: in the OpenFeature SDK, the provider's hooks and the event hand-off. The PRs stacked on this one report their client-path impact with it:

Baseline on #13485 (JMH -prof gc, 1 fork, 3 × 2 s warm-up, 5 × 2 s measurement, 256 MB heap, JDK 21, 5-vCPU sandbox with up to ~28% steal at 4 threads, so the times are rough and the bytes are exact):

Case, unique subjects Threads Time per evaluation Allocation per evaluation Young GCs GC time
scale/flags 1 0.89 µs 1,098 B 91 321 ms
scale/splits 1 23.9 µs 40,485 B 123 554 ms
scale/wide-context 1 6.25 µs 7,790 B 91 212 ms
scale/semver 1 12.2 µs 22,562 B 129 235 ms
scale/flags 4 2.27 µs 1,056 B 134 353 ms
scale/splits 4 45.7 µs 39,475 B 251 752 ms

Additional Notes

  • -prof gc counts the allocation of every thread, including the runtime's background workers, so the bytes include aggregation and serialization.
  • Run all cases: ./gradlew :dd-openfeature:jmh -Pjmh.includes=ClientEvaluationBenchmark -Pjmh.profilers=gc. One case at a given thread count: java -jar dd-openfeature/build/libs/dd-openfeature-*-jmh.jar ClientEvaluationBenchmark -p cases=scale/splits -p subjects=unique -t 4 -jvmArgsAppend "-Xms256m -Xmx256m" -prof gc.
  • Time differences under about 15% need 3 forks (-f 3) on a quiet machine.

Relation to Leo's FFE stack (#13484 and #13486–#13490, on #13485)

Contributor Checklist

Jira ticket: [PROJ-IDENT]

Review follow-ups

  • Each benchmark thread now starts at its own case and subject (Fibonacci-hashed offset). Before this, threads ran in lockstep, all evaluating the same flag for the same subject at once. Multi-threaded numbers taken before this change, including Bits' 4-thread runs of the stacked PRs, carry that bias.

🤖 Generated with Claude Code

@dougqh dougqh added type: feature Enhancements and improvements tag: performance Performance related changes tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 18, 2026
@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 60.63% (-0.03%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8525804 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.01 s 13.94 s [-0.2%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 12.92 s 12.89 s [-0.3%; +0.8%] (no difference)
startup:petclinic:appsec:Agent 17.16 s 17.16 s [-1.0%; +1.0%] (no difference)
startup:petclinic:iast:Agent 17.02 s 17.08 s [-0.9%; +0.2%] (no difference)
startup:petclinic:profiling:Agent 16.69 s 16.29 s [-1.8%; +6.7%] (no difference)
startup:petclinic:sca:Agent 17.20 s 17.14 s [-0.3%; +1.1%] (no difference)
startup:petclinic:tracing:Agent 15.82 s 16.24 s [-6.8%; +1.6%] (no difference)

Commit: 85258040 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Evaluate the scale suite cases through a client, with the provider's
hooks and a started runtime posting to a transport that drops events,
for subjects that repeat or are new on every evaluation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: dougqh <dougqh@gmail.com>
@dougqh
dougqh force-pushed the dougqh/ffe-client-benchmark branch from 1964dba to 8670aed Compare November 19, 2026 15:16
@dougqh dougqh changed the title Benchmark flag evaluation through the OpenFeature client Benchmark flag evaluation through the OpenFeature client (FFE optimization) Nov 19, 2026
dougqh and others added 2 commits November 19, 2026 17:31
Threads that start together stay in lockstep, all evaluating the same flag
for the same subject at once, which shares cache lines and bucket state in a
way production traffic does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Above a few hundred thousand evaluations per second the flag evaluation
queue overflows, so unpaced runs measured a cost that left part of the
work out. The teardown now prints the dropped share, and a pace param
spends CPU before each evaluation to keep the rate under the overflow.
Adds scale/variations, which logs exposures, and starts each thread on a
distinct case.

From Bits' review of the benchmark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants