Repository navigation
Conversation
This was referenced Nov 18, 2026
Draft
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
This was referenced Nov 18, 2026
Base automatically changed from
leo.romanovsky/ffe-scale-suite-benchmark
to
master
November 19, 2026 03:09
Evaluate the scale suite cases through a client, with the provider's hooks and a started runtime posting to a transport that drops events, for subjects that repeat or are new on every evaluation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: dougqh <dougqh@gmail.com>
dougqh
force-pushed
the
dougqh/ffe-client-benchmark
branch
from
November 19, 2026 15:16
1964dba to
8670aed
Compare
Threads that start together stay in lockstep, all evaluating the same flag for the same subject at once, which shares cache lines and bucket state in a way production traffic does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Above a few hundred thousand evaluations per second the flag evaluation queue overflows, so unpaced runs measured a cost that left part of the work out. The teardown now prints the dropped share, and a pace param spends CPU before each evaluation to keep the rate under the overflow. Adds scale/variations, which logs exposures, and starts each thread on a distinct case. From Bits' review of the benchmark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Adds
ClientEvaluationBenchmark, which evaluates the scale suite cases the way an application does: through an OpenFeature client, with the provider's hooks and a started runtime. That covers evaluation metrics, evaluation counts and exposure logging.ScaleSuiteBenchmarkcalls the provider directly and measures none of those.subjects=repeatedevaluates each case with its own targeting key, so exposures are deduplicated after the first one.subjects=uniquegives every evaluation a new subject, like a service with many users: every exposure is new, and evaluation counts open a bucket per subject up to the per-flag cap. The 2^18 subject keys are built in the setup, and the context wrapper is reused per thread, so the benchmark allocates nothing per operation.ScaleCasegains a client-pathevaluate.This PR has no change to shipped code.
Motivation
Applications evaluate through the client. For a simple flag, about two thirds of the per-evaluation allocation happens outside the provider: in the OpenFeature SDK, the provider's hooks and the event hand-off. The PRs stacked on this one report their client-path impact with it:
Baseline on #13485 (JMH
-prof gc, 1 fork, 3 × 2 s warm-up, 5 × 2 s measurement, 256 MB heap, JDK 21, 5-vCPU sandbox with up to ~28% steal at 4 threads, so the times are rough and the bytes are exact):scale/flagsscale/splitsscale/wide-contextscale/semverscale/flagsscale/splitsAdditional Notes
-prof gccounts the allocation of every thread, including the runtime's background workers, so the bytes include aggregation and serialization../gradlew :dd-openfeature:jmh -Pjmh.includes=ClientEvaluationBenchmark -Pjmh.profilers=gc. One case at a given thread count:java -jar dd-openfeature/build/libs/dd-openfeature-*-jmh.jar ClientEvaluationBenchmark -p cases=scale/splits -p subjects=unique -t 4 -jvmArgsAppend "-Xms256m -Xmx256m" -prof gc.-f 3) on a quiet machine.Relation to Leo's FFE stack (#13484 and #13486–#13490, on #13485)
FlagEvaluatorand the UFC parser and model. These PRs change different code: the OpenFeature client hook (Stop taking the OpenTelemetry global lock per feature flag evaluation (FFE optimization) #13530) and the long-lived state of the event pipelines, meaning the exposure cache, the evaluation aggregator and the event payloads (Keep one object per subject in the feature flag exposure cache (FFE optimization) #13533–Build feature flag event payloads in chunks instead of large arrays (FFE optimization) #13535). Benchmark flag evaluation through the OpenFeature client (FFE optimization) #13529 adds a client-path benchmark on top of Benchmark FFE evaluation on the scale suites #13485's scale suites, covering the hooks and pipelines thatScaleSuiteBenchmarkskips.scale/splitsfrom about 40 KB to under 1 KB per evaluation, which changes the GC picture for that case.dd-openfeature/build.gradle.ktsandgradle.lockfile(Shade the perf toolbox into dd-openfeature (FFE optimization) #13532, Keep one object per subject in the feature flag exposure cache (FFE optimization) #13533), andScaleCase.java(Benchmark flag evaluation through the OpenFeature client (FFE optimization) #13529). These are textual conflicts. Whichever merges second rebases and regenerates the lockfile. Evaluate feature flag regex conditions in linear time #13487 also addsre2jas animplementationdependency. Once Shade the perf toolbox into dd-openfeature (FFE optimization) #13532 lands,re2jcould be relocated into the SDK jar the same way as the toolbox, if we want the SDK to carry no third-party runtime dependencies beyond its API contracts. That would be a follow-up, not a blocker.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Review follow-ups
🤖 Generated with Claude Code