Repository navigation
Benchmark FFE evaluation on the scale suites #13485
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
gh-worker-dd-mergequeue-cf854d
merged 16 commits into
master
from
leo.romanovsky/ffe-scale-suite-benchmark
Nov 19, 2026
Merged
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
92ed06e
Benchmark the FFE scale evaluation suite
leoromanovsky c18784a
Benchmark every scale dimension on master, and the configuration load
leoromanovsky a887657
Check the scale suites in the unit tests and harden the benchmarks
leoromanovsky 09289ff
Pin the fixtures to the latest head of the scale suite PR
leoromanovsky 30df999
Benchmark one-of lists of small numbers
leoromanovsky 8d0f39c
Benchmark the scale cases with the client's layered context
leoromanovsky 72d60ca
Pin the fixtures with a 4 MB largest JSON value
leoromanovsky e69a0af
Give each benchmark thread its own case cursor
leoromanovsky 911924f
Pin the fixtures with first-match cases and check selection metadata
leoromanovsky 89a6035
Pin the fixtures to the merged scale suites
leoromanovsky 89120c7
Move the shared fixtures into dd-openfeature
leoromanovsky 6f3c1dd
Ignore the JVM settings in the scale suites
leoromanovsky 9b7f7de
Parse each scale suite once in the test
leoromanovsky 109bd9f
Check the benchmark's own case-file list
leoromanovsky 53243ac
Merge branch 'master' into leo.romanovsky/ffe-scale-suite-benchmark
leoromanovsky 971fcdf
Parenthesize the cursor wrap condition
leoromanovsky File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| [submodule "dd-java-agent/agent-jmxfetch/integrations-core"] | ||
| path = dd-java-agent/agent-jmxfetch/integrations-core | ||
| url = https://github.com/DataDog/integrations-core.git | ||
| [submodule "dd-smoke-tests/openfeature/src/test/resources/ffe-system-test-data"] | ||
| path = dd-smoke-tests/openfeature/src/test/resources/ffe-system-test-data | ||
| [submodule "dd-openfeature/src/test/fixtures/ffe-system-test-data"] | ||
| path = dd-openfeature/src/test/fixtures/ffe-system-test-data | ||
| url = https://github.com/DataDog/ffe-system-test-data.git |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
153 changes: 153 additions & 0 deletions
153
dd-openfeature/src/jmh/java/com/datadog/openfeature/ScaleSuiteBenchmark.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,153 @@ | ||
| package com.datadog.openfeature; | ||
|
|
||
| import static java.util.concurrent.TimeUnit.NANOSECONDS; | ||
| import static java.util.concurrent.TimeUnit.SECONDS; | ||
|
|
||
| import de.thetaphi.forbiddenapis.SuppressForbidden; | ||
| import dev.openfeature.sdk.ImmutableContext; | ||
| import dev.openfeature.sdk.LayeredEvaluationContext; | ||
| import dev.openfeature.sdk.ProviderEvaluation; | ||
| import java.io.IOException; | ||
| import java.util.List; | ||
| import org.openjdk.jmh.annotations.Benchmark; | ||
| import org.openjdk.jmh.annotations.BenchmarkMode; | ||
| import org.openjdk.jmh.annotations.Fork; | ||
| import org.openjdk.jmh.annotations.Level; | ||
| import org.openjdk.jmh.annotations.Measurement; | ||
| import org.openjdk.jmh.annotations.Mode; | ||
| import org.openjdk.jmh.annotations.OutputTimeUnit; | ||
| import org.openjdk.jmh.annotations.Param; | ||
| import org.openjdk.jmh.annotations.Scope; | ||
| import org.openjdk.jmh.annotations.Setup; | ||
| import org.openjdk.jmh.annotations.State; | ||
| import org.openjdk.jmh.annotations.TearDown; | ||
| import org.openjdk.jmh.annotations.Warmup; | ||
|
|
||
| /** | ||
| * Evaluation cost on the opt-in scale suites of the shared FFE fixtures ({@code | ||
| * evaluation-suites/scale} and {@code evaluation-suites/scale-json}): large lists, regex, rules, | ||
| * splits and shards, time windows, many variations and flags, and JSON values up to 4 MB. The | ||
| * dimensions are listed in {@code evaluation-suites/scale/AGENTS.md} of the fixtures. | ||
| * | ||
| * <p>Each operation evaluates a case of the selected case file through the provider. With {@code | ||
| * index=all}, each benchmark thread goes through the cases of the file in turn, so a row is the | ||
| * unweighted mean over every case of the file, and its slowest case dominates it. Select one case | ||
| * to measure one shape: the setup prints the index of each case. | ||
| * | ||
| * <p>The setup evaluates every case once and the teardown again after the measurement. Both fail if | ||
| * a value, reason, or expected selection metadata (serial ID, {@code doLog}, split features) | ||
| * differs from the expected one, or if an evaluation reports an error, so a run that resolves a | ||
| * case wrongly fails instead of reporting a number. JSON values are compared by numeric value: a | ||
| * JSON integer can come back as an integer or as a double. This check guards the measurement and is | ||
| * not the conformance check: {@code CanonicalFixtureTest} checks the types. | ||
| * | ||
| * <p>The provider is called directly: the OpenFeature client, its hooks and exposure logging are | ||
| * not measured. {@code evaluate} passes a reused context. {@code evaluateLayered} passes a fresh | ||
| * {@code LayeredEvaluationContext} around it on each call, as the client does, so it also shows the | ||
| * cost of reading attributes through the client's context, such as a rule on {@code id} when the | ||
| * context has no {@code id}. Select one method with {@code | ||
| * -Pjmh.includes=ScaleSuiteBenchmark.evaluate$}. | ||
| * | ||
| * <p>The cases and their check are shared with {@code ScaleSuiteTest}, which runs them in the unit | ||
| * tests. One fork is enough for allocation and for large time differences. For a time difference | ||
| * under about 15%, run three forks: {@code -Pjmh.forks=3} or {@code -f 3}. | ||
| * | ||
| * <p>Run all case files: {@code ./gradlew :dd-openfeature:jmh -Pjmh.includes=ScaleSuiteBenchmark | ||
| * -Pjmh.profilers=gc}. Run one case: {@code ./gradlew :dd-openfeature:jmhJar}, then {@code java | ||
| * -jar dd-openfeature/build/libs/*-jmh.jar ScaleSuiteBenchmark -p cases=scale/splits -p index=0 | ||
| * -prof gc}. | ||
| */ | ||
| @State(Scope.Benchmark) | ||
| @Warmup(iterations = 3, time = 2, timeUnit = SECONDS) | ||
| @Measurement(iterations = 5, time = 2, timeUnit = SECONDS) | ||
| @BenchmarkMode(Mode.AverageTime) | ||
| @OutputTimeUnit(NANOSECONDS) | ||
| @Fork(value = 1) | ||
| public class ScaleSuiteBenchmark { | ||
|
|
||
| /** | ||
| * A case file, as {@code <suite>/<file>} without the {@code .json} extension. {@code | ||
| * ScaleSuiteTest} reads this list and fails unless it names every case file of the fixtures, in | ||
| * sorted order. | ||
| */ | ||
| @Param({ | ||
| "scale/flags", | ||
| "scale/numeric", | ||
| "scale/one-of", | ||
| "scale/one-of-many", | ||
| "scale/one-of-small-numbers", | ||
| "scale/regex", | ||
| "scale/regex-many", | ||
| "scale/rules", | ||
| "scale/rules-sweep", | ||
| "scale/semver", | ||
| "scale/splits", | ||
| "scale/time-windows", | ||
| "scale/variations", | ||
| "scale/wide-context", | ||
| "scale-json/json-shapes", | ||
| "scale-json/json-sizes", | ||
| "scale-json/json-values" | ||
| }) | ||
| public String cases; | ||
|
|
||
| /** The index of one case in the file, or {@code all} to go through every case in turn. */ | ||
| @Param({"all"}) | ||
| public String index; | ||
|
|
||
| private Provider provider; | ||
| private ScaleCase[] scaleCases; | ||
|
|
||
| /** The position of one benchmark thread in the cases, so threads share no mutable state. */ | ||
| @State(Scope.Thread) | ||
| public static class Cursor { | ||
| private int next; | ||
|
|
||
| ScaleCase next(final ScaleCase[] scaleCases) { | ||
| final ScaleCase scaleCase = scaleCases[next]; | ||
| next = (next + 1 == scaleCases.length) ? 0 : next + 1; | ||
| return scaleCase; | ||
| } | ||
| } | ||
|
|
||
| @Setup(Level.Trial) | ||
| @SuppressForbidden | ||
| public void setUp() throws IOException { | ||
| final int slash = cases.indexOf('/'); | ||
| final ScaleSuites.Suite suite = ScaleSuites.load(cases.substring(0, slash)); | ||
| provider = suite.provider(); | ||
| final List<ScaleCase> loaded = suite.cases(cases.substring(slash + 1)); | ||
| for (int i = 0; i < loaded.size(); i++) { | ||
| final ScaleCase scaleCase = loaded.get(i); | ||
| scaleCase.check(scaleCase.evaluate(provider)); | ||
| System.out.println("[" + cases + "] case " + i + ": " + scaleCase.describe()); | ||
| } | ||
| scaleCases = | ||
| "all".equals(index) | ||
| ? loaded.toArray(new ScaleCase[0]) | ||
| : new ScaleCase[] {loaded.get(Integer.parseInt(index))}; | ||
| } | ||
|
|
||
| @TearDown(Level.Trial) | ||
| public void tearDown() { | ||
| for (final ScaleCase scaleCase : scaleCases) { | ||
| scaleCase.check(scaleCase.evaluate(provider)); | ||
| } | ||
| provider.shutdown(); | ||
| } | ||
|
|
||
| @Benchmark | ||
| public ProviderEvaluation<?> evaluate(final Cursor cursor) { | ||
| return cursor.next(scaleCases).evaluate(provider); | ||
| } | ||
|
|
||
| /** As {@link #evaluate}, with the fresh layered context that the OpenFeature client builds. */ | ||
| @Benchmark | ||
| public ProviderEvaluation<?> evaluateLayered(final Cursor cursor) { | ||
| final ScaleCase scaleCase = cursor.next(scaleCases); | ||
| return scaleCase.evaluate( | ||
| provider, | ||
| new LayeredEvaluationContext( | ||
| ImmutableContext.EMPTY, ImmutableContext.EMPTY, null, scaleCase.context())); | ||
| } | ||
| } | ||
90 changes: 90 additions & 0 deletions
90
dd-openfeature/src/jmh/java/com/datadog/openfeature/ScaleSuiteLoadBenchmark.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| package com.datadog.openfeature; | ||
|
|
||
| import static java.util.concurrent.TimeUnit.MILLISECONDS; | ||
| import static java.util.concurrent.TimeUnit.SECONDS; | ||
|
|
||
| import com.datadog.openfeature.internal.ufc.ServerConfiguration; | ||
| import com.datadog.openfeature.internal.ufc.UniversalFlagConfigParser; | ||
| import de.thetaphi.forbiddenapis.SuppressForbidden; | ||
| import java.io.IOException; | ||
| import java.lang.management.ManagementFactory; | ||
| import java.lang.management.MemoryMXBean; | ||
| import java.lang.ref.Reference; | ||
| import org.openjdk.jmh.annotations.Benchmark; | ||
| import org.openjdk.jmh.annotations.BenchmarkMode; | ||
| import org.openjdk.jmh.annotations.Fork; | ||
| import org.openjdk.jmh.annotations.Level; | ||
| import org.openjdk.jmh.annotations.Measurement; | ||
| import org.openjdk.jmh.annotations.Mode; | ||
| import org.openjdk.jmh.annotations.OutputTimeUnit; | ||
| import org.openjdk.jmh.annotations.Param; | ||
| import org.openjdk.jmh.annotations.Scope; | ||
| import org.openjdk.jmh.annotations.Setup; | ||
| import org.openjdk.jmh.annotations.State; | ||
| import org.openjdk.jmh.annotations.Warmup; | ||
|
|
||
| /** | ||
| * Configuration load cost of the opt-in scale suites: the time to parse each suite's {@code | ||
| * ufc-config.json}. Work that moves into the parser, such as indexes, compiled patterns or prebuilt | ||
| * JSON values, shows here. | ||
| * | ||
| * <p>The setup also prints the heap that a parsed configuration retains. It is the used heap after | ||
| * a full collection, with and without the parsed configuration, as the smallest of three | ||
| * measurements, so it is approximate. | ||
| * | ||
| * <p>Run: {@code ./gradlew :dd-openfeature:jmh -Pjmh.includes=ScaleSuiteLoadBenchmark | ||
| * -Pjmh.profilers=gc}. | ||
| */ | ||
| @State(Scope.Benchmark) | ||
| @Warmup(iterations = 3, time = 2, timeUnit = SECONDS) | ||
| @Measurement(iterations = 5, time = 2, timeUnit = SECONDS) | ||
| @BenchmarkMode(Mode.AverageTime) | ||
| @OutputTimeUnit(MILLISECONDS) | ||
| @Fork(value = 1) | ||
| public class ScaleSuiteLoadBenchmark { | ||
|
|
||
| @Param({"scale", "scale-json"}) | ||
| public String suite; | ||
|
|
||
| private byte[] config; | ||
|
|
||
| @Setup(Level.Trial) | ||
| @SuppressForbidden | ||
| public void setUp() throws IOException { | ||
| config = ScaleSuites.load(suite).config(); | ||
| // The first parse initializes the parser, so it is left out of the measurement. | ||
| ServerConfiguration retained = UniversalFlagConfigParser.parse(config); | ||
| long retainedBytes = Long.MAX_VALUE; | ||
| for (int i = 0; i < 3; i++) { | ||
| retained = null; | ||
| final long before = usedHeapAfterCollection(); | ||
| retained = UniversalFlagConfigParser.parse(config); | ||
| final long after = usedHeapAfterCollection(); | ||
| Reference.reachabilityFence(retained); | ||
| retainedBytes = Math.min(retainedBytes, after - before); | ||
| } | ||
| System.out.println( | ||
| "[" | ||
| + suite | ||
| + "] config " | ||
| + config.length / 1024 | ||
| + " KB, " | ||
| + retained.flags.size() | ||
| + " flags, retained heap about " | ||
| + retainedBytes / 1024 | ||
| + " KB"); | ||
| } | ||
|
|
||
| @Benchmark | ||
| public ServerConfiguration parse() throws IOException { | ||
| return UniversalFlagConfigParser.parse(config); | ||
| } | ||
|
|
||
| private static long usedHeapAfterCollection() { | ||
| final MemoryMXBean memory = ManagementFactory.getMemoryMXBean(); | ||
| for (int i = 0; i < 3; i++) { | ||
| System.gc(); | ||
| } | ||
| return memory.getHeapMemoryUsage().getUsed(); | ||
| } | ||
| } |
File renamed without changes.
File renamed without changes.
File renamed without changes.
Submodule ffe-system-test-data
added at
508a34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.