Repository navigation
Add opt-in scale, scale-json, and regex evaluation suites - #78
Merged
Merged
Conversation
The suite uses only root-configuration syntax at sizes that production configurations reach: 10,000-entry ONE_OF and NOT_ONE_OF lists with string and number attributes, regex conditions on 2 KB inputs, 300 conditions that share 10 patterns, 20 allocations of 5 conditions, a 200-attribute context, and JSON values with 719 nodes, 2,000 keys and 16 levels of nesting. It is a separate suite because its configuration is about 1.3 MB and some regex cases are slow for backtracking engines. The cases check behavior only; they define no time or memory limits. No flag uses shards, and all values are synthetic.
Add evaluation-suites/regex with 39 MATCHES and NOT_MATCHES flags and 63 cases in the dialect of the Rust regex crate: literal patterns, dialect differences, inputs that make a backtracking engine slow, and syntax that the dialect rejects. Document the suite in the README.
1 task done
- Add ci/generate-scale-suite.py with --write and --check modes, and run --check in the fixture validation workflow. The suite files are no longer hand-maintained. - Give each of the 300 duplicate-pattern allocations its own variation, so a result shows which allocation won. Add cases for the first and last pattern groups. - Move the condition that tells the 20 rules apart to the end of each rule, so an evaluator runs all 5 conditions of each rule. - README: say that suites also isolate fixtures that are too large or slow for every core run, drop the production-size claim, state that JSON integers must stay integers, and document the generator. - CONTRIBUTING: describe when a standard-syntax suite is the right place.
The regex and scale suites are both opt-in performance suites, so one pull request carries both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scale suite covered one size of each dimension and no shards. It now sweeps allocations x rules x conditions, per-tenant lists, many attributes, 1,000 distinct regex patterns, SemVer and numeric ladders, time windows, rollouts and splits with derived shard results, 1,000 variations, and 10,000 flags. JSON values move to a new scale-json suite, which adds records at 100 KB, 1 MB, and 10 MB and parser-heavy shapes, so a targeting benchmark does not parse them. Each suite has an AGENTS.md that lists the dimensions and the rules for adding one. The validator accepts AGENTS.md and README.md in a suite, and .gitattributes marks the generated files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--check proved only that the files match the generator, so a wrong reason, shard key, or ordering went out to every SDK unchecked. The generator now evaluates every case with a small reference evaluator, written apart from the builders, and checks list order, number range, and domains on its output. --write and --check both fail on a mismatch, so CI verifies the expected results without an SDK. The README compares scale JSON values by value, as the core cases do, because SDKs with one number type, and dd-trace-java master, return integers as floats. AGENTS.md no longer points to a core case that is not on main, and its SDK step uses a working dd-trace-java benchmark command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only one scale ONE_OF list held numbers, and its entries were large and evenly spaced. Lists of the integers 1 to 1,000 and of the powers of two to 2^52 are dense or regular keys that defeat a weak hash in a lookup table, so an evaluator with that bug now shows it at scale. Cases cover an integer attribute at the last entry in byte order, the first entry, a numeric-string attribute, and a miss. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 10 MB record list made the scale-json configuration 14.1 MB, larger than the 10 MiB response that dd-trace-go and libdatadog accept, so it measured a payload no SDK can receive. It was also most of the suite's size, which every consumer of this repository downloads. A 4 MB tier keeps a multi-megabyte value and the 100 KB, 1 MB, 4 MB progression, and the configuration is now 8.1 MB. The generator fails if a suite configuration reaches 10 MiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Rust regex crate matches \d and \s against Unicode digits and white space. A dialect that matches only ASCII would return a different result for the same flag. The cases cover Arabic-Indic and fullwidth digits, a no-break space and an ideographic space, and a zero-width space, which is not white space. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sameerank
reviewed
Nov 17, 2026
sameerank
reviewed
Nov 17, 2026
sameerank
reviewed
Nov 17, 2026
The fixture workflow checked the regex suite only for structure, so a wrong expected value or reason could merge with green checks. ci/regex-conformance evaluates all 69 cases with the regex crate at 1.13.1, pinned by its Cargo.lock, and fails on a different value, reason, error code, or case count. The README now says which cases libdatadog cross-checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AGENTS.md asks for a first match, a last match, and a miss in each ordered dimension. 14 cases were missing, including both targeted-rollout successes. The scale-many-variations cases now assert serialId, doLog, and the 200 split features, and the self-check derives them from the selected split, so a consumer that drops the metadata fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sameerank
approved these changes
Nov 17, 2026
sameerank
left a comment
There was a problem hiding this comment.
Thanks for addressing the feedback!
1 task
This was referenced Nov 18, 2026
Draft
leoromanovsky
added a commit
to DataDog/dd-trace-java
that referenced
this pull request
Nov 18, 2026
DataDog/ffe-system-test-data#78 merged as 508a342 with the same suites as its head. Main also brings mutually exclusive layer cases and precomputed fixture corrections, which the unit and smoke tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 18, 2026
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/dd-trace-java
that referenced
this pull request
Nov 19, 2026
Benchmark the FFE scale evaluation suite Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head. Benchmark every scale dimension on master, and the configuration load The scale benchmark sat on the JSON value change and checked JSON integers by type, so it could not set a baseline on master. It now compares JSON numbers by value, covers every case file of the scale and scale-json suites, can isolate one case with the index parameter, and prints each case's index. A load benchmark measures parsing each suite's configuration and prints the heap it retains, so changes that move work to load time show their cost. The submodule points to the scale-suite fixture branch, which adds the N x M dimensions and JSON values up to 10 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Check the scale suites in the unit tests and harden the benchmarks The scale benchmarks failed the forbidden API check on System.out, and nothing outside a manual JMH run checked the scale cases, so a sibling change that resolves a large configuration wrongly would go unnoticed. - Add ScaleSuiteTest: evaluate every case of the scale and scale-json suites twice, and fail when the benchmark misses a case file. The cases, their check and the suite reader move to the test sources and the benchmarks reuse them; the reader uses JsonReading. - Allow System.out in the benchmark setups (@SuppressForbidden). - Check every case again after the measurement, fail on an error code, and reject expectations the check does not handle. - Rerun the jmh task when the fixtures change. - Measure retained heap after a first parse, as the smallest of three signed samples. - Describe a row as the unweighted mean over the cases of a file, say that hooks are not measured and when to run three forks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Pin the fixtures to the latest head of the scale suite PR The fixture PR now checks that the committed scale files match the generator and fixes its docs. The case and configuration files are unchanged, so the benchmark numbers do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Benchmark one-of lists of small numbers The fixtures add a case file with ONE_OF lists of the integers 1 to 1000 and of the powers of two up to 2^52, which an index keyed on exact doubles must still match. Pin them and benchmark the file; its 8 cases pass on master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Benchmark the scale cases with the client's layered context The OpenFeature client passes the provider a new LayeredEvaluationContext on each call, while the benchmark reused one MutableContext. A rule on `id` when the context has no `id` reads the layered context's key set, which it builds again on each call, so the benchmark did not show what applications pay for it. evaluateLayered wraps each case's context the way the client does, as ExperimentationOptOutBenchmark.evaluateFreshLayeredContext does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Pin the fixtures with a 4 MB largest JSON value The fixtures replace the 10 MB JSON flag with a 4 MB one, which brings the scale-json configuration from 14 MB down to 8 MB. The other cases are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Give each benchmark thread its own case cursor With several JMH threads and index=all, the threads shared one unsynchronized cursor, so a thread could advance it past the last case and fail the run with ArrayIndexOutOfBoundsException. The cursor is now thread-scoped state, and the provider and cases stay shared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Pin the fixtures with first-match cases and check selection metadata The pinned scale suite adds first-match cases and expects the selected serialId, doLog, and split features on scale-many-variations. ScaleCase now checks those fields from the flag metadata, as CanonicalFixtureTest does. The scale suites run without span enrichment, so an allocation that does not log carries no doLog key, and a missing key means false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Pin the fixtures to the merged scale suites DataDog/ffe-system-test-data#78 merged as 508a342 with the same suites as its head. Main also brings mutually exclusive layer cases and precomputed fixture corrections, which the unit and smoke tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Move the shared fixtures into dd-openfeature The SDK is the main reader of the ffe-system-test-data submodule and the only reader of the vendored experimentation opt-out cases, so both move from the smoke-test resources to dd-openfeature/src/test/fixtures. They stay outside any source set: the tests and benchmarks read them from disk, and the smoke tests copy ufc-config.json and evaluation-cases into their resources. The jmh task declares the scale suites as a file tree, which is empty rather than missing when the submodule is not initialized, so a jmh run on such a checkout no longer fails before it starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Ignore the JVM settings in the scale suites The suites read the system properties and environment, so setting DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED turned span enrichment on: the benchmarks measured the enriched metadata path and the expected selection metadata no longer described it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Parse each scale suite once in the test Each case file parsed its suite again, 14 times for the 7.8 MB scale configuration. One provider per suite, shut down after all cases, parses each configuration once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Check the benchmark's own case-file list ScaleSuiteTest compared the fixtures with a copy of the benchmark's @PARAM list, so a case file missing from the benchmark went unnoticed. The test now reads the @PARAM by reflection, with the jmh classes on its runtime classpath. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Merge branch 'master' into leo.romanovsky/ffe-scale-suite-benchmark Keep the fixtures submodule at dd-openfeature/src/test/fixtures, pinned to 508a342. Master bumped it at the old smoke-test path to 29ba6bf, which is 4 commits behind that pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Parenthesize the cursor wrap condition Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The core fixtures use small lists, short attribute values, a few allocations, and small JSON values. Customer configurations reach much larger sizes, and the cost of an evaluation grows with them: thousands of list entries, hundreds of allocations, rollouts with many splits and shards, long attribute values on regex conditions, and JSON dynamic configs of megabytes. Small fixtures do not show these costs or the bugs that appear at these sizes. For example, a backtracking regex engine takes milliseconds for
.*@(a|b)$on a 2 KB input that does not match, and an SDK that reads JSON integers as doubles returns5.0for5.These suites give every SDK the same large inputs, with expected results. SDK test suites can check behavior at scale, and benchmarks can use the same files as their input. The first consumer is the dd-trace-java benchmark in DataDog/dd-trace-java#13485, which sets a baseline on every dimension; three stacked Java PRs show their gains against it.
This PR now also contains the
regexsuite from #79, so one PR carries the opt-in performance suites. #79 is superseded.Changes
Three opt-in suites. All use only syntax from the root configuration, except that
regexexpects the dialect of the Rustregexcrate.scalescale-jsonregex\dand\s), backtracking inputs, unsupported syntax (from #79)Where the suites are used
flowchart LR subgraph fixtures["ffe-system-test-data"] gen["ci/generate-scale-suite.py"] -->|"--write"| scale["evaluation-suites/scale"] gen -->|"--write"| sjson["evaluation-suites/scale-json"] regex["evaluation-suites/regex"] core["evaluation-cases/"] check["Fixture validation workflow"] end check -->|"--check"| scale check -->|"--check"| sjson check -->|"regex-conformance"| regex subgraph java["dd-trace-java, as a submodule"] test["ScaleSuiteTest: every case"] bench["ScaleSuiteBenchmark: evaluation"] load["ScaleSuiteLoadBenchmark: loading"] canon["CanonicalFixtureTest"] end scale --> test sjson --> test scale --> bench sjson --> bench scale --> load sjson --> load regex -->|"with PR 13487"| test regex -->|"with PR 13487"| bench core --> canon other["Other SDKs: downstream compatibility workflow"] core --> other scale -.->|"not wired yet"| other sjson -.-> other regex -.-> otherdd-trace-java is the only consumer so far: DataDog/dd-trace-java#13485 runs every
scaleandscale-jsoncase in its unit tests and benchmarks them, and DataDog/dd-trace-java#13487 addsregex. The downstream compatibility workflow still loads onlyevaluation-cases/, so the other SDKs read none of the new suites until their own test runners load them.The
scaleandscale-jsondimensions, one case file each:ONE_OF/NOT_ONE_OFwith string and number attributes; the integers 1 to 1,000 and the powers of two to 2^52, as numeric strings in byte order; 500 allocations × 100 UUIDs (per-tenant targeting); 1,000 entries of 256 characters; 50 attributes × 200 entries in one rule.doLog, serial IDs, and 200 split features; 10,000 flags, every 10th disabled.id(the targeting key fallback) with that context.scale); a record list at 100 KB, 1 MB, and 4 MB behind a targeting rule; a 100,000-number array; a 1 MB string; 5,000 strings with escapes; 500 × 50 nested arrays.Files:
ci/generate-scale-suite.pygeneratesscaleandscale-json.--writeregenerates them;--checkfails when they differ, and the fixture workflow runs it. Both first run a self-check: a small reference evaluator, written apart from the builders, evaluates every case, and the list order, number range, and domain rules are checked on the output. Each ordered dimension has a first match, a last match, and a miss. Thescale-many-variationscases also assert the selectedserialId,doLog, and split features, and the self-check derives them from the selected split.ci/regex-conformanceevaluates everyregexcase with the Rustregexcrate at 1.13.1, pinned by itsCargo.lock, and the fixture workflow runs it. It fails on a different value, reason, error code, or case count. It also fails on a flag outside the suite's shape: more or fewer than one allocation, rule, condition, or split; a time window; a split with shards; or an operator other thanMATCHESandNOT_MATCHES.evaluation-suites/scale/AGENTS.mdlists every dimension and the rules for adding one: worst case last, one variation for each allocation, derived shard results, the reason rules, byte-order lists, seeded randomness, portable syntax and numbers.scale-json/AGENTS.mdpoints to it.ci/validate-fixtures.pyacceptsAGENTS.mdandREADME.mdin a suite directory, with a unit test..gitattributesmarks the generated files, so GitHub collapses them in the diff.README.mdandCONTRIBUTING.md: the suites in the tree, the suite table, and their sections.Decisions
md5("<salt>-<targetingKey>")modulo the shard count) and searches for a targeting key with the property it needs, such as "outside every earlier rollout". A new SDK still does not need a reference implementation to check the results.--checkproved only that the files match the generator, so a wrong reason or shard key would reach every SDK. Generator mutants that change an expected result (hash, reasons, SemVer, number lists, disabled flags) now fail--check.regexcrate, so a pinned copy of that crate checks the promise directly. It builds in seconds and does not tie fixture CI to libdatadog's evaluator API. The cost is a second, small evaluator: it supports only the suite's one flag shape and fails on any other.scale-jsonis separate, so a benchmark of targeting does not parse 8 MB of JSON values, and a consumer can enable one without the other.ufc-config.jsonstays below that (scale-jsonis 8.1 MB), and the generator fails if one reaches it. Every consumer that checks out this repository downloads the files.AGENTS.mdasks for a new shape before a bigger copy of an existing one..test.Verification
python3 ci/generate-scale-suite.py --check: the files match the generator, and the reference evaluator agrees with all 122scaleandscale-jsonexpected results, including theserialId,doLog, and features expectations. Generator changes that shift every serial ID, drop one split feature, flipdoLog, move a targeted rollout's shard range, or change a first-match result each fail it.cargo run --locked --manifest-path ci/regex-conformance/Cargo.toml -- evaluation-suites/regex 69: all 69regexcases match the Rustregexcrate. Changing a match to no-match, flipping a Unicode\dresult, removing aPARSE_ERRORcode, adding one to a valid case, deleting a case, or changing the expected count each fail it.af0188a, before the small-integer dimension, two more evaluators agreed with everyscaleandscale-jsonresult: libdatadog'slibdd-ffeat2c25337(100 of 100; it also matches all 63regexcases from Add an opt-in regex evaluation suite #79), and dd-trace-go's evaluator (all cases except SemVer, which that checkout predates). libdatadogmainreportsSTATICfor the full-range rollout, as it does for the core casetest-case-flag-with-empty-string.json.evaluation-cases/.python3 ci/validate-fixtures.py: 6 suites, 644 cases in 81 files.python3 -m unittest discover -s ci -p 'test_*.py': 83 tests pass.scaleandscale-jsoncase through the provider onmasterand fails on the first wrong value, reason, or expectedserialId,doLog, or features. All 122 cases at52205a7pass, including the sharded ones, at Benchmark FFE evaluation on the scale suites dd-trace-java#13485911924f. With the linear-time matcher in Evaluate feature flag regex conditions in linear time dd-trace-java#13487, theregexliteral and backtracking cases pass the same check, and that PR reports all 63regexcases from Add an opt-in regex evaluation suite #79 match libdatadog. The 6 Unicode\dand\scases added later were checked against the Rustregexcrate 1.13.1, not yet against an SDK.java.util.regexwith default flags and Go's RE2 treat\dand\sas ASCII, so they fail the 4 matching cases;java.util.regexwithUNICODE_CHARACTER_CLASSmatches Rust on all 6.Checklist
ufc-config.jsoncases/