Skip to content

Add opt-in scale, scale-json, and regex evaluation suites - #78

Merged
leoromanovsky merged 12 commits into
mainfrom
leo.romanovsky/scale-suite
Nov 18, 2026
Merged

leoromanovsky merged 12 commits into
mainfrom
leo.romanovsky/scale-suite

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

The core fixtures use small lists, short attribute values, a few allocations, and small JSON values. Customer configurations reach much larger sizes, and the cost of an evaluation grows with them: thousands of list entries, hundreds of allocations, rollouts with many splits and shards, long attribute values on regex conditions, and JSON dynamic configs of megabytes. Small fixtures do not show these costs or the bugs that appear at these sizes. For example, a backtracking regex engine takes milliseconds for .*@(a|b)$ on a 2 KB input that does not match, and an SDK that reads JSON integers as doubles returns 5.0 for 5.

These suites give every SDK the same large inputs, with expected results. SDK test suites can check behavior at scale, and benchmarks can use the same files as their input. The first consumer is the dd-trace-java benchmark in DataDog/dd-trace-java#13485, which sets a baseline on every dimension; three stacked Java PRs show their gains against it.

This PR now also contains the regex suite from #79, so one PR carries the opt-in performance suites. #79 is superseded.

Changes

Three opt-in suites. All use only syntax from the root configuration, except that regex expects the dialect of the Rust regex crate.

Suite Flags Cases What it holds
scale 10,030 109 in 14 files Targeting structure at large N and N × M, with small values
scale-json 10 13 in 3 files JSON variant values from 719 nodes to 4 MB
regex 41 69 in 4 files Literal patterns, dialect differences (including Unicode \d and \s), backtracking inputs, unsupported syntax (from #79)

Where the suites are used

flowchart LR
  subgraph fixtures["ffe-system-test-data"]
    gen["ci/generate-scale-suite.py"] -->|"--write"| scale["evaluation-suites/scale"]
    gen -->|"--write"| sjson["evaluation-suites/scale-json"]
    regex["evaluation-suites/regex"]
    core["evaluation-cases/"]
    check["Fixture validation workflow"]
  end
  check -->|"--check"| scale
  check -->|"--check"| sjson
  check -->|"regex-conformance"| regex
  subgraph java["dd-trace-java, as a submodule"]
    test["ScaleSuiteTest: every case"]
    bench["ScaleSuiteBenchmark: evaluation"]
    load["ScaleSuiteLoadBenchmark: loading"]
    canon["CanonicalFixtureTest"]
  end
  scale --> test
  sjson --> test
  scale --> bench
  sjson --> bench
  scale --> load
  sjson --> load
  regex -->|"with PR 13487"| test
  regex -->|"with PR 13487"| bench
  core --> canon
  other["Other SDKs: downstream compatibility workflow"]
  core --> other
  scale -.->|"not wired yet"| other
  sjson -.-> other
  regex -.-> other
Loading

dd-trace-java is the only consumer so far: DataDog/dd-trace-java#13485 runs every scale and scale-json case in its unit tests and benchmarks them, and DataDog/dd-trace-java#13487 adds regex. The downstream compatibility workflow still loads only evaluation-cases/, so the other SDKs read none of the new suites until their own test runners load them.

The scale and scale-json dimensions, one case file each:

  • Lists: 10,000-entry ONE_OF/NOT_ONE_OF with string and number attributes; the integers 1 to 1,000 and the powers of two to 2^52, as numeric strings in byte order; 500 allocations × 100 UUIDs (per-tenant targeting); 1,000 entries of 256 characters; 50 attributes × 200 entries in one rule.
  • Regex: 2 KB inputs; 300 conditions that share 10 patterns; 1,000 distinct patterns; one pattern with 500 alternatives; counted repetition.
  • Rules: allocations × rules × conditions at 1×1×50, 1×50×1, 50×1×1, 10×10×10, and 50×10×5, and 20 × 5 with mixed operators.
  • Ladders: 50 allocations × 2 SemVer conditions, and 50 × 2 numeric conditions, on one attribute.
  • Time windows: 100 expired and 100 future allocations before the active one.
  • Splits and shards: 50 rollout allocations of one shard each; 100 splits × 2 shards; one shard with 1,000 ranges; 20 targeted rollouts whose rule matches but whose shard does not.
  • Variations and flags: 1,000 variations × 1,000 allocations with doLog, serial IDs, and 200 split features; 10,000 flags, every 10th disabled.
  • Context: 200 attributes; 20 rules on id (the targeting key fallback) with that context.
  • JSON: 719 nodes, 2,000 keys, and 16 levels (moved from scale); a record list at 100 KB, 1 MB, and 4 MB behind a targeting rule; a 100,000-number array; a 1 MB string; 5,000 strings with escapes; 500 × 50 nested arrays.

Files:

  • ci/generate-scale-suite.py generates scale and scale-json. --write regenerates them; --check fails when they differ, and the fixture workflow runs it. Both first run a self-check: a small reference evaluator, written apart from the builders, evaluates every case, and the list order, number range, and domain rules are checked on the output. Each ordered dimension has a first match, a last match, and a miss. The scale-many-variations cases also assert the selected serialId, doLog, and split features, and the self-check derives them from the selected split.
  • ci/regex-conformance evaluates every regex case with the Rust regex crate at 1.13.1, pinned by its Cargo.lock, and the fixture workflow runs it. It fails on a different value, reason, error code, or case count. It also fails on a flag outside the suite's shape: more or fewer than one allocation, rule, condition, or split; a time window; a split with shards; or an operator other than MATCHES and NOT_MATCHES.
  • evaluation-suites/scale/AGENTS.md lists every dimension and the rules for adding one: worst case last, one variation for each allocation, derived shard results, the reason rules, byte-order lists, seeded randomness, portable syntax and numbers. scale-json/AGENTS.md points to it.
  • ci/validate-fixtures.py accepts AGENTS.md and README.md in a suite directory, with a unit test.
  • .gitattributes marks the generated files, so GitHub collapses them in the diff.
  • README.md and CONTRIBUTING.md: the suites in the tree, the suite table, and their sections.

Decisions

  • Generated, and reviewed through the generator. The output is about 24 MB of JSON. The generator is about 690 lines, and each expected result follows from how the generator builds the flag. Review the generator; the collapsed output is for consumers.
  • Shard results are derived, not recorded. The generator computes the UFC shard (the first 8 hex digits of md5("<salt>-<targetingKey>") modulo the shard count) and searches for a targeting key with the property it needs, such as "outside every earlier rollout". A new SDK still does not need a reference implementation to check the results.
  • The generator checks its own results. Without the self-check, --check proved only that the files match the generator, so a wrong reason or shard key would reach every SDK. Generator mutants that change an expected result (hash, reasons, SemVer, number lists, disabled flags) now fail --check.
  • The regex suite is checked with the Rust crate, not libdatadog. The suite promises the dialect of the Rust regex crate, so a pinned copy of that crate checks the promise directly. It builds in seconds and does not tie fixture CI to libdatadog's evaluator API. The cost is a second, small evaluator: it supports only the suite's one flag shape and fails on any other.
  • Two suites for size. scale-json is separate, so a benchmark of targeting does not parse 8 MB of JSON values, and a consumer can enable one without the other.
  • One flag or case per line. The files are compact JSON with one flag or case on each line, so a diff names the flags that changed, at about half the size of indented JSON.
  • 4 MB is the largest value. dd-trace-go and libdatadog reject a configuration response of 10 MiB or more, so each suite's ufc-config.json stays below that (scale-json is 8.1 MB), and the generator fails if one reaches it. Every consumer that checks out this repository downloads the files. AGENTS.md asks for a new shape before a bigger copy of an existing one.
  • Behavior only, no time or memory limits. Timing depends on the machine, so a limit here would fail at random in downstream CI.
  • JSON integers stay within ±2^53, so a platform with only double-precision numbers can represent every value. JSON values are compared by value, as the core cases do, so an SDK may return an integer as a float of equal value. Integers above 2^53 are the core case in Add JSON number precision fixture #77, which stays a separate PR because it changes the core contract for every SDK.
  • Synthetic data only, with reserved domains such as .test.

Verification

  • python3 ci/generate-scale-suite.py --check: the files match the generator, and the reference evaluator agrees with all 122 scale and scale-json expected results, including the serialId, doLog, and features expectations. Generator changes that shift every serial ID, drop one split feature, flip doLog, move a targeted rollout's shard range, or change a first-match result each fail it.
  • cargo run --locked --manifest-path ci/regex-conformance/Cargo.toml -- evaluation-suites/regex 69: all 69 regex cases match the Rust regex crate. Changing a match to no-match, flipping a Unicode \d result, removing a PARSE_ERROR code, adding one to a valid case, deleting a case, or changing the expected count each fail it.
  • At af0188a, before the small-integer dimension, two more evaluators agreed with every scale and scale-json result: libdatadog's libdd-ffe at 2c25337 (100 of 100; it also matches all 63 regex cases from Add an opt-in regex evaluation suite #79), and dd-trace-go's evaluator (all cases except SemVer, which that checkout predates). libdatadog main reports STATIC for the full-range rollout, as it does for the core case test-case-flag-with-empty-string.json.
  • The downstream compatibility jobs are not evidence for these suites: their runners load only evaluation-cases/.
  • python3 ci/validate-fixtures.py: 6 suites, 644 cases in 81 files.
  • python3 -m unittest discover -s ci -p 'test_*.py': 83 tests pass.
  • dd-trace-java (Benchmark FFE evaluation on the scale suites dd-trace-java#13485) evaluates every scale and scale-json case through the provider on master and fails on the first wrong value, reason, or expected serialId, doLog, or features. All 122 cases at 52205a7 pass, including the sharded ones, at Benchmark FFE evaluation on the scale suites dd-trace-java#13485 911924f. With the linear-time matcher in Evaluate feature flag regex conditions in linear time dd-trace-java#13487, the regex literal and backtracking cases pass the same check, and that PR reports all 63 regex cases from Add an opt-in regex evaluation suite #79 match libdatadog. The 6 Unicode \d and \s cases added later were checked against the Rust regex crate 1.13.1, not yet against an SDK. java.util.regex with default flags and Go's RE2 treat \d and \s as ASCII, so they fail the 4 matching cases; java.util.regex with UNICODE_CHARACTER_CLASS matches Rust on all 6.

Checklist

  • JSON files are valid (no syntax errors)
  • New test cases reference existing flags in the suite's ufc-config.json
  • New flags have corresponding test cases in the suite's cases/
  • Expected results match the flag configuration logic

The suite uses only root-configuration syntax at sizes that production
configurations reach: 10,000-entry ONE_OF and NOT_ONE_OF lists with
string and number attributes, regex conditions on 2 KB inputs, 300
conditions that share 10 patterns, 20 allocations of 5 conditions, a
200-attribute context, and JSON values with 719 nodes, 2,000 keys and
16 levels of nesting.

It is a separate suite because its configuration is about 1.3 MB and
some regex cases are slow for backtracking engines. The cases check
behavior only; they define no time or memory limits. No flag uses
shards, and all values are synthetic.
Add evaluation-suites/regex with 39 MATCHES and NOT_MATCHES flags and
63 cases in the dialect of the Rust regex crate: literal patterns,
dialect differences, inputs that make a backtracking engine slow, and
syntax that the dialect rejects. Document the suite in the README.
@leoromanovsky leoromanovsky added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
@leoromanovsky leoromanovsky removed the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
leoromanovsky and others added 3 commits November 16, 2026 19:19
- Add ci/generate-scale-suite.py with --write and --check modes, and run
  --check in the fixture validation workflow. The suite files are no
  longer hand-maintained.
- Give each of the 300 duplicate-pattern allocations its own variation,
  so a result shows which allocation won. Add cases for the first and
  last pattern groups.
- Move the condition that tells the 20 rules apart to the end of each
  rule, so an evaluator runs all 5 conditions of each rule.
- README: say that suites also isolate fixtures that are too large or
  slow for every core run, drop the production-size claim, state that
  JSON integers must stay integers, and document the generator.
- CONTRIBUTING: describe when a standard-syntax suite is the right place.
The regex and scale suites are both opt-in performance suites, so one
pull request carries both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scale suite covered one size of each dimension and no shards. It now
sweeps allocations x rules x conditions, per-tenant lists, many attributes,
1,000 distinct regex patterns, SemVer and numeric ladders, time windows,
rollouts and splits with derived shard results, 1,000 variations, and
10,000 flags. JSON values move to a new scale-json suite, which adds
records at 100 KB, 1 MB, and 10 MB and parser-heavy shapes, so a targeting
benchmark does not parse them.

Each suite has an AGENTS.md that lists the dimensions and the rules for
adding one. The validator accepts AGENTS.md and README.md in a suite, and
.gitattributes marks the generated files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky leoromanovsky changed the title Add an opt-in scale evaluation suite Add opt-in scale, scale-json, and regex evaluation suites Nov 17, 2026
@leoromanovsky leoromanovsky added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026 — with ddtool CLI
@leoromanovsky leoromanovsky removed the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
leoromanovsky and others added 2 commits November 17, 2026 04:04
--check proved only that the files match the generator, so a wrong
reason, shard key, or ordering went out to every SDK unchecked. The
generator now evaluates every case with a small reference evaluator,
written apart from the builders, and checks list order, number range,
and domains on its output. --write and --check both fail on a mismatch,
so CI verifies the expected results without an SDK.

The README compares scale JSON values by value, as the core cases do,
because SDKs with one number type, and dd-trace-java master, return
integers as floats. AGENTS.md no longer points to a core case that is not
on main, and its SDK step uses a working dd-trace-java benchmark command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only one scale ONE_OF list held numbers, and its entries were large and
evenly spaced. Lists of the integers 1 to 1,000 and of the powers of two
to 2^52 are dense or regular keys that defeat a weak hash in a lookup
table, so an evaluator with that bug now shows it at scale. Cases cover
an integer attribute at the last entry in byte order, the first entry,
a numeric-string attribute, and a miss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky and others added 2 commits November 17, 2026 04:50
The 10 MB record list made the scale-json configuration 14.1 MB, larger
than the 10 MiB response that dd-trace-go and libdatadog accept, so it
measured a payload no SDK can receive. It was also most of the suite's
size, which every consumer of this repository downloads. A 4 MB tier
keeps a multi-megabyte value and the 100 KB, 1 MB, 4 MB progression,
and the configuration is now 8.1 MB. The generator fails if a suite
configuration reaches 10 MiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Rust regex crate matches \d and \s against Unicode digits and
white space. A dialect that matches only ASCII would return a different
result for the same flag. The cases cover Arabic-Indic and fullwidth
digits, a no-break space and an ideographic space, and a zero-width
space, which is not white space.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky
leoromanovsky marked this pull request as ready for review November 17, 2026 16:20
@leoromanovsky
leoromanovsky requested a review from a team as a code owner November 17, 2026 16:20
Comment thread README.md Resolved
Comment thread evaluation-suites/scale/AGENTS.md Resolved
Comment thread ci/generate-scale-suite.py Resolved
leoromanovsky and others added 2 commits November 17, 2026 23:13
The fixture workflow checked the regex suite only for structure, so a
wrong expected value or reason could merge with green checks.
ci/regex-conformance evaluates all 69 cases with the regex crate at
1.13.1, pinned by its Cargo.lock, and fails on a different value, reason,
error code, or case count. The README now says which cases libdatadog
cross-checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AGENTS.md asks for a first match, a last match, and a miss in each
ordered dimension. 14 cases were missing, including both targeted-rollout
successes. The scale-many-variations cases now assert serialId, doLog, and
the 200 split features, and the self-check derives them from the selected
split, so a consumer that drops the metadata fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@sameerank sameerank left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the feedback!

@leoromanovsky
leoromanovsky merged commit 508a342 into main Nov 18, 2026
10 checks passed
@leoromanovsky
leoromanovsky deleted the leo.romanovsky/scale-suite branch November 18, 2026 02:11
leoromanovsky added a commit to DataDog/dd-trace-java that referenced this pull request Nov 18, 2026
DataDog/ffe-system-test-data#78 merged as 508a342 with the same suites
as its head. Main also brings mutually exclusive layer cases and
precomputed fixture corrections, which the unit and smoke tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/dd-trace-java that referenced this pull request Nov 19, 2026
Benchmark the FFE scale evaluation suite

Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.

Benchmark every scale dimension on master, and the configuration load

The scale benchmark sat on the JSON value change and checked JSON
integers by type, so it could not set a baseline on master. It now
compares JSON numbers by value, covers every case file of the scale and
scale-json suites, can isolate one case with the index parameter, and
prints each case's index. A load benchmark measures parsing each suite's
configuration and prints the heap it retains, so changes that move work
to load time show their cost.

The submodule points to the scale-suite fixture branch, which adds the
N x M dimensions and JSON values up to 10 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Check the scale suites in the unit tests and harden the benchmarks

The scale benchmarks failed the forbidden API check on System.out, and
nothing outside a manual JMH run checked the scale cases, so a sibling
change that resolves a large configuration wrongly would go unnoticed.

- Add ScaleSuiteTest: evaluate every case of the scale and scale-json
  suites twice, and fail when the benchmark misses a case file. The
  cases, their check and the suite reader move to the test sources and
  the benchmarks reuse them; the reader uses JsonReading.
- Allow System.out in the benchmark setups (@SuppressForbidden).
- Check every case again after the measurement, fail on an error code,
  and reject expectations the check does not handle.
- Rerun the jmh task when the fixtures change.
- Measure retained heap after a first parse, as the smallest of three
  signed samples.
- Describe a row as the unweighted mean over the cases of a file, say
  that hooks are not measured and when to run three forks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pin the fixtures to the latest head of the scale suite PR

The fixture PR now checks that the committed scale files match the
generator and fixes its docs. The case and configuration files are
unchanged, so the benchmark numbers do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Benchmark one-of lists of small numbers

The fixtures add a case file with ONE_OF lists of the integers 1 to
1000 and of the powers of two up to 2^52, which an index keyed on exact
doubles must still match. Pin them and benchmark the file; its 8 cases
pass on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Benchmark the scale cases with the client's layered context

The OpenFeature client passes the provider a new LayeredEvaluationContext
on each call, while the benchmark reused one MutableContext. A rule on
`id` when the context has no `id` reads the layered context's key set,
which it builds again on each call, so the benchmark did not show what
applications pay for it.

evaluateLayered wraps each case's context the way the client does, as
ExperimentationOptOutBenchmark.evaluateFreshLayeredContext does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pin the fixtures with a 4 MB largest JSON value

The fixtures replace the 10 MB JSON flag with a 4 MB one, which brings
the scale-json configuration from 14 MB down to 8 MB. The other cases
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Give each benchmark thread its own case cursor

With several JMH threads and index=all, the threads shared one
unsynchronized cursor, so a thread could advance it past the last case
and fail the run with ArrayIndexOutOfBoundsException. The cursor is now
thread-scoped state, and the provider and cases stay shared.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pin the fixtures with first-match cases and check selection metadata

The pinned scale suite adds first-match cases and expects the selected
serialId, doLog, and split features on scale-many-variations. ScaleCase
now checks those fields from the flag metadata, as CanonicalFixtureTest
does. The scale suites run without span enrichment, so an allocation
that does not log carries no doLog key, and a missing key means false.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pin the fixtures to the merged scale suites

DataDog/ffe-system-test-data#78 merged as 508a342 with the same suites
as its head. Main also brings mutually exclusive layer cases and
precomputed fixture corrections, which the unit and smoke tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Move the shared fixtures into dd-openfeature

The SDK is the main reader of the ffe-system-test-data submodule and the
only reader of the vendored experimentation opt-out cases, so both move
from the smoke-test resources to dd-openfeature/src/test/fixtures. They
stay outside any source set: the tests and benchmarks read them from
disk, and the smoke tests copy ufc-config.json and evaluation-cases into
their resources.

The jmh task declares the scale suites as a file tree, which is empty
rather than missing when the submodule is not initialized, so a jmh run
on such a checkout no longer fails before it starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Ignore the JVM settings in the scale suites

The suites read the system properties and environment, so setting
DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED turned span
enrichment on: the benchmarks measured the enriched metadata path and
the expected selection metadata no longer described it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Parse each scale suite once in the test

Each case file parsed its suite again, 14 times for the 7.8 MB scale
configuration. One provider per suite, shut down after all cases, parses
each configuration once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Check the benchmark's own case-file list

ScaleSuiteTest compared the fixtures with a copy of the benchmark's
@PARAM list, so a case file missing from the benchmark went unnoticed.
The test now reads the @PARAM by reflection, with the jmh classes on
its runtime classpath.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge branch 'master' into leo.romanovsky/ffe-scale-suite-benchmark

Keep the fixtures submodule at dd-openfeature/src/test/fixtures, pinned
to 508a342. Master bumped it at the old smoke-test path to 29ba6bf,
which is 4 commits behind that pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Parenthesize the cursor wrap condition

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants