Repository navigation
Read flag attributes once per evaluation and cache split feature metadata - #13489
leoromanovsky wants to merge 18 commits into
Conversation
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON integers by type, so it could not set a baseline on master. It now compares JSON numbers by value, covers every case file of the scale and scale-json suites, can isolate one case with the index parameter, and prints each case's index. A load benchmark measures parsing each suite's configuration and prints the heap it retains, so changes that move work to load time show their cost. The submodule points to the scale-suite fixture branch, which adds the N x M dimensions and JSON values up to 10 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… at load A flag with many allocations re-read the same context attributes in every allocation: a context lookup and a value conversion each time, and the conversion boxes numbers. Each evaluation now keeps the attributes it has resolved, so a flag with 1,000 allocations reads its attribute once. A matched split's features were turned into metadata entries on every evaluation, concatenating a prefixed key for each feature and destination. The split now resolves its entries once, when it is built, and an evaluation only copies them. Unknown destinations are now reported when the configuration loads, still once per name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The split caches what the evaluator derives from its features instead of building it in its constructor, so the configuration model does not depend on the evaluator and an unknown destination is still reported when a split is served, as before. The allocation and rule scans keep their for-each loops. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most flags read one attribute, so the evaluation scope keeps it in fields and allocates its arrays only for a second attribute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
The scale benchmarks failed the forbidden API check on System.out, and nothing outside a manual JMH run checked the scale cases, so a sibling change that resolves a large configuration wrongly would go unnoticed. - Add ScaleSuiteTest: evaluate every case of the scale and scale-json suites twice, and fail when the benchmark misses a case file. The cases, their check and the suite reader move to the test sources and the benchmarks reuse them; the reader uses JsonReading. - Allow System.out in the benchmark setups (@SuppressForbidden). - Check every case again after the measurement, fail on an error code, and reject expectations the check does not handle. - Rerun the jmh task when the fixtures change. - Measure retained heap after a first parse, as the smallest of three signed samples. - Describe a row as the unweighted mean over the cases of a file, say that hooks are not measured and when to run three forks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture PR now checks that the committed scale files match the generator and fixes its docs. The case and configuration files are unchanged, so the benchmark numbers do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to 1000 and of the powers of two up to 2^52, which an index keyed on exact doubles must still match. Pin them and benchmark the file; its 8 cases pass on master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
A few ideas from an experiment on top of #13485 (measured against #13529, the client-path benchmark stacked on #13485; 1 fork on a noisy 5-vCPU box, so the bytes are exact and the times are rough): 1. final Value value = context.getValue(name);
if (value == null) {
return ID.equals(name) ? context.getTargetingKey() : null;
}
return context.convertValue(value);On the layered wide context, this cut 7.0 KB to 416 B per evaluation (−94%), and the time by about 90%. One edge case changes: a raw Java 2. 3. Sharing the variant metadata instead of caching features on the split. The remaining 28 KB on the 200-feature case is the
It also avoids caching on each |
Very cool thinking! Trying it out myself to get a feel for the cache behavior. The shared metadata table is what I’m most curious about. I’m trying it out myself as a separate PR against DataDog/dd-trace-java#13485 · Benchmark FFE evaluation on the…, with you as co-author, so its gain shows up apart from this PR’s. I want to get a feel for how the table behaves under load. One thing I’m exploring is a wider time key than 1 ms. Exposures take their own timestamp, and the evaluation events only report first_evaluation/last_evaluation per 10 s window. So a bucket of, say, 100 ms would only skew last_evaluation by that much, and each entry would get 100x longer to be reused. I’m experimenting with bucket width, table size, thread count and how skewed flag traffic is, to find where hit rate and timestamp precision balance out. Was 1 ms keying just about keeping the timestamp exact, or did you see something else that pushed you there? |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The id fallback tested context.keySet() for one key. A layered context builds that set from every layer on each call, and the OpenFeature client creates a new layered context per evaluation. getValue returns null only for an absent key, so it answers the same question without the copy. A raw Java null stored under id, which a context built from a Java map can hold, now falls back to the targeting key as an absent id does. A null Value behaves as before. Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A default, disabled or error result carries only the consent flag, which has two values. Two shared instances replace a new builder and map per result. Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A flag whose conditions each read a different attribute gained nothing from the per-evaluation cache and paid for its arrays and map: a 50-attribute rule took about 2.7 times as long. The flag now records, the first time it is evaluated, whether two conditions read the same attribute, and only such flags keep the resolved attributes. A condition without an attribute is read as before and never kept, as it could break later lookups. The feature destination constants move to SplitFeatures, their only user, and the tests evaluate whole flags to check what is read and when unknown destinations warn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SpotBugs reports ES_COMPARING_PARAMETER_STRING_WITH_EQ for the identity check before equals in EvaluationScope.attribute. String.equals already compares the references first, so the same name still returns at once. Allocation and time are unchanged on the wide-context, variations and rules-sweep scale cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The id fallback that reads getValue instead of the context's keySet() now lands with the opt-out fix of the same kind, apart from this change, so each pull request carries one idea. This restores the keySet() check in EvaluationScope until that change reaches master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sharing one metadata instance per consent value is a form of sharing built metadata, so it belongs with the change that shares variant metadata between evaluations rather than with the attribute reads of this one. Results without a variant build their metadata as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What Does This Do
Evaluating a flag with many allocations costs less time and memory. The results are the same.
__dd_feature.*,__dd_exposure_feature.*and__dd_evaluation_feature.*entries are built the first time the split is served and cached on the split. Before, each evaluation that served the split built a new key for each feature and destination by concatenation.The served allocation, variant, reason, error codes and metadata do not change.
Two related changes moved out of this PR so that each one carries one idea:
idwithout copying the context's keys is in Read the id and opt-out attributes without copying the context's keys #13563, with the opt-out lookup of the same kind. This branch picks it up frommasteronce it lands.Motivation
We need to win a performance-sensitive customer. The scale benchmarks of #13485 are the basis for this work. In them, a flag with 1,000 allocations (
scale/variations) costs about 55 µs and 28 KB per evaluation, and its worst case costs 112 µs and 83 KB.flowchart LR subgraph Before A1[allocation 1..N] --> C1[each condition: getValue + convertValue] S1[matched split] --> F1[each evaluation: prefix + key per feature and destination] end subgraph After A2[allocation 1..N] --> E2[EvaluationScope: resolve once when reused] S2[matched split] --> F2[SplitFeatures, built on first serve] endAdditional Notes
Scale suite results
The base is 4075f1f, this PR's merge base with
master, which includes #13485. Host: AWS workspace, Intel Xeon Platinum 8175M, 16 vCPUs, JDK 21. Settings: JMH AverageTime, 1 fork, 3 warm-up and 5 measurement iterations of 2 s,-prof gc. The two jars ran in turn, twice, and the table gives both runs. Other sessions shared the machine, with a load average of 2–10, so allocation is exact but time differences of about 10% or less are within noise. Each row is the mean over the cases of the file, evaluated in turn.scale/variationsscale/rulesscale/rules-sweepscale/wide-contextscale/flagsscale/time-windowsscale/wide-context, layered contextThe layered context still copies its keys for the
idcondition on this branch; #13563 removes that copy (8,345 ns and 11,360 B to 456 ns and 416 B for that case).When the two
time-windowscases alternate, the mean is slower on this branch in both runs. An earlier measurement found no difference when each case is benchmarked alone, and the cause is not identified.The remaining bytes on
scale/variationsare mostly theHashMapbehindImmutableMetadatafor the 200-feature case: about 600 entries and its resizes. The OpenFeature builder takes entries one at a time and cannot be presized.ScaleSuiteLoadBenchmarkis not affected: the configuration parse does no new work.Decisions
EvaluationScopeholds the context and the attributes resolved so far. It is created when the first allocation with rules is reached, andevaluateRulesandevaluateConditiontake it in place of the context. A flag without rules pays nothing.Flag.preparedRules). Otherwise the scope reads each attribute directly. Without that check, a rule with 50 different attributes was 2.7 times slower. When caching, the first attribute is kept in fields, attributes 2 to 9 in two arrays of 8, and further ones in aHashMap. The scope then costs 40 B for one attribute and up to about 180 B for 2 to 9. That cost accounts for most of the added bytes above; the50x10x5first-allocation match pays it without reusing much.ufc) does not depend on the evaluator, soSplitandFlageach hold one lazily computedvolatilefield of what the evaluator derives. That is 8 B per split and per flag, plus the prepared entries of each split that is served. The entries have the same keys, the same value types (numbers become doubles, as before) and the same first-key-wins rule. The warning for an unknown destination still fires when a split is served, once per destination name.SplitFeaturescache.Testing
EvaluationScopeTest: covers four areas.idfallback, including an explicit nullValue.EvaluationScopeFlagTest: evaluates whole flags to check that each attribute is read once across allocations and again on the next evaluation, that a condition without an attribute still serves the right variant, and that an unknown destination warns once, on first serve and not at parse.SplitFeaturesTest: covers these behaviours.CanonicalFixtureTestcompares the served value, reason, error code and feature metadata with every shared fixture case. Thedd-openfeaturemodule tests pass.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issue🤖 Generated with Claude Code