Skip to content

Read flag attributes once per evaluation and cache split feature metadata - #13489

Draft
leoromanovsky wants to merge 18 commits into
masterfrom
leo.romanovsky/ffe-allocation-scan
Draft

leoromanovsky wants to merge 18 commits into
masterfrom
leo.romanovsky/ffe-allocation-scan

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Evaluating a flag with many allocations costs less time and memory. The results are the same.

  • Each evaluation reads a context attribute once when two or more conditions of the flag read it. Before, every condition read the attribute from the context again and converted its value. A flag whose conditions each read a different attribute reads them directly, as before.
  • A split's feature metadata is built once. The __dd_feature.*, __dd_exposure_feature.* and __dd_evaluation_feature.* entries are built the first time the split is served and cached on the split. Before, each evaluation that served the split built a new key for each feature and destination by concatenation.

The served allocation, variant, reason, error codes and metadata do not change.

Two related changes moved out of this PR so that each one carries one idea:

Motivation

We need to win a performance-sensitive customer. The scale benchmarks of #13485 are the basis for this work. In them, a flag with 1,000 allocations (scale/variations) costs about 55 µs and 28 KB per evaluation, and its worst case costs 112 µs and 83 KB.

flowchart LR
  subgraph Before
    A1[allocation 1..N] --> C1[each condition: getValue + convertValue]
    S1[matched split] --> F1[each evaluation: prefix + key per feature and destination]
  end
  subgraph After
    A2[allocation 1..N] --> E2[EvaluationScope: resolve once when reused]
    S2[matched split] --> F2[SplitFeatures, built on first serve]
  end
Loading

Additional Notes

Scale suite results

The base is 4075f1f, this PR's merge base with master, which includes #13485. Host: AWS workspace, Intel Xeon Platinum 8175M, 16 vCPUs, JDK 21. Settings: JMH AverageTime, 1 fork, 3 warm-up and 5 measurement iterations of 2 s, -prof gc. The two jars ran in turn, twice, and the table gives both runs. Other sessions shared the machine, with a load average of 2–10, so allocation is exact but time differences of about 10% or less are within noise. Each row is the mean over the cases of the file, evaluated in turn.

Case Base This PR Change
scale/variations 55.7 / 55.3 µs, 28,068 B 36.7 / 35.0 µs, 9,635 B 1.5× faster, 66% less allocation
scale/rules 2.77 / 2.75 µs, 1,622 B 2.54 / 2.46 µs, 1,747 B 8–11% faster, +125 B
scale/rules-sweep 22.3 / 22.0 µs, 368 B 22.8 / 23.4 µs, 492 B within noise, +124 B
scale/wide-context 636 / 606 ns, 368 B 549 / 543 ns, 416 B within noise, +48 B
scale/flags 215 / 217 ns, 330 B 231 / 238 ns, 366 B within noise, +36 B
scale/time-windows 793 / 802 ns, 368 B 1,010 / 984 ns, 416 B 23–27% slower, +48 B
scale/wide-context, layered context 5,494 / 5,482 ns, 6,944 B 5,839 / 5,900 ns, 7,030 B within noise, +86 B

The layered context still copies its keys for the id condition on this branch; #13563 removes that copy (8,345 ns and 11,360 B to 456 ns and 416 B for that case).

When the two time-windows cases alternate, the mean is slower on this branch in both runs. An earlier measurement found no difference when each case is benchmarked alone, and the cause is not identified.

The remaining bytes on scale/variations are mostly the HashMap behind ImmutableMetadata for the 200-feature case: about 600 entries and its resizes. The OpenFeature builder takes entries one at a time and cannot be presized. ScaleSuiteLoadBenchmark is not affected: the configuration parse does no new work.

Decisions

  • One per-evaluation object. EvaluationScope holds the context and the attributes resolved so far. It is created when the first allocation with rules is reached, and evaluateRules and evaluateCondition take it in place of the context. A flag without rules pays nothing.
  • Cache only where an attribute is reused. The first time a flag is evaluated, it records whether two of its conditions read the same attribute (Flag.preparedRules). Otherwise the scope reads each attribute directly. Without that check, a rule with 50 different attributes was 2.7 times slower. When caching, the first attribute is kept in fields, attributes 2 to 9 in two arrays of 8, and further ones in a HashMap. The scope then costs 40 B for one attribute and up to about 180 B for 2 to 9. That cost accounts for most of the added bytes above; the 50x10x5 first-allocation match pays it without reusing much.
  • Same attribute semantics otherwise. An attribute that changes in the context during the evaluation is not read again. A condition without an attribute is read every time and never cached.
  • Feature metadata cached on the split, not in the parser. The configuration model (ufc) does not depend on the evaluator, so Split and Flag each hold one lazily computed volatile field of what the evaluator derives. That is 8 B per split and per flag, plus the prepared entries of each split that is served. The entries have the same keys, the same value types (numbers become doubles, as before) and the same first-key-wins rule. The warning for an unknown destination still fires when a split is served, once per destination name.
  • The split cache may be replaced. A separate PR, Share flag metadata across evaluations of a split in the same time bucket #13541, shares the built variant metadata between evaluations. If it lands, it can replace the per-split SplitFeatures cache.

Testing

  • EvaluationScopeTest: covers four areas.
    • Each attribute is read from the context once, below and above 8 attributes, and a missing attribute is cached as missing.
    • The id fallback, including an explicit null Value.
    • A condition without an attribute.
    • The uncached path and when a flag reuses an attribute.
  • EvaluationScopeFlagTest: evaluates whole flags to check that each attribute is read once across allocations and again on the next evaluation, that a condition without an attribute still serves the right variant, and that an unknown destination warns once, on first serve and not at parse.
  • SplitFeaturesTest: covers these behaviours.
    • The entries for each destination.
    • First-key-wins deduplication.
    • Incomplete and unsupported features.
    • Numbers widened to doubles.
    • The evaluation-features marker and the shared empty preparation.
    • That the split computes its entries once.
  • CanonicalFixtureTest compares the served value, reason, error code and feature metadata with every shared fixture case. The dd-openfeature module tests pass.

Contributor Checklist

🤖 Generated with Claude Code

leoromanovsky and others added 5 commits November 17, 2026 00:47
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON
integers by type, so it could not set a baseline on master. It now
compares JSON numbers by value, covers every case file of the scale and
scale-json suites, can isolate one case with the index parameter, and
prints each case's index. A load benchmark measures parsing each suite's
configuration and prints the heap it retains, so changes that move work
to load time show their cost.

The submodule points to the scale-suite fixture branch, which adds the
N x M dimensions and JSON values up to 10 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… at load

A flag with many allocations re-read the same context attributes in every
allocation: a context lookup and a value conversion each time, and the
conversion boxes numbers. Each evaluation now keeps the attributes it has
resolved, so a flag with 1,000 allocations reads its attribute once.

A matched split's features were turned into metadata entries on every
evaluation, concatenating a prefixed key for each feature and destination.
The split now resolves its entries once, when it is built, and an
evaluation only copies them. Unknown destinations are now reported when
the configuration loads, still once per name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The split caches what the evaluator derives from its features instead of
building it in its constructor, so the configuration model does not
depend on the evaluator and an unknown destination is still reported
when a split is served, as before. The allocation and rule scans keep
their for-each loops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most flags read one attribute, so the evaluation scope keeps it in fields
and allocates its arrays only for a second attribute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky leoromanovsky added type: feature Enhancements and improvements tag: performance Performance related changes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 17, 2026
@leoromanovsky leoromanovsky added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026 — with ddtool CLI
@datadog-official

datadog-official Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 97.06%
• Overall Coverage: 60.66% (-0.02%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b69ca4c | Docs | Give us feedback!

@leoromanovsky leoromanovsky removed the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
@dd-octo-sts

dd-octo-sts Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.88 s 14.83 s [-0.5%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 13.64 s 13.66 s [-0.9%; +0.7%] (no difference)
startup:petclinic:appsec:Agent 17.72 s 17.41 s [+1.0%; +2.5%] (significantly worse)
startup:petclinic:iast:Agent 17.53 s 17.58 s [-1.1%; +0.5%] (no difference)
startup:petclinic:profiling:Agent 17.18 s 16.70 s [-1.6%; +7.3%] (no difference)
startup:petclinic:sca:Agent 17.69 s 17.67 s [-0.6%; +0.8%] (no difference)
startup:petclinic:tracing:Agent 16.66 s 16.76 s [-1.4%; +0.2%] (no difference)

Commit: b69ca4c2 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

leoromanovsky and others added 6 commits November 17, 2026 04:00
The scale benchmarks failed the forbidden API check on System.out, and
nothing outside a manual JMH run checked the scale cases, so a sibling
change that resolves a large configuration wrongly would go unnoticed.

- Add ScaleSuiteTest: evaluate every case of the scale and scale-json
  suites twice, and fail when the benchmark misses a case file. The
  cases, their check and the suite reader move to the test sources and
  the benchmarks reuse them; the reader uses JsonReading.
- Allow System.out in the benchmark setups (@SuppressForbidden).
- Check every case again after the measurement, fail on an error code,
  and reject expectations the check does not handle.
- Rerun the jmh task when the fixtures change.
- Measure retained heap after a first parse, as the smallest of three
  signed samples.
- Describe a row as the unweighted mean over the cases of a file, say
  that hooks are not measured and when to run three forks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture PR now checks that the committed scale files match the
generator and fixes its docs. The case and configuration files are
unchanged, so the benchmark numbers do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to
1000 and of the powers of two up to 2^52, which an index keyed on exact
doubles must still match. Pin them and benchmark the file; its 8 cases
pass on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh

dougqh commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

A few ideas from an experiment on top of #13485 (measured against #13529, the client-path benchmark stacked on #13485; 1 fork on a noisy 5-vCPU box, so the bytes are exact and the times are rough):

1. id still copies the context's keys once per evaluation. EvaluationScope.resolve checks context.keySet().contains("id"). On the client's LayeredEvaluationContext, keySet() builds a HashSet of every layer's keys, and the client creates a new context on each call. That's the 10× cost in the #13485 table (scale-id-wide-context: 1.24 µs, 368 B with evaluate vs 11.4 µs, 11,296 B with evaluateLayered). getValue returns null only when the key is absent, so it can replace the check:

final Value value = context.getValue(name);
if (value == null) {
  return ID.equals(name) ? context.getTargetingKey() : null;
}
return context.convertValue(value);

On the layered wide context, this cut 7.0 KB to 416 B per evaluation (−94%), and the time by about 90%. One edge case changes: a raw Java null stored under id now falls back to the targeting key, as an absent id does. A Value holding null, which is what the OpenFeature contexts store, behaves as before. ExperimentationOptOut.cutoff has the same keySet().contains pattern, which runs on every evaluation when opt-out is enabled.

2. consentMetadata has only two possible values. It builds a new ImmutableMetadata (builder plus HashMap) for every default, disabled and error result. Two static final instances, one for each consent value, would remove that.

3. Sharing the variant metadata instead of caching features on the split. The remaining 28 KB on the 200-feature case is the ImmutableMetadata map itself. The experiment kept a fixed table of 1,024 slots holding the metadata built last for each (flag, variation type, allocation, serial id, features, consent, exposure decision, millisecond). Evaluations of the same split in the same millisecond share one immutable instance. Entries have only final fields and are published with a single reference store, so a lost race only costs a rebuild. Results:

Case Before After
scale/flags 210 ns, 330 B 145 ns, 84 B
scale/variations 48.0 µs, 28,069 B 34.4 µs, 2,931 B

It also avoids caching on each Split, which grows with the configuration (an estimated 4–8 MB of retained heap on the 10,000-flag scale config). The hit rate depends on how often a split is served within one millisecond, so it helps most for hot flags. The table is direct-mapped, without locking, and its size never grows. Happy to share the patch if you want to try it on your stack.

@leoromanovsky

Copy link
Copy Markdown
Contributor Author

3. Sharing the variant metadata instead of caching features on the split. The remaining 28 KB on the 200-feature case is the ImmutableMetadata map itself. The experiment kept a fixed table of 1,024 slots holding the metadata built last for each (flag, variation type, allocation, serial id, features, consent, exposure decision, millisecond). Evaluations of the same split in the same millisecond share one immutable instance. Entries have only final fields and are published with a single reference store, so a lost race only costs a rebuild. Results:

Case Before After
scale/flags 210 ns, 330 B 145 ns, 84 B
scale/variations 48.0 µs, 28,069 B 34.4 µs, 2,931 B
It also avoids caching on each Split, which grows with the configuration (an estimated 4–8 MB of retained heap on the 10,000-flag scale config). The hit rate depends on how often a split is served within one millisecond, so it helps most for hot flags. The table is direct-mapped, without locking, and its size never grows. Happy to share the patch if you want to try it on your stack.

Very cool thinking! Trying it out myself to get a feel for the cache behavior.

The shared metadata table is what I’m most curious about. I’m trying it out myself as a separate PR against DataDog/dd-trace-java#13485 · Benchmark FFE evaluation on the…, with you as co-author, so its gain shows up apart from this PR’s. I want to get a feel for how the table behaves under load.

One thing I’m exploring is a wider time key than 1 ms. Exposures take their own timestamp, and the evaluation events only report first_evaluation/last_evaluation per 10 s window. So a bucket of, say, 100 ms would only skew last_evaluation by that much, and each entry would get 100x longer to be reused. I’m experimenting with bucket width, table size, thread count and how skewed flag traffic is, to find where hit rate and timestamp precision balance out.

Was 1 ms keying just about keeping the timestamp exact, or did you see something else that pushed you there?

leoromanovsky and others added 3 commits November 19, 2026 03:42
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The id fallback tested context.keySet() for one key. A layered context
builds that set from every layer on each call, and the OpenFeature
client creates a new layered context per evaluation. getValue returns
null only for an absent key, so it answers the same question without
the copy. A raw Java null stored under id, which a context built from a
Java map can hold, now falls back to the targeting key as an absent id
does. A null Value behaves as before.

Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A default, disabled or error result carries only the consent flag, which
has two values. Two shared instances replace a new builder and map per
result.

Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A flag whose conditions each read a different attribute gained nothing
from the per-evaluation cache and paid for its arrays and map: a
50-attribute rule took about 2.7 times as long. The flag now records,
the first time it is evaluated, whether two conditions read the same
attribute, and only such flags keep the resolved attributes.

A condition without an attribute is read as before and never kept, as
it could break later lookups. The feature destination constants move to
SplitFeatures, their only user, and the tests evaluate whole flags to
check what is read and when unknown destinations warn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky leoromanovsky changed the title Resolve feature flag attributes once per evaluation and cache split feature metadata Read flag attributes once per evaluation and cache split feature metadata Nov 19, 2026
leoromanovsky and others added 3 commits November 19, 2026 17:24
SpotBugs reports ES_COMPARING_PARAMETER_STRING_WITH_EQ for the identity
check before equals in EvaluationScope.attribute. String.equals already
compares the references first, so the same name still returns at once.
Allocation and time are unchanged on the wide-context, variations and
rules-sweep scale cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The id fallback that reads getValue instead of the context's keySet()
now lands with the opt-out fix of the same kind, apart from this
change, so each pull request carries one idea. This restores the
keySet() check in EvaluationScope until that change reaches master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sharing one metadata instance per consent value is a form of sharing
built metadata, so it belongs with the change that shares variant
metadata between evaluations rather than with the attribute reads of
this one. Results without a variant build their metadata as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants