Skip to content

Share flag metadata across evaluations of a split in the same time bucket - #13541

Draft
leoromanovsky wants to merge 21 commits into
masterfrom
leo.romanovsky/ffe-metadata-table
Draft

leoromanovsky wants to merge 21 commits into
masterfrom
leo.romanovsky/ffe-metadata-table

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Successful flag evaluations that resolve the same split in the same time bucket (by default, the same 10 ms) now share one immutable ImmutableMetadata instead of each building its own. The metadata keeps the same keys and values, except that __dd_eval_timestamp_ms is the time of the evaluation that built the shared entry, up to 9 ms earlier than the evaluation that reuses it. Exposures and span enrichment are unaffected; see Choosing the time bucket below.

Two new settings control it:

  • DD_FEATURE_FLAGS_EVALUATION_METADATA_TABLE_SIZE (default 1024, rounded up to a power of two, at most 65,536) sets the number of slots. 0 turns sharing off and builds the metadata on every evaluation, as before.
  • DD_FEATURE_FLAGS_EVALUATION_METADATA_TIME_BUCKET_MS (default 10, at most 10,000) sets the width W of the time bucket in which evaluations of a split share their metadata. With W > 1, evaluations that reuse an entry report the time of the evaluation that built it, up to W − 1 ms early. 0 leaves the timestamp out of the shared metadata, so FlagEvalLoggingHook uses its own clock, and the entry is reused until its slot is replaced. 1 keeps today's exact timestamps. The default is a reviewer question; see Choosing the time bucket below.

The idea and its first measurements come from @dougqh in his review of #13489 (idea 3). He is credited as co-author of the commits.

Motivation

On a latency- and allocation-sensitive service, the metadata map was most of what a successful evaluation allocated: 330 B of a scale/flags evaluation, and 28 KB on the 200-feature scale/variations split. Almost all of it is fixed per split. The flag key, variation type, allocation key, serial ID, doLog, consent and split features never change for a split. The timestamp changes only once per millisecond. A service that evaluates the same flags for every request was rebuilding the same map thousands of times per millisecond.

flowchart LR
  E[Evaluation resolves a split] --> X[Exposure lookup, per subject]
  X --> L{"Slot holds the same Split, Allocation, Flag,<br/>consent, exposure state and time bucket?"}
  L -- hit --> S[Return the shared metadata]
  L -- miss --> B[Build the metadata] --> P[Publish it with one reference store] --> S
Loading

Additional Notes

A burst coalescer, not a cache. With a bounded bucket, an entry can only be reused within the bucket it was built in (10 ms by default). After that, the next evaluation of the split builds a new entry and replaces the old one. Only the unbounded bucket (0) turns it into a cache of static per-split metadata. The table never grows, holds at most one entry per slot, and does not keep a configuration alive beyond the last bucket in which it was used.

The key compares references, not strings. It holds the Split, Allocation and Flag objects plus consent, exposure state and time bucket. A reloaded configuration parses new objects, so it can never be served an entry from the old one, and a metadata field cannot be missed from the key. The exposure lookup still runs on every evaluation, because its result depends on the subject. Its result is part of the key and picks the slot, so an already-exposed subject and a new one don't evict each other.

Lock-free. Entries have only final fields and are published with a single plain reference store. When two threads miss the same slot at once, both build the metadata and the last store wins, so a lost race only costs one extra build.

Benchmarks. Baseline: master at 5e0370d, which this branch now merges. Host: a shared Datadog workspace with a 16 vCPU Xeon Platinum 8175M (8 cores, 2 hardware threads each) and JDK 21.0.10. Other sessions were active, and the 1-minute load average was 2.3 to 4.7 during the runs. Single-threaded runs were pinned to CPUs 4–7, with 3 forks of 5 × 2 s iterations, at this PR's defaults (1,024 slots, W = 10 ms). ± is JMH's 99.9% interval. Allocation is gc.alloc.rate.norm per evaluation.

ScaleSuiteBenchmark master This PR
evaluate, scale/flags 214 ± 10 ns, 330 B 158 ± 4 ns, 84 B
evaluate, scale/splits 20.7 ± 0.8 µs, 31,528 B 19.5 ± 0.4 µs, 31,778 ± 441 B
evaluate, scale/variations 54.4 ± 2.9 µs, 28,068 B 34.7 ± 1.7 µs, 357 B
evaluate, scale/wide-context 618 ± 38 ns, 368 B 561 ± 25 ns, 40 B
evaluateLayered, scale/flags 242 ± 12 ns, 378 B 184 ± 5 ns, 132 B
evaluateLayered, scale/splits 20.5 ± 0.7 µs, 31,858 ± 454 B 20.4 ± 0.6 µs, 31,528 ± 454 B
evaluateLayered, scale/variations 56.3 ± 2.8 µs, 28,116 B 34.6 ± 1.4 µs, 380 B
evaluateLayered, scale/wide-context 5.68 ± 0.08 µs, 6,944 B 5.49 ± 0.05 µs, 6,617 B

evaluateLayered is the client path, through the OpenFeature context layering. scale/splits is unchanged within noise: its cost is the split selection itself, not the metadata. On scale/wide-context, the client path's allocation is mostly the layered context, so the table saves the same ~330 B there as on the direct path.

Exposures switch. Merging master brought in the provider's exposuresEnabled switch (#12387). The table key uses the effective exposure state, so with exposures disabled every subject shares one entry per split, and the metadata carries doLog and the exposure cache hit exactly as on master.

Choosing the time bucket

What a wider bucket changes. The only consumer of __dd_eval_timestamp_ms is FlagEvalLoggingHook, which feeds the flag evaluation events. Those events aggregate evaluations per 10 s flush and report only first_evaluation and last_evaluation per bucket. Exposures stamp their own time and are not affected. Events are grouped by arrival, not by timestamp, so an entry built just before a flush can make both first_evaluation and last_evaluation up to W − 1 ms early, not only the last one. With 0 (unbounded), the hook stamps the time it runs, which is after the evaluation rather than before. That reverses the deliberate choice of evaluation time over hook time that the timestamp was added for.

What it buys. EvaluationMetadataTableBenchmark sweeps table size × W on 1 and 12 threads. Each request evaluates 10 or 50 flags drawn from 1,000 active flags, either uniformly or Zipf-skewed (exponent 1), with a new subject and cohort per request. A few cells from the full matrix below:

Shape Slots No table W = 1 W = 10 W = 100 W = 1,000 Unbounded
1 thread, 10 flags, uniform 1,024 368 B 41% hit, 261 B 62%, 185 B 64%, 175 B 64%, 174 B 65%, 153 B
1 thread, 10 flags, uniform 4,096 368 B 52% hit, 221 B 85%, 98 B 89%, 83 B 89%, 81 B 89%, 74 B
12 threads, 50 flags, skewed 4,096 368 B, 31.1 µs 92%, 70 B, 22.9 µs 96%, 57 B, 26.2 µs 95%, 60 B 96%, 55 B 96%, 54 B, 22.4 µs
Reported time early by (p99 / max) 0 0 9 / 9 ms ~95 / 99 ms ~950 / 999 ms 0 (hook time instead)

Bytes are per evaluation, times are per request, and hit rate is the share of successful evaluations that reused metadata.

What the data shows:

  • Most of the bucket's gain comes by 10 ms. Going from 1 to 10 ms is the big step when a single thread spreads requests across many flags: at 4,096 slots, the hit rate goes from 52% to 85% and allocation per evaluation halves. Under 12 threads, even 1 ms already reuses 92%.
  • Wider buckets add little. 100 ms, 1,000 ms and unbounded add at most about 5 more points, for 10 to 100 times the timestamp error.
  • Table size matters more than W. The table is direct-mapped and 1,000 active flags collide in 1,024 slots, which caps the hit rate at about 65% for any W. 4,096 slots lift that cap to about 89% (uniform) and 95% (skewed). A two-way probe (also checking the neighboring slot) would recover much of the collision loss for a second read on a miss. It is left out so this PR measures the design as proposed.
  • Slot writes barely contend. With 12 threads, hit rates match or beat one thread at the same size and W, so threads racing to rebuild the same entry at a bucket boundary cost little. Nothing waits on a lock.
  • Latency moves less than allocation. The table cuts request time where the hit rate is high (31 → 22–26 µs at 12 threads, 50 skewed flags). Across W, the time differences are mostly within this host's noise.

Reviewer question: is 10 ms the right default? This PR sets W to 10 ms. It keeps evaluation-time semantics and captures most of the reuse, at a cost of reporting first_evaluation and last_evaluation at most 9 ms early in a 10 s window. Unbounded reuses slightly more and is never early, but it replaces evaluation time with hook time. A wider bucket helps little unless the table is also large enough for the number of active flags, so also consider raising the default table size to 4,096. When every slot is full, that retains about 1.5 MB for metadata shaped like the benchmark flags. Splits with many features retain more: the metadata of the 200-feature scale/variations split is about 25 KB. This PR keeps the table at 1,024 slots. Set W to 1 to keep today's exact timestamps.

Benchmark limits. Host: the same shared workspace, a 16 vCPU Xeon Platinum 8175M with JDK 21.0.10. Each cell ran 2 × 1 s of warmup and 3 × 1 s of measurement in SampleTime mode, with 1 fork. 1-thread runs were pinned to CPUs 4–7, and 12-thread runs to 12 logical CPUs (2–7, 10–15). Other sessions' builds and benchmarks shared the host. Load averages ran from about 3 to 15. The first 12-thread, 50-flag pass ran at load 20–27 and was discarded; it was re-run after waiting for the shared JMH lock. Treat mean and p99 times as rough; bytes, hit rates and timestamp errors are exact. To keep the matrix finishing, the bucket width was not swept with the table off, since it has no effect there. Timestamp error was sampled on every 8th request, as the clock just before the evaluation minus the reported time.

Full matrix (128 cells)

1 thread

Flags Popularity Slots W (ms) B/eval Mean µs/req p99 µs/req Hit rate Early p99 / max (ms)
10 uniform 0 — 368 5.34 17.4 0.0% 0 / 0
10 uniform 256 1 340 6.05 18.3 20.3% 0 / 0
10 uniform 256 10 325 5.51 17.7 24.2% 3 / 9
10 uniform 256 100 324 5.03 13.9 24.6% 6 / 99
10 uniform 256 1000 324 5.14 13.9 24.6% 8 / 999
10 uniform 256 unbounded 281 4.74 12.1 24.7% 0 / 0
10 uniform 1,024 1 261 5.55 17.5 41.3% 0 / 0
10 uniform 1,024 10 185 5.51 17.6 61.5% 9 / 9
10 uniform 1,024 100 175 5.03 14.9 64.1% 94 / 99
10 uniform 1,024 1000 174 5.18 14.6 64.4% 938 / 999
10 uniform 1,024 unbounded 153 4.75 13.8 64.6% 0 / 0
10 uniform 4,096 1 221 5.26 15.7 51.9% 0 / 0
10 uniform 4,096 10 98 4.77 15.8 84.5% 9 / 9
10 uniform 4,096 100 83 4.35 11.4 88.5% 98 / 99
10 uniform 4,096 1000 81 4.43 12.5 89.1% 984 / 999
10 uniform 4,096 unbounded 74 4.10 9.6 89.3% 0 / 0
10 skewed 0 — 368 3.80 8.1 0.0% 0 / 0
10 skewed 256 1 188 3.77 9.3 60.7% 0 / 0
10 skewed 256 10 175 4.31 12.1 64.1% 3 / 9
10 skewed 256 100 173 4.08 11.2 64.5% 5 / 99
10 skewed 256 1000 173 3.89 10.0 64.6% 5 / 999
10 skewed 256 unbounded 153 3.63 9.1 64.7% 0 / 0
10 skewed 1,024 1 146 3.82 9.9 71.8% 0 / 0
10 skewed 1,024 10 105 3.85 10.5 82.7% 9 / 9
10 skewed 1,024 100 98 3.63 8.5 84.6% 94 / 99
10 skewed 1,024 1000 97 3.67 9.1 84.8% 948 / 999
10 skewed 1,024 unbounded 89 3.49 8.9 84.8% 0 / 0
10 skewed 4,096 1 132 3.66 9.9 75.5% 0 / 0
10 skewed 4,096 10 75 3.71 10.1 90.8% 9 / 9
10 skewed 4,096 100 63 3.42 10.3 93.8% 98 / 99
10 skewed 4,096 1000 62 3.40 8.7 94.2% 985 / 999
10 skewed 4,096 unbounded 59 3.21 13.1 94.2% 0 / 0
50 uniform 0 — 368 22.59 37.3 0.0% 0 / 0
50 uniform 256 1 337 23.27 39.6 21.0% 0 / 0
50 uniform 256 10 325 27.57 54.6 24.2% 3 / 9
50 uniform 256 100 323 24.92 49.3 24.6% 9 / 99
50 uniform 256 1000 323 24.58 42.8 24.7% 9 / 999
50 uniform 256 unbounded 281 22.51 37.2 24.8% 0 / 0
50 uniform 1,024 1 257 24.01 44.8 42.4% 0 / 0
50 uniform 1,024 10 184 25.98 48.8 61.8% 9 / 9
50 uniform 1,024 100 174 25.45 44.5 64.4% 94 / 99
50 uniform 1,024 1000 173 24.95 41.1 64.7% 937 / 999
50 uniform 1,024 unbounded 153 22.75 38.1 64.6% 0 / 0
50 uniform 4,096 1 213 22.90 43.3 54.0% 0 / 0
50 uniform 4,096 10 99 25.75 53.4 84.4% 9 / 9
50 uniform 4,096 100 82 22.50 38.9 88.7% 98 / 99
50 uniform 4,096 1000 80 21.40 35.3 89.3% 984 / 999
50 uniform 4,096 unbounded 74 20.61 35.0 89.3% 0 / 0
50 skewed 0 — 368 17.95 31.8 0.0% 0 / 0
50 skewed 256 1 187 18.21 34.2 61.1% 0 / 0
50 skewed 256 10 174 20.39 40.9 64.3% 3 / 9
50 skewed 256 100 173 19.98 38.8 64.7% 5 / 99
50 skewed 256 1000 173 18.91 36.9 64.8% 5 / 998
50 skewed 256 unbounded 152 18.54 32.8 64.9% 0 / 0
50 skewed 1,024 1 144 17.50 33.2 72.4% 0 / 0
50 skewed 1,024 10 105 20.23 40.0 82.6% 9 / 9
50 skewed 1,024 100 98 19.08 36.5 84.7% 95 / 99
50 skewed 1,024 1000 97 18.63 34.4 84.9% 930 / 999
50 skewed 1,024 unbounded 88 16.60 30.0 85.1% 0 / 0
50 skewed 4,096 1 131 17.69 32.8 75.7% 0 / 0
50 skewed 4,096 10 74 17.86 36.2 91.0% 9 / 9
50 skewed 4,096 100 63 18.41 35.6 93.9% 98 / 99
50 skewed 4,096 1000 61 16.19 30.3 94.4% 984 / 999
50 skewed 4,096 unbounded 58 14.79 27.8 94.3% 0 / 0

12 threads

Flags Popularity Slots W (ms) B/eval Mean µs/req p99 µs/req Hit rate Early p99 / max (ms)
10 uniform 0 — 368 8.18 23.0 0.0% 0 / 0
10 uniform 256 1 325 8.09 15.5 24.1% 0 / 0
10 uniform 256 10 324 8.91 24.7 24.6% 1 / 9
10 uniform 256 100 325 8.21 15.9 24.3% 1 / 99
10 uniform 256 1000 324 7.89 15.2 24.5% 1 / 996
10 uniform 256 unbounded 282 7.57 14.6 24.6% 0 / 0
10 uniform 1,024 1 194 7.68 22.9 59.1% 0 / 0
10 uniform 1,024 10 177 7.53 22.2 63.7% 9 / 9
10 uniform 1,024 100 175 7.28 14.9 64.1% 92 / 99
10 uniform 1,024 1000 183 7.30 15.0 62.0% 923 / 999
10 uniform 1,024 unbounded 160 6.84 14.9 62.7% 0 / 0
10 uniform 4,096 1 101 7.30 22.9 83.9% 0 / 0
10 uniform 4,096 10 89 6.67 21.8 87.0% 9 / 9
10 uniform 4,096 100 89 6.27 13.3 87.1% 98 / 99
10 uniform 4,096 1000 87 5.83 12.3 87.5% 983 / 999
10 uniform 4,096 unbounded 75 5.82 12.4 89.0% 0 / 0
10 skewed 0 — 368 7.03 20.7 0.0% 0 / 0
10 skewed 256 1 178 7.97 23.2 63.4% 0 / 0
10 skewed 256 10 177 6.47 21.1 63.7% 1 / 9
10 skewed 256 100 188 6.40 13.6 60.8% 1 / 99
10 skewed 256 1000 174 6.18 13.1 64.4% 4 / 999
10 skewed 256 unbounded 163 6.46 13.7 61.8% 0 / 0
10 skewed 1,024 1 117 9.90 25.9 79.7% 0 / 0
10 skewed 1,024 10 95 5.90 14.4 85.5% 9 / 9
10 skewed 1,024 100 98 5.57 12.8 84.7% 92 / 99
10 skewed 1,024 1000 100 5.84 12.4 84.1% 913 / 999
10 skewed 1,024 unbounded 87 6.46 21.1 85.4% 0 / 0
10 skewed 4,096 1 75 9.05 24.2 90.8% 0 / 0
10 skewed 4,096 10 58 5.95 18.6 95.1% 9 / 9
10 skewed 4,096 100 60 5.66 13.6 94.6% 98 / 99
10 skewed 4,096 1000 60 6.10 21.2 94.8% 983 / 999
10 skewed 4,096 unbounded 55 8.37 23.1 95.5% 0 / 0
50 uniform 0 — 368 34.21 51.4 0.0% 0 / 0
50 uniform 256 1 325 38.89 56.7 24.2% 0 / 0
50 uniform 256 10 322 40.04 57.5 24.9% 1 / 9
50 uniform 256 100 323 42.05 60.1 24.7% 1 / 96
50 uniform 256 1000 323 42.27 60.4 24.7% 1 / 999
50 uniform 256 unbounded 281 36.57 54.5 24.7% 0 / 0
50 uniform 1,024 1 185 32.28 50.3 61.4% 0 / 0
50 uniform 1,024 10 187 33.64 52.2 60.9% 9 / 9
50 uniform 1,024 100 175 35.61 54.6 64.2% 93 / 99
50 uniform 1,024 1000 179 40.39 57.2 63.1% 928 / 999
50 uniform 1,024 unbounded 159 41.99 55.7 62.8% 0 / 0
50 uniform 4,096 1 99 30.55 50.1 84.3% 0 / 0
50 uniform 4,096 10 83 29.15 47.4 88.6% 9 / 9
50 uniform 4,096 100 82 28.85 47.2 88.8% 98 / 99
50 uniform 4,096 1000 77 29.32 47.2 90.1% 985 / 999
50 uniform 4,096 unbounded 81 27.54 45.3 87.3% 0 / 0
50 skewed 0 — 368 31.06 48.0 0.0% 0 / 0
50 skewed 256 1 171 29.12 46.8 65.3% 0 / 0
50 skewed 256 10 176 31.60 49.7 63.8% 1 / 9
50 skewed 256 100 167 30.06 48.1 66.2% 2 / 99
50 skewed 256 1000 176 30.53 48.2 63.8% 1 / 916
50 skewed 256 unbounded 170 27.74 45.4 59.4% 0 / 0
50 skewed 1,024 1 112 25.57 43.4 80.8% 0 / 0
50 skewed 1,024 10 99 29.94 47.7 84.3% 9 / 9
50 skewed 1,024 100 102 27.68 45.3 83.5% 91 / 99
50 skewed 1,024 1000 98 25.77 43.5 84.5% 960 / 999
50 skewed 1,024 unbounded 93 25.38 43.1 83.5% 0 / 0
50 skewed 4,096 1 70 22.89 40.4 92.0% 0 / 0
50 skewed 4,096 10 57 26.22 42.2 95.5% 9 / 9
50 skewed 4,096 100 60 25.46 42.6 94.6% 98 / 99
50 skewed 4,096 1000 55 23.08 40.3 96.2% 982 / 999
50 skewed 4,096 unbounded 54 22.36 39.6 95.7% 0 / 0

Relation to #13489. This PR does not depend on #13489, build on it, or include any of it. Its gain is measured against master alone. If it lands, #13489 can drop its per-split SplitFeatures cache: this table covers the same allocation without retaining state on every Split, which @dougqh estimated at 4–8 MB of retained heap on the 10,000-flag configuration.

Commit list. #13485 was squash-merged, so its 16 original branch commits still show in this PR's commit list, along with merges of master. The diff against master contains only this change.

Contributor Checklist

leoromanovsky and others added 18 commits November 17, 2026 00:47
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON
integers by type, so it could not set a baseline on master. It now
compares JSON numbers by value, covers every case file of the scale and
scale-json suites, can isolate one case with the index parameter, and
prints each case's index. A load benchmark measures parsing each suite's
configuration and prints the heap it retains, so changes that move work
to load time show their cost.

The submodule points to the scale-suite fixture branch, which adds the
N x M dimensions and JSON values up to 10 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scale benchmarks failed the forbidden API check on System.out, and
nothing outside a manual JMH run checked the scale cases, so a sibling
change that resolves a large configuration wrongly would go unnoticed.

- Add ScaleSuiteTest: evaluate every case of the scale and scale-json
  suites twice, and fail when the benchmark misses a case file. The
  cases, their check and the suite reader move to the test sources and
  the benchmarks reuse them; the reader uses JsonReading.
- Allow System.out in the benchmark setups (@SuppressForbidden).
- Check every case again after the measurement, fail on an error code,
  and reject expectations the check does not handle.
- Rerun the jmh task when the fixtures change.
- Measure retained heap after a first parse, as the smallest of three
  signed samples.
- Describe a row as the unweighted mean over the cases of a file, say
  that hooks are not measured and when to run three forks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture PR now checks that the committed scale files match the
generator and fixes its docs. The case and configuration files are
unchanged, so the benchmark numbers do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to
1000 and of the powers of two up to 2^52, which an index keyed on exact
doubles must still match. Pin them and benchmark the file; its 8 cases
pass on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OpenFeature client passes the provider a new LayeredEvaluationContext
on each call, while the benchmark reused one MutableContext. A rule on
`id` when the context has no `id` reads the layered context's key set,
which it builds again on each call, so the benchmark did not show what
applications pay for it.

evaluateLayered wraps each case's context the way the client does, as
ExperimentationOptOutBenchmark.evaluateFreshLayeredContext does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures replace the 10 MB JSON flag with a 4 MB one, which brings
the scale-json configuration from 14 MB down to 8 MB. The other cases
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With several JMH threads and index=all, the threads shared one
unsynchronized cursor, so a thread could advance it past the last case
and fail the run with ArrayIndexOutOfBoundsException. The cursor is now
thread-scoped state, and the provider and cases stay shared.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pinned scale suite adds first-match cases and expects the selected
serialId, doLog, and split features on scale-many-variations. ScaleCase
now checks those fields from the flag metadata, as CanonicalFixtureTest
does. The scale suites run without span enrichment, so an allocation
that does not log carries no doLog key, and a missing key means false.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DataDog/ffe-system-test-data#78 merged as 508a342 with the same suites
as its head. Main also brings mutually exclusive layer cases and
precomputed fixture corrections, which the unit and smoke tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SDK is the main reader of the ffe-system-test-data submodule and the
only reader of the vendored experimentation opt-out cases, so both move
from the smoke-test resources to dd-openfeature/src/test/fixtures. They
stay outside any source set: the tests and benchmarks read them from
disk, and the smoke tests copy ufc-config.json and evaluation-cases into
their resources.

The jmh task declares the scale suites as a file tree, which is empty
rather than missing when the submodule is not initialized, so a jmh run
on such a checkout no longer fails before it starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The suites read the system properties and environment, so setting
DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED turned span
enrichment on: the benchmarks measured the enriched metadata path and
the expected selection metadata no longer described it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each case file parsed its suite again, 14 times for the 7.8 MB scale
configuration. One provider per suite, shut down after all cases, parses
each configuration once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ScaleSuiteTest compared the fixtures with a copy of the benchmark's
@PARAM list, so a case file missing from the benchmark went unnoticed.
The test now reads the @PARAM by reflection, with the jmh classes on
its runtime classpath.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the fixtures submodule at dd-openfeature/src/test/fixtures, pinned
to 508a342. Master bumped it at the old smoke-test path to 29ba6bf,
which is 4 commits behind that pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cond

FlagEvaluator built a new ImmutableMetadata on every successful
evaluation, although everything in it but the timestamp is fixed by the
split, and the timestamp has millisecond resolution. A fixed,
direct-mapped, lock-free table now keeps the metadata built last for a
key of the Split, Allocation and Flag references, the consent, the
exposure state and the evaluation millisecond, so evaluations of the
same split in the same millisecond share one immutable instance.

It is a burst coalescer, not a cache: an entry is reusable only within
its millisecond. Keying on references means a reloaded configuration
never matches an entry of the previous one. Entries have only final
fields and are published with a single reference store, so a lost race
only costs a rebuild.

DD_FEATURE_FLAGS_EVALUATION_METADATA_TABLE_SIZE sets the number of slots
(1024 by default, rounded up to a power of two, at most 65536); 0
builds the metadata of every evaluation, as before.

Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky leoromanovsky added type: feature Enhancements and improvements tag: performance Performance related changes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 19, 2026
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 97.62%
• Overall Coverage: 60.95% (-0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 99df09e | Docs | Give us feedback!

DD_FEATURE_FLAGS_EVALUATION_METADATA_TIME_BUCKET_MS sets the width of
the time bucket in which evaluations of a split share their metadata.
The default, 1 ms, keeps __dd_eval_timestamp_ms exact. A width of W ms
lets an evaluation reuse metadata built earlier in its bucket, so the
flag evaluation events report it up to W - 1 ms early. 0 leaves the
timestamp out of the metadata, so the flag evaluation hook stamps its
own time and a split's metadata is shared until its slot is replaced.
Widths are capped at the 10 s flush interval of the evaluation events.
Exposures stamp their own time and are not affected.

The table benchmark now sweeps the table size and bucket width, with
uniform and Zipf-skewed flag popularity, and reports hit rate and how
early the reported timestamps are.

Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.10 s 14.12 s [-0.9%; +0.6%] (no difference)
startup:insecure-bank:tracing:Agent 13.05 s 13.09 s [-1.1%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 17.82 s 17.60 s [+0.2%; +2.2%] (maybe worse)
startup:petclinic:iast:Agent 17.46 s 17.66 s [-1.8%; -0.4%] (maybe better)
startup:petclinic:profiling:Agent 16.86 s 17.26 s [-6.6%; +1.9%] (no difference)
startup:petclinic:sca:Agent 17.90 s 17.68 s [+0.1%; +2.4%] (maybe worse)
startup:petclinic:tracing:Agent 16.74 s 16.92 s [-1.8%; -0.3%] (maybe better)

Commit: 99df09eb · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@leoromanovsky leoromanovsky changed the title Share flag metadata across evaluations of a split in the same millisecond Share flag metadata across evaluations of a split in the same time bucket Nov 19, 2026
leoromanovsky and others added 2 commits November 21, 2026 14:44
Master added a switch that turns exposures off. The evaluator now takes
it with the metadata table settings, and computes doLog from it before
choosing the table key, so with exposures off every subject shares one
entry and the metadata carries no exposure keys.

Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
The sweep in EvaluationMetadataTableBenchmark shows a 10 ms bucket
shares most of what wider buckets share: at 4,096 slots on one thread,
the hit rate rises from 52% to 85% over 1 ms, against at most 4 more
points for 100 ms or more. Evaluations then report first_evaluation and
last_evaluation at most 9 ms early in a 10 s flush interval. A 1 ms
bucket still keeps the timestamps exact.

Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant