Repository navigation
Share flag metadata across evaluations of a split in the same time bucket - #13541
Draft
leoromanovsky wants to merge 21 commits into
Draft
leoromanovsky wants to merge 21 commits into
leoromanovsky wants to merge 21 commits into
Conversation
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON integers by type, so it could not set a baseline on master. It now compares JSON numbers by value, covers every case file of the scale and scale-json suites, can isolate one case with the index parameter, and prints each case's index. A load benchmark measures parsing each suite's configuration and prints the heap it retains, so changes that move work to load time show their cost. The submodule points to the scale-suite fixture branch, which adds the N x M dimensions and JSON values up to 10 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scale benchmarks failed the forbidden API check on System.out, and nothing outside a manual JMH run checked the scale cases, so a sibling change that resolves a large configuration wrongly would go unnoticed. - Add ScaleSuiteTest: evaluate every case of the scale and scale-json suites twice, and fail when the benchmark misses a case file. The cases, their check and the suite reader move to the test sources and the benchmarks reuse them; the reader uses JsonReading. - Allow System.out in the benchmark setups (@SuppressForbidden). - Check every case again after the measurement, fail on an error code, and reject expectations the check does not handle. - Rerun the jmh task when the fixtures change. - Measure retained heap after a first parse, as the smallest of three signed samples. - Describe a row as the unweighted mean over the cases of a file, say that hooks are not measured and when to run three forks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture PR now checks that the committed scale files match the generator and fixes its docs. The case and configuration files are unchanged, so the benchmark numbers do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to 1000 and of the powers of two up to 2^52, which an index keyed on exact doubles must still match. Pin them and benchmark the file; its 8 cases pass on master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OpenFeature client passes the provider a new LayeredEvaluationContext on each call, while the benchmark reused one MutableContext. A rule on `id` when the context has no `id` reads the layered context's key set, which it builds again on each call, so the benchmark did not show what applications pay for it. evaluateLayered wraps each case's context the way the client does, as ExperimentationOptOutBenchmark.evaluateFreshLayeredContext does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures replace the 10 MB JSON flag with a 4 MB one, which brings the scale-json configuration from 14 MB down to 8 MB. The other cases are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With several JMH threads and index=all, the threads shared one unsynchronized cursor, so a thread could advance it past the last case and fail the run with ArrayIndexOutOfBoundsException. The cursor is now thread-scoped state, and the provider and cases stay shared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pinned scale suite adds first-match cases and expects the selected serialId, doLog, and split features on scale-many-variations. ScaleCase now checks those fields from the flag metadata, as CanonicalFixtureTest does. The scale suites run without span enrichment, so an allocation that does not log carries no doLog key, and a missing key means false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DataDog/ffe-system-test-data#78 merged as 508a342 with the same suites as its head. Main also brings mutually exclusive layer cases and precomputed fixture corrections, which the unit and smoke tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SDK is the main reader of the ffe-system-test-data submodule and the only reader of the vendored experimentation opt-out cases, so both move from the smoke-test resources to dd-openfeature/src/test/fixtures. They stay outside any source set: the tests and benchmarks read them from disk, and the smoke tests copy ufc-config.json and evaluation-cases into their resources. The jmh task declares the scale suites as a file tree, which is empty rather than missing when the submodule is not initialized, so a jmh run on such a checkout no longer fails before it starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The suites read the system properties and environment, so setting DD_EXPERIMENTAL_FLAGGING_PROVIDER_SPAN_ENRICHMENT_ENABLED turned span enrichment on: the benchmarks measured the enriched metadata path and the expected selection metadata no longer described it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each case file parsed its suite again, 14 times for the 7.8 MB scale configuration. One provider per suite, shut down after all cases, parses each configuration once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the fixtures submodule at dd-openfeature/src/test/fixtures, pinned to 508a342. Master bumped it at the old smoke-test path to 29ba6bf, which is 4 commits behind that pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cond FlagEvaluator built a new ImmutableMetadata on every successful evaluation, although everything in it but the timestamp is fixed by the split, and the timestamp has millisecond resolution. A fixed, direct-mapped, lock-free table now keeps the metadata built last for a key of the Split, Allocation and Flag references, the consent, the exposure state and the evaluation millisecond, so evaluations of the same split in the same millisecond share one immutable instance. It is a burst coalescer, not a cache: an entry is reusable only within its millisecond. Keying on references means a reloaded configuration never matches an entry of the previous one. Entries have only final fields and are published with a single reference store, so a lost race only costs a rebuild. DD_FEATURE_FLAGS_EVALUATION_METADATA_TABLE_SIZE sets the number of slots (1024 by default, rounded up to a power of two, at most 65536); 0 builds the metadata of every evaluation, as before. Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
DD_FEATURE_FLAGS_EVALUATION_METADATA_TIME_BUCKET_MS sets the width of the time bucket in which evaluations of a split share their metadata. The default, 1 ms, keeps __dd_eval_timestamp_ms exact. A width of W ms lets an evaluation reuse metadata built earlier in its bucket, so the flag evaluation events report it up to W - 1 ms early. 0 leaves the timestamp out of the metadata, so the flag evaluation hook stamps its own time and a split's metadata is shared until its slot is replaced. Widths are capped at the 10 s flush interval of the evaluation events. Exposures stamp their own time and are not affected. The table benchmark now sweeps the table size and bucket width, with uniform and Zipf-skewed flag popularity, and reports hit rate and how early the reported timestamps are. Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Master added a switch that turns exposures off. The evaluator now takes it with the metadata table settings, and computes doLog from it before choosing the table key, so with exposures off every subject shares one entry and the metadata carries no exposure keys. Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
The sweep in EvaluationMetadataTableBenchmark shows a 10 ms bucket shares most of what wider buckets share: at 4,096 slots on one thread, the hit rate rises from 52% to 85% over 1 ms, against at most 4 more points for 100 ms or more. Evaluations then report first_evaluation and last_evaluation at most 9 ms early in a 10 s flush interval. A 1 ms bucket still keeps the timestamps exact. Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Successful flag evaluations that resolve the same split in the same time bucket (by default, the same 10 ms) now share one immutable
ImmutableMetadatainstead of each building its own. The metadata keeps the same keys and values, except that__dd_eval_timestamp_msis the time of the evaluation that built the shared entry, up to 9 ms earlier than the evaluation that reuses it. Exposures and span enrichment are unaffected; see Choosing the time bucket below.Two new settings control it:
DD_FEATURE_FLAGS_EVALUATION_METADATA_TABLE_SIZE(default1024, rounded up to a power of two, at most 65,536) sets the number of slots.0turns sharing off and builds the metadata on every evaluation, as before.DD_FEATURE_FLAGS_EVALUATION_METADATA_TIME_BUCKET_MS(default10, at most 10,000) sets the width W of the time bucket in which evaluations of a split share their metadata. With W > 1, evaluations that reuse an entry report the time of the evaluation that built it, up to W − 1 ms early.0leaves the timestamp out of the shared metadata, soFlagEvalLoggingHookuses its own clock, and the entry is reused until its slot is replaced.1keeps today's exact timestamps. The default is a reviewer question; see Choosing the time bucket below.The idea and its first measurements come from @dougqh in his review of #13489 (idea 3). He is credited as co-author of the commits.
Motivation
On a latency- and allocation-sensitive service, the metadata map was most of what a successful evaluation allocated: 330 B of a
scale/flagsevaluation, and 28 KB on the 200-featurescale/variationssplit. Almost all of it is fixed per split. The flag key, variation type, allocation key, serial ID,doLog, consent and split features never change for a split. The timestamp changes only once per millisecond. A service that evaluates the same flags for every request was rebuilding the same map thousands of times per millisecond.flowchart LR E[Evaluation resolves a split] --> X[Exposure lookup, per subject] X --> L{"Slot holds the same Split, Allocation, Flag,<br/>consent, exposure state and time bucket?"} L -- hit --> S[Return the shared metadata] L -- miss --> B[Build the metadata] --> P[Publish it with one reference store] --> SAdditional Notes
A burst coalescer, not a cache. With a bounded bucket, an entry can only be reused within the bucket it was built in (10 ms by default). After that, the next evaluation of the split builds a new entry and replaces the old one. Only the unbounded bucket (
0) turns it into a cache of static per-split metadata. The table never grows, holds at most one entry per slot, and does not keep a configuration alive beyond the last bucket in which it was used.The key compares references, not strings. It holds the
Split,AllocationandFlagobjects plus consent, exposure state and time bucket. A reloaded configuration parses new objects, so it can never be served an entry from the old one, and a metadata field cannot be missed from the key. The exposure lookup still runs on every evaluation, because its result depends on the subject. Its result is part of the key and picks the slot, so an already-exposed subject and a new one don't evict each other.Lock-free. Entries have only final fields and are published with a single plain reference store. When two threads miss the same slot at once, both build the metadata and the last store wins, so a lost race only costs one extra build.
Benchmarks. Baseline:
masterat5e0370d, which this branch now merges. Host: a shared Datadog workspace with a 16 vCPU Xeon Platinum 8175M (8 cores, 2 hardware threads each) and JDK 21.0.10. Other sessions were active, and the 1-minute load average was 2.3 to 4.7 during the runs. Single-threaded runs were pinned to CPUs 4–7, with 3 forks of 5 × 2 s iterations, at this PR's defaults (1,024 slots, W = 10 ms). ± is JMH's 99.9% interval. Allocation isgc.alloc.rate.normper evaluation.ScaleSuiteBenchmarkmasterevaluate,scale/flagsevaluate,scale/splitsevaluate,scale/variationsevaluate,scale/wide-contextevaluateLayered,scale/flagsevaluateLayered,scale/splitsevaluateLayered,scale/variationsevaluateLayered,scale/wide-contextevaluateLayeredis the client path, through the OpenFeature context layering.scale/splitsis unchanged within noise: its cost is the split selection itself, not the metadata. Onscale/wide-context, the client path's allocation is mostly the layered context, so the table saves the same ~330 B there as on the direct path.Exposures switch. Merging
masterbrought in the provider'sexposuresEnabledswitch (#12387). The table key uses the effective exposure state, so with exposures disabled every subject shares one entry per split, and the metadata carriesdoLogand the exposure cache hit exactly as onmaster.Choosing the time bucket
What a wider bucket changes. The only consumer of
__dd_eval_timestamp_msisFlagEvalLoggingHook, which feeds the flag evaluation events. Those events aggregate evaluations per 10 s flush and report onlyfirst_evaluationandlast_evaluationper bucket. Exposures stamp their own time and are not affected. Events are grouped by arrival, not by timestamp, so an entry built just before a flush can make bothfirst_evaluationandlast_evaluationup to W − 1 ms early, not only the last one. With0(unbounded), the hook stamps the time it runs, which is after the evaluation rather than before. That reverses the deliberate choice of evaluation time over hook time that the timestamp was added for.What it buys.
EvaluationMetadataTableBenchmarksweeps table size × W on 1 and 12 threads. Each request evaluates 10 or 50 flags drawn from 1,000 active flags, either uniformly or Zipf-skewed (exponent 1), with a new subject and cohort per request. A few cells from the full matrix below:Bytes are per evaluation, times are per request, and hit rate is the share of successful evaluations that reused metadata.
What the data shows:
Reviewer question: is 10 ms the right default? This PR sets W to 10 ms. It keeps evaluation-time semantics and captures most of the reuse, at a cost of reporting
first_evaluationandlast_evaluationat most 9 ms early in a 10 s window. Unbounded reuses slightly more and is never early, but it replaces evaluation time with hook time. A wider bucket helps little unless the table is also large enough for the number of active flags, so also consider raising the default table size to 4,096. When every slot is full, that retains about 1.5 MB for metadata shaped like the benchmark flags. Splits with many features retain more: the metadata of the 200-featurescale/variationssplit is about 25 KB. This PR keeps the table at 1,024 slots. Set W to1to keep today's exact timestamps.Benchmark limits. Host: the same shared workspace, a 16 vCPU Xeon Platinum 8175M with JDK 21.0.10. Each cell ran 2 × 1 s of warmup and 3 × 1 s of measurement in
SampleTimemode, with 1 fork. 1-thread runs were pinned to CPUs 4–7, and 12-thread runs to 12 logical CPUs (2–7, 10–15). Other sessions' builds and benchmarks shared the host. Load averages ran from about 3 to 15. The first 12-thread, 50-flag pass ran at load 20–27 and was discarded; it was re-run after waiting for the shared JMH lock. Treat mean and p99 times as rough; bytes, hit rates and timestamp errors are exact. To keep the matrix finishing, the bucket width was not swept with the table off, since it has no effect there. Timestamp error was sampled on every 8th request, as the clock just before the evaluation minus the reported time.Full matrix (128 cells)
1 thread
12 threads
Relation to #13489. This PR does not depend on #13489, build on it, or include any of it. Its gain is measured against
masteralone. If it lands, #13489 can drop its per-splitSplitFeaturescache: this table covers the same allocation without retaining state on everySplit, which @dougqh estimated at 4–8 MB of retained heap on the 10,000-flag configuration.Commit list. #13485 was squash-merged, so its 16 original branch commits still show in this PR's commit list, along with merges of
master. The diff againstmastercontains only this change.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issue