Repository navigation
Conversation
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
|
Local measurements. The thread scaling is the clearest signal for this PR: with the global lock gone, time per evaluation on a simple flag stops roughly doubling from 4 to 8 threads. Allocation and GC changes also include the toolbox stack (#13531–#13535). Relation to Leo's FFE stack (#13484 and #13486–#13490, on #13485)
Method. Measured locally on an M-series laptop (10 cores, 8 performance), Temurin 21.0.4, using Time per evaluation (avgt, 3 forks, 256 MB heap,
GC time at 256 MB dropped 13–42% at 1 and 4 threads (for example, Small heap: 128 MB, G1, 4 threads (throughput, 2 forks, GC logs; pauses and promotion are per million evaluations over the 25 s measurement window; humongous and Full GC counts are per fork, over the whole run)
The remaining 2–3 humongous-triggered GCs happen during setup, when the 7.8 MB scale configuration is parsed, and appear on both sides. Concurrent marking runs almost constantly on both sides at 128 MB, because the scale configuration alone retains about 24 MB. That's sediment this stack doesn't address. |
1964dba to
8670aed
Compare
ff97017 to
9c4f395
Compare
9c4f395 to
bbbe280
Compare
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3621ab8bda
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (now - lastNoopCheckNanos < noopRecheckNanos) { | ||
| return null; | ||
| } | ||
| lastNoopCheckNanos = now; |
There was a problem hiding this comment.
Throttle lookups only after observing a no-op provider
With an SDK already registered, concurrent initial evaluations can now be silently dropped. The first thread updates lastNoopCheckNanos before building the counter; while it initializes the meter/counter, another thread can observe counter == null, pass this interval check, and return without recording its evaluation. Previously, concurrent callers waited in synchronized initializeCounter() and then recorded. This is separate from the documented late-SDK registration delay. Update the throttle timestamp only after confirming a no-op provider, or serialize initialization so concurrent evaluations wait for the recording counter.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch, fixed in 209f4e6. The recheck timestamp is now set only when the lookup finds a no-op provider, so an evaluation racing the first one's counter initialization waits in initializeCounter and records. FlagEvalMetricsLateSdkForkedTest.evaluationsRacingCounterInitializationAreStillRecorded holds the first evaluation inside meterBuilder while a second one records; before the fix only 1 of the 2 evaluations was recorded.
GlobalOpenTelemetry.getOrNoop() locks a global mutex on every call. Keep the counter of a registered SDK, which the global cannot replace, look for an SDK at most every 100 ms until one is registered, and skip the attributes while OpenTelemetry is the no-op one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: dougqh <dougqh@gmail.com>
With dd-java-agent and DD_METRICS_OTEL_ENABLED, the agent makes the no-op OpenTelemetry instance return a meter provider that records, so skipping the no-op instance dropped feature_flag.evaluations entirely. Checking the meter provider instead also skips the wrapped no-op that older APIs return from GlobalOpenTelemetry.get(), which still built attributes per evaluation. Also drop the telemetry field the keep-forever counter made redundant, lower-case with Locale.ROOT, and read the allocation key through one MetadataKeys helper instead of four copies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3621ab8 to
f0678eb
Compare
The recheck timestamp was set before the counter was built, so an evaluation racing the first one's counter initialization returned early and went unrecorded. It's now set only when the lookup finds a no-op provider, so racing evaluations wait in initializeCounter and record. From Codex's review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What Does This Do
Stops the evaluation metrics hook from taking a global lock on every evaluation:
GlobalOpenTelemetry.getOrNoop()(andisSet()) lock a static mutex on every call, even after a global is set, and the hook called it on every evaluation from every thread. OpenTelemetry lets the global be set only once, so the counter resolved from a registered SDK stays valid and is kept.FlagEvalMetricsLateSdkForkedTestkeeps covering that, with the interval set to 0.Attributes(builder, list, sorted arrays) and lowercased its reason.ImmutableMetadatagetter, which builds a debug message for every absent key.Motivation
With 4 threads evaluating a simple flag, JFR showed 5,235 contended enters on the
GlobalOpenTelemetrymutex in 11 s, about 1.6 s of blocked time, or roughly 15% of thread time. After this change: 3 enters, 0 ms. A global monitor gets worse as cores are added, so a larger machine should gain more than this 5-vCPU sandbox shows.Measured on its own: this PR's head (
9c4f395eb0) against #13529's head (8670aedfe4), the branch it stacks on.ClientEvaluationBenchmark, unique subjects, 256 MB heap (G1), JDK 21.0.12 HotSpot, 5-vCPU sandbox, 3 forks per side with base and PR forks alternating, 3×2 s warmup and 5×2 s measurement,-prof gc. "Noise" means the fork means of the two sides overlap.scale/flagsscale/flagsscale/flagsscale/splitsscale/splitsscale/splitsscale/flags, the saving is about 260–285 B per evaluation at every thread count, and the time gains are well outside noise: the fork ranges are far apart.scale/splits, the evaluation itself allocates about 40 KB, so the same ~260 B saving disappears into the ±200 B spread between forks. Time and GC are within noise.scale/flags(92 → 103 at 1 thread), because the PR runs 44% more evaluations in the same 10 s. Per evaluation, they go down.Paced re-measure (Bits, after the benchmark fix): unpaced runs overflow the flag-evaluation queue and drop 53–63% of events, so they leave part of the work out. With
paceset so both sides drop only 2–5% (scale/flags, unique subjects, 256 MB, 3 forks per side, alternating):So the honest per-evaluation saving is about −245 B (−20%) and −330 ns. The unpaced −24.5% bytes and −35% time above overstate the share a little. The paced time percentage is diluted by the pace itself, so quote the absolute saving.
Additional Notes
getOrNoop()read the volatile field before locking, asget()does, would help every library that calls it per operation. It would need the OpenTelemetry API version that ships it.Attributesper flag, variant, reason, error and allocation is a possible follow-up for applications that do register an SDK.Relation to Leo's FFE stack (#13484 and #13486–#13490, on #13485)
FlagEvaluatorand the UFC parser and model. These PRs change different code: the OpenFeature client hook (Stop taking the OpenTelemetry global lock per feature flag evaluation (FFE optimization) #13530) and the long-lived state of the event pipelines, meaning the exposure cache, the evaluation aggregator and the event payloads (Keep one object per subject in the feature flag exposure cache (FFE optimization) #13533–Build feature flag event payloads in chunks instead of large arrays (FFE optimization) #13535). Benchmark flag evaluation through the OpenFeature client (FFE optimization) #13529 adds a client-path benchmark on top of Benchmark FFE evaluation on the scale suites #13485's scale suites, covering the hooks and pipelines thatScaleSuiteBenchmarkskips.scale/splitsfrom about 40 KB to under 1 KB per evaluation, which changes the GC picture for that case.dd-openfeature/build.gradle.ktsandgradle.lockfile(Shade the perf toolbox into dd-openfeature (FFE optimization) #13532, Keep one object per subject in the feature flag exposure cache (FFE optimization) #13533), andScaleCase.java(Benchmark flag evaluation through the OpenFeature client (FFE optimization) #13529). These are textual conflicts. Whichever merges second rebases and regenerates the lockfile. Evaluate feature flag regex conditions in linear time #13487 also addsre2jas animplementationdependency. Once Shade the perf toolbox into dd-openfeature (FFE optimization) #13532 lands,re2jcould be relocated into the SDK jar the same way as the toolbox, if we want the SDK to carry no third-party runtime dependencies beyond its API contracts. That would be a follow-up, not a blocker.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Review follow-ups
OpenTelemetry.noop(). Withdd-java-agentandDD_METRICS_OTEL_ENABLED, the agent makes the no-op instance's meter provider record, so the skip droppedfeature_flag.evaluationsentirely. It now skips only whengetMeterProvider() == MeterProvider.noop(). That also skips the wrapped no-op that OpenTelemetry API versions before 1.57 return fromGlobalOpenTelemetry.get(), which still built attributes on every evaluation. Covered byskipsOnlyAnOpenTelemetryWhoseMeterProviderRecordsNothing.telemetryfield, which the keep-forever counter made redundant; lower-case withLocale.ROOT; added Javadoc for the test constructor; and the four hooks now read the allocation key through oneMetadataKeys.allocationKeyhelper.🤖 Generated with Claude Code