Skip to content

Stop taking the OpenTelemetry global lock per feature flag evaluation (FFE optimization) - #13530

Draft
dougqh wants to merge 3 commits into
dougqh/ffe-client-benchmarkfrom
dougqh/ffe-otel-lock
Draft

dougqh wants to merge 3 commits into
dougqh/ffe-client-benchmarkfrom
dougqh/ffe-otel-lock

Conversation

@dougqh

@dougqh dougqh commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Stops the evaluation metrics hook from taking a global lock on every evaluation:

  • Once an SDK is registered, it is never looked up again. GlobalOpenTelemetry.getOrNoop() (and isSet()) lock a static mutex on every call, even after a global is set, and the hook called it on every evaluation from every thread. OpenTelemetry lets the global be set only once, so the counter resolved from a registered SDK stays valid and is kept.
  • Until an SDK is registered, it is looked up at most every 100 ms. An application that registers its SDK later is still picked up, and FlagEvalMetricsLateSdkForkedTest keeps covering that, with the interval set to 0.
  • The attributes are skipped while OpenTelemetry is the no-op one. There is nothing to record into, but every evaluation still built an Attributes (builder, list, sorted arrays) and lowercased its reason.
  • The reason and error type are lowercased once. The values of the standard reasons and error codes are computed once, and the allocation key is read without the ImmutableMetadata getter, which builds a debug message for every absent key.

Motivation

With 4 threads evaluating a simple flag, JFR showed 5,235 contended enters on the GlobalOpenTelemetry mutex in 11 s, about 1.6 s of blocked time, or roughly 15% of thread time. After this change: 3 enters, 0 ms. A global monitor gets worse as cores are added, so a larger machine should gain more than this 5-vCPU sandbox shows.

Measured on its own: this PR's head (9c4f395eb0) against #13529's head (8670aedfe4), the branch it stacks on. ClientEvaluationBenchmark, unique subjects, 256 MB heap (G1), JDK 21.0.12 HotSpot, 5-vCPU sandbox, 3 forks per side with base and PR forks alternating, 3×2 s warmup and 5×2 s measurement, -prof gc. "Noise" means the fork means of the two sides overlap.

Case Threads Time per evaluation Bytes per evaluation GC ms per million evaluations
scale/flags 1 886 → 615 ns (−31%) 1,108 → 849 (−23%) 24.7 → 19.0 (−23%)
scale/flags 4 2.35 → 1.57 µs (−33%) 1,097 → 812 (−26%) 23.5 → 17.9 (−24%)
scale/flags 8 4.52 → 2.96 µs (−35%) 1,068 → 785 (−27%) 24.3 → 20.5 (−16%)
scale/splits 1 22.3 → 22.7 µs (noise) 39,989 → 39,888 (noise) 1,294 → 1,395 (noise)
scale/splits 4 49.3 → 47.4 µs (noise) 40,057 → 39,718 (noise) 846 → 854
scale/splits 8 99.3 → 98.6 µs (noise) 39,753 → 39,661 (noise) 816 → 788
  • On scale/flags, the saving is about 260–285 B per evaluation at every thread count, and the time gains are well outside noise: the fork ranges are far apart.
  • On scale/splits, the evaluation itself allocates about 40 KB, so the same ~260 B saving disappears into the ±200 B spread between forks. Time and GC are within noise.
  • Raw GC counts per fork go up on scale/flags (92 → 103 at 1 thread), because the PR runs 44% more evaluations in the same 10 s. Per evaluation, they go down.
  • 8 threads oversubscribe the sandbox's 5 vCPUs, so the PR's 8-thread spread is wider (12% coefficient of variation). The direction holds in every fork.

Paced re-measure (Bits, after the benchmark fix): unpaced runs overflow the flag-evaluation queue and drop 53–63% of events, so they leave part of the work out. With pace set so both sides drop only 2–5% (scale/flags, unique subjects, 256 MB, 3 forks per side, alternating):

Threads, pace Drops base / PR Time per evaluation Bytes per evaluation
1, pace 1000 2.0% / 2.3% 3.02 → 2.67 µs (−350 ns) 1,249 → 1,002 (−247 B, −19.8%)
4, pace 4000 4.0% / 4.4% 15.06 → 14.41 µs (−650 ns) 1,257 → 1,012 (−245 B, −19.5%)

So the honest per-evaluation saving is about −245 B (−20%) and −330 ns. The unpaced −24.5% bytes and −35% time above overstate the share a little. The paced time percentage is diluted by the pace itself, so quote the absolute saving.

Additional Notes

  • With an SDK registered, metrics are recorded as before. The only behavior change: if the application registers an SDK after evaluations started, evaluations in the up to 100 ms before the next lookup are not counted.
  • An upstream change that makes getOrNoop() read the volatile field before locking, as get() does, would help every library that calls it per operation. It would need the OpenTelemetry API version that ships it.
  • Caching Attributes per flag, variant, reason, error and allocation is a possible follow-up for applications that do register an SDK.

Relation to Leo's FFE stack (#13484 and #13486–#13490, on #13485)

Contributor Checklist

Jira ticket: [PROJ-IDENT]

Review follow-ups

  • Fixed (P1 from Bits' sdk-review): the no-op skip compared the OpenTelemetry instance with OpenTelemetry.noop(). With dd-java-agent and DD_METRICS_OTEL_ENABLED, the agent makes the no-op instance's meter provider record, so the skip dropped feature_flag.evaluations entirely. It now skips only when getMeterProvider() == MeterProvider.noop(). That also skips the wrapped no-op that OpenTelemetry API versions before 1.57 return from GlobalOpenTelemetry.get(), which still built attributes on every evaluation. Covered by skipsOnlyAnOpenTelemetryWhoseMeterProviderRecordsNothing.
  • Dropped the telemetry field, which the keep-forever counter made redundant; lower-case with Locale.ROOT; added Javadoc for the test constructor; and the four hooks now read the allocation key through one MetadataKeys.allocationKey helper.

🤖 Generated with Claude Code

@dougqh dougqh added type: feature Enhancements and improvements tag: performance Performance related changes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 18, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 94.87%
• Overall Coverage: 60.64% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 209f4e6 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.08 s 14.03 s [-0.3%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent 12.99 s 12.98 s [-0.5%; +0.7%] (no difference)
startup:petclinic:appsec:Agent 17.81 s 17.63 s [+0.2%; +1.9%] (maybe worse)
startup:petclinic:iast:Agent 17.56 s 17.70 s [-1.5%; -0.1%] (maybe better)
startup:petclinic:profiling:Agent 17.53 s 17.27 s [+0.1%; +2.8%] (maybe worse)
startup:petclinic:sca:Agent 17.82 s 17.70 s [-0.4%; +1.8%] (no difference)
startup:petclinic:tracing:Agent 16.80 s 16.90 s [-1.7%; +0.4%] (no difference)

Commit: 209f4e69 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh

dougqh commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Local measurements. The thread scaling is the clearest signal for this PR: with the global lock gone, time per evaluation on a simple flag stops roughly doubling from 4 to 8 threads. Allocation and GC changes also include the toolbox stack (#13531–#13535).

Relation to Leo's FFE stack (#13484 and #13486–#13490, on #13485)

Method. Measured locally on an M-series laptop (10 cores, 8 performance), Temurin 21.0.4, using ClientEvaluationBenchmark from #13529 with unique subjects and consent off unless noted. Both sides are #13485 + #13529 merged with today's master. "After" adds #13530 and #13531–#13535, including #13535's benchmark transport override. Before and after runs alternated per configuration. The machine was shared (load average 5–11), so times are noisy, and differences under about 10% are inconclusive. These numbers are for the combined stack; they aren't split per PR. Leo's evaluator PRs (#13488–#13490) are in neither side, which is why scale/splits barely moves in time or bytes: its cost is the per-shard MessageDigest.

Time per evaluation (avgt, 3 forks, 256 MB heap, -prof gc)

Case Threads Before After Change Bytes per evaluation
scale/flags 1 378 ± 6 ns 248 ± 3 ns −34% 1,035 → 760 (−27%)
scale/flags 4 787 ± 35 ns 759 ± 70 ns −4% (noise) 1,071 → 792 (−26%)
scale/flags 6 1,725 ± 97 ns 806 ± 34 ns −53%
scale/flags 8 2,094 ± 201 ns 971 ± 42 ns −54%
scale/splits 1 18.8 ± 0.3 µs 17.6 ± 0.6 µs −6% 37.8 → 37.0 KB
scale/splits 4 20.0 ± 0.5 µs 19.5 ± 0.4 µs −2% 37.2 → 36.7 KB
scale/splits 6 21.0 ± 0.5 µs 21.0 ± 0.5 µs 0%
scale/splits 8 22.3 ± 0.6 µs 22.0 ± 1.0 µs −1%

GC time at 256 MB dropped 13–42% at 1 and 4 threads (for example, scale/splits at 1 thread went from 585 to 339 ms), even where bytes per evaluation barely changed.

Small heap: 128 MB, G1, 4 threads (throughput, 2 forks, GC logs; pauses and promotion are per million evaluations over the 25 s measurement window; humongous and Full GC counts are per fork, over the whole run)

Case Throughput Pause time / Mops Promoted / Mops Old gen max Humongous-triggered GCs Full GCs
scale/flags, consent off +21% −20% −54% 101 → 71 MB 15 → 3 2 → 0
scale/flags, consent on +42% −27% −58% 104 → 93 MB 16 → 3 7 → 0
scale/splits, consent off +6% −28% −34% 98 → 88 MB 27 → 2 2 → 0
scale/splits, consent on +4% −16% −52% 86 → 76 MB 21 → 2 0 → 0

The remaining 2–3 humongous-triggered GCs happen during setup, when the 7.8 MB scale configuration is parsed, and appear on both sides. Concurrent marking runs almost constantly on both sides at 128 MB, because the scale configuration alone retains about 24 MB. That's sediment this stack doesn't address.

@dougqh
dougqh force-pushed the dougqh/ffe-client-benchmark branch from 1964dba to 8670aed Compare November 19, 2026 15:16
@dougqh
dougqh force-pushed the dougqh/ffe-otel-lock branch from ff97017 to 9c4f395 Compare November 19, 2026 15:16
@dougqh dougqh changed the title Stop taking the OpenTelemetry global lock per feature flag evaluation Stop taking the OpenTelemetry global lock per feature flag evaluation (FFE optimization) Nov 19, 2026
@dougqh
dougqh force-pushed the dougqh/ffe-otel-lock branch from 9c4f395 to bbbe280 Compare November 19, 2026 22:32
@dougqh

dougqh commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T12:49:00.359086Z 3621ab8 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3621ab8bda

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +91 to +94
if (now - lastNoopCheckNanos < noopRecheckNanos) {
return null;
}
lastNoopCheckNanos = now;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Throttle lookups only after observing a no-op provider

With an SDK already registered, concurrent initial evaluations can now be silently dropped. The first thread updates lastNoopCheckNanos before building the counter; while it initializes the meter/counter, another thread can observe counter == null, pass this interval check, and return without recording its evaluation. Previously, concurrent callers waited in synchronized initializeCounter() and then recorded. This is separate from the documented late-SDK registration delay. Update the throttle timestamp only after confirming a no-op provider, or serialize initialization so concurrent evaluations wait for the recording counter.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed in 209f4e6. The recheck timestamp is now set only when the lookup finds a no-op provider, so an evaluation racing the first one's counter initialization waits in initializeCounter and records. FlagEvalMetricsLateSdkForkedTest.evaluationsRacingCounterInitializationAreStillRecorded holds the first evaluation inside meterBuilder while a second one records; before the fix only 1 of the 2 evaluations was recorded.

datadog-bits and others added 2 commits November 20, 2026 10:38
GlobalOpenTelemetry.getOrNoop() locks a global mutex on every call. Keep
the counter of a registered SDK, which the global cannot replace, look
for an SDK at most every 100 ms until one is registered, and skip the
attributes while OpenTelemetry is the no-op one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: dougqh <dougqh@gmail.com>
With dd-java-agent and DD_METRICS_OTEL_ENABLED, the agent makes the no-op
OpenTelemetry instance return a meter provider that records, so skipping
the no-op instance dropped feature_flag.evaluations entirely. Checking the
meter provider instead also skips the wrapped no-op that older APIs return
from GlobalOpenTelemetry.get(), which still built attributes per
evaluation.

Also drop the telemetry field the keep-forever counter made redundant,
lower-case with Locale.ROOT, and read the allocation key through one
MetadataKeys helper instead of four copies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/ffe-otel-lock branch from 3621ab8 to f0678eb Compare November 20, 2026 15:38
The recheck timestamp was set before the counter was built, so an
evaluation racing the first one's counter initialization returned early
and went unrecorded. It's now set only when the lookup finds a no-op
provider, so racing evaluations wait in initializeCounter and record.

From Codex's review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants