Skip to content

Shade the perf toolbox into dd-openfeature (FFE optimization) - #13532

Open
dougqh wants to merge 4 commits into
dougqh/toolbox-componentfrom
dougqh/ffe-toolbox-shading
Open

dougqh wants to merge 4 commits into
dougqh/toolbox-componentfrom
dougqh/ffe-toolbox-shading

Conversation

@dougqh

@dougqh dougqh commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

dd-openfeature's published jar can carry the toolbox classes the SDK uses, relocated into com.datadog.openfeature.internal.shaded. Classes the SDK doesn't reach are dropped (class-level tree shaking with Shadow's minimize()).

Nothing in the SDK uses the toolbox yet, so the jar's contents don't change. The PRs stacked on this one are the first users.

Motivation

dd-openfeature is published on its own and has to keep working with any agent version attached. It can't share toolbox classes with the agent, and it can't depend on an unpublished module. A relocated private copy keeps that contract and lets the SDK use the toolbox's tables and buffers for its long-lived state.

Additional Notes

  • The POM doesn't change. Applying the Shadow plugin would make publish.gradle list only api dependencies and drop opentelemetry-api, an implementation dependency. So the plugin isn't applied. Instead, a ShadowJar task, toolboxJar, replaces the artifact of the java component's apiElements and runtimeElements. The generated POM is the same before and after.
  • Jars and CI:
    • The plain jar moves to the unbundled classifier in build/libs-unbundled, as dd-trace-ot does.
    • jar is finalized by toolboxJar, so the :dd-openfeature:jar that CI runs still leaves exactly one jar in build/libs: the shaded one.
    • The shaded jar takes the plain jar's manifest, Implementation-* attributes and Automatic-Module-Name included.
  • Relocation check: checkToolboxRelocation (part of check) fails if an unrelocated toolbox class reaches the jar, since it would be shared with the attached agent's copy. One toolboxPackages list drives both the relocations and the check.
  • Compatibility checks:
    • japicmp and the reproducible-build check (compareToReferenceJar) compare the shaded jar. japicmp's package includes already leave out the relocated package.
    • The openfeature smoke tests resolve the shaded jar.
    • dd-openfeature-1.69 compiles against it and its tests pass.
    • test-published-dependencies uses the unchanged POM.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

🤖 Generated with Claude Code

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 60.79% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 676974d | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.90 s 14.82 s [-0.2%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent 13.64 s 13.74 s [-1.5%; +0.0%] (no difference)
startup:petclinic:appsec:Agent 17.77 s 17.69 s [-0.4%; +1.4%] (no difference)
startup:petclinic:iast:Agent 17.57 s 17.60 s [-1.2%; +0.8%] (no difference)
startup:petclinic:profiling:Agent 17.34 s 17.38 s [-1.4%; +0.9%] (no difference)
startup:petclinic:sca:Agent 17.64 s 17.68 s [-1.1%; +0.7%] (no difference)
startup:petclinic:tracing:Agent 16.20 s 16.77 s [-7.6%; +0.8%] (no difference)

Commit: 676974da · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh

dougqh commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Local measurements of this stack together with #13530. The small-heap table is the clearest signal for the toolbox changes: promotion per evaluation fell 34–58%, Full GCs went to zero, and almost all humongous-triggered GCs disappeared. Time-per-evaluation gains on the simple flag are mostly #13530's. Nothing here shows #13535's possible +9% single-thread regression, but it wasn't isolated, so it still needs a per-PR check.

Relation to Leo's FFE stack (#13484 and #13486–#13490, on #13485)

Method. Measured locally on an M-series laptop (10 cores, 8 performance), Temurin 21.0.4, using ClientEvaluationBenchmark from #13529 with unique subjects and consent off unless noted. Both sides are #13485 + #13529 merged with today's master. "After" adds #13530 and #13531–#13535, including #13535's benchmark transport override. Before and after runs alternated per configuration. The machine was shared (load average 5–11), so times are noisy, and differences under about 10% are inconclusive. These numbers are for the combined stack; they aren't split per PR. Leo's evaluator PRs (#13488–#13490) are in neither side, which is why scale/splits barely moves in time or bytes: its cost is the per-shard MessageDigest.

Time per evaluation (avgt, 3 forks, 256 MB heap, -prof gc)

Case Threads Before After Change Bytes per evaluation
scale/flags 1 378 ± 6 ns 248 ± 3 ns −34% 1,035 → 760 (−27%)
scale/flags 4 787 ± 35 ns 759 ± 70 ns −4% (noise) 1,071 → 792 (−26%)
scale/flags 6 1,725 ± 97 ns 806 ± 34 ns −53%
scale/flags 8 2,094 ± 201 ns 971 ± 42 ns −54%
scale/splits 1 18.8 ± 0.3 µs 17.6 ± 0.6 µs −6% 37.8 → 37.0 KB
scale/splits 4 20.0 ± 0.5 µs 19.5 ± 0.4 µs −2% 37.2 → 36.7 KB
scale/splits 6 21.0 ± 0.5 µs 21.0 ± 0.5 µs 0%
scale/splits 8 22.3 ± 0.6 µs 22.0 ± 1.0 µs −1%

GC time at 256 MB dropped 13–42% at 1 and 4 threads (for example, scale/splits at 1 thread went from 585 to 339 ms), even where bytes per evaluation barely changed.

Small heap: 128 MB, G1, 4 threads (throughput, 2 forks, GC logs; pauses and promotion are per million evaluations over the 25 s measurement window; humongous and Full GC counts are per fork, over the whole run)

Case Throughput Pause time / Mops Promoted / Mops Old gen max Humongous-triggered GCs Full GCs
scale/flags, consent off +21% −20% −54% 101 → 71 MB 15 → 3 2 → 0
scale/flags, consent on +42% −27% −58% 104 → 93 MB 16 → 3 7 → 0
scale/splits, consent off +6% −28% −34% 98 → 88 MB 27 → 2 2 → 0
scale/splits, consent on +4% −16% −52% 86 → 76 MB 21 → 2 0 → 0

The remaining 2–3 humongous-triggered GCs happen during setup, when the 7.8 MB scale configuration is parsed, and appear on both sides. Concurrent marking runs almost constantly on both sides at 128 MB, because the scale configuration alone retains about 24 MB. That's sediment this stack doesn't address.

@dougqh dougqh changed the title Shade the perf toolbox into dd-openfeature Shade the perf toolbox into dd-openfeature (FFE optimization) Nov 19, 2026
@dougqh
dougqh force-pushed the dougqh/toolbox-component branch from 81636d4 to c76183c Compare November 19, 2026 21:24
@dougqh
dougqh force-pushed the dougqh/ffe-toolbox-shading branch 3 times, most recently from 89895cd to 8c81722 Compare November 20, 2026 13:39
@pr-commenter

pr-commenter Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

LLM Validation

LLM Validation Gate — java-agent-instructions

✅ PASS

  • No blocking-case regressions; the quality change is within noise (baseline/candidate confidence intervals overlap).

6 case(s) · quality 81.6 → 83.1 (+1.6) · win-rate 51% [49%–54%]

Details — analysis and per-case results

Analysis

Changed instruction file(s): AGENTS.md.

No safety or blocking-case regressions across 6 case(s). Overall pairwise win-rate 51% [49%–54%], quality +1.6 — see the verdict above for whether that clears the noise band.

Results

  • Pairwise win-rate: 51% [49%–54%] — candidate's share of blind comparisons (90% CI; spanning 50% = no clear difference)
  • Overall quality: 81.6 → 83.1 (/100, +1.6)
  • Bad signals introduced (advisory): 0
  • Candidate criteria coverage (advisory): 24/27 (89%) — expected_criteria the candidate met; does not affect the gate
  • Blocking-case regressions: 0

Cases

Case Mode Quality Δ Win-rate (90% CI) Safety
java-core-validation-001 warn +3.1 58% [45%–72%] ok
java-unit-test-conventions-001 warn +0.8 50% [50%–50%] ok
java-premain-safety-001 warn -1.9 50% [50%–50%] ok
java-ci-download-guidance-001 warn +1.1 50% [50%–50%] ok
java-documentation-only-validation-001 warn +5.8 50% [50%–50%] ok
java-existing-spock-maintenance-001 warn +0.6 50% [50%–50%] ok

Per-dimension scores, token usage, latency, and estimated cost are in the CI job logs.

@dougqh
dougqh requested review from amarziali and removed request for a team November 20, 2026 20:46
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T20:48:40.736455Z 8c81722 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Publication, project consumers, and compatibility checks consistently target the shaded JAR while preserving existing POM dependency scopes. Gradle execution remains unverified.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 8c81722

@dougqh
dougqh force-pushed the dougqh/ffe-toolbox-shading branch from 8c81722 to 6ac4db5 Compare November 20, 2026 20:50

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Confidence is limited to static inspection of the artifact replacement and relocation configuration; the shaded jar’s contents and generated publication metadata were not validated by running Gradle.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 6ac4db5

@dougqh
dougqh force-pushed the dougqh/ffe-toolbox-shading branch from 6ac4db5 to 0e0d85e Compare November 20, 2026 21:32

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The static review found no reportable defect in the shaded-jar publication wiring. Generated artifact contents and POM equivalence were not verified by an executed build.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 0e0d85e

dougqh and others added 4 commits November 20, 2026 22:53
dd-openfeature is published on its own and must work with any attached agent version, so it can't
depend on the toolbox module. Instead its published jar carries a relocated copy of the toolbox
classes it uses, under com.datadog.openfeature.internal.shaded, and Shadow's minimize() drops the
classes it doesn't reach.

The Shadow plugin is not applied: publish.gradle would then list only api dependencies in the POM
and drop opentelemetry-api. A ShadowJar task replaces the jar of the java component's apiElements
and runtimeElements instead, so the publication and its POM are unchanged. The plain jar moves to
the "unbundled" classifier, and the japicmp checks compare the published jar.

checkToolboxRelocation, run by check, fails if an unrelocated toolbox class reaches the jar.

Nothing in the SDK uses the toolbox yet, so the published jar has the same entries as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compareToReferenceJar (jardiff) compares the module's jar task output unless the Shadow plugin
is applied. Here the plain jar moved to the "unbundled" classifier and toolboxJar became the
published jar, so the check looked for an unbundled jar the reference set doesn't have. It now
compares the published, shaded jar; a clean rebuild matches it byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI runs :dd-openfeature:jar and collects build/libs/*.jar, but jar now
builds the unbundled jar into build/libs-unbundled, which left nothing for
validate_build, the S3 upload or system-tests. jar is now finalized by
toolboxJar, which writes the shaded jar to build/libs.

The shaded jar also takes the plain jar's manifest, so it keeps the
Implementation-* attributes the jar convention sets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/ffe-toolbox-shading branch from 0e0d85e to 676974d Compare November 21, 2026 03:53

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Static review supports preserving the existing POM while directing publication and compatibility checks to the shaded jar. Actual artifact generation and publication behavior were not verified by executing a build.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 676974d

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant