Skip to content

Compare FFE SemVer attributes without parsing them - #13488

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 14 commits into
masterfrom
leo.romanovsky/ffe-semver-parse-once
Nov 20, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 14 commits into
masterfrom
leo.romanovsky/ffe-semver-parse-once

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

SEMVER_* conditions compare the attribute string with the pre-parsed comparand without parsing it or allocating. Before this change, each condition parsed the value into a new ParsedSemver. On the scale/semver suite an evaluation is 2× faster and allocates 97.5% less. Its worst case, a long prerelease and build version, is 2.4× faster and allocates 99% less.

The results do not change, including for invalid versions (no match) and prerelease precedence. The error codes and messages of numeric comparisons do not change either.

Motivation

We need to win a performance-sensitive customer. The scale benchmarks of #13485, now on master, are the foundation for this work.

A rollout ladder targets version ranges, and each step is an allocation with SEMVER_GTE and SEMVER_LT on the same attribute. The scale-semver-ladder fixture has 50 steps, so one evaluation parsed the same app_version string up to 100 times. A version such as 1.49.3-beta.11.rc-7+build.20261117.sha.0123456789abcdef costs 35 µs and 42 KB of garbage per evaluation on master.

Additional Notes

Approach

ParsedSemver.compare(String, ParsedSemver) validates the version and compares it with the comparand without allocating. It reads the core components digit by digit, with the same leading-zero and unsigned 64-bit overflow checks as parse. It compares the prerelease as a range of the string, and validates the build metadata without keeping it. parse and this method share one helper that validates the prerelease and build metadata. It returns exactly compare(parse(version), comparand), or INVALID_ORDERING when parse would reject the version. @dougqh suggested this approach in review and is credited as co-author of the commit.

Alternative considered: a per-evaluation parse cache

An earlier revision of this PR parsed each attribute value once per evaluation and cached the result. That needed a SemverParseCache and a hasSemverConditions marker on Flag. It was faster but allocated more:

scale/semver master Parse cache This PR (one pass)
Mean of 6 cases 13.36 µs ± 2.08, 15,971 B 3.37 µs ± 0.29, 501 B 6.73 µs ± 0.64, 392 B
Case 1, 1.49.3 13.05 µs ± 1.65, 13,992 B 4.53 µs ± 0.62, 488 B 5.55 µs ± 0.27, 392 B
Case 2, long prerelease and build 34.75 µs ± 10.4, 42,392 B 4.56 µs ± 0.33, 544 B 14.51 µs ± 3.15, 392 B

We chose the one-pass compare because it allocates the least: at most 392 B per evaluation, however long the version. It also adds no per-evaluation state and no field on Flag. The cost is time on long versions in large ladders, where each condition scans the string again. Case 2 takes about 3× as long as with the cache, and is still 2.4× faster than on master.

AWS workspace, Intel Xeon Platinum 8175M, 16 vCPUs, JDK 21.0.10, JMH AverageTime, 3 warm-up and 5 measurement iterations of 2 s, -prof gc. The master column (4075f1fd4a) is from 1 fork, and the other two columns from 2 forks. Other sessions shared the machine, so the time errors are wider than usual; the B/op values repeated exactly across runs.

Other decisions

  • Numeric conditions. The numeric ladder's condition values are already Double after parsing, so converting operands without boxing saves only 24 B per evaluation, within run-to-run variance. It stays because it follows the same pattern, costs nothing, and does not change any error. Most of the numeric ladder's remaining ~1.5 KB is the attribute lookup itself: OpenFeature's convertValue boxes the numeric attribute again for every condition that reads it.
  • Error messages. getErrorMessage() exposes exception messages when full evaluation data is observed. Operands that are not numbers still go through mapValue, and tests compare the numeric errors with mapValue's for "abc", "", a boolean, "1,5", and a null ONE_OF element.
  • Interaction with the allocation scan PR. Read flag attributes once per evaluation and cache split feature metadata #13489 adds a per-evaluation EvaluationScope passed to evaluateRules and evaluateCondition. This PR no longer changes those signatures, so the two do not overlap there.

Testing

ParsedSemverTest checks that the in-place comparison returns exactly compare(parse(version), comparand), or INVALID_ORDERING, over every pair of the valid, invalid, spec-ordering and edge-case versions. Edge cases include missing and extra components, leading zeros, overflow, empty identifiers and hyphenated identifiers. It also checks 20,000 seeded random near-valid strings against 8 comparands each, and 20,000 random suffixes on equal cores, which reach the prerelease comparison. Separate tests check SemVer precedence through the String overload and reject non-ASCII digits in both paths. A review-only check also compared the String overload with master's parse and compare over 800,116 mutated version strings, and all agreed. Five deliberate mutations of the new method (no overflow check, leading zeros allowed, omitted components ignored, an unbounded prerelease search, no build validation) each fail these tests. FlagEvaluatorTest evaluates a SemVer ladder whose rules test two attributes, including prerelease precedence, an omitted patch, and invalid versions. It also covers SEMVER operator edge cases (an invalid suffix after a decided core, extra core components, a dotted build) and a fractional numeric comparison. The null-element message test runs on JDKs with helpful NullPointerException messages (15 and later) and is skipped on older ones. CanonicalFixtureTest, ScaleSuiteTest and the rest of the dd-openfeature tests pass.

Contributor Checklist

🤖 Generated with Claude Code

leoromanovsky and others added 4 commits November 17, 2026 00:47
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON
integers by type, so it could not set a baseline on master. It now
compares JSON numbers by value, covers every case file of the scale and
scale-json suites, can isolate one case with the index parameter, and
prints each case's index. A load benchmark measures parsing each suite's
configuration and prints the heap it retains, so changes that move work
to load time show their cost.

The submodule points to the scale-suite fixture branch, which adds the
N x M dimensions and JSON values up to 10 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A flag with many SemVer conditions on the same attribute parsed the
attribute value once per condition: the scale-semver-ladder fixture
parses it up to 100 times per evaluation. The parser now marks flags
with SemVer conditions, and their evaluations keep a small cache of
attribute parses, invalid versions included. Other flags allocate
nothing for it.

ParsedSemver.parse no longer copies the core components, the build
metadata or the remainder, and prerelease comparison compares
identifiers in place instead of copying them. Numeric conditions
convert operands to double without boxing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Numeric comparisons convert numbers without boxing and send every
other operand through mapValue, so an unconvertible or null operand
still fails with the same exception and message, which reaches
getErrorMessage() when full evaluation data is observed. Tests compare
those messages with mapValue's for a few bad inputs.

Flag.hasSemverConditions documents that only the parser sets it,
before the configuration is published. The parse cache derives its
count from its entries. The ladder tests move into FlagEvaluatorTest
and the parser flag test into UniversalFlagConfigParserTest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky leoromanovsky added type: feature Enhancements and improvements tag: performance Performance related changes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 17, 2026
@leoromanovsky leoromanovsky added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026 — with ddtool CLI
@leoromanovsky leoromanovsky removed the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
leoromanovsky and others added 6 commits November 17, 2026 04:00
The scale benchmarks failed the forbidden API check on System.out, and
nothing outside a manual JMH run checked the scale cases, so a sibling
change that resolves a large configuration wrongly would go unnoticed.

- Add ScaleSuiteTest: evaluate every case of the scale and scale-json
  suites twice, and fail when the benchmark misses a case file. The
  cases, their check and the suite reader move to the test sources and
  the benchmarks reuse them; the reader uses JsonReading.
- Allow System.out in the benchmark setups (@SuppressForbidden).
- Check every case again after the measurement, fail on an error code,
  and reject expectations the check does not handle.
- Rerun the jmh task when the fixtures change.
- Measure retained heap after a first parse, as the smallest of three
  signed samples.
- Describe a row as the unweighted mean over the cases of a file, say
  that hooks are not measured and when to run three forks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture PR now checks that the committed scale files match the
generator and fixes its docs. The case and configuration files are
unchanged, so the benchmark numbers do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test now counts the SemVer parses of one evaluation's rules, so
bypassing the cache fails. New precedence rows cover a numeric
prerelease identifier that is not the last one. Core components use
Long.parseUnsignedLong on the version's range instead of a hand-written
parser. The null-element message test is skipped on JDKs without
helpful NullPointerException messages instead of comparing null with
null, and the toDouble Javadoc states what it converts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to
1000 and of the powers of two up to 2^52, which an index keyed on exact
doubles must still match. Pin them and benchmark the file; its 8 cases
pass on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 60.65% (-0.03%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 1986a92 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.85 s 14.70 s [+0.2%; +1.8%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.67 s 13.71 s [-1.0%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 17.21 s 17.18 s [-0.7%; +1.1%] (no difference)
startup:petclinic:iast:Agent 17.10 s 16.65 s [-1.9%; +7.2%] (no difference)
startup:petclinic:profiling:Agent 16.74 s 16.77 s [-1.2%; +0.8%] (no difference)
startup:petclinic:sca:Agent 17.23 s 17.13 s [-0.3%; +1.4%] (no difference)
startup:petclinic:tracing:Agent 16.33 s 16.26 s [-0.5%; +1.3%] (no difference)

Commit: 1986a92c · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh

dougqh commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

An alternative to consider, from an experiment on top of #13485: compare the attribute string against the pre-parsed comparand in place, without parsing it at all.

ParsedSemver.compare(String, ParsedSemver) validates and compares in a single pass:

  • It reads the core components digit by digit, with the same unsigned 64-bit overflow check.
  • It treats the prerelease and build identifiers as ranges of the string.
  • It returns an INVALID_ORDERING sentinel for any version parse would reject.

A test checks that, for every pair from the valid, invalid and spec-ordering lists plus edge cases (missing or extra components, leading zeros, build metadata, hyphenated identifiers), it returns exactly compare(parse(version), comparand), or INVALID_ORDERING when parse fails.

On scale/semver it went from 10.8 µs, 21,595 B to 6.1 µs, 392 B. That's close to what this PR reaches (501 B). The numbers came from a different, noisier machine, so the times can't be compared directly. The main difference is simplicity: there's no per-evaluation cache and no SemVer marker on Flag. Each condition costs one linear scan of the attribute instead of a cache lookup, though. On the 50-step ladder that's up to 100 scans of a short string, against one parse plus cache hits. It may not be worth switching, but I wanted you to have the option. Happy to share the patch.

leoromanovsky and others added 3 commits November 19, 2026 03:42
Takes master's version of every #13485 file; only this branch's SemVer
changes differ from master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SpotBugs reports ES_COMPARING_PARAMETER_STRING_WITH_EQ for the
`cached == value || cached.equals(value)` fast path. String.equals
already returns early on identity, so the explicit == adds nothing;
scale/semver time and allocation are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An evaluation now validates and compares each SemVer attribute against the
pre-parsed comparand in one pass over the string, without building a
ParsedSemver. This removes the per-evaluation SemverParseCache and the
hasSemverConditions marker on Flag, which only existed to feed that cache.

ParsedSemver.compare(String, ParsedSemver) returns exactly
compare(parse(version), comparand), or INVALID_ORDERING when parse would
reject the version. Tests check that over every pair of the valid, invalid,
spec-ordering and edge-case versions, and over seeded random strings.

Suggested by Doug Hawkins in the review of #13488.

Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky leoromanovsky changed the title Parse each FFE SemVer attribute once per evaluation Compare FFE SemVer attributes without parsing them Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Comment from Claude working with @leoromanovsky:

Thanks Doug, we adopted your approach in e006963, with you credited as co-author. The simpler implementation is a big part of why: there's no per-evaluation cache and no marker on Flag, only one more method on ParsedSemver.

scale/semver on one host, with -prof gc. The machine was shared, so the time errors are wide, but the B/op values repeated exactly:

master Parse cache One pass
Mean of 6 cases 13.36 µs, 15,971 B 3.37 µs, 501 B 6.73 µs, 392 B
Case 2, long prerelease and build 34.75 µs, 42,392 B 4.56 µs, 544 B 14.51 µs, 392 B

The cache stays faster on long versions in the 50-step ladder, where each condition scans the string again. The one-pass version allocates the least and is much simpler, so we went with it. The tests check it against compare(parse(version), comparand) over every pair of the valid, invalid, spec-ordering and edge-case versions, and over seeded random strings. The PR description has the details.

parse and compare(String, ParsedSemver) now validate the prerelease and
build metadata through one helper, so a grammar change lands in one place.
The helper allocates nothing and returns the end of the prerelease.

New tests reject non-ASCII digits in both paths, fuzz the suffixes of
equal cores, check SemVer precedence through the String overload, cover
SEMVER operator edge cases in the evaluator, and catch a truncating
numeric conversion. The javadocs now say the attribute is compared
without allocating rather than in one pass, and describe how null fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky
leoromanovsky marked this pull request as ready for review November 19, 2026 18:22
@leoromanovsky
leoromanovsky requested review from a team as code owners November 19, 2026 18:22
@leoromanovsky
leoromanovsky requested review from AlexeyKuznetsov-DD, greghuels and typotter and removed request for a team and AlexeyKuznetsov-DD November 19, 2026 18:22
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T18:26:21.484029Z 1986a92 Draft marked ready
🔒 Security Review ✅ Completed 2026-11-19T18:28:28.412270Z 1986a92 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Static review identified no actionable discrepancy in SemVer validation, prerelease precedence, or numeric comparison error handling. Repeated scans of long version strings are an explicitly accepted trade-off for lower allocation.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 1986a92

@greghuels greghuels left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Reviewed this commit; no merge blockers found.

@leoromanovsky
leoromanovsky added this pull request to the merge queue Nov 20, 2026
@dd-octo-sts

dd-octo-sts Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-11-20 01:43:52 UTC ℹ️ Start processing command /merge


2026-11-20 01:43:56 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-11-20 02:26:18 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 20, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 1ab1eac into master Nov 20, 2026
697 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the leo.romanovsky/ffe-semver-parse-once branch November 20, 2026 02:26
@github-actions github-actions Bot added this to the 1.70.0 milestone Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants