Repository navigation
Compare FFE SemVer attributes without parsing them - #13488
gh-worker-dd-mergequeue-cf854d[bot] merged 14 commits into
Conversation
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON integers by type, so it could not set a baseline on master. It now compares JSON numbers by value, covers every case file of the scale and scale-json suites, can isolate one case with the index parameter, and prints each case's index. A load benchmark measures parsing each suite's configuration and prints the heap it retains, so changes that move work to load time show their cost. The submodule points to the scale-suite fixture branch, which adds the N x M dimensions and JSON values up to 10 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A flag with many SemVer conditions on the same attribute parsed the attribute value once per condition: the scale-semver-ladder fixture parses it up to 100 times per evaluation. The parser now marks flags with SemVer conditions, and their evaluations keep a small cache of attribute parses, invalid versions included. Other flags allocate nothing for it. ParsedSemver.parse no longer copies the core components, the build metadata or the remainder, and prerelease comparison compares identifiers in place instead of copying them. Numeric conditions convert operands to double without boxing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Numeric comparisons convert numbers without boxing and send every other operand through mapValue, so an unconvertible or null operand still fails with the same exception and message, which reaches getErrorMessage() when full evaluation data is observed. Tests compare those messages with mapValue's for a few bad inputs. Flag.hasSemverConditions documents that only the parser sets it, before the configuration is published. The parse cache derives its count from its entries. The ladder tests move into FlagEvaluatorTest and the parser flag test into UniversalFlagConfigParserTest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scale benchmarks failed the forbidden API check on System.out, and nothing outside a manual JMH run checked the scale cases, so a sibling change that resolves a large configuration wrongly would go unnoticed. - Add ScaleSuiteTest: evaluate every case of the scale and scale-json suites twice, and fail when the benchmark misses a case file. The cases, their check and the suite reader move to the test sources and the benchmarks reuse them; the reader uses JsonReading. - Allow System.out in the benchmark setups (@SuppressForbidden). - Check every case again after the measurement, fail on an error code, and reject expectations the check does not handle. - Rerun the jmh task when the fixtures change. - Measure retained heap after a first parse, as the smallest of three signed samples. - Describe a row as the unweighted mean over the cases of a file, say that hooks are not measured and when to run three forks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture PR now checks that the committed scale files match the generator and fixes its docs. The case and configuration files are unchanged, so the benchmark numbers do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A test now counts the SemVer parses of one evaluation's rules, so bypassing the cache fails. New precedence rows cover a numeric prerelease identifier that is not the last one. Core components use Long.parseUnsignedLong on the version's range instead of a hand-written parser. The null-element message test is skipped on JDKs without helpful NullPointerException messages instead of comparing null with null, and the toDouble Javadoc states what it converts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to 1000 and of the powers of two up to 2^52, which an index keyed on exact doubles must still match. Pin them and benchmark the file; its 8 cases pass on master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
|
An alternative to consider, from an experiment on top of #13485: compare the attribute string against the pre-parsed comparand in place, without parsing it at all.
A test checks that, for every pair from the valid, invalid and spec-ordering lists plus edge cases (missing or extra components, leading zeros, build metadata, hyphenated identifiers), it returns exactly On |
Takes master's version of every #13485 file; only this branch's SemVer changes differ from master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SpotBugs reports ES_COMPARING_PARAMETER_STRING_WITH_EQ for the `cached == value || cached.equals(value)` fast path. String.equals already returns early on identity, so the explicit == adds nothing; scale/semver time and allocation are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An evaluation now validates and compares each SemVer attribute against the pre-parsed comparand in one pass over the string, without building a ParsedSemver. This removes the per-evaluation SemverParseCache and the hasSemverConditions marker on Flag, which only existed to feed that cache. ParsedSemver.compare(String, ParsedSemver) returns exactly compare(parse(version), comparand), or INVALID_ORDERING when parse would reject the version. Tests check that over every pair of the valid, invalid, spec-ordering and edge-case versions, and over seeded random strings. Suggested by Doug Hawkins in the review of #13488. Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🤖 Comment from Claude working with @leoromanovsky: Thanks Doug, we adopted your approach in e006963, with you credited as co-author. The simpler implementation is a big part of why: there's no per-evaluation cache and no marker on
The cache stays faster on long versions in the 50-step ladder, where each condition scans the string again. The one-pass version allocates the least and is much simpler, so we went with it. The tests check it against |
parse and compare(String, ParsedSemver) now validate the prerelease and build metadata through one helper, so a grammar change lands in one place. The helper allocates nothing and returns the end of the prerelease. New tests reject non-ASCII digits in both paths, fuzz the suffixes of equal cores, check SemVer precedence through the String overload, cover SEMVER operator edge cases in the evaluator, and catch a truncating numeric conversion. The javadocs now say the attribute is compared without allocating rather than in one pass, and describe how null fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
More details
Static review identified no actionable discrepancy in SemVer validation, prerelease precedence, or numeric comparison error handling. Repeated scans of long version strings are an explicitly accepted trade-off for lower allocation.
🤖 Bits Code Review · Commit 1986a92
greghuels
left a comment
There was a problem hiding this comment.
🤖 Reviewed this commit; no merge blockers found.
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
What Does This Do
SEMVER_*conditions compare the attribute string with the pre-parsed comparand without parsing it or allocating. Before this change, each condition parsed the value into a newParsedSemver. On thescale/semversuite an evaluation is 2× faster and allocates 97.5% less. Its worst case, a long prerelease and build version, is 2.4× faster and allocates 99% less.The results do not change, including for invalid versions (no match) and prerelease precedence. The error codes and messages of numeric comparisons do not change either.
Motivation
We need to win a performance-sensitive customer. The scale benchmarks of #13485, now on
master, are the foundation for this work.A rollout ladder targets version ranges, and each step is an allocation with
SEMVER_GTEandSEMVER_LTon the same attribute. Thescale-semver-ladderfixture has 50 steps, so one evaluation parsed the sameapp_versionstring up to 100 times. A version such as1.49.3-beta.11.rc-7+build.20261117.sha.0123456789abcdefcosts 35 µs and 42 KB of garbage per evaluation onmaster.Additional Notes
Approach
ParsedSemver.compare(String, ParsedSemver)validates the version and compares it with the comparand without allocating. It reads the core components digit by digit, with the same leading-zero and unsigned 64-bit overflow checks asparse. It compares the prerelease as a range of the string, and validates the build metadata without keeping it.parseand this method share one helper that validates the prerelease and build metadata. It returns exactlycompare(parse(version), comparand), orINVALID_ORDERINGwhenparsewould reject the version. @dougqh suggested this approach in review and is credited as co-author of the commit.Alternative considered: a per-evaluation parse cache
An earlier revision of this PR parsed each attribute value once per evaluation and cached the result. That needed a
SemverParseCacheand ahasSemverConditionsmarker onFlag. It was faster but allocated more:scale/semvermaster1.49.3We chose the one-pass compare because it allocates the least: at most 392 B per evaluation, however long the version. It also adds no per-evaluation state and no field on
Flag. The cost is time on long versions in large ladders, where each condition scans the string again. Case 2 takes about 3× as long as with the cache, and is still 2.4× faster than onmaster.AWS workspace, Intel Xeon Platinum 8175M, 16 vCPUs, JDK 21.0.10, JMH AverageTime, 3 warm-up and 5 measurement iterations of 2 s,
-prof gc. Themastercolumn (4075f1fd4a) is from 1 fork, and the other two columns from 2 forks. Other sessions shared the machine, so the time errors are wider than usual; the B/op values repeated exactly across runs.Other decisions
Doubleafter parsing, so converting operands without boxing saves only 24 B per evaluation, within run-to-run variance. It stays because it follows the same pattern, costs nothing, and does not change any error. Most of the numeric ladder's remaining ~1.5 KB is the attribute lookup itself: OpenFeature'sconvertValueboxes the numeric attribute again for every condition that reads it.getErrorMessage()exposes exception messages when full evaluation data is observed. Operands that are not numbers still go throughmapValue, and tests compare the numeric errors withmapValue's for"abc","", a boolean,"1,5", and anullONE_OFelement.EvaluationScopepassed toevaluateRulesandevaluateCondition. This PR no longer changes those signatures, so the two do not overlap there.Testing
ParsedSemverTestchecks that the in-place comparison returns exactlycompare(parse(version), comparand), orINVALID_ORDERING, over every pair of the valid, invalid, spec-ordering and edge-case versions. Edge cases include missing and extra components, leading zeros, overflow, empty identifiers and hyphenated identifiers. It also checks 20,000 seeded random near-valid strings against 8 comparands each, and 20,000 random suffixes on equal cores, which reach the prerelease comparison. Separate tests check SemVer precedence through the String overload and reject non-ASCII digits in both paths. A review-only check also compared the String overload withmaster'sparseandcompareover 800,116 mutated version strings, and all agreed. Five deliberate mutations of the new method (no overflow check, leading zeros allowed, omitted components ignored, an unbounded prerelease search, no build validation) each fail these tests.FlagEvaluatorTestevaluates a SemVer ladder whose rules test two attributes, including prerelease precedence, an omitted patch, and invalid versions. It also covers SEMVER operator edge cases (an invalid suffix after a decided core, extra core components, a dotted build) and a fractional numeric comparison. The null-element message test runs on JDKs with helpfulNullPointerExceptionmessages (15 and later) and is skipped on older ones.CanonicalFixtureTest,ScaleSuiteTestand the rest of thedd-openfeaturetests pass.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issue🤖 Generated with Claude Code