Repository navigation
Conversation
…I perf review) Allocation claims need a gc-profiler before/after naming both SHAs; timing claims need a controlled same-machine run with error bars; changes that only remove work may claim the reduction without numbers. A claim missing its evidence is an advisory finding scoped to the unsupported part, not a merge blocker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LLM ValidationLLM Validation Gate — java-agent-instructions✅ PASS
6 case(s) · quality 80.0 → 81.9 (+1.9) · win-rate 50% [47%–53%] Details — analysis and per-case resultsAnalysisChanged instruction file(s): No safety or blocking-case regressions across 6 case(s). Overall pairwise win-rate 50% [47%–53%], quality +1.9 — see the verdict above for whether that clears the noise band. Results
Cases
Per-dimension scores, token usage, latency, and estimated cost are in the CI job logs. |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
There was a problem hiding this comment.
More details
Allocation evidence no longer has to support timing claims; speedups still require controlled comparisons with error bars, and unsupported claims are advisory in both guidance files.
🤖 Bits Code Review · Commit 4b3db89
bric3
left a comment
There was a problem hiding this comment.
Could we require matching JVM and workload settings for allocation comparisons? One run per revision is enough when both runs measure the same work. I'd keep the full rule in AGENTS.md so reviewers apply it consistently.
Also, the PR title should be : Set benchmark requirements for performance claims.
…vidence # Conflicts: # .agents/skills/perf-review/SKILL.md
Both sides of a before/after must use the same JDK, JVM flags, benchmark settings and workload; one run per side is still enough for allocation. The run-to-run-variance test for timing claims moves into AGENTS.md too, and perf-review's posture.md now points there instead of restating the rule, so reviewers following either document reach the same conclusion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What Does This Do
Changes the core performance rule in
AGENTS.mdand theperf-reviewskill so the benchmark evidence a PR must show depends on what it claims:-Pjmh.profilers=gcbefore/after that names both commit SHAs. One run per side is enough.A claim without its matching evidence becomes an advisory finding. It covers only the unsupported part of the claim, it doesn't block the merge, and it has no severity rating. The author either adds the numbers or narrows the claim.
Motivation
The current rule asks for full controlled before/after time and allocation results for every claim. On #13481, a clear allocation win with credible
gc.alloc.rate.normnumbers was flagged P1 by the Codex reviewer and flagged again by/perf-review. Its timing table really was too noisy to support its numbers (an unchanged control moved −18% to +39% between runs), but its allocation numbers were fine. The rule should catch the timing gap without treating the whole PR as unmeasured.Additional Notes
gc.alloc.rate.normbarely varies between runs, so allocation claims don't need the forks and error bars that timing claims do.SKILL.mdpoints toAGENTS.mdfor this rule, so both are updated together to keep them consistent.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: N/A
🤖 Generated with Claude Code