Skip to content

Set benchmark requirements for performance claims (AI perf review) - #13502

Open
dougqh wants to merge 3 commits into
masterfrom
dougqh/perf-claim-evidence
Open

dougqh wants to merge 3 commits into
masterfrom
dougqh/perf-claim-evidence

Conversation

@dougqh

@dougqh dougqh commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Changes the core performance rule in AGENTS.md and the perf-review skill so the benchmark evidence a PR must show depends on what it claims:

  • Allocates less: a JMH -Pjmh.profilers=gc before/after that names both commit SHAs. One run per side is enough.
  • Faster: a before/after on the same machine, JDK, JVM flags and benchmark settings, with error bars.
  • Structural (only removes work, such as an allocation, a lookup or a re-encode): may claim the reduction without numbers, but not a speedup.

A claim without its matching evidence becomes an advisory finding. It covers only the unsupported part of the claim, it doesn't block the merge, and it has no severity rating. The author either adds the numbers or narrows the claim.

Motivation

The current rule asks for full controlled before/after time and allocation results for every claim. On #13481, a clear allocation win with credible gc.alloc.rate.norm numbers was flagged P1 by the Codex reviewer and flagged again by /perf-review. Its timing table really was too noisy to support its numbers (an unchanged control moved −18% to +39% between runs), but its allocation numbers were fine. The rule should catch the timing gap without treating the whole PR as unmeasured.

Additional Notes

  • gc.alloc.rate.norm barely varies between runs, so allocation claims don't need the forks and error bars that timing claims do.
  • SKILL.md points to AGENTS.md for this rule, so both are updated together to keep them consistent.

Contributor Checklist

Jira ticket: N/A

🤖 Generated with Claude Code

…I perf review)

Allocation claims need a gc-profiler before/after naming both SHAs;
timing claims need a controlled same-machine run with error bars; changes
that only remove work may claim the reduction without numbers. A claim
missing its evidence is an advisory finding scoped to the unsupported
part, not a merge blocker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added type: documentation tag: performance Performance related changes tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling tag: ai generated Largely based on code generated by an AI or LLM labels Nov 17, 2026
@pr-commenter

pr-commenter Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

LLM Validation

LLM Validation Gate — java-agent-instructions

✅ PASS

  • No blocking-case regressions; the quality change is within noise (baseline/candidate confidence intervals overlap).

6 case(s) · quality 80.0 → 81.9 (+1.9) · win-rate 50% [47%–53%]

Details — analysis and per-case results

Analysis

Changed instruction file(s): AGENTS.md.

No safety or blocking-case regressions across 6 case(s). Overall pairwise win-rate 50% [47%–53%], quality +1.9 — see the verdict above for whether that clears the noise band.

Results

  • Pairwise win-rate: 50% [47%–53%] — candidate's share of blind comparisons (90% CI; spanning 50% = no clear difference)
  • Overall quality: 80.0 → 81.9 (/100, +1.9)
  • Bad signals introduced (advisory): 0
  • Candidate criteria coverage (advisory): 24/27 (89%) — expected_criteria the candidate met; does not affect the gate
  • Blocking-case regressions: 0

Cases

Case Mode Quality Δ Win-rate (90% CI) Safety
java-core-validation-001 warn +5.8 58% [45%–72%] ok
java-unit-test-conventions-001 warn +1.7 50% [50%–50%] ok
java-premain-safety-001 warn 0.0 42% [28%–55%] ok
java-ci-download-guidance-001 warn +1.4 50% [50%–50%] ok
java-documentation-only-validation-001 warn +1.7 50% [50%–50%] ok
java-existing-spock-maintenance-001 warn +0.6 50% [50%–50%] ok

Per-dimension scores, token usage, latency, and estimated cost are in the CI job logs.

@dougqh
dougqh marked this pull request as ready for review November 17, 2026 15:38
@dougqh
dougqh requested a review from a team as a code owner November 17, 2026 15:38
@dougqh
dougqh requested review from cecile75 and removed request for a team November 17, 2026 15:38
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T15:40:46.389908Z 4b3db89 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@dd-octo-sts

dd-octo-sts Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.89 s 14.76 s [+0.1%; +1.8%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.64 s 13.77 s [-1.8%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent 17.26 s 17.24 s [-0.7%; +0.9%] (no difference)
startup:petclinic:iast:Agent 17.15 s 17.09 s [-0.5%; +1.2%] (no difference)
startup:petclinic:profiling:Agent 16.98 s 16.82 s [-0.1%; +2.0%] (no difference)
startup:petclinic:sca:Agent 17.28 s 17.17 s [-0.3%; +1.6%] (no difference)
startup:petclinic:tracing:Agent 16.34 s 16.31 s [-0.8%; +1.1%] (no difference)

Commit: 38ac6f2f · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Allocation evidence no longer has to support timing claims; speedups still require controlled comparisons with error bars, and unsupported claims are advisory in both guidance files.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 4b3db89

@bric3 bric3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we require matching JVM and workload settings for allocation comparisons? One run per revision is enough when both runs measure the same work. I'd keep the full rule in AGENTS.md so reviewers apply it consistently.

Also, the PR title should be : Set benchmark requirements for performance claims.

Comment thread AGENTS.md Resolved
Comment thread .agents/skills/perf-review/SKILL.md Resolved
dougqh and others added 2 commits November 20, 2026 09:37
…vidence

# Conflicts:
#	.agents/skills/perf-review/SKILL.md
Both sides of a before/after must use the same JDK, JVM flags, benchmark
settings and workload; one run per side is still enough for allocation.
The run-to-run-variance test for timing claims moves into AGENTS.md too, and
perf-review's posture.md now points there instead of restating the rule, so
reviewers following either document reach the same conclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh changed the title Match required benchmark evidence to the kind of performance claim (AI perf review) Set benchmark requirements for performance claims (AI perf review) Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: tooling Build & Tooling tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes tag: performance Performance related changes type: documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants