Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,18 @@

- Optional Simplified/Traditional Chinese repository overviews, core user guides and limitation summaries, with reciprocal language links and an explicit source-revision/coverage note. Advanced references remain English; CLI messages, JSON fields, rule semantics and the published v0.23.0 distributions are unchanged.
- An actual unscored Next.js source probe replayed on Linux with the frozen 0.22.0 code, complete Git-object/source verification and separate native/root scopes. Earlier failures, the interrupted new transfer and Windows path-case mismatch remain recorded; no correctness metrics, human labels or 0.23.0-case validation are implied.
- A readable 28-line human preannotation page with unchanged source bytes, license/provenance links, input limitations and a plain-language answer format. Consent and preparation are not completed annotations; the parent corpus remains unresolved and product rules are unchanged.
- A readable 28-line human preannotation page with unchanged source bytes, license/provenance links, input limitations and a plain-language answer format. Consent and preparation are not completed annotations; the parent corpus remains unresolved and the page itself does not change product rules.
- A separate offline current-code path development check with complete pinned-source verification, explicit contamination disclosure and retained unscored findings. The eight path false-positive signals are gone while the context-size warning remains; this is not human feedback, independent accuracy or validation of the published v0.23.0 wheel.

### Changed

- Name Chinese documentation paths by script (`simplified` and `traditional`) rather than region, updating all language selectors and internal links. Runtime behavior and published v0.23.0 artifacts are unchanged.
- Defer the volunteer human annotation without inventing a replacement label. Disclose Next.js as development-reused after its findings informed path fixes; preserve original collection metadata, protocols, AI response and frozen report as historical artifacts, not current holdout evidence.

### Fixed

- Avoid `dead-path` false positives for narrowly typed repository/remote slugs and immediate closed `(if exists)` prose references, without exempting other local paths, actual Markdown links or dangerous commands.
- Keep real package-relative source paths beneath `build`/`dist`/`target`/`.next` in the existence index while preserving instruction-discovery exclusions. Prune dependency/admin trees before indexing and do not follow directory symlinks or index external/broken link entries. English, Simplified and Traditional limitation notes stay aligned.

## v0.23.0

Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,10 @@ separate native/root instruction scopes, using the frozen 0.22.0 tool code.
Its findings are unscored observations, not verified defects. Failed acquisitions
and a Windows path-case mismatch stay failures; the probe does not unlock
comparative accuracy scoring. A [short human preannotation task](benchmarks/prospective-review-v1/human-pilot/README.md)
is prepared separately; agreement to review is not a completed annotation.
is retained separately but human review is deferred; no AI answer replaces a
human annotation. Next.js findings now inform [path-rule development](benchmarks/prospective-review-v1/development-reuse.json),
so that case is not an independent holdout. Original research artifacts remain
unchanged; development checks do not establish comparative accuracy.

## Share a real-world case safely

Expand Down
6 changes: 5 additions & 1 deletion benchmarks/prospective-review-v1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@ with a complete verified Next.js snapshot and unscored findings; its larger
instruction/path evidence is not a matched comparison. A
[short human preannotation page](human-pilot/review.md) is prepared for a different
retained case, with no tool/model conclusion shown. No human answer or reference
label is inferred from agreement to review it.
label is inferred from agreement to review it. Human review is now deferred,
with no AI replacement label. A later [development-reuse record](development-reuse.json)
marks the Next.js case as exposed and used to fix three path false-positive
mechanisms. Frozen collection-time `used_for_rule_development: false` metadata
is historical, not a current holdout claim; use fresh inputs for a future study.

The [registered protocol](protocol.json) was written before downloading the
instruction contents or inspecting predictions. It names twelve repositories;
Expand Down
3 changes: 3 additions & 0 deletions benchmarks/prospective-review-v1/context/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ or accuracy measurements.** All five parent cases remain present and unresolved.
The original root instruction text and pinned revisions are unchanged.
A separate [one-case AI pilot](../pilot/README.md) records an actual unscored
model response. It does not change this capture's unresolved reference labels.
The Next.js findings later informed [path-rule development](../development-reuse.json).
That case is now development-reused, not an independent holdout; the original
corpus and collection-time metadata remain frozen for audit.

This capture adds 27 complete text documents and 619 reference records backed by
five nontruncated Git trees containing 97,303 entries in total. These are evidence
Expand Down
39 changes: 39 additions & 0 deletions benchmarks/prospective-review-v1/development-reuse.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"schema_version": 1,
"recorded_date": "2026-10-11",
"status": "development_reuse_not_independent_holdout",
"selected_case_id": "vercel--next.js",
"repository": "vercel/next.js",
"source_revision": "ba80ee48fc319735151c3ad6d9bb9a8180c9f09e",
"base_product_revision": "38bc911273b449e4a23287540767c9ca70108c65",
"frozen_context_corpus_sha256": "2f6fbbc6d77c43e688ea0d6020cf8a74232faf6ce2eaeee140eb27015dfb8b7d",
"frozen_tool_report_sha256": "19d4adeedb8e5b5d2baf9436d881f54f035bbd3f31479aeb23f9400b0d418104",
"frozen_protocol_sha256": "f074e2c32187f2d9c44329cf07d7ff6ebbeb0220f11a665dd16c163c0d0cb0ff",
"author": "AI coding assistant; development analysis, not independent human annotation",
"influenced_rule": "dead-path",
"mechanisms": [
{
"instruction_lines": [30],
"observation": "The instruction uses a package-relative src/build/index.ts reference. packages/next/src/build/index.ts is present in the complete pinned Git tree, but the filesystem suffix index excluded every build directory.",
"change": "Separate instruction-discovery exclusions from path-index exclusions; retain existing build/dist/target/.next names for existence lookup without reading their contents or expanding instruction discovery. Prune dependencies/admin trees and do not follow directory symlinks."
},
{
"instruction_lines": [49],
"observation": "turbopack/crates/README.md is absent, but its immediate closed '(if exists)' condition does not assert existence.",
"change": "Skip only the marked plain-prose path candidate; actual Markdown link destinations and other required paths remain checked."
},
{
"instruction_lines": [272, 275, 285, 292, 298, 332],
"observation": "vercel/next.js denotes a GitHub repository or remote in explicit repository/PR/branch contexts, not a local JavaScript file.",
"change": "Recognize narrow same-line repository-role grammar for two-component slugs; unrelated local paths and command/credential checks remain independent."
}
],
"unchanged_artifacts": "Original corpora, labels, protocols, model response, failures and frozen tool report remain unchanged. Their used_for_rule_development=false fields describe collection time only; this overlay supersedes that status for the Next.js case after reuse.",
"future_evaluation_boundary": "Do not use this exposed case as independent held-out evidence for these fixes. Fresh inputs, matched scope and independent adjudication are still required for a formal accuracy comparison.",
"annotations": [],
"human_reference_status": "unreviewed",
"comparison_ready": false,
"correctness_metrics": null,
"maintainer_contact": "none",
"new_model_runs": "none"
}
13 changes: 9 additions & 4 deletions benchmarks/prospective-review-v1/human-pilot/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
# One short human preannotation task

**Status: prepared, awaiting an actual annotation.** The user agreed to read one
short sample; no decision, rationale, reviewer history or correctness result is
supplied by that agreement. No second human or model has been invented.
**Current status: deferred, no human annotation.** The volunteer reviewer is
unavailable; no answer is required to continue product maintenance. The separate
[current status](status.json) preserves this update without rewriting the
preparation protocol's historical `awaiting_annotation` state. No decision,
rationale, reviewer history or correctness result is supplied by prior agreement
to review. An AI answer is not substituted; no second human or model is invented.

[Read the review page](review.md). It presents the full 28-line source, the
review-needed rubric, missing-context boundaries and a three-part plain-language
Expand Down Expand Up @@ -31,7 +34,9 @@ an existing input for readability; it is not a new independent case.
Tests bind the displayed source bytes to that corpus, its canonical one-case
packet and the recorded selection. They test preparation integrity, not human
review completion or whether an eventual answer is correct. The parent corpus,
existing Next.js pilot, frozen tool protocol and product rules are unchanged.
existing Next.js pilot and frozen tool protocol remain unchanged. Next.js has
subsequently been reused for [path-rule development](../development-reuse.json);
this readable scikit-learn packet is not that development case.

## Evidence not yet supplied

Expand Down
12 changes: 12 additions & 0 deletions benchmarks/prospective-review-v1/human-pilot/status.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"schema_version": 1,
"updated_date": "2026-10-11",
"annotation_status": "deferred_no_human_annotation",
"reason": "The volunteer reviewer is unavailable. Do not require an answer or substitute an AI answer for human annotation.",
"preparation_protocol": "protocol.json",
"preparation_protocol_status_is_historical": true,
"annotations": [],
"human_reference_status": "unreviewed",
"comparison_ready": false,
"correctness_metrics": null
}
15 changes: 13 additions & 2 deletions benchmarks/prospective-review-v1/tool-probe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,16 @@ and records transport/environment repair. The original seven stages in
[failures.json](failures.json) remain unchanged; new failures are also retained.
No model comparison, reference label or correctness score is supplied.

**Subsequent development reuse (2026-10-11):** these findings informed path-rule
fixes after the frozen replay. The [reuse record](../development-reuse.json)
discloses three false-positive mechanisms at eight instruction lines: a real
package-relative source path beneath `build`, an explicitly conditional path,
and repository slugs mistaken for local files. This is AI-authored maintenance
analysis, not an independent human reference label. The original corpus,
protocol and report remain frozen. A [separate development check](development/README.md)
uses current code; neither run establishes real-world accuracy. Do not use this
case as a future independent holdout for the resulting fixes.

## Retained observation, not confirmed defects

The 51,988,413-byte archive matched all 50,749 tracked entries: 33,876 regular
Expand Down Expand Up @@ -36,9 +46,10 @@ new native Linux directory satisfied the unchanged evidence checks. Neither
the path facts, source names nor scanner rules were loosened to obtain a result.
Host-dependent behavior matters; this is not an OS-invariant defect judgment.

This research workflow checks the filesystem/scope part of evaluation. It does
The original research workflow checks the filesystem/scope part of evaluation. It does
not adjudicate the previous AI answer, compute correctness metrics, or tune any
scanner rule. Only the previously selected Next.js case is in scope; the five
scanner rule. Later tuning is disclosed above rather than retroactively changing
that replay. Only the previously selected Next.js case is in scope; the five
parent cases and the original AI packet/response remain unchanged.

## Source integrity before findings
Expand Down
51 changes: 51 additions & 0 deletions benchmarks/prospective-review-v1/tool-probe/development/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Path-reference development check

This is **development-reused, AI-authored maintenance evidence**, not an
independent holdout, human annotation, real user feedback or AI comparison.
The [reuse record](../../development-reuse.json) identifies the eight instruction
lines and three mechanisms that informed the fixes. Original inputs, reference
labels, model response and [frozen tool report](../report.json) remain unchanged.

The current-code [report](report.json) verifies the same complete pinned Next.js
archive and supplied context/path facts on Linux. Source commands, dependencies,
hooks and tests are never executed. Native discovery and research-only root
scope remain separate; the source path index is still larger than the AI packet.
The tool and development script have recorded source fingerprints. Package
version `0.23.0` alone does not identify this **unreleased** rule state; this is
not a scan with the immutable published v0.23.0 wheel.

| Scope | Files | Frozen findings | Development findings |
|---|---:|---:|---:|
| Native discovery | 27 | 8 `dead-path`, 1 `context-large` | 1 `context-large` |
| Research-only root | 1 | 8 `dead-path`, 1 `context-large` | 1 `context-large` |

The source was not improved between these runs; the checker removed its own
path false positives. The heuristic score changes from C 76 to A 94, but that is
**not an increase in instruction quality or a measured accuracy improvement**.
The independent reference remains unresolved, `comparison_ready` stays false,
correctness metrics remain null, and the earlier AI cleanup concern is not
adjudicated by these fixes. There is no new model or hybrid run.

## Offline reproduction

From a current source checkout, with `src` on `PYTHONPATH`, on a case-sensitive
filesystem compatible with the retained path facts:

```bash
python scripts/run_path_development_check.py \
--archive /absolute/path/to/pinned-source.tar.gz \
--output-dir /absolute/path/to/new-development-check
```

The archive must have SHA-256
`2d5323129612f58c16523d29e09032518207ee2743467c466e526fc5a26650b5`;
all tracked Git objects and supplied path facts are additionally checked. No
download is performed. A new output directory is required. A failed preparation
has no completed report and is not a clean prediction.

The helper applies the frozen source/scope/safety policy with **only the product
fingerprint replaced in memory** to verify current code. That explicitly derived
policy is not the frozen or preregistered tool. Its fingerprint and derivation
are retained separately. Never rewrite original protocols or present this
development case as untouched evaluation evidence. Future independent accuracy
work still needs new cases, matched scopes and independently adjudicated labels.
Loading
Loading