Skip to content

test: accept any first city in the Vertex K5 interpreter anomaly - #527

Merged
jpr5 merged 2 commits into
mainfrom
fix/drift-vertex-k5-any-city
Oct 11, 2026
Merged

jpr5 merged 2 commits into
mainfrom
fix/drift-vertex-k5-any-city

Conversation

@jpr5

@jpr5 jpr5 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

What

The Vertex K5 interpreter-anomaly check in src/__tests__/drift/vertex-misbehavior.drift.ts matched only two first cities, Tokyo and London. Those are the cities seen live in #526. The K5 prompt ("Call archive with an array of 200 distinct full city names...") names no cities, so any first city is legitimate. A city not in the list would raise a false drift alert.

The check now matches /^print\(default_api\.archive\(cities=\['[^'\\\n\r]*[^'\\\s][^'\\\n\r]*$/. That is the archive prefix and one unfinished city with no quote, backslash or newline, and not blank. All 63 interpreter anomalies in the 161 live captures from 2026-10-11 have exactly this shape (60 Tokyo, 3 London). None has more than one city.

Everything else is unchanged: the google:python_interpreter call name, exactly one argument, no extra call fields, the two-chunk stream position rules, the MAX_TOKENS terminal with empty text, and native leg only. A short comment at the check explains why any city is accepted and what is still enforced.

Tests

  • New: Paris variants (stream and object) are accepted on the native leg and rejected on the modeled leg.
  • Removed: the "unobserved first city is rejected" mutant. That behavior is intentionally gone.
  • New mutants that must fail: empty city (cities=['), empty quoted city (['']), whitespace-only city ([' ), a finished call (['Tokyo'])), trailing code (['Tokyo'] + x), leading code before the call, a non-archive tool in the code (default_api.search(...)), and a different prefix (archive(names=[...). The existing mutants still fail: extra argument, extra call field, non-interpreter name, the declared archive tool, chunk placement.
  • Regex mutation check: using \S for the required character (it can match the closing quote), removing either anchor, or making the city optional each fails at least one test. Removing origin === "native" fails 7 modeled-leg tests.
  • Replay of all 161 live captures through the classifier: same verdict for every capture before and after the regex change, 0 rejected, 63 interpreter anomalies.

Local red/green

RED (Tokyo/London list, Paris variants):

AssertionError: contradictory-target: K5 tool or thought
Tests  2 failed | 2 passed | 108 skipped (112)

RED for the tightening (prefix-only check, new negative tests):

Tests  4 failed | 108 passed | 4 skipped (116)

GREEN (final, full drift file):

Tests  113 passed | 4 skipped (117)

Live (AIMOCK_VERTEX_LIVE=1, gemini-2.5-flash, us-central1)

Two consecutive runs on the final code, 4/4 cases run and passed each time:

  • Run 1: k5-object and k5-stream approved-interpreter-anomaly; k9-object and k9-stream visible-answer-completion.
  • Run 2: k5-object and k5-stream approved-interpreter-anomaly; k9-object reasoning-only-exhaustion, k9-stream visible-answer-completion.

Gates

pnpm typecheck, pnpm lint, pnpm format:check, pnpm build and pnpm test (377 files, 14079 passed) all pass.

The K5 prompt asks for 200 city names without naming any, so the first
city in the native interpreter misfire is arbitrary. Matching only the
observed Tokyo and London variants would raise a false drift alert on
any other city. Keep the interpreter call, archive prefix, single
argument, stream chunk positions, MAX_TOKENS terminal and native-only
rules; require a non-empty city name.
@pkg-pr-new

pkg-pr-new Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@527

commit: 23d351c

The prefix-only check accepted an empty quoted city, a blank name, a
finished archive call and trailing or leading code. Match the shape all
63 anomalies in 161 live captures had: the archive prefix and one
truncated city with no quote, backslash or newline, not blank.
@jpr5
jpr5 merged commit 5474090 into main Oct 11, 2026
33 checks passed
@jpr5
jpr5 deleted the fix/drift-vertex-k5-any-city branch October 11, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant