Repository navigation
test: accept native Vertex malformed-function terminal with empty content - #524
Merged
Merged
Conversation
…tent
In object mode Vertex can serve finishReason MALFORMED_FUNCTION_CALL with
an empty content object ({}) instead of omitting content. The comparator
already approves that terminal as NOT_TRIGGERED when content is absent, but
rejected the empty object as invalid content, failing k5-object and leaving
the remaining live cases unattempted.
On the native leg only, an empty content object with a MALFORMED_FUNCTION_CALL
terminal is now treated as absent content. Any other content still needs role
model plus a parts array, and the modeled leg stays strict. The CI capture is
retained byte-exact with its sha256.
commit: |
Make the parts mutants reach the parts check (role model with non-array or missing parts), pin stream-mode acceptance of empty content with a malformed-function terminal, reject the capture without a finish message, and run the CI capture through runVertexRecurrence so the later cases are proven to still run.
Contributor
Author
|
Merging with admin override over |
jpr5
added a commit
that referenced
this pull request
Oct 11, 2026
…e leg (#525) ## Problem Two Cohere live-drift failures on `main` (9420afa). They have the same root cause: - **`cohere.drift.ts:688` `expected object`** in `cohereRecord` ← `cohereUsage` ← `readCohereK5:884` (stream branch). Seen in both legs of PR #524's drift-live-pr run 38157499382. - **`cohere.drift.ts:324` `expected 422 to be 200`** in the non-streaming shape test. Seen in post-merge Drift Tests run 38158387279. Neither appears in any main drift report before 2026-10-11 ~17:00 UTC. I scanned the artifacts of every main/PR drift run back to 37580386296. ## Diagnosis: upstream nondeterminism, not a contract change Sometimes Cohere generates nothing. Live probes of the exact test requests on 2026-10-11 (trial key; 429s excluded): | Request | Outcome | Count | |---|---|---| | K5 object (`tool_choice: REQUIRED`, 128 tok) | 200 `MAX_TOKENS`, normal usage | 15 | | K5 object | **422** `{"error_type":"NO_TOOL_CALL_OR_RESPONSE_GENERATED","id":…,"message":…}` | 5 | | K5 stream | 200 `TOOL_CALL`, normal usage | 29 | | K5 stream | **200 SSE, `message-end` = `{"finish_reason":"ERROR","usage":{},"error":"No tool calls or response was generated…","error_type":"NO_TOOL_CALL_OR_RESPONSE_GENERATED"}`** | 8 | | Shape leg `Say hello` (object) | 200 | 13 | | Shape leg `Say hello` (object) | **422** `{"error_type":"NO_VALID_RESPONSE_GENERATED",…}` | 2 | | Shape leg `Say hello` (stream) | 200 | 12 | | Shape leg `Say hello` (stream) | **422** `NO_VALID_RESPONSE_GENERATED` | 3 | The `usage: {}` on the stream ERROR terminal is exactly what fails at line 688: `usage.tokens` is undefined. The 422 is what fails at line 324. Successful responses kept the same shape throughout, so aimock's Cohere builder has nothing to follow. No `src/` code changes; only test files. ## Fix (live leg only; aimock and replay legs stay strict) - `cohere-model.ts`: `cohereNoGenerationError(status, body, allowed)` matches only a 422 whose keys are exactly `{error_type,id,message}` (non-empty strings) and whose code is in `allowed`. It returns `{errorType, id}`. The codes are per request: the no-tools shape legs (`COHERE_SHAPE_LEG_NO_GENERATION`) accept only `NO_VALID_RESPONSE_GENERATED`, and the K5 tool request (`COHERE_K5_NO_GENERATION`) accepts only `NO_TOOL_CALL_OR_RESPONSE_GENERATED`. `isCohereNoGenerationStreamEnd(delta)` matches only the exact observed ERROR terminal. - K5: `readCohereK5`/`compareCohereK5`/`runCohereK5Modes` take a `{ live }` option. Only the live test sets it. The exact variants become `UPSTREAM_NO_GENERATION`: a 422 on **either** send, or the stream ERROR terminal. The stream variant must not have emitted any tool or content first, and the SDK error must be a `CohereError` with status 422. Anything else rethrows. A refused 422 rethrows with `cause: { sdk, classifier }` so the classifier's reason is kept. - **A K5 run where every send is `UPSTREAM_NO_GENERATION` is a skip, not a pass.** Its reason cites each upstream code and id. One compared send is still a pass. - The shape legs (non-streaming and streaming) skip on the exact 422 envelope. Every `ctx.skip` carries a reason with the upstream code or status and the response id. - Genuine live captures are kept as hashed fixtures. The 422 on the K5 stream send is a constructed replay (not yet observed live) and is marked as such. ## Local red/green **RED: unmodified main @9420afa1, same filtered live command, 8 runs:** run 5 reproduced the CI stack trace exactly: ``` × P1 Cohere native K5 object modeled and stream observed comparison AssertionError: expected object ❯ cohereRecord cohere.drift.ts:688:3 ❯ cohereUsage cohere.drift.ts:698:19 ❯ readCohereK5 cohere.drift.ts:884:13 ``` Replay RED: the new strict-leg test asserts that the unmodified reader throws `expected object` on the captured stream and `HTTP failure` on the captured 422. **GREEN: this branch, same command, 8 consecutive live runs, all passed.** They included real occurrences of the variants: ``` run 2: [object UPSTREAM_NO_GENERATION, stream UPSTREAM_NO_GENERATION] ✓ run 5: [object UPSTREAM_NO_GENERATION, stream TARGET] ✓ run 6: [object UPSTREAM_NO_GENERATION, stream TARGET] ✓ run 7: [object NOT_TRIGGERED, stream UPSTREAM_NO_GENERATION] ✓ runs 1,3,4,8: [object NOT_TRIGGERED, stream TARGET] ✓ ``` **Mutants, round 1 (5/5 killed):** removing the live gate on stream, removing the live gate on object, making the stream-end classifier always true, making the 422 classifier accept any error type, and dropping the tool-before-terminal check. **Mutants, round 2 after review (12/12 killed):** all-no-generation skip disabled · 422 accepted on the object send only · shape leg allows the tool code · `allowed` filter dropped · bare rethrow instead of rethrow-with-cause · `contentCount` guard deleted · `typeof` deleted on id, message and stream `error` · `null`-body guard deleted · 422 `complete` guard deleted · 422 content-type guard deleted. **Live after review, 6 consecutive runs, 0 failures.** Run 1 showed the new skip working live: ``` ↓ P1 Cohere native K5 … [Cohere upstream generated nothing (NO_TOOL_CALL_OR_RESPONSE_GENERATED id=53e39cc7-…; NO_TOOL_CALL_OR_RESPONSE_GENERATED id=88013390-…); no success envelope to compare] ↓ Cohere drift > non-streaming /v2/chat shape matches [Cohere upstream generated nothing (NO_VALID_RESPONSE_GENERATED id=7d6ebc99-…); …] ``` Runs 2–6: K5 passed every time (3× stream TARGET, 2× stream UPSTREAM_NO_GENERATION with the object compared). Two shape-leg sends were skipped with reasons. **Gates (after review):** `pnpm typecheck` ✓, `pnpm lint` ✓, `pnpm format:check` ✓, `pnpm build` ✓, `pnpm test` ✓ (377 files, 14079 passed). Cohere drift file offline: 23 passed / 3 skipped. ## Tradeoff When the stream send hits no-generation (about 22% of probes), that run does not observe K5 `TARGET`. If the object send is compared, the run passes. If both sends generate nothing, the run is reported as skipped. This follows the precedent approved today for genuine upstream nondeterminism: accept the exact variants on the live leg rather than retry. ## Backlog (out of scope) - `parseDataOnlySSE` drops `event:` frames (pre-existing). - The K5 422 path has no body-size check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Drift Tests run 38104122278 on
main@ b9cff07 failed the live Vertex casek5-object(expected [ 'TARGET_COMPARED', 'NOT_TRIGGERED' ] to include 'FAILURE'). Because of that,k5-stream,k9-objectandk9-streamwere not attempted (earlier-case-failed).Vertex returned HTTP 200 from
gemini-2.5-flash:The comparator already approves a
MALFORMED_FUNCTION_CALLterminal asNOT_TRIGGERED(reasonmalformed-function-terminal). The retainedk5-streamcapture passes this way. The difference is the shape: in object mode Vertex sent an emptycontent: {}object, while the stream capture has nocontentkey. Anycontentpresent had to haverole: "model"plus apartsarray, so the check threwinvalid content. This is a deterministic shape-handling bug in the test comparator, not a new upstream behavior class.Change (test-only; no
src/change)src/__tests__/drift/vertex-misbehavior.drift.ts:finishReasonisMALFORMED_FUNCTION_CALL, an emptycontentobject (no keys) is treated the same as absent content.contentmeans no content in either mode.role: "model"plus apartsarray. The modeled (aimock) leg stays strict.4c38ba91…477eb), with the same request hash as the existing k5 captures.withVertexSequencetakes an optional list of served captures, so the recurrence test can serve the CI capture first.Local red/green
Real surface: the captured raw bytes are replayed through the actual
@google/genaiSDK over an owned localhost HTTP server, then throughcompareVertexObservation. Separately, the CI capture runs throughrunVertexRecurrence, the live path that failed in run 38104122278, against a localhost origin.RED (new tests, with the comparator's content check restored to the b9cff07 line):
GREEN (after fix):
Mutation check: I replaced each guard clause with
true, one at a time, and each mutant was caught:origin === "native"fails the modeled-leg testMALFORMED_FUNCTION_CALLcheck fails the STOP and MAX_TOKENS testsArray.isArray(content.parts)check fails the 'non-array parts' and 'missing parts' testsLive verification: not run locally
I could not run Vertex live locally. CI gets its token through GitHub workload identity for the
mainschedule/dispatch runs only. The local gcloud credentials need interactive re-authentication, and no Vertex credential is in the local secret store. The live confirmation is a manual Drift Tests run onmainafter merge.Gates
pnpm typecheck,pnpm lint,pnpm format:check,pnpm build: pass.pnpm test: 377 files / 14073 passed, 3 skipped.