Skip to content

test: accept native Vertex malformed-function terminal with empty content - #524

Merged
jpr5 merged 2 commits into
mainfrom
fix/drift-vertex-k5-malformed-call
Oct 11, 2026
Merged

jpr5 merged 2 commits into
mainfrom
fix/drift-vertex-k5-malformed-call

Conversation

@jpr5

@jpr5 jpr5 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The Drift Tests run 38104122278 on main @ b9cff07 failed the live Vertex case k5-object (expected [ 'TARGET_COMPARED', 'NOT_TRIGGERED' ] to include 'FAILURE'). Because of that, k5-stream, k9-object and k9-stream were not attempted (earlier-case-failed).

Vertex returned HTTP 200 from gemini-2.5-flash:

"candidates":[{"content":{},"finishReason":"MALFORMED_FUNCTION_CALL","finishMessage":"Malformed function call: print(default_api.archive(cities=['Tokyo"}]

The comparator already approves a MALFORMED_FUNCTION_CALL terminal as NOT_TRIGGERED (reason malformed-function-terminal). The retained k5-stream capture passes this way. The difference is the shape: in object mode Vertex sent an empty content: {} object, while the stream capture has no content key. Any content present had to have role: "model" plus a parts array, so the check threw invalid content. This is a deterministic shape-handling bug in the test comparator, not a new upstream behavior class.

Change (test-only; no src/ change)

src/__tests__/drift/vertex-misbehavior.drift.ts:

  • On the native leg only, when the candidate's finishReason is MALFORMED_FUNCTION_CALL, an empty content object (no keys) is treated the same as absent content.
  • This covers object and stream mode on purpose: an empty content means no content in either mode.
  • Any other content still needs role: "model" plus a parts array. The modeled (aimock) leg stays strict.
  • The CI capture is retained byte-exact as a fixture with its sha256 (4c38ba91…477eb), with the same request hash as the existing k5 captures.
  • withVertexSequence takes an optional list of served captures, so the recurrence test can serve the CI capture first.

Local red/green

Real surface: the captured raw bytes are replayed through the actual @google/genai SDK over an owned localhost HTTP server, then through compareVertexObservation. Separately, the CI capture runs through runVertexRecurrence, the live path that failed in run 38104122278, against a localhost origin.

RED (new tests, with the comparator's content check restored to the b9cff07 line):

× ... classifies the captured empty-content object as an approved non-trigger      AssertionError: invalid content
× ... rejects the captured malformed-function terminal without a finish message    (got 'invalid content' before reaching the terminal check)
× ... accepts empty content with a malformed-function terminal in stream mode      AssertionError: invalid content
× Vertex recurrence continues past the captured empty-content malformed-function terminal
    → expected [ … ] to have a length of 4 but got 1   (k5-object FAILURE; later cases earlier-case-failed)
Tests  4 failed | 6 passed | 72 skipped (82)

GREEN (after fix):

✓ Vertex retained native dispositions > 'k5-object' | 'k5-stream' | 'k9-object' | 'k9-stream' preserve genuine wire bytes and approved non-trigger
✓ ... classifies the captured empty-content object as an approved non-trigger     (NOT_TRIGGERED / malformed-function-terminal)
✓ ... accepts empty content with a malformed-function terminal in stream mode      (NOT_TRIGGERED / malformed-function-terminal)
✓ ... rejects the same empty content on the modeled leg
✓ ... rejects malformed-function content with 'non-model role'   ({role:"user"})
✓ ... rejects malformed-function content with 'non-array parts'  ({role:"model", parts:"x"})
✓ ... rejects malformed-function content with 'missing parts'    ({role:"model"})
✓ ... rejects empty content with a STOP terminal
✓ ... rejects empty content with a MAX_TOKENS terminal
✓ ... rejects the captured malformed-function terminal without a finish message   (invalid malformed-function terminal)
✓ Vertex recurrence continues past the captured empty-content malformed-function terminal
    (4 requests; all four NOT_TRIGGERED; no earlier-case-failed)
Whole file: Tests  78 passed | 4 skipped (82)   (the 4 skipped are the live-gated cases)

Mutation check: I replaced each guard clause with true, one at a time, and each mutant was caught:

  • dropping origin === "native" fails the modeled-leg test
  • dropping the MALFORMED_FUNCTION_CALL check fails the STOP and MAX_TOKENS tests
  • dropping the empty-object check fails the role and parts tests
  • dropping the existing Array.isArray(content.parts) check fails the 'non-array parts' and 'missing parts' tests

Live verification: not run locally

I could not run Vertex live locally. CI gets its token through GitHub workload identity for the main schedule/dispatch runs only. The local gcloud credentials need interactive re-authentication, and no Vertex credential is in the local secret store. The live confirmation is a manual Drift Tests run on main after merge.

Gates

pnpm typecheck, pnpm lint, pnpm format:check, pnpm build: pass. pnpm test: 377 files / 14073 passed, 3 skipped.

…tent

In object mode Vertex can serve finishReason MALFORMED_FUNCTION_CALL with
an empty content object ({}) instead of omitting content. The comparator
already approves that terminal as NOT_TRIGGERED when content is absent, but
rejected the empty object as invalid content, failing k5-object and leaving
the remaining live cases unattempted.

On the native leg only, an empty content object with a MALFORMED_FUNCTION_CALL
terminal is now treated as absent content. Any other content still needs role
model plus a parts array, and the modeled leg stays strict. The CI capture is
retained byte-exact with its sha256.
@pkg-pr-new

pkg-pr-new Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@524

commit: 1dab38b

Make the parts mutants reach the parts check (role model with non-array or
missing parts), pin stream-mode acceptance of empty content with a
malformed-function terminal, reject the capture without a finish message,
and run the CI capture through runVertexRecurrence so the later cases are
proven to still run.
@jpr5

jpr5 commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

Merging with admin override over drift-live-pr. Its only failure is cohere.drift.ts:688 AssertionError: expected object, which also fails in the job's base-main leg, so it happens on main without this PR. This PR changes only vertex-misbehavior.drift.ts, and Vertex doesn't run live on PR legs. All other checks pass. The live Vertex confirmation is the post-merge Drift Tests run on main.

@jpr5
jpr5 merged commit 9420afa into main Oct 11, 2026
35 of 37 checks passed
@jpr5
jpr5 deleted the fix/drift-vertex-k5-malformed-call branch October 11, 2026 17:08
jpr5 added a commit that referenced this pull request Oct 11, 2026
…e leg (#525)

## Problem

Two Cohere live-drift failures on `main` (9420afa). They have the same
root cause:

- **`cohere.drift.ts:688` `expected object`** in `cohereRecord` ←
`cohereUsage` ← `readCohereK5:884` (stream branch). Seen in both legs of
PR #524's drift-live-pr run 38157499382.
- **`cohere.drift.ts:324` `expected 422 to be 200`** in the
non-streaming shape test. Seen in post-merge Drift Tests run
38158387279.

Neither appears in any main drift report before 2026-10-11 ~17:00 UTC. I
scanned the artifacts of every main/PR drift run back to 37580386296.

## Diagnosis: upstream nondeterminism, not a contract change

Sometimes Cohere generates nothing. Live probes of the exact test
requests on 2026-10-11 (trial key; 429s excluded):

| Request | Outcome | Count |
|---|---|---|
| K5 object (`tool_choice: REQUIRED`, 128 tok) | 200 `MAX_TOKENS`,
normal usage | 15 |
| K5 object | **422**
`{"error_type":"NO_TOOL_CALL_OR_RESPONSE_GENERATED","id":…,"message":…}`
| 5 |
| K5 stream | 200 `TOOL_CALL`, normal usage | 29 |
| K5 stream | **200 SSE, `message-end` =
`{"finish_reason":"ERROR","usage":{},"error":"No tool calls or response
was generated…","error_type":"NO_TOOL_CALL_OR_RESPONSE_GENERATED"}`** |
8 |
| Shape leg `Say hello` (object) | 200 | 13 |
| Shape leg `Say hello` (object) | **422**
`{"error_type":"NO_VALID_RESPONSE_GENERATED",…}` | 2 |
| Shape leg `Say hello` (stream) | 200 | 12 |
| Shape leg `Say hello` (stream) | **422** `NO_VALID_RESPONSE_GENERATED`
| 3 |

The `usage: {}` on the stream ERROR terminal is exactly what fails at
line 688: `usage.tokens` is undefined. The 422 is what fails at line
324. Successful responses kept the same shape throughout, so aimock's
Cohere builder has nothing to follow. No `src/` code changes; only test
files.

## Fix (live leg only; aimock and replay legs stay strict)

- `cohere-model.ts`: `cohereNoGenerationError(status, body, allowed)`
matches only a 422 whose keys are exactly `{error_type,id,message}`
(non-empty strings) and whose code is in `allowed`. It returns
`{errorType, id}`. The codes are per request: the no-tools shape legs
(`COHERE_SHAPE_LEG_NO_GENERATION`) accept only
`NO_VALID_RESPONSE_GENERATED`, and the K5 tool request
(`COHERE_K5_NO_GENERATION`) accepts only
`NO_TOOL_CALL_OR_RESPONSE_GENERATED`.
`isCohereNoGenerationStreamEnd(delta)` matches only the exact observed
ERROR terminal.
- K5: `readCohereK5`/`compareCohereK5`/`runCohereK5Modes` take a `{ live
}` option. Only the live test sets it. The exact variants become
`UPSTREAM_NO_GENERATION`: a 422 on **either** send, or the stream ERROR
terminal. The stream variant must not have emitted any tool or content
first, and the SDK error must be a `CohereError` with status 422.
Anything else rethrows. A refused 422 rethrows with `cause: { sdk,
classifier }` so the classifier's reason is kept.
- **A K5 run where every send is `UPSTREAM_NO_GENERATION` is a skip, not
a pass.** Its reason cites each upstream code and id. One compared send
is still a pass.
- The shape legs (non-streaming and streaming) skip on the exact 422
envelope. Every `ctx.skip` carries a reason with the upstream code or
status and the response id.
- Genuine live captures are kept as hashed fixtures. The 422 on the K5
stream send is a constructed replay (not yet observed live) and is
marked as such.

## Local red/green

**RED: unmodified main @9420afa1, same filtered live command, 8 runs:**
run 5 reproduced the CI stack trace exactly:
```
× P1 Cohere native K5 object modeled and stream observed comparison
AssertionError: expected object
 ❯ cohereRecord cohere.drift.ts:688:3
 ❯ cohereUsage cohere.drift.ts:698:19
 ❯ readCohereK5 cohere.drift.ts:884:13
```
Replay RED: the new strict-leg test asserts that the unmodified reader
throws `expected object` on the captured stream and `HTTP failure` on
the captured 422.

**GREEN: this branch, same command, 8 consecutive live runs, all
passed.** They included real occurrences of the variants:
```
run 2: [object UPSTREAM_NO_GENERATION, stream UPSTREAM_NO_GENERATION]  ✓
run 5: [object UPSTREAM_NO_GENERATION, stream TARGET]                  ✓
run 6: [object UPSTREAM_NO_GENERATION, stream TARGET]                  ✓
run 7: [object NOT_TRIGGERED, stream UPSTREAM_NO_GENERATION]           ✓
runs 1,3,4,8: [object NOT_TRIGGERED, stream TARGET]                    ✓
```

**Mutants, round 1 (5/5 killed):** removing the live gate on stream,
removing the live gate on object, making the stream-end classifier
always true, making the 422 classifier accept any error type, and
dropping the tool-before-terminal check.

**Mutants, round 2 after review (12/12 killed):** all-no-generation skip
disabled · 422 accepted on the object send only · shape leg allows the
tool code · `allowed` filter dropped · bare rethrow instead of
rethrow-with-cause · `contentCount` guard deleted · `typeof` deleted on
id, message and stream `error` · `null`-body guard deleted · 422
`complete` guard deleted · 422 content-type guard deleted.

**Live after review, 6 consecutive runs, 0 failures.** Run 1 showed the
new skip working live:
```
↓ P1 Cohere native K5 … [Cohere upstream generated nothing (NO_TOOL_CALL_OR_RESPONSE_GENERATED id=53e39cc7-…; NO_TOOL_CALL_OR_RESPONSE_GENERATED id=88013390-…); no success envelope to compare]
↓ Cohere drift > non-streaming /v2/chat shape matches [Cohere upstream generated nothing (NO_VALID_RESPONSE_GENERATED id=7d6ebc99-…); …]
```
Runs 2–6: K5 passed every time (3× stream TARGET, 2× stream
UPSTREAM_NO_GENERATION with the object compared). Two shape-leg sends
were skipped with reasons.

**Gates (after review):** `pnpm typecheck` ✓, `pnpm lint` ✓, `pnpm
format:check` ✓, `pnpm build` ✓, `pnpm test` ✓ (377 files, 14079
passed). Cohere drift file offline: 23 passed / 3 skipped.

## Tradeoff

When the stream send hits no-generation (about 22% of probes), that run
does not observe K5 `TARGET`. If the object send is compared, the run
passes. If both sends generate nothing, the run is reported as skipped.
This follows the precedent approved today for genuine upstream
nondeterminism: accept the exact variants on the live leg rather than
retry.

## Backlog (out of scope)

- `parseDataOnlySSE` drops `event:` frames (pre-existing).
- The K5 422 path has no body-size check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant