Repository navigation
test(drift): accept Cohere upstream no-generation outcomes on the live leg - #525
Merged
Merged
Conversation
…e leg
Cohere intermittently generates nothing for both the K5 tool request and
the plain shape-leg request. Live probes on 2026-10-11 saw it as a 422
{error_type,id,message} envelope (NO_TOOL_CALL_OR_RESPONSE_GENERATED or
NO_VALID_RESPONSE_GENERATED), or mid-stream as a 200 SSE whose message-end
is finish_reason ERROR with usage {}. The empty usage is what failed
cohereUsage at line 688.
The live K5 leg now classifies those exact shapes as UPSTREAM_NO_GENERATION,
and the shape legs skip on the exact 422 envelope. The aimock and replay
legs stay strict. Genuine captures are kept as fixtures with replay,
strict-rejection and mutant tests.
commit: |
…cceptance - A K5 live run where every send generated nothing is now a skip that cites each upstream code and id, not a pass. - Accept the 422 no-generation envelope on the K5 stream send too. - Restrict the codes per request: the no-tools shape legs accept only NO_VALID_RESPONSE_GENERATED, the K5 tool request only NO_TOOL_CALL_OR_RESPONSE_GENERATED. - Every skip carries a reason with the upstream code or status and id. A refused 422 keeps both the SDK error and the classifier reason as cause. - Pin the content-before-terminal, typeof, null-body, 422 completeness and content-type guards with tests.
This was referenced Oct 11, 2026
jpr5
added a commit
that referenced
this pull request
Oct 11, 2026
…528) ## Problem The streaming `/v2/chat` shape leg in `cohere.drift.ts` never compared anything. It parsed both bodies with `parseDataOnlySSE`, which keeps only frames that *start* with `data: `. Every Cohere v2 frame is `event: <type>` + `data: <json>`, so both lists came out empty, the `if (realChunks.length > 0 && mockChunks.length > 0)` guard was false, and the test always passed. (A non-200 mock reply was also let through without a comparison.) ## Fix (test-only; no `src/` behavior change) - `parseCohereSSE`: strict parser. Each frame must be exactly `event:` + `data:` with `data.type` equal to the event name. A trailing `data: [DONE]` is dropped. Any other frame throws instead of being dropped. - The leg now requires mock status 200 and non-empty event lists on both sides, opening with `message-start` and closing with `message-end`. It compares every event type with `compareSSESequences` against SDK shapes for message-start / content-start / content-delta / content-end / message-end (the same pattern the Anthropic stream leg uses). - Live no-generation: the 422 envelope skip is unchanged. A 200 stream whose `message-end` is the no-generation terminal (`finish_reason: "ERROR"`, empty usage) is now a skip that gives the upstream code and id (`NO_VALID_RESPONSE_GENERATED id=...`), the same as #525. `isCohereNoGenerationStreamEnd` now takes the allowed error types, like `cohereNoGenerationError`. K5 passes `COHERE_K5_NO_GENERATION`, so K5 behavior is unchanged. - Offline tests: the parser on the recorded live K5 stream fixture, on aimock's real stream, and rejection of malformed frames. Classifier test for the `allowed` param. ## Local RED (current main, live Cohere) Temporary probe in the leg: ``` RED-PROBE realChunks=0 mockChunks=0 realBytes=1607 mockBytes=709 ✓ Cohere drift > streaming /v2/chat shape matches ``` Mutant in `src/cohere.ts` (text stream `message-end`: `finish_reason` renamed to `finishReason`). The test **still passes**: ``` RED-PROBE realChunks=0 mockChunks=0 realBytes=1607 mockBytes=708 Tests 2 passed | 24 skipped (26) ``` ## Local GREEN (this branch, live Cohere) ``` GREEN-PROBE realEvents=13 [message-start,content-start,content-delta x9,content-end,message-end] mockEvents=5 [message-start,content-start,content-delta,content-end,message-end] ``` Two consecutive live runs, probe removed: ``` run 1: ✓ Cohere drift > streaming /v2/chat shape matches 304ms run 2: ✓ Cohere drift > streaming /v2/chat shape matches 317ms ``` The same `finishReason` mutant now **fails**: ``` × Cohere drift > streaming /v2/chat shape matches [critical] LLMOCK DRIFT — field in SDK + real API but missing from mock Path: SSE:message-end.delta.finish_reason ``` A second mutant (drop the `content-end` event) also **fails**: ``` [critical] LLMOCK DRIFT — real API emits event type "content-end" but mock does not ``` Both mutants reverted. ## Non-critical differences now visible (not fixed here) These come from the comparison, but the existing triangulation rules do not rate them critical: live `usage.cached_tokens`, live `content-start` carries `text: ""`, and aimock adds `content.type` on deltas plus `billed_units.search_units/classifications`. Outside the shape comparison: live ends with `data: [DONE]` and aimock's Cohere stream does not. ## Gates `pnpm typecheck`, `pnpm lint`, `pnpm format:check`, `pnpm build`, `pnpm test` (377 files, 14080 passed) all pass. ## Follow-up: parse by the SSE spec (review finding) `parseCohereSSE` threw on legal SSE: a single trailing `\n`, `event:X` with no space, `:` keepalive comments, and `id:`/`retry:` lines. It now follows the WHATWG event-stream rules: any line ending (`\n`, `\r\n`, `\r`), the space after the colon is optional, comments and `id`/`retry`/unknown fields are ignored, multi-line `data` is joined, and a final event is dispatched without a trailing blank line. These still throw: data that is not JSON, data that is not an object, an event name that does not match `data.type` (an event without `event:` is named `message` by the spec), and any event after `[DONE]`. RED: the new offline tests run against the previous parser: ``` × parseCohereSSE > accepts legal SSE variants a server or proxy may send Error: Unsupported Cohere SSE frame: "event: message-end\ndata: {...}\n" × parseCohereSSE > rejects malformed data, mismatched names and events after [DONE] ``` GREEN: all 4 parser tests pass. Live streaming leg passed 2 consecutive times. The `finishReason` mutant still fails on `SSE:message-end.delta.finish_reason`. Gates pass again (377 files, 14080 tests).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two Cohere live-drift failures on
main(9420afa). They have the same root cause:cohere.drift.ts:688expected objectincohereRecord←cohereUsage←readCohereK5:884(stream branch). Seen in both legs of PR test: accept native Vertex malformed-function terminal with empty content #524's drift-live-pr run 38157499382.cohere.drift.ts:324expected 422 to be 200in the non-streaming shape test. Seen in post-merge Drift Tests run 38158387279.Neither appears in any main drift report before 2026-10-11 ~17:00 UTC. I scanned the artifacts of every main/PR drift run back to 37580386296.
Diagnosis: upstream nondeterminism, not a contract change
Sometimes Cohere generates nothing. Live probes of the exact test requests on 2026-10-11 (trial key; 429s excluded):
tool_choice: REQUIRED, 128 tok)MAX_TOKENS, normal usage{"error_type":"NO_TOOL_CALL_OR_RESPONSE_GENERATED","id":…,"message":…}TOOL_CALL, normal usagemessage-end={"finish_reason":"ERROR","usage":{},"error":"No tool calls or response was generated…","error_type":"NO_TOOL_CALL_OR_RESPONSE_GENERATED"}Say hello(object)Say hello(object){"error_type":"NO_VALID_RESPONSE_GENERATED",…}Say hello(stream)Say hello(stream)NO_VALID_RESPONSE_GENERATEDThe
usage: {}on the stream ERROR terminal is exactly what fails at line 688:usage.tokensis undefined. The 422 is what fails at line 324. Successful responses kept the same shape throughout, so aimock's Cohere builder has nothing to follow. Nosrc/code changes; only test files.Fix (live leg only; aimock and replay legs stay strict)
cohere-model.ts:cohereNoGenerationError(status, body, allowed)matches only a 422 whose keys are exactly{error_type,id,message}(non-empty strings) and whose code is inallowed. It returns{errorType, id}. The codes are per request: the no-tools shape legs (COHERE_SHAPE_LEG_NO_GENERATION) accept onlyNO_VALID_RESPONSE_GENERATED, and the K5 tool request (COHERE_K5_NO_GENERATION) accepts onlyNO_TOOL_CALL_OR_RESPONSE_GENERATED.isCohereNoGenerationStreamEnd(delta)matches only the exact observed ERROR terminal.readCohereK5/compareCohereK5/runCohereK5Modestake a{ live }option. Only the live test sets it. The exact variants becomeUPSTREAM_NO_GENERATION: a 422 on either send, or the stream ERROR terminal. The stream variant must not have emitted any tool or content first, and the SDK error must be aCohereErrorwith status 422. Anything else rethrows. A refused 422 rethrows withcause: { sdk, classifier }so the classifier's reason is kept.UPSTREAM_NO_GENERATIONis a skip, not a pass. Its reason cites each upstream code and id. One compared send is still a pass.ctx.skipcarries a reason with the upstream code or status and the response id.Local red/green
RED: unmodified main @9420afa1, same filtered live command, 8 runs: run 5 reproduced the CI stack trace exactly:
Replay RED: the new strict-leg test asserts that the unmodified reader throws
expected objecton the captured stream andHTTP failureon the captured 422.GREEN: this branch, same command, 8 consecutive live runs, all passed. They included real occurrences of the variants:
Mutants, round 1 (5/5 killed): removing the live gate on stream, removing the live gate on object, making the stream-end classifier always true, making the 422 classifier accept any error type, and dropping the tool-before-terminal check.
Mutants, round 2 after review (12/12 killed): all-no-generation skip disabled · 422 accepted on the object send only · shape leg allows the tool code ·
allowedfilter dropped · bare rethrow instead of rethrow-with-cause ·contentCountguard deleted ·typeofdeleted on id, message and streamerror·null-body guard deleted · 422completeguard deleted · 422 content-type guard deleted.Live after review, 6 consecutive runs, 0 failures. Run 1 showed the new skip working live:
Runs 2–6: K5 passed every time (3× stream TARGET, 2× stream UPSTREAM_NO_GENERATION with the object compared). Two shape-leg sends were skipped with reasons.
Gates (after review):
pnpm typecheck✓,pnpm lint✓,pnpm format:check✓,pnpm build✓,pnpm test✓ (377 files, 14079 passed). Cohere drift file offline: 23 passed / 3 skipped.Tradeoff
When the stream send hits no-generation (about 22% of probes), that run does not observe K5
TARGET. If the object send is compared, the run passes. If both sends generate nothing, the run is reported as skipped. This follows the precedent approved today for genuine upstream nondeterminism: accept the exact variants on the live leg rather than retry.Backlog (out of scope)
parseDataOnlySSEdropsevent:frames (pre-existing).