[donei003]

Overview

Implements the SDK side of the idempotent offline-eval create plan (docs/idempotent-eval-create.md).

run(key=…) now reconciles via public …/offline-evaluations:

1. GET …/offline-evaluations/{key}
2. 404 → POST create-only with required key + name (name defaults to key); on 409 race, re-GET and continue
3. 200 → PATCH full config snapshot only when desired config differs (no-op when equal)
4. POST …/evaluations/{id}/runs (runs stay on the legacy by-id prefix until nested offline-evaluations runs ship server-side)

No PUT. Legacy POST …/evaluations create-always remains on the server for SDK 1.0 clients; this release no longer calls it.

Depends on server stack: ai-evaluator#922 (schema), ai-evaluator#923 (API).

Contract update: companion draft on ai-sdks-monorepo for TESTING.md §8.4–§8.5.

Version / docs

• README: stable keys; document >=1.1.0 for this reconcile behavior (release-please will bump from feat(evaluations): on merge).
• Examples use key="support-qa" instead of timestamp-suffixed keys.

Testing strategy

uv run pytest \
  packages/client/tests/test_evaluations_run.py \
  packages/client/tests/test_evaluations.py \
  packages/client/tests/test_evaluations_spans.py -q

265 passed, including new coverage for:

• first mint (GET 404 → POST)
• reuse without PATCH when config matches
• PATCH when config drifts
• POST 409 → re-GET → PATCH race
• LDApiClient.patch JSON body

Other

Out of scope here: moving runs/summary/upload under …/offline-evaluations/{id}/… (server follow-on); soft-hide null-key UI; Mintlify docs.

  

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 64b20b6. Configure here.}

[cursor]

Config match treats extras as drift

Medium Severity

_evaluation_config_matches compares GET payloads to the harness wire objects with exact !=. LaunchDarkly injects successDirection onto judge criteria at create time, and GET bodies can also add ids or other server fields on criteria, tools, or messages. A semantically unchanged config then looks different, so every later run() with judges PATCHes and bumps version instead of taking the documented no-op path.

 

^{Reviewed by Cursor Bugbot for commit 64b20b6. Configure here.}

[cursor]

PATCH omits emptied config fields

Medium Severity

_evaluation_config_body drops empty tools, criteria, parameters, and promptSnippets from the PATCH snapshot, while _evaluation_config_matches treats those omissions as “must be empty on the server” and triggers a PATCH. messages is always sent, including as []. A later run() that removes judges, tools, or parameters therefore PATCHes a body that does not name the fields being cleared, so stale config can remain and every following run PATCHes again.

Additional Locations (1)

• packages/client/src/launchdarkly_ai_server/evaluations/runner.py#L634-L640

 

^{Reviewed by Cursor Bugbot for commit 64b20b6. Configure here.}