Skip to content

test(evals): accept handleAuthkitProxy and app-owned AuthKitProvider modules - #262

Merged
nicknisi merged 2 commits into
mainfrom
test/eval-graders-accept-alternatives
Oct 9, 2026
Merged

nicknisi merged 2 commits into
mainfrom
test/eval-graders-accept-alternatives

Conversation

@nicknisi

@nicknisi nicknisi commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

What

Two eval graders were rejecting valid integrations. Both false negatives came from the Opus 5.5 eval run in #261:

  • Next.js: accept handleAuthkitProxy( as well as handleAuthkitHeaders(. In @workos-inc/authkit-nextjs 4.4.0, handleAuthkitHeaders is a @deprecated alias of handleAuthkitProxy.
  • React: look for AuthKitProvider (and the env-based config next to it) anywhere in src/**/*.tsx, not only in src/{main,index}.tsx. Wrapping the provider in an app-owned provider module is valid, and the React skill doesn't require a specific file.

The companion skills change (workos/skills#53) updates the Next.js reference to recommend handleAuthkitProxy, so this grader change is needed before that lands.

Verification

…modules

handleAuthkitHeaders is a deprecated alias of handleAuthkitProxy in
authkit-nextjs 4.4. AuthKitProvider can live in a provider module, not
only src/main.tsx or src/index.tsx.
@greptile-apps

greptile-apps Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

[Medium impact] Updates evaluation tests for authentication provider detection.

The PR still needs to check that the app actually renders the React provider before it appears safe to merge.

Findings

  1. P1 Unused provider passes grading ▶
Fix with agent prompt
### Issue 1
tests/evals/graders/react.grader.ts:undefined-32
If `src/auth/Providers.tsx` imports and renders `AuthKitProvider` and reads the environment config, this search accepts it even when the entry file never mounts it. A component using `useAuth` and a successful build can satisfy the remaining checks, so an app without the provider can receive a passing grade. Check that the matched provider is connected to the entry file.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR accepts both Next.js helper names and allows React’s AuthKitProvider to live in an app-owned module.

  • Next.js evals accept both AuthKit handler names.
  • React evals recognize provider modules imported by app code.

Reviews (2) · Last reviewed commit: "test(evals): require mounted AuthKitProv..." · Reviewed by Greptile

Comment thread tests/evals/graders/react.grader.ts Outdated
checks.push(
await this.fileGrader.checkFileWithPattern(
'src/{main,index}.tsx',
'src/**/*.tsx',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Unused provider passes grading

If src/auth/Providers.tsx imports and renders AuthKitProvider and reads the environment config, this search accepts it even when the entry file never mounts it. A component using useAuth and a successful build can satisfy the remaining checks, so an app without the provider can receive a passing grade. Check that the matched provider is connected to the entry file.

Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/evals/graders/react.grader.ts
Line: 32

Comment:
**Unused provider passes grading**

If `src/auth/Providers.tsx` imports and renders `AuthKitProvider` and reads the environment config, this search accepts it even when the entry file never mounts it. A component using `useAuth` and a successful build can satisfy the remaining checks, so an app without the provider can receive a passing grade. Check that the matched provider is connected to the entry file.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Comment thread tests/evals/graders/react.grader.ts Outdated
Comment thread tests/evals/graders/nextjs.grader.ts Outdated
- react: AuthKitProvider must be in the entry file or a module imported
  elsewhere in src/; client ID env may be read in any src file
- nextjs: match handleAuthkitProxy/handleAuthkitHeaders by name so aliased
  imports and spaced calls still pass
@nicknisi
nicknisi merged commit 7eb33eb into main Oct 9, 2026
6 checks passed
@nicknisi
nicknisi deleted the test/eval-graders-accept-alternatives branch October 9, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant