Skip to content

fix: update skills for current SDKs and frontier models - #53

Merged
nicknisi merged 3 commits into
mainfrom
chore/skills-current-models
Oct 19, 2026
Merged

nicknisi merged 3 commits into
mainfrom
chore/skills-current-models

Conversation

@nicknisi

@nicknisi nicknisi commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

What

A pass over the skills for current frontier models (Opus 4.5 / 5.5), in two commits.

1. fix: correct stale SDK APIs in skill references (f0c6715)

Every claim below was checked against the published SDKs (node 11.0.0, authkit-nextjs 4.4.0, widgets 1.20.0, Ruby gem 10.4.0, workos-go v6.5.0, com.workos:workos 7.4.0, @workos/openapi-spec 0.123.0).

File Was Now
authkit-nextjs "Always return via handleAuthkitHeaders()" handleAuthkitProxy (handleAuthkitHeaders is a deprecated alias)
radar "no workos.radar.* namespace" Node v11 has createAttempt / updateAttempt / addListEntry / removeListEntry
directory-sync, management workos.webhooks.verifyEvent (doesn't exist) constructEvent
go, widgets/go workos-go/v4 workos-go/v6
ruby, sso, widgets/ruby module-level calls, config.key gem 7+ WorkOS.client.*, config.api_key, widgets.create_token; note for gems below 7
kotlin, widgets/java com.workos:workos-kotlin:4.18.1 (not on Maven Central), builder API com.workos:workos 7.x, AuthKitAuthorizationUrlOptions, widgets.createToken
migrate-standalone-sso clientId requirement reversed getProfileAndToken requires it; authenticateWithCode falls back to the client's
migrate-* hash list pbkdf2 is kept (the API accepts it), ssha256 added, and a note that Node's PasswordHashType omits both
widgets <UserManagement /> <UsersManagement />, plus the WorkOsWidgets provider, peer deps, and authToken

2. refactor: prune skill references for current models (dde8efd)

Net about −2,900 lines. Removed: STOP/BLOCKING/CRITICAL/ALL MUST PASS banners, package-manager and framework detection trees, generic error-recovery tables, duplicated checklists, and long code blocks the agent copied verbatim. Kept: WorkOS-specific gotchas, env vars, security rules, doc URLs, and every string the tests assert. The 9 backend references now share one short shape (setup / gotchas / existing auth).

Installer prompt size for Next.js (base + nextjs + setup): 41.2 KB → 28.2 KB. Backend references went from about 6 KB to about 2 KB each.

Out of scope, flagged for follow-up:

  • Widgets architecture. The skill teaches custom UIs over /_widgets plus an 11k-line spec and bundled parser, instead of the shipped components. I only fixed names here.
  • Backend logout. The backend references clear only the local session; they don't end the AuthKit session. This was already the case before this PR.
  • Webhook timeout. events.md said 5s and directory-sync said 10s, and the docs give no number. Both now say "quickly".

Verification

Opus 4.5 before Opus 4.5 after Opus 5.5 before Opus 5.5 after
Passed 34/34 34/34 32/34¹ 34/34
First-attempt pass 64.7% 73.5% 76.5% 85.3%
Duration p50 / p95 177s / 298s 142s / 208s 164s / 249s 167s / 293s
TTFT p50 8.7s 13.1s 6.1s 7.1s
Tool calls (total) 1047 996 1178 1074

¹ Both failures were grader false negatives, fixed in workos/cli#262.

One run per model per condition, so treat the deltas as directional.

- nextjs: handleAuthkitHeaders is a deprecated alias of handleAuthkitProxy; recommend the current name and stop forbidding it
- radar: Node SDK v11 ships workos.radar.{createAttempt,updateAttempt,addListEntry,removeListEntry}
- webhooks: verifyEvent does not exist; use constructEvent
- go: module path is workos-go/v6
- ruby: gem 7+ uses WorkOS.client.* (config.api_key, get_authorization_url, widgets.create_token)
- kotlin/java: artifact is com.workos:workos (7.x); update getAuthorizationUrl and widgets token APIs
- standalone SSO migration: clientId requirement was reversed
- migrations: API accepts pbkdf2/ssha256 but Node SDK PasswordHashType omits them
- widgets: package exports UsersManagement; note WorkOsWidgets provider, peer deps, authToken
@greptile-apps

greptile-apps Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5 Tier: apex

[Low impact] The PR appears safe to merge; the remaining Radar evaluation mismatch is non-blocking.

Findings

  1. P2 Correct Radar answers lose points ▶
Fix with agent prompt
### Issue 1
plugins/workos/skills/workos/references/workos-radar.md:undefined-22
The guide now recommends `workos.radar.updateAttempt`, but `scripts/eval/cases/radar.yaml` still lists that exact method under `hallucinations`. `scoreOutput()` uses that list, so an answer following the updated guide gets marked as an invented API and loses points. Update the Radar cases' methods, parameters, and flow expectations alongside this SDK correction.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Updates WorkOS SDK examples and shortens the skill references for coding agents.

  • Corrects webhook, Radar, Ruby, Go, JVM, and packaged widget API guidance.
  • Keeps shared AuthKit URL and flow checks and updates the Next.js test’s section names.
  • nicknisi deferred replacing the custom widget architecture and ending backend AuthKit sessions because both are existing behavior outside this pass.
  • nicknisi replaced conflicting webhook timeout numbers with “quickly” because the docs provide no limit.

Reviews (2) · Last reviewed commit: "fix: align eval cases with corrected API..." · Reviewed by Greptile

Comment thread node_modules Outdated
- `auth_method` must be one of: `Password`, `Passkey`, `Authenticator`, `SMS_OTP`, `Email_OTP`, `Social`, `SSO`, `Other`. Claude tends to use lowercase or invented values.
- `action` accepts: `login`, `signup` (and variants like `sign-in`, `sign_up`). Use the simplest form.
- After a successful authentication, call `PUT /radar/attempts/:id` with `attempt_status: "success"` to improve Radar's model (enables impossible travel detection).
- After a successful authentication, update the attempt with `attempt_status: "success"` (`updateAttempt({ id, attemptStatus: 'success' })`) to improve Radar's model (enables impossible travel detection).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Correct Radar answers lose points

The guide now recommends workos.radar.updateAttempt, but scripts/eval/cases/radar.yaml still lists that exact method under hallucinations. scoreOutput() uses that list, so an answer following the updated guide gets marked as an invented API and loses points. Update the Radar cases' methods, parameters, and flow expectations alongside this SDK correction.

Prompt To Fix With AI
This is a comment left during a code review.
Path: plugins/workos/skills/workos/references/workos-radar.md
Line: 22

Comment:
**Correct Radar answers lose points**

The guide now recommends `workos.radar.updateAttempt`, but `scripts/eval/cases/radar.yaml` still lists that exact method under `hallucinations`. `scoreOutput()` uses that list, so an answer following the updated guide gets marked as an invented API and loses points. Update the Radar cases' methods, parameters, and flow expectations alongside this SDK correction.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Comment thread plugins/workos/skills/workos/references/workos-authkit-tanstack-start.md Outdated
Remove scaffolding that frontier models no longer need: STOP/BLOCKING/
CRITICAL banners, package-manager and framework detection trees, generic
error-recovery tables, duplicated checklists, and long verbatim code
blocks. Keep WorkOS-specific gotchas, env vars, security rules, and doc
URLs. Backend SDK references now share one short shape (setup, gotchas,
existing auth). No API facts changed.
… start path

- directory-sync cases expect workos.webhooks.constructEvent
- radar: workos.radar.updateAttempt is a real Node v11 method, not a hallucination
- tanstack-start: legacy app/-rooted projects use app/start.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant