feat!: Add support for real time judge evals #969
Conversation
|
@launchdarkly/browser size report |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
@launchdarkly/js-sdk-common size report |
|
@launchdarkly/js-client-sdk size report |
|
@launchdarkly/js-client-sdk-common size report |
|
Will |
If this is set: Then it will only be a minor. |
| })); | ||
| } | ||
| const config = await this._evaluate(key, context, defaultValue, 'completion', variables); | ||
| return this._addVercelAISDKSupport(config as LDAIConversationConfig); |
There was a problem hiding this comment.
It seems like the core AI SDK impl shouldn't know about the specifics of an extending SDK.
There was a problem hiding this comment.
This was previously supported and just left "unchanged" in this PR other than moving it into a helper function. We plan to deprecate this and remove it in support of the new vercel provider package.
There was a problem hiding this comment.
@abarker-launchdarkly do you have any reservations if we remove this now since it will be a major breaking change anyways?
I only want it to be a minor, but I want the proper change logs to show breaking even for the minor bump. |
feat!: AI Config defaults require the "enabled" attribute
feat!: Renamed LDAIAgentConfig to LDAIAgentConfigRequest for improved clarity
feat!: Renamed LDAIAgent to LDAIAgentConfig *note the previous use of this name
feat!: Removed LDAIAgentDefault in favor of LDAIAgentConfig
feat!: Removed LDAIDefaults in favor of LDAIConfig
feat: Added judge method to AI SDK to retrieve an AI Judge Config
feat: Added initJudge method to create a Judge based on the judge key provided
feat: Added trackEvalScores method to config tracker
Note
Introduces Judge-based real-time evaluations with structured output and metrics, refactors config/types and client API (new completion/agent/judge config and createChat/createJudge), integrates judge scoring into TrackedChat, and enhances tracking/provider abstractions.
Judgewith structured evaluations, sampling, and parsing; newEvaluationSchemaBuilderand judgetypes.TrackedChattriggers judge evaluations per response, tracks scores viatracker.trackEvalScores, and exposesresponse.evaluations.completionConfig,agentConfig,judgeConfig,createChat,createJudge, andagentConfigs(plural).config,agent,agents,initChat(kept as thin wrappers).judgeConfiguration; adds tracking event keys.LDAIConfigmonolith withtypes(LDAIConversationConfig,LDAIAgentConfig,LDAIJudgeConfig, defaults,LDJudgeConfiguration,LDAIConfigMode).LDAIConfigUtilsandLDAIConfigFlagValuefor flag conversion and disabled fallbacks.createChatusage; defaults includeenabled).LDAIConfigTracker: exposegetTrackData()and addtrackEvalScores(); corresponding impl updates.AIProvider: add defaultinvokeModeland newinvokeStructuredModel;AIProviderFactorynow acceptsLDAIConfigKind.Judgetests; expandLDAIClientImplandTrackedChattests for new APIs and behaviors.Written by Cursor Bugbot for commit b83c1e3. This will update automatically on new commits. Configure here.