feat!: Add support for real time judge evals #969
Conversation
|
@launchdarkly/browser size report |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
@launchdarkly/js-sdk-common size report |
|
@launchdarkly/js-client-sdk size report |
|
@launchdarkly/js-client-sdk-common size report |
|
Will |
If this is set: Then it will only be a minor. |
I only want it to be a minor, but I want the proper change logs to show breaking even for the minor bump. |
| this._ldClient.track(TRACK_CONFIG_SINGLE, context, key, 1); | ||
|
|
||
| const config = await this._evaluate(key, context, defaultValue, 'completion', variables); | ||
| return this._addVercelAISDKSupport(config as LDAICompletionConfig); |
There was a problem hiding this comment.
Bug
When completionConfig receives a disabled config from _evaluate (due to mode mismatch), it still calls _addVercelAISDKSupport on it. However, the disabled config returned by LDAIConfigUtils.createDisabledConfig is cast to LDAICompletionConfig but doesn't actually have the proper structure. The _addVercelAISDKSupport method will try to access config.messages which will be undefined for disabled configs, and create a mapper with undefined messages. While this may not crash, it's inconsistent behavior - disabled configs shouldn't have the toVercelAISDK method added since they can't be used anyway. The test at line 127 in the diff expects toVercelAISDK to be present even for disabled configs, but this creates a misleading API where a disabled config appears to have functionality it shouldn't use.
There was a problem hiding this comment.
This method is deprecated and will be removed in a separate PR.
feat: Added judgeConfig method to AI SDK to retrieve an AI Judge Config
feat: Added createJudge method to create a Judge based on the judge key provided
feat: Added trackEvalScores method to config tracker
feat: Chat will evaluate responses with configured judges
fix!: AI Config defaults require the "enabled" attribute
fix!: Renamed LDAIAgentConfig to LDAIAgentConfigRequest for improved clarity
fix!: Renamed LDAIAgent to LDAIAgentConfig *note the previous use of this name
fix!: Renamed LDAIAgentDefault to LDAIAgentConfigDefault for improved clarity
fix!: Renamed LDAIDefaults to LDAICompletionConfigDefault for improved clarity
Note
Introduces Judge evaluations (judgeConfig/createJudge) with structured outputs and automatic chat scoring, refactors config types/modes, updates tracking, and deprecates older APIs.
Judgewith structured output (invokeStructuredModel) and schema builder; newjudgeConfigandcreateJudgeAPIs.TrackedChatcan attach judges and asynchronously evaluate responses; results tracked viatrackEvalScores.completionConfig,agentConfig,judgeConfig,agentConfigs,createChat,createJudgemethods; legacyconfig,agent,agents,initChatdeprecated (now wrappers).LDAIConfigUtilsand disabled-return on mode mismatch.api/config/typeswith modes:completion | agent | judgeand new defaults (LDAICompletionConfigDefault,LDAIAgentConfigDefault,LDAIJudgeConfigDefault).LDAIConfig/agent types; removeapi/agentsmodule; add judge attachment (LDJudgeConfiguration).AIProviderwith defaultinvokeModeland newinvokeStructuredModel; updateAIProviderFactoryto accept union config kinds.getTrackData; addtrackEvalScores; new tracking event keys; minor tracker refactors.createChat, enabled checks); comprehensive tests forJudge, client config APIs, andTrackedChat.Written by Cursor Bugbot for commit a187fcc. This will update automatically on new commits. Configure here.