Skip to content

Commit 2810850

Browse files
felixarntzfelixarntz
authored
fix(ai): improve type validation error messages with field paths and entity identifiers (#12106)
## Background Validation errors from `safeValidateUIMessages` and streaming validation lacked context about what was being validated, making debugging difficult. **Before:** ``` Type validation failed: Value: undefined. Error message: Required ``` **After:** ``` Type validation failed for messages[0].metadata (id: "msg-123"): Value: undefined. Error message: Required ``` Alternative to #10154: This PR focuses on addressing the lack of context closer to the root, and includes the full field identifier to help finding the culprit more quickly. ## Summary Added optional context tracking to `TypeValidationError` to include field paths (e.g., `messages[0].parts[1].data`) and entity identifiers (e.g., message ID, tool call ID) in validation error messages. **Approach:** - Introduced `TypeValidationContext` interface with `field`, `entityName`, and `entityId` properties - Extended `TypeValidationError` and validation utilities to accept and format context - Updated all validation call sites in `validate-ui-messages.ts` and `process-ui-message-stream.ts` to provide context with index tracking The change is backward compatible (context is optional) and includes a minor performance improvement by consolidating data/tool validation into a single loop. ## Manual Verification To verify the improved error messages: ```typescript import { validateUIMessages } from 'ai'; import { z } from 'zod'; try { await validateUIMessages({ messages: [{ id: 'msg-123', role: 'user', metadata: { userId: 123 }, // wrong type parts: [{ type: 'text', text: 'Hello' }], }], metadataSchema: z.object({ userId: z.string() }), }); } catch (error) { console.log(error.message); // Shows: "Type validation failed for messages[0].metadata (id: "msg-123"): ..." } ``` ## Checklist - [x] Tests have been added / updated (for bug fixes / features) - [ ] Documentation has been added / updated (for bug fixes / features) - [x] A _patch_ changeset for relevant packages has been added (for bug fixes / features - run `pnpm changeset` in the project root) - [x] I have reviewed this pull request (self-review) ## Future Work Issue #10137 also mentions fixing documentation for metadata schema examples. That's not the primary issue raised though, and will be addressed in a separate PR. ## Related Issues Fixes #10137
1 parent 8f1bd75 commit 2810850

File tree

7 files changed

+195
-76
lines changed

7 files changed

+195
-76
lines changed

‎.changeset/early-elephants-play.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
'@ai-sdk/provider-utils': patch
3+
'@ai-sdk/provider': patch
4+
'ai': patch
5+
---
6+
7+
fix(ai): improve type validation error messages with field paths and entity identifiers

‎packages/ai/src/ui/process-ui-message-stream.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import { TypeValidationContext } from '@ai-sdk/provider';
12
import { FlexibleSchema, validateTypes } from '@ai-sdk/provider-utils';
23
import { UIMessageStreamError } from '../error/ui-message-stream-error';
34
import { ProviderMetadata } from '../types';
@@ -288,6 +289,10 @@ export function processUIMessageStream<UI_MESSAGE extends UIMessage>({
288289
await validateTypes({
289290
value: mergedMetadata,
290291
schema: messageMetadataSchema,
292+
context: {
293+
field: 'message.metadata',
294+
entityId: state.message.id,
295+
},
291296
});
292297
}
293298

@@ -710,9 +715,24 @@ export function processUIMessageStream<UI_MESSAGE extends UIMessage>({
710715
if (isDataUIMessageChunk(chunk)) {
711716
// validate data chunk if dataPartSchemas is provided
712717
if (dataPartSchemas?.[chunk.type] != null) {
718+
const partIdx = state.message.parts.findIndex(
719+
p =>
720+
'id' in p &&
721+
'data' in p &&
722+
p.id === chunk.id &&
723+
p.type === chunk.type,
724+
);
725+
const actualPartIdx =
726+
partIdx >= 0 ? partIdx : state.message.parts.length;
727+
713728
await validateTypes({
714729
value: chunk.data,
715730
schema: dataPartSchemas[chunk.type],
731+
context: {
732+
field: `message.parts[${actualPartIdx}].data`,
733+
entityName: chunk.type,
734+
entityId: chunk.id,
735+
},
716736
});
717737
}
718738

‎packages/ai/src/ui/validate-ui-messages.test.ts‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -172,7 +172,7 @@ describe('validateUIMessages', () => {
172172
}),
173173
}),
174174
).rejects.toThrowErrorMatchingInlineSnapshot(`
175-
[AI_TypeValidationError: Type validation failed: Value: {"foo":123}.
175+
[AI_TypeValidationError: Type validation failed for messages[0].metadata (id: "1"): Value: {"foo":123}.
176176
Error message: [
177177
{
178178
"expected": "string",
@@ -512,7 +512,7 @@ describe('validateUIMessages', () => {
512512
},
513513
}),
514514
).rejects.toThrowErrorMatchingInlineSnapshot(`
515-
[AI_TypeValidationError: Type validation failed: Value: {"foo":123}.
515+
[AI_TypeValidationError: Type validation failed for messages[0].parts[0].data (foo): Value: {"foo":123}.
516516
Error message: [
517517
{
518518
"expected": "string",
@@ -541,7 +541,7 @@ describe('validateUIMessages', () => {
541541
},
542542
}),
543543
).rejects.toThrowErrorMatchingInlineSnapshot(`
544-
[AI_TypeValidationError: Type validation failed: Value: {"foo":"bar"}.
544+
[AI_TypeValidationError: Type validation failed for messages[0].parts[0].data (bar): Value: {"foo":"bar"}.
545545
Error message: No data schema found for data part bar]
546546
`);
547547
});
@@ -1142,7 +1142,7 @@ describe('validateUIMessages', () => {
11421142
},
11431143
}),
11441144
).rejects.toThrowErrorMatchingInlineSnapshot(`
1145-
[AI_TypeValidationError: Type validation failed: Value: {"foo":"bar"}.
1145+
[AI_TypeValidationError: Type validation failed for messages[0].parts[0].input (bar, id: "1"): Value: {"foo":"bar"}.
11461146
Error message: No tool schema found for tool part bar]
11471147
`);
11481148
});
@@ -1170,7 +1170,7 @@ describe('validateUIMessages', () => {
11701170
},
11711171
}),
11721172
).rejects.toThrowErrorMatchingInlineSnapshot(`
1173-
[AI_TypeValidationError: Type validation failed: Value: {"foo":123}.
1173+
[AI_TypeValidationError: Type validation failed for messages[0].parts[0].input (foo, id: "1"): Value: {"foo":123}.
11741174
Error message: [
11751175
{
11761176
"expected": "string",
@@ -1208,7 +1208,7 @@ describe('validateUIMessages', () => {
12081208
},
12091209
}),
12101210
).rejects.toThrowErrorMatchingInlineSnapshot(`
1211-
[AI_TypeValidationError: Type validation failed: Value: {"result":123}.
1211+
[AI_TypeValidationError: Type validation failed for messages[0].parts[0].output (foo, id: "1"): Value: {"result":123}.
12121212
Error message: [
12131213
{
12141214
"expected": "string",

‎packages/ai/src/ui/validate-ui-messages.ts‎

Lines changed: 87 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { TypeValidationError } from '@ai-sdk/provider';
1+
import { TypeValidationContext, TypeValidationError } from '@ai-sdk/provider';
22
import {
33
FlexibleSchema,
44
lazySchema,
@@ -346,79 +346,107 @@ export async function safeValidateUIMessages<UI_MESSAGE extends UIMessage>({
346346
});
347347

348348
if (metadataSchema) {
349-
for (const message of validatedMessages) {
349+
for (const [msgIdx, message] of validatedMessages.entries()) {
350350
await validateTypes({
351351
value: message.metadata,
352352
schema: metadataSchema,
353+
context: {
354+
field: `messages[${msgIdx}].metadata`,
355+
entityId: message.id,
356+
},
353357
});
354358
}
355359
}
356360

357-
if (dataSchemas) {
358-
for (const message of validatedMessages) {
359-
const dataParts = message.parts.filter(part =>
360-
part.type.startsWith('data-'),
361-
) as DataUIPart<InferUIMessageData<UI_MESSAGE>>[];
361+
if (dataSchemas || tools) {
362+
for (const [msgIdx, message] of validatedMessages.entries()) {
363+
for (const [partIdx, part] of message.parts.entries()) {
364+
// Data part validation
365+
if (dataSchemas && part.type.startsWith('data-')) {
366+
const dataPart = part as DataUIPart<InferUIMessageData<UI_MESSAGE>>;
367+
const dataName = dataPart.type.slice(5);
368+
const dataSchema = dataSchemas[dataName];
362369

363-
for (const dataPart of dataParts) {
364-
const dataName = dataPart.type.slice(5);
365-
const dataSchema = dataSchemas[dataName];
370+
if (!dataSchema) {
371+
return {
372+
success: false,
373+
error: new TypeValidationError({
374+
value: dataPart.data,
375+
cause: `No data schema found for data part ${dataName}`,
376+
context: {
377+
field: `messages[${msgIdx}].parts[${partIdx}].data`,
378+
entityName: dataName,
379+
entityId: dataPart.id,
380+
},
381+
}),
382+
};
383+
}
366384

367-
if (!dataSchema) {
368-
return {
369-
success: false,
370-
error: new TypeValidationError({
371-
value: dataPart.data,
372-
cause: `No data schema found for data part ${dataName}`,
373-
}),
374-
};
385+
await validateTypes({
386+
value: dataPart.data,
387+
schema: dataSchema,
388+
context: {
389+
field: `messages[${msgIdx}].parts[${partIdx}].data`,
390+
entityName: dataName,
391+
entityId: dataPart.id,
392+
},
393+
});
375394
}
376395

377-
await validateTypes({
378-
value: dataPart.data,
379-
schema: dataSchema,
380-
});
381-
}
382-
}
383-
}
384-
385-
if (tools) {
386-
for (const message of validatedMessages) {
387-
const toolParts = message.parts.filter(part =>
388-
part.type.startsWith('tool-'),
389-
) as ToolUIPart<InferUIMessageTools<UI_MESSAGE>>[];
396+
// Tool part validation
397+
if (tools && part.type.startsWith('tool-')) {
398+
const toolPart = part as ToolUIPart<
399+
InferUIMessageTools<UI_MESSAGE>
400+
>;
401+
const toolName = toolPart.type.slice(5);
402+
const tool = tools[toolName];
390403

391-
for (const toolPart of toolParts) {
392-
const toolName = toolPart.type.slice(5);
393-
const tool = tools[toolName];
404+
// TODO support dynamic tools
405+
if (!tool) {
406+
return {
407+
success: false,
408+
error: new TypeValidationError({
409+
value: toolPart.input,
410+
cause: `No tool schema found for tool part ${toolName}`,
411+
context: {
412+
field: `messages[${msgIdx}].parts[${partIdx}].input`,
413+
entityName: toolName,
414+
entityId: toolPart.toolCallId,
415+
},
416+
}),
417+
};
418+
}
394419

395-
// TODO support dynamic tools
396-
if (!tool) {
397-
return {
398-
success: false,
399-
error: new TypeValidationError({
420+
// Tool input validation
421+
if (
422+
toolPart.state === 'input-available' ||
423+
toolPart.state === 'output-available' ||
424+
(toolPart.state === 'output-error' &&
425+
toolPart.input !== undefined)
426+
) {
427+
await validateTypes({
400428
value: toolPart.input,
401-
cause: `No tool schema found for tool part ${toolName}`,
402-
}),
403-
};
404-
}
429+
schema: tool.inputSchema,
430+
context: {
431+
field: `messages[${msgIdx}].parts[${partIdx}].input`,
432+
entityName: toolName,
433+
entityId: toolPart.toolCallId,
434+
},
435+
});
436+
}
405437

406-
if (
407-
toolPart.state === 'input-available' ||
408-
toolPart.state === 'output-available' ||
409-
(toolPart.state === 'output-error' && toolPart.input !== undefined)
410-
) {
411-
await validateTypes({
412-
value: toolPart.input,
413-
schema: tool.inputSchema,
414-
});
415-
}
416-
417-
if (toolPart.state === 'output-available' && tool.outputSchema) {
418-
await validateTypes({
419-
value: toolPart.output,
420-
schema: tool.outputSchema,
421-
});
438+
// Tool output validation
439+
if (toolPart.state === 'output-available' && tool.outputSchema) {
440+
await validateTypes({
441+
value: toolPart.output,
442+
schema: tool.outputSchema,
443+
context: {
444+
field: `messages[${msgIdx}].parts[${partIdx}].output`,
445+
entityName: toolName,
446+
entityId: toolPart.toolCallId,
447+
},
448+
});
449+
}
422450
}
423451
}
424452
}

‎packages/provider-utils/src/validate-types.ts‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { TypeValidationError } from '@ai-sdk/provider';
1+
import { TypeValidationContext, TypeValidationError } from '@ai-sdk/provider';
22
import { FlexibleSchema, asSchema } from './schema';
33

44
/**
@@ -8,19 +8,22 @@ import { FlexibleSchema, asSchema } from './schema';
88
* @template T - The type of the object to validate.
99
* @param {string} options.value - The object to validate.
1010
* @param {Validator<T>} options.schema - The schema to use for validating the JSON.
11+
* @param {TypeValidationContext} options.context - Optional context about what is being validated.
1112
* @returns {Promise<T>} - The typed object.
1213
*/
1314
export async function validateTypes<OBJECT>({
1415
value,
1516
schema,
17+
context,
1618
}: {
1719
value: unknown;
1820
schema: FlexibleSchema<OBJECT>;
21+
context?: TypeValidationContext;
1922
}): Promise<OBJECT> {
20-
const result = await safeValidateTypes({ value, schema });
23+
const result = await safeValidateTypes({ value, schema, context });
2124

2225
if (!result.success) {
23-
throw TypeValidationError.wrap({ value, cause: result.error });
26+
throw TypeValidationError.wrap({ value, cause: result.error, context });
2427
}
2528

2629
return result.value;
@@ -33,14 +36,17 @@ export async function validateTypes<OBJECT>({
3336
* @template T - The type of the object to validate.
3437
* @param {string} options.value - The JSON object to validate.
3538
* @param {Validator<T>} options.schema - The schema to use for validating the JSON.
39+
* @param {TypeValidationContext} options.context - Optional context about what is being validated.
3640
* @returns An object with either a `success` flag and the parsed and typed data, or a `success` flag and an error object.
3741
*/
3842
export async function safeValidateTypes<OBJECT>({
3943
value,
4044
schema,
45+
context,
4146
}: {
4247
value: unknown;
4348
schema: FlexibleSchema<OBJECT>;
49+
context?: TypeValidationContext;
4450
}): Promise<
4551
| {
4652
success: true;
@@ -68,13 +74,13 @@ export async function safeValidateTypes<OBJECT>({
6874

6975
return {
7076
success: false,
71-
error: TypeValidationError.wrap({ value, cause: result.error }),
77+
error: TypeValidationError.wrap({ value, cause: result.error, context }),
7278
rawValue: value,
7379
};
7480
} catch (error) {
7581
return {
7682
success: false,
77-
error: TypeValidationError.wrap({ value, cause: error }),
83+
error: TypeValidationError.wrap({ value, cause: error, context }),
7884
rawValue: value,
7985
};
8086
}

‎packages/provider/src/errors/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,6 @@ export { LoadSettingError } from './load-setting-error';
1111
export { NoContentGeneratedError } from './no-content-generated-error';
1212
export { NoSuchModelError } from './no-such-model-error';
1313
export { TooManyEmbeddingValuesForCallError } from './too-many-embedding-values-for-call-error';
14+
export type { TypeValidationContext } from './type-validation-error';
1415
export { TypeValidationError } from './type-validation-error';
1516
export { UnsupportedFunctionalityError } from './unsupported-functionality-error';

0 commit comments

Comments
 (0)