-
Notifications
You must be signed in to change notification settings - Fork 6.1k
Document strategies for handling invalid AI function call input #50896
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Copilot
wants to merge
7
commits into
main
Choose a base branch
from
copilot/document-invalid-input-strategies
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+259
−0
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
99a59a9
Initial plan
Copilot d7daa90
Add how-to article for handling invalid tool input with code snippets
Copilot 8e24605
Fix snippet paths and update target framework to net10.0
Copilot 0e2a197
Fix retry logic and clarify error handling approach
Copilot c512958
Update StrictSchema to use generic gpt-4o model with explanatory comment
Copilot 066948f
human edits
gewarren 8166dff
shorten snippoets
gewarren File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| --- | ||
| title: Handle invalid tool input from AI models | ||
| description: Learn strategies to handle invalid tool input when AI models provide incorrect or malformed function call parameters. | ||
| ms.date: 01/05/2026 | ||
| ai-usage: ai-assisted | ||
| --- | ||
|
|
||
| # Handle invalid tool input from AI models | ||
|
|
||
| When AI models call functions in your .NET code, they might sometimes provide invalid input that doesn't match the expected schema. The `Microsoft.Extensions.AI` library provides several strategies to handle these scenarios gracefully. | ||
|
|
||
| ## Common scenarios for invalid input | ||
|
|
||
| AI models can provide invalid function call input in several ways: | ||
|
|
||
| - Missing required parameters. | ||
| - Incorrect data types (for example, sending a string when an integer is expected). | ||
| - Malformed JSON that can't be deserialized. | ||
|
|
||
| Without proper error handling, these issues can cause your application to fail or provide poor user experiences. | ||
|
|
||
| ## Enable detailed error messages | ||
|
|
||
| By default, when a function invocation fails, the AI model receives a generic error message. You can enable detailed error reporting using the <xref:Microsoft.Extensions.AI.FunctionInvokingChatClient.IncludeDetailedErrors?displayProperty=nameWithType> property. When this property is set to `true` and an error occurs during function invocation, the full exception message is added to the chat history. This allows the AI model to see what went wrong and potentially self-correct in subsequent attempts. | ||
|
|
||
| :::code language="csharp" source="snippets/handle-invalid-tool-input/csharp/IncludeDetailedErrors.cs" id="BasicUsage"::: | ||
|
|
||
| > [!NOTE] | ||
| > Setting `IncludeDetailedErrors` to `true` can expose internal system details to the AI model and potentially to end users. Ensure exception messages don't contain secrets, connection strings, or other sensitive information. To avoid leaking sensitive information, consider disabling detailed errors in production environments. | ||
|
|
||
| ## Implement custom error handling | ||
|
|
||
| For more control over error handling, you can set a custom <xref:Microsoft.Extensions.AI.FunctionInvokingChatClient.FunctionInvoker?displayProperty=nameWithType> delegate. This allows you to intercept function calls, catch exceptions, and return custom error messages to the AI model. | ||
|
|
||
| The following example shows how to implement a custom function invoker that catches serialization errors and provides helpful feedback: | ||
|
|
||
| :::code language="csharp" source="snippets/handle-invalid-tool-input/csharp/FunctionInvoker.cs" id="BasicInvoker"::: | ||
|
|
||
| By returning descriptive error messages instead of throwing exceptions, you allow the AI model to see what went wrong and try again with corrected input. | ||
|
|
||
| ### Best practices for error messages | ||
|
|
||
| When returning error messages to enable AI self-correction, provide clear, actionable feedback: | ||
|
|
||
| - **Be specific**: Explain exactly what was wrong with the input. | ||
| - **Provide examples**: Show the expected format or valid values. | ||
| - **Use consistent format**: Help the AI model learn from patterns. | ||
| - **Log errors**: Track error patterns for debugging and monitoring. | ||
|
|
||
| ## Use strict JSON schema (OpenAI only) | ||
|
|
||
| When using OpenAI models, you can enable strict JSON schema mode to enforce that the model's output strictly adheres to your function's schema. This helps prevent type mismatches and missing required fields. | ||
|
|
||
| Enable strict mode using the `Strict` additional property on your function metadata. When enabled, OpenAI models try to ensure their output matches your schema exactly: | ||
|
|
||
| :::code language="csharp" source="snippets/handle-invalid-tool-input/csharp/StrictSchema.cs" id="StrictMode"::: | ||
|
|
||
| For the latest list of models that support strict JSON schema, check the [OpenAI documentation](https://platform.openai.com/docs/guides/structured-outputs). | ||
|
|
||
| ### Limitations | ||
|
|
||
| While strict mode significantly improves schema adherence, keep these limitations in mind: | ||
|
|
||
| - Not all JSON Schema features are supported in strict mode. | ||
| - Complex schemas might still produce occasional errors. | ||
| - Always validate outputs even with strict mode enabled. | ||
| - Strict mode is OpenAI-specific and doesn't apply to other AI providers. | ||
|
|
||
| ## Next steps | ||
|
|
||
| - [Access data in AI functions](access-data-in-functions.md) | ||
| - [Execute a local .NET function](../quickstarts/use-function-calling.md) | ||
| - [Build a chat app](../quickstarts/build-chat-app.md) |
65 changes: 65 additions & 0 deletions
65
docs/ai/how-to/snippets/handle-invalid-tool-input/csharp/FunctionInvoker.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| using Microsoft.Extensions.AI; | ||
| using Microsoft.Extensions.Configuration; | ||
| using OpenAI; | ||
| using System.Text.Json; | ||
|
|
||
| class FunctionInvoker | ||
| { | ||
| static async Task Main() | ||
| { | ||
| IConfigurationRoot config = new ConfigurationBuilder().AddUserSecrets<FunctionInvoker>().Build(); | ||
| string? model = config["ModelName"]; | ||
| string? key = config["OpenAIKey"]; | ||
|
|
||
| // <BasicInvoker> | ||
| IChatClient chatClient = new OpenAIClient(key).GetChatClient(model ?? "gpt-4o").AsIChatClient(); | ||
|
|
||
| var functionInvokingClient = new FunctionInvokingChatClient(chatClient) | ||
| { | ||
| FunctionInvoker = async (context, cancellationToken) => | ||
| { | ||
| try | ||
| { | ||
| // Invoke the function normally. | ||
| return await context.Function.InvokeAsync(context.Arguments, cancellationToken); | ||
| } | ||
| catch (JsonException ex) | ||
| { | ||
| // Catch JSON serialization errors and provide helpful feedback. | ||
| return $"Error: Unable to parse the function arguments. {ex.Message}. " + | ||
| "Check the parameter types and try again."; | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| // Catch all other errors. | ||
| return $"Error: Function execution failed - {ex.Message}"; | ||
| } | ||
| } | ||
| }; | ||
| // </BasicInvoker> | ||
|
|
||
| IChatClient client = new ChatClientBuilder(functionInvokingClient).Build(); | ||
|
|
||
| var chatOptions = new ChatOptions | ||
| { | ||
| Tools = [AIFunctionFactory.Create((string location, int temperature) => | ||
| { | ||
| if (string.IsNullOrWhiteSpace(location)) | ||
| { | ||
| throw new ArgumentException("Location cannot be empty", nameof(location)); | ||
| } | ||
| return $"Recorded temperature of {temperature}°C for {location}"; | ||
| }, | ||
| "record_temperature", | ||
| "Records the temperature for a location")] | ||
| }; | ||
|
|
||
| List<ChatMessage> chatHistory = [ | ||
| new(ChatRole.System, "You are a weather data recorder."), | ||
| new(ChatRole.User, "Record the temperature as 25 degrees for London") | ||
| ]; | ||
|
|
||
| ChatResponse response = await client.GetResponseAsync(chatHistory, chatOptions); | ||
| Console.WriteLine($"Assistant >>> {response.Text}"); | ||
| } | ||
| } | ||
17 changes: 17 additions & 0 deletions
17
docs/ai/how-to/snippets/handle-invalid-tool-input/csharp/HandleInvalidToolInput.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <OutputType>Library</OutputType> | ||
| <TargetFramework>net10.0</TargetFramework> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="Microsoft.Extensions.AI" Version="10.1.1" /> | ||
| <PackageReference Include="Microsoft.Extensions.AI.OpenAI" Version="9.4.0-preview.1.25207.5" /> | ||
| <PackageReference Include="Microsoft.Extensions.Configuration" Version="10.0.1" /> | ||
| <PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" Version="10.0.1" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
49 changes: 49 additions & 0 deletions
49
docs/ai/how-to/snippets/handle-invalid-tool-input/csharp/IncludeDetailedErrors.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,49 @@ | ||||||
| using Microsoft.Extensions.AI; | ||||||
| using Microsoft.Extensions.Configuration; | ||||||
| using OpenAI; | ||||||
|
|
||||||
| class IncludeDetailedErrors | ||||||
| { | ||||||
| static async Task Main() | ||||||
| { | ||||||
| IConfigurationRoot config = new ConfigurationBuilder().AddUserSecrets<IncludeDetailedErrors>().Build(); | ||||||
| string? model = config["ModelName"]; | ||||||
| string? key = config["OpenAIKey"]; | ||||||
|
|
||||||
| IChatClient client = new ChatClientBuilder( | ||||||
| new OpenAIClient(key).GetChatClient(model ?? "gpt-4o").AsIChatClient()) | ||||||
| .UseFunctionInvocation() | ||||||
| .Build(); | ||||||
|
|
||||||
| // <BasicUsage> | ||||||
| // Enable detailed error messages to help the AI model self-correct. | ||||||
| FunctionInvokingChatClient? functionInvokingClient = client as FunctionInvokingChatClient; | ||||||
| functionInvokingClient?.IncludeDetailedErrors = true; | ||||||
| // </BasicUsage> | ||||||
|
|
||||||
| var chatOptions = new ChatOptions | ||||||
| { | ||||||
| Tools = [AIFunctionFactory.Create((int temperature, string location) => | ||||||
| { | ||||||
| // Validate temperature is in a reasonable range. | ||||||
| if (temperature < -50 || temperature > 50) | ||||||
| { | ||||||
| throw new ArgumentOutOfRangeException( | ||||||
| nameof(temperature), | ||||||
| "Temperature must be between -50 and 50 degrees Celsius."); | ||||||
| } | ||||||
| return $"The temperature in {location} is {temperature}°C"; | ||||||
| }, | ||||||
| "get_temperature", | ||||||
| "Gets the current temperature for a location")] | ||||||
| }; | ||||||
|
|
||||||
| List<ChatMessage> chatHistory = [ | ||||||
| new(ChatRole.System, "You are a helpful weather assistant."), | ||||||
| new(ChatRole.User, "What's the temperature in Paris?") | ||||||
|
||||||
| new(ChatRole.User, "What's the temperature in Paris?") | |
| new(ChatRole.User, "What's the temperature in Paris?") |
52 changes: 52 additions & 0 deletions
52
docs/ai/how-to/snippets/handle-invalid-tool-input/csharp/StrictSchema.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| using Microsoft.Extensions.AI; | ||
| using Microsoft.Extensions.Configuration; | ||
| using OpenAI; | ||
|
|
||
| class StrictSchema | ||
| { | ||
| static async Task Main() | ||
| { | ||
| IConfigurationRoot config = new ConfigurationBuilder().AddUserSecrets<StrictSchema>().Build(); | ||
| string? model = config["ModelName"]; | ||
| string? key = config["OpenAIKey"]; | ||
|
|
||
| // Ensure your model configuration uses a compatible version (e.g., gpt-4o-2024-08-06 or later). | ||
| IChatClient client = new ChatClientBuilder( | ||
| new OpenAIClient(key).GetChatClient(model ?? "gpt-4o").AsIChatClient()) | ||
| .UseFunctionInvocation() | ||
| .Build(); | ||
|
|
||
| // <StrictMode> | ||
| AIFunction weatherFunction = AIFunctionFactory.Create( | ||
| (string location, int temperature, bool isRaining) => | ||
| { | ||
| string weather = isRaining ? "rainy" : "clear"; | ||
| return $"Weather in {location}: {temperature}°C and {weather}"; | ||
| }, | ||
| new AIFunctionFactoryOptions | ||
| { | ||
| Name = "report_weather", | ||
| Description = "Reports the weather for a location with temperature and rain status", | ||
| // Enable strict mode by adding the Strict property (OpenAI only). | ||
| AdditionalProperties = new Dictionary<string, object?> | ||
| { | ||
| { "Strict", true } | ||
| } | ||
| } | ||
| ); | ||
| // </StrictMode> | ||
|
|
||
| var chatOptions = new ChatOptions | ||
| { | ||
| Tools = [weatherFunction] | ||
| }; | ||
|
|
||
| List<ChatMessage> chatHistory = [ | ||
| new(ChatRole.System, "You are a weather reporting assistant."), | ||
| new(ChatRole.User, "What's the weather in Tokyo? It's 22 degrees and not raining.") | ||
| ]; | ||
|
|
||
| ChatResponse response = await client.GetResponseAsync(chatHistory, chatOptions); | ||
| Console.WriteLine($"Assistant >>> {response.Text}"); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The indentation is inconsistent. Lines 46-54 should be properly indented to match the AIFunctionFactory.Create call structure, and line 59 should align with line 58.