Support native JSON output and strict tool calls for Anthropic #3457

dsfaccini · 2025-11-17T19:18:05Z

Fixes #3428

work in progress:

check CI tests and coverage
double check code to clarify

…c model - addresses pydantic#3428

docs/models/anthropic.md

docs/output.md

pydantic_ai_slim/pydantic_ai/models/anthropic.py

DouweM · 2025-11-17T21:26:59Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+        tools, strict_tools_requested = self._get_tools(model_request_parameters, model_settings)
        tools, mcp_servers, beta_features = self._add_builtin_tools(tools, model_request_parameters)
+        output_format = self._build_output_format(model_request_parameters)
+        structured_output_beta_required = strict_tools_requested or bool(output_format)


We could simplify the code below by adding a value to the beta_headers list here right?

pydantic_ai_slim/pydantic_ai/models/anthropic.py

pydantic_ai_slim/pydantic_ai/profiles/anthropic.py

tests/models/test_anthropic.py

DouweM · 2025-11-17T21:51:15Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

            'input_schema': f.parameters_json_schema,
        }
+        if f.strict is not None:
+            tool_param['strict'] = f.strict  # type: ignore[assignment]


Look at how the OpenAI model uses is_strict_compatible. If the user didn't explicitly say strict=True on their tool, it'll be strict=None, so we check if the schema is strict-compatible, and if so we set strict=True.

So to get the same behavior with Anthropic, we should check if the schema can be transformed successfully (losslessly), and set strict=True unless it's explicitly strict=False

… with cassette

- Resolved conflicts in anthropic.py by keeping our strict tools and native JSON implementation - Resolved conflicts in test_anthropic.py by merging strict=True with TTL='5m' in assertions - Kept our test_anthropic_mixed_strict_tool_run test - Added upstream's new features: - count_tokens() method for token counting - TTL support for cache control (5m and 1h) - New helper methods (_infer_tool_choice, _map_extra_headers) - test_anthropic_cache_with_custom_ttl test

- makes schema tests clearer by showing original and transofrmed - handles multiple beta-heards (wasn't hadnled previously) - unifies betas logic around sdk's betas parameter

pydantic_ai_slim/pydantic_ai/models/anthropic.py

DouweM · 2025-11-20T22:08:29Z

pydantic_ai_slim/pydantic_ai/profiles/anthropic.py

+        all_of = node.pop('allOf', None)
+
+        node.pop('description', None)
+        node.pop('title', None)


Why are we doing all of these pops? And why are we modifying the schema in place if the method sounds like it just does a check?

DouweM · 2025-11-20T22:10:50Z

pydantic_ai_slim/pydantic_ai/profiles/anthropic.py

+            # check compatibility before calling anthropic's transformer
+            # so we don't auto-enable strict when the SDK would drop constraints
+            self.is_strict_compatible = False
+        transformed = transform_schema(schema)


If you look at how OpenAiJsonSchemaTransformer uses is_strict_compatible, you'll see that it depends on self.strict when it find something in compatible with strict mode:

If self.strict is True, it will modify the schema to somehow make it work, even if this is lossy

If self.strict is None, it will NOT modify the schema and set is_strict_compatible = False

If self.strict is False, it won't do anything

I don't think that's the behavior we currently have, as we also transform the schema, not just when self.strict is True

By the way I think that means that for tool defs, we should respect their own strict property, as it's valid to a send non-strict-compatible schema.

But for native output, I think we should always force strict=True into this validator (by setting it on the output_object?) because Anthropic requires the schema to be strict.

With OpenAI it's different as their "native" json schema output mode also allows strict=False.

DouweM · 2025-11-20T22:15:36Z

tests/CLAUDE.md

+
+### about static typing
+
+- other codebases don't use types in their test files


Let's move claude changes to a separate PR so we can be a bit more nitpicky on teh details of what we teach the LLM. For example, I don't think it's necessary/helpful to claim this as if it's a fact :)

Please move to a new PR!

tests/models/anthropic/conftest.py

Co-authored-by: Douwe Maan <[email protected]>

- set strict=True when output_mode=True

DouweM · 2025-11-21T17:40:13Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

            ):  # pragma: no branch
                # This would result in `tool_choice=required`, which Anthropic does not support with thinking.
                raise UserError(
                    'Anthropic does not support thinking and output tools at the same time. Use `output_type=PromptedOutput(...)` instead.'


The mode we recommend should be dynamic based on supports_json_schema_output; see Google where we do the same thing

DouweM · 2025-11-21T17:41:26Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+
+        if (
+            model_request_parameters.output_mode == 'native' and model_request_parameters.output_object is not None
+        ):  # pragma: no branch


This pragma: no branch is weird, as it means we never get here with an output mode other than native. There should definitely be cases where we get here with tool or prompted

DouweM · 2025-11-21T17:42:51Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+
        try:
-            extra_headers = self._map_extra_headers(beta_features, model_settings)
+            betas_list, extra_headers = self._prepare_betas_and_headers(betas, model_settings)


Having both betas and betas_list is a bit weird. I'd rather have just betas as a set, and then turn it into a list when we pass it into the method below

DouweM · 2025-11-21T17:44:18Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

                tools=tools or OMIT,
                tool_choice=tool_choice or OMIT,
                mcp_servers=mcp_servers or OMIT,
+                betas=betas_list or OMIT,


Weird that we don't have to pass output_format here, as it does contribute to token usage. Can you make an explicit comment about that, so it doesn't look like an oversight?

DouweM · 2025-11-21T17:48:04Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+            tool_def.strict for tool_def in model_request_parameters.tool_defs.values()
+        )
+
+        if has_strict_tools or model_request_parameters.output_mode == 'native':


I think there's a scenario where we can send a tool def with strict=True, without also sending the structured output beta: if ToolDefinition.strict is None (by default), has_strict_tools will be False, but customize_request_parameters will set strict=schema_transformer.is_strict_compatible, which maybe True.

So we should really add this beta depending on the result of _get_tools/_map_tool_definition, not the original ToolDefinitions.

That means we also don't need to check self.profile.supports_json_schema_output here anymore, as the tool dicts only get strict=True if that value is enabled

DouweM · 2025-11-21T17:57:24Z

pydantic_ai_slim/pydantic_ai/profiles/anthropic.py

+            transformed = transform_schema(schema)
+            if before != transformed:
+                self.is_strict_compatible = False
+            return transformed


Shouldn't we return the unmodified before if self.is_strict_compatible is False?

DouweM · 2025-11-21T17:58:38Z

tests/CLAUDE.md

+
+### about static typing
+
+- other codebases don't use types in their test files


Please move to a new PR!

DouweM · 2025-11-21T18:02:45Z

tests/models/anthropic/test_output.py

+    result = await agent.run('What is the capital of the user country?')
+    # Should return CityLocation since we asked about capital
+    assert isinstance(result.output, city_location_schema | country_language_schema)
+    if isinstance(result.output, city_location_schema):  # pragma: no branch


We shouldn't need this check if we make the assert explicitly verify city_location_schema

DouweM · 2025-11-21T18:07:04Z

tests/profiles/test_anthropic.py

+
+
+def test_lossless_simple_model():
+    """Simple BaseModel with basic types should be lossless."""


This test doesn't test anything anymore, as passing strict=True will always result in is_strict_compatible is True. I want to make sure that passing basic models like this with strict=None will result in is_strict_compatible is True, so that we use strict as much as possible, except in specific cases where the user is really doing something incompatible.

I'm a little worried that transform_schema is going to return something slightly different than the input even in cases where the conversion is lossless, and we end up incorrectly having is_strict_compatible is False and thus not passing strict=True, even though we could have.

For example, turning {type: 'string', nullable: true} into type: ['string', 'null'] would be a lossless conversion, but our logic would treat it as strict-incompatible because it's not identical. This example may not apply to transform_schema, but you get the idea: inequality does not always mean lossiness. So we should have a bunch of tests for standard scenarios to ensure it works as expected.

I believe the openai tests have a big parameterized test for testing how strict ends up already.

DouweM · 2025-11-21T18:08:45Z

tests/profiles/test_anthropic.py

+    assert transformer.is_strict_compatible is False
+
+
+def test_lossy_nested_defs():


We're now testing a lot of things that we know require lossy transformation, but I think it'll be more valuable to test for things that should not require lossy transformation (i.e. no transformation at all, or lossless transformation) and ensure that those get is_strict_compatible is True (+ possibly a transformed schema).

dsfaccini added 2 commits November 17, 2025 14:10

adds native json output and strict tool call support for the anthropi…

b26f93b

…c model - addresses pydantic#3428

update tests to use new transformer

04a9b3b

DouweM requested changes Nov 17, 2025

View reviewed changes

DouweM self-assigned this Nov 17, 2025

DouweM added the awaiting author revision label Nov 17, 2025

dsfaccini added 4 commits November 18, 2025 18:39

restore docs and bump anthropic sdk and simplify model - add new test…

b5243d0

… with cassette

validate strict compatibility

4a51289

Merge branch 'main' into anthropic-native-json

41598ec

Anas20001 mentioned this pull request Nov 19, 2025

feat(anthropic): Support JSON schema transformer and strict tool calls #3471

Closed

dsfaccini added 9 commits November 19, 2025 17:34

add model-based support and add tests

1578993

update snapshots for coverage

0b27ecf

rerun anthropic tests against api

eb6edc6

updated respective cassette

2446e6f

add tests

c2aa94f

check compatibility for strict tool defs

dea5f0f

Merge branch 'main' into anthropic-native-json

e3b67f7

- adds pragmas to transformer

cbcb783

- makes schema tests clearer by showing original and transofrmed - handles multiple beta-heards (wasn't hadnled previously) - unifies betas logic around sdk's betas parameter

fix coverage and fix cassette

b25da57

DouweM changed the title ~~Support native json output and strict tool calls for anthropic~~ Support native JSON output and strict tool calls for Anthropic Nov 20, 2025

DouweM requested changes Nov 20, 2025

View reviewed changes

dsfaccini and others added 6 commits November 20, 2025 17:55

beautify beta header merging

8f278ed

Co-authored-by: Douwe Maan <[email protected]>

coverage

9ab40a3

apply review changes

8aa4d59

remove transform

1ea365b

- don't transform for strict=False

d2eecd6

- set strict=True when output_mode=True

Merge branch 'main' into anthropic-native-json

5011da8

DouweM requested changes Nov 21, 2025

View reviewed changes


		### about static typing

		- other codebases don't use types in their test files



		def test_lossless_simple_model():
		"""Simple BaseModel with basic types should be lossless."""

		assert transformer.is_strict_compatible is False


		def test_lossy_nested_defs():

Support native JSON output and strict tool calls for Anthropic #3457

Are you sure you want to change the base?

Support native JSON output and strict tool calls for Anthropic #3457

Conversation

dsfaccini commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dsfaccini commented Nov 17, 2025 •

edited

Loading