Add spec parameter to agent.override() and agent.run()#4769
Add spec parameter to agent.override() and agent.run()#4769DouweM wants to merge 7 commits intocapabilitiesfrom
spec parameter to agent.override() and agent.run()#4769Conversation
Allows passing an AgentSpec (dict or object) at override time (full replacement semantics) and run time (additive merge semantics), enabling agent optimization workflows where candidate specs can be serialized, loaded, and tried without reconstructing the agent. Key changes: - Make AgentSpec.model optional so partial specs work - Add _resolve_spec() helper that validates spec, instantiates capabilities, and extracts contributions - override(spec=...): spec values as defaults, explicit params win, ContextVars for root_capability and builtin_tools - iter(spec=...): additive — model as fallback, instructions/metadata merged, capabilities combined via CombinedCapability - spec param threaded through all run methods, WrapperAgent, and durable agents (Temporal, DBOS, Prefect) - Unsupported spec fields (retries, end_strategy, etc.) warn at run/override time Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Docs Preview
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The durable agents (Temporal, DBOS, Prefect) explicitly re-declare run/run_sync/run_stream/run_stream_events/iter with overloads. Adding `spec` to AbstractAgent's signatures without also adding it to these overloads caused pyright "incompatible method override" errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| def _merged_meta(ctx: RunContext[AgentDepsT]) -> dict[str, Any]: | ||
| return {**(_spec_meta or {}), **metadata(ctx)} # type: ignore[operator] | ||
|
|
||
| metadata = _merged_meta |
There was a problem hiding this comment.
Devin correctly identified this: _merged_meta captures metadata by reference, and then metadata = _merged_meta on this line means the closure now recursively calls itself. The fix is to bind the original callable to a local variable before defining the closure:
_orig_metadata = metadata
def _merged_meta(ctx: RunContext[AgentDepsT]) -> dict[str, Any]:
return {**(_spec_meta or {}), **_orig_metadata(ctx)} # type: ignore[operator]
metadata = _merged_meta
| self, | ||
| spec: dict[str, Any] | AgentSpec | None, | ||
| custom_capability_types: Sequence[type[AbstractCapability[Any]]] = (), | ||
| ) -> _ResolvedSpec | None: |
There was a problem hiding this comment.
There's significant code duplication between _resolve_spec() and from_spec() — specifically the registry building, capability instantiation, and template context setup. The coding guidelines call for extracting duplicated logic into shared helpers after 2+ occurrences. Consider extracting the shared parts (registry building + _instantiate_cap + capability loading loop) into a helper that both from_spec() and _resolve_spec() can call.
|
|
||
| # Set capability and builtin_tools from spec | ||
| if resolved is not None and resolved.capability is not None: | ||
| cap_token = self._override_root_capability.set(_utils.Some(resolved.capability)) |
There was a problem hiding this comment.
When override(spec=...) provides capabilities, _override_root_capability is set to ONLY the spec's CombinedCapability, which completely replaces the agent's original _root_capability (see line 1130: base_capability = override_cap.value if override_cap is not None else self._root_capability). The PR description says override uses "defaults" semantics, but for capabilities this is a full replacement — an agent constructed with capabilities=[Thinking()] loses Thinking when override(spec={'capabilities': ['Instructions']}) is used.
By contrast, the iter() path at lines 1133-1134 merges additively via CombinedCapability([base_capability, resolved.capability]). The two code paths should have consistent semantics — probably the override should also combine the agent's base capability with the spec capability, rather than replacing.
@DouweM is the replacement behavior intentional for override(), or should this be additive like at run time?
| builtin_tools=[ | ||
| *self._builtin_tools, | ||
| *cap_builtin_tools, | ||
| *(override_bt.value if (override_bt := self._override_builtin_tools.get()) is not None else []), |
There was a problem hiding this comment.
When override(spec=...) provides capabilities, the builtin tools from those capabilities will be included twice:
cap_builtin_toolsat line 1230: extracted fromeffective_capabilityat line 1150, which is the override capability (sinceoverride_cap is not None)override_bt.valueat line 1231: set from the same override capability'sget_builtin_tools()at line 1677
The _override_builtin_tools context var seems redundant with the existing cap_builtin_tools extraction that already handles the override case at lines 1147-1153. The simplest fix would be to remove _override_builtin_tools entirely and rely on the capability-based extraction.
| infer_name: bool = True, | ||
| toolsets: Sequence[AbstractToolset[AgentDepsT]] | None = None, | ||
| builtin_tools: Sequence[AbstractBuiltinTool | BuiltinToolFunc[AgentDepsT]] | None = None, | ||
| spec: dict[str, Any] | AgentSpec | None = None, |
There was a problem hiding this comment.
PrefectAgent.iter() accepts spec but doesn't forward it in the super().iter() call at lines 840-854. This means spec is silently dropped for all PrefectAgent runs.
The same issue affects run() (line ~268, doesn't forward spec to super(WrapperAgent, self).run()), run_sync() (line ~403), run_stream() (line ~535), and run_stream_events() (line ~679). All accept spec in their signature but never pass it through.
Either forward spec=spec in all these super calls, or (if spec is intentionally not supported for durable agents) don't add the parameter and let it be handled through the **_deprecated_kwargs mechanism like DBOS/Temporal do.
|
|
||
| spec: AgentSpec | ||
| capability: CombinedCapability[Any] | None | ||
| instructions: list[Any] |
There was a problem hiding this comment.
instructions: list[Any] is too loose — the type should match what _instructions.normalize_instructions() returns. Per the coding guidelines, avoid Any type annotations; use the actual type for precision.
tests/test_capabilities.py
Outdated
| with agent.override(spec={'capabilities': []}, model='test'): | ||
| # Override with empty caps - just make sure it works | ||
| result = await agent.run('hello') | ||
| assert result.output is not None |
There was a problem hiding this comment.
Several tests in both TestOverrideWithSpec and TestRunWithSpec only assert result.output is not None (here, and at lines 2433, 2458, 2480, 2494). The test guidelines say to assert meaningful behavior, not just execution or type checks. At minimum, these should verify that the capabilities/spec actually took effect (e.g. check that instructions were applied, that the model was used, etc.), and ideally snapshot result.all_messages() to validate the complete execution trace.
The existing test_override_with_spec_instructions_and_model and test_run_with_spec_instructions_added are good examples of meaningful assertions — the other tests should follow that pattern.
|
|
||
| json_schema_path: str | None = Field(default=None, alias='$schema') | ||
| model: str | ||
| model: str | None = None |
There was a problem hiding this comment.
Making model optional changes the contract of AgentSpec — specs that were previously valid (always had a model) now allow None. While the from_spec() method has a runtime check (line 697-700), this is a breaking change for existing code that relies on spec.model being a str. Any serialized specs without a model field will now silently validate as None rather than failing at parse time.
Is this intentional? If model should only be optional at run/override time (not for from_spec()/from_file()), consider keeping model: str on AgentSpec and using a separate type or a model: str | None only in the _resolve_spec path.
| toolsets: Sequence[AbstractToolset[AgentDepsT]] | None = None, | ||
| builtin_tools: Sequence[AbstractBuiltinTool | BuiltinToolFunc[AgentDepsT]] | None = None, | ||
| event_stream_handler: EventStreamHandler[AgentDepsT] | None = None, | ||
| spec: dict[str, Any] | AgentSpec | None = None, |
There was a problem hiding this comment.
Same issue as flagged on PrefectAgent: spec is accepted as a parameter but never forwarded in the self.dbos_wrapped_run_workflow() call at lines 366-383. The same applies to run_sync(), run_stream(), run_stream_events(), iter(), and override() throughout this file and the Temporal agent file — spec is silently dropped in all cases.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…t class Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
| # Apply spec values as defaults where explicit params are not set | ||
| if resolved is not None: | ||
| if not _utils.is_set(name) and resolved.name is not None: | ||
| name = resolved.name | ||
| if not _utils.is_set(model) and resolved.model is not None: | ||
| model = resolved.model | ||
| if not _utils.is_set(instructions) and resolved.instructions: | ||
| instructions = resolved.instructions | ||
| if not _utils.is_set(model_settings) and resolved.model_settings is not None: | ||
| model_settings = resolved.model_settings | ||
| if not _utils.is_set(metadata) and resolved.metadata is not None: | ||
| metadata = resolved.metadata |
There was a problem hiding this comment.
🚩 Semantic difference between iter(spec=...) and override(spec=...)
There's a notable design asymmetry in how spec is handled:
- In
iter()(pydantic_ai_slim/pydantic_ai/agent/__init__.py:1041-1048): instructions from spec are additive — they extend the existing instructions. - In
override()(pydantic_ai_slim/pydantic_ai/agent/__init__.py:1625-1626): instructions from spec replace the agent's instructions (they're only applied if the explicitinstructionsparam is unset, but when applied, they become the sole instructions override).
This is a meaningful behavioral difference that could confuse users. The same asymmetry applies to model settings and metadata. This may be intentional (override = replace, run = extend), but it's worth documenting explicitly.
Was this helpful? React with 👍 or 👎 to provide feedback.
- Fix infinite recursion when merging callable metadata with spec metadata (bind original callable before closure) - Remove _override_builtin_tools ContextVar (was duplicating builtin tools from capabilities already extracted via cap_builtin_tools) - Make override(spec=...) combine capabilities additively with agent's root capability instead of replacing it - Fix _ResolvedSpec.instructions type from list[Any] to proper type - Forward spec=spec in all durable agent super() calls (Temporal, DBOS, Prefect) so spec is not silently dropped - Strengthen tests to assert meaningful behavior instead of just `is not None` Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…pec/resolve_spec Both from_spec() and _resolve_spec() had duplicated logic for building the capability registry, defining _instantiate_cap, and loading capabilities from the registry. Extracted into a shared module-level helper function. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
override(spec=...) should replace the agent's root capability, not combine with it — that's the distinction from iter(spec=...) which is additive. Updated test to verify replacement behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
| metadata=metadata, | ||
| infer_name=infer_name, | ||
| toolsets=toolsets, | ||
| spec=spec, | ||
| ) as run: |
There was a problem hiding this comment.
🔴 PrefectAgent.iter() silently drops builtin_tools parameter
The PrefectAgent.iter() method accepts builtin_tools in its signature (line 781) but does not pass it through to super().iter() in the call at lines 866–881. This means any builtin_tools provided to PrefectAgent.iter() are silently ignored. This is a pre-existing bug, but the PR modified this exact call site (adding spec=spec) while following the same pattern of forwarding all parameters — making it a directly related omission. For comparison, DBOSAgent.iter() (pydantic_ai_slim/pydantic_ai/durable_exec/dbos/_agent.py:961) and TemporalAgent.iter() (pydantic_ai_slim/pydantic_ai/durable_exec/temporal/_agent.py:1000) both correctly pass builtin_tools=builtin_tools.
(Refers to lines 866-881)
Was this helpful? React with 👍 or 👎 to provide feedback.
| _unsupported_fields = { | ||
| 'end_strategy': 'early', | ||
| 'retries': 1, | ||
| 'output_retries': None, | ||
| 'tool_timeout': None, | ||
| 'output_schema': None, | ||
| 'deps_schema': None, | ||
| } |
There was a problem hiding this comment.
🚩 Spec fields description and instrument are silently ignored at run/override time
The _resolve_spec() method (pydantic_ai_slim/pydantic_ai/agent/__init__.py:1468-1525) captures model, name, instructions, model_settings, metadata, and capability from the spec into _ResolvedSpec. However, the description and instrument fields from AgentSpec are neither captured nor included in the _unsupported_fields warning dict (lines 1496-1503). If a user passes spec={'description': 'foo', 'instrument': True} at run/override time, these values are silently ignored without any warning. Other unsupported fields like end_strategy, retries, etc. correctly produce UserWarning. This inconsistency may confuse users.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
spec: dict[str, Any] | AgentSpec | Noneparameter toagent.override()and allagent.run*()/agent.iter()methodsCombinedCapabilityAgentSpec.modeloptional (str | None = None) so partial specs work at run/override time_resolve_spec()helper sharing capability instantiation logic withfrom_spec()retries,end_strategy, etc.) at run/override timespecthroughWrapperAgentand durable agents (Temporal, DBOS, Prefect)Test plan
test_override_with_spec_instructions_and_model— spec instructions replace agent's via overridetest_override_with_spec_explicit_param_wins— explicit override param beats spec valuetest_override_with_spec_capabilities— override with spec capabilities workstest_run_with_spec_instructions_added— spec instructions added additively at run timetest_run_with_spec_model_as_fallback— spec model used when agent has nonetest_run_with_spec_model_settings_merged— spec model_settings merged with runtest_run_with_spec_partial_no_model— partial spec without model workstest_run_with_spec_capabilities— combined root_capability from spec + agenttest_run_with_spec_metadata_merged— metadata merge with precedencetest_spec_unsupported_fields_warns— warning for non-default unsupported fields🤖 Generated with Claude Code