Skip to content

Conversation

jerry-heygen
Copy link
Contributor

This PR fixes #2489.

Problem

When processing video content that lacks an audio track through Google's Gemini models, the metadata parsing fails with a KeyError. The API response includes an AUDIO modality entry in the metadata details, but without the token_count field, causing the code to crash when trying to access it.

This can be reproduced by adding the following line in pydantic_ai_slim/pydantic_ai/models/google.py#L606:

for key, metadata_details in metadata.items():
    if key.endswith('_details') and metadata_details:
        suffix = key.removesuffix('_details')
        for detail in metadata_details:
            details[f'{detail["modality"].lower()}_{suffix}'] = detail.get('token_count', 0)
            print("DEBUG: detail: ", detail)  # <-- Add this debug line

processing a video file without an audio track. Debug output shows:

DEBUG: detail:  {'modality': <MediaModality.TEXT: 'TEXT'>, 'token_count': 1201}
DEBUG: detail:  {'modality': <MediaModality.VIDEO: 'VIDEO'>, 'token_count': 2064}
DEBUG: detail:  {'modality': <MediaModality.TEXT: 'TEXT'>, 'token_count': 1717}
DEBUG: detail:  {'modality': <MediaModality.AUDIO: 'AUDIO'>}

Solution

Use .get('token_count', 0) instead of direct key access to safely handle missing token_count fields:

# Before (causes KeyError):
details[f'{detail["modality"].lower()}_{suffix}'] = detail['token_count']

# After (safe with default):
details[f'{detail["modality"].lower()}_{suffix}'] = detail.get('token_count', 0)

This ensures that modalities without token counts (like audio tracks in silent videos) default to 0 tokens rather than crashing the application.

Testing

  • Verified the fix resolves the issue with video files lacking audio tracks
  • All existing Google model tests pass without regression

@DouweM
Copy link
Collaborator

DouweM commented Aug 12, 2025

@jerry-heygen Thank you!

@DouweM DouweM merged commit e81ff69 into pydantic:main Aug 12, 2025
33 checks passed
ethanabrooks added a commit to reflectionai/pydantic-ai that referenced this pull request Aug 20, 2025
* Add `priority` `service_tier` to `OpenAIModelSettings` and respect it in `OpenAIResponsesModel` (pydantic#2368)

* Add an example of using RunContext to pass data among tools (pydantic#2316)

Co-authored-by: Douwe Maan <[email protected]>

* Rename gemini-2.5-flash-lite-preview-06-17 to gemini-2.5-flash-lite as it's out of preview (pydantic#2387)

* Fix toggleable toolset example so toolset state is not shared across agent runs (pydantic#2396)

* Support custom thinking tags specified on the model profile (pydantic#2364)

Co-authored-by: jescudero <[email protected]>
Co-authored-by: Douwe Maan <[email protected]>

* Add convenience functions to handle AG-UI requests with request-specific deps (pydantic#2397)

* docs: add missing optional packages in `install.md` (pydantic#2412)

* Include default values in tool arguments JSON schema (pydantic#2418)

* Fix "test_download_item_no_content_type test fails on macOS" (pydantic#2404)

* Allow string format, pattern and others in OpenAI strict JSON mode (pydantic#2420)

* Let more `BaseModel`s use OpenAI strict JSON mode by defaulting to `additionalProperties=False` (pydantic#2419)

* BREAKING CHANGE: Change type of 'source' field on EvaluationResult (pydantic#2388)

Co-authored-by: Douwe Maan <[email protected]>

* Fix ImageUrl, VideoUrl, AudioUrl and DocumentUrl not being serializable (pydantic#2422)

* BREAKING CHANGE: Support printing reasons in the console output for pydantic-evals (pydantic#2163)

* Document performance implications of async vs sync tools (pydantic#2298)

Co-authored-by: Douwe Maan <[email protected]>

* Mention that tools become toolset internally (pydantic#2395)

Co-authored-by: Douwe Maan <[email protected]>

* Fix tests for Logfire>=3.22.0 (pydantic#2346)

* tests: speed up the test suite (pydantic#2414)

* google: add more information about schema on union (pydantic#2426)

* typo in output docs (pydantic#2427)

* Deprecate `GeminiModel` in favor of `GoogleModel` (pydantic#2416)

* Use `httpx` on `GoogleProvider` (pydantic#2438)

* Remove older deprecated models and add new model of Anthropic (pydantic#2435)

* Remove `next()` method from `Graph` (pydantic#2440)

* BREAKING CHANGE: Remove `data` from `FinalResult` (pydantic#2443)

* BREAKING CHANGE: Remove `get_data` and `validate_structured_result` from `StreamedRunResult` (pydantic#2445)

* docs: add `griffe_warnings_deprecated` (pydantic#2444)

* BREAKING CHANGE: Remove `format_as_xml` module (pydantic#2446)

* BREAKING CHANGE: Remove `result_type` parameter and similar from `Agent` (pydantic#2441)

* Deprecate `GoogleGLAProvider` and `GoogleVertexProvider` (pydantic#2450)

* BREAKING CHANGE: drop 4 months old deprecation warnings (pydantic#2451)

* Automatically use OpenAI strict mode for strict-compatible native output types (pydantic#2447)

* Make `InlineDefsJsonSchemaTransformer` public (pydantic#2455)

* Send `ThinkingPart`s back to Anthropic used through Bedrock (pydantic#2454)

* Bump boto3 to support `AWS_BEARER_TOKEN_BEDROCK` API key env var (pydantic#2456)

* Add new Heroku models (pydantic#2459)

* Add `builtin_tools` to `Agent` (pydantic#2102)

Co-authored-by: Marcelo Trylesinski <[email protected]>
Co-authored-by: Douwe Maan <[email protected]>

* Bump mcp-run-python (pydantic#2470)

* Remove fail_under from top-level coverage config so <100% html-coverage step doesn't end CI run (pydantic#2475)

* Add AbstractAgent, WrapperAgent, Agent.event_stream_handler, Toolset.id, Agent.override(tools=...) in preparation for Temporal (pydantic#2458)

* Let toolsets be built dynamically based on run context (pydantic#2366)

Co-authored-by: Douwe Maan <[email protected]>

* Add ToolsetFunc to API docs (fix CI) (pydantic#2486)

* tests: change time of evals example (pydantic#2501)

* ci: remove html and xml reports (pydantic#2491)

* fix: Add gpt-5 models to reasoning model detection for temperature parameter handling (pydantic#2483)

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Douwe Maan <[email protected]>
Co-authored-by: Marcelo Trylesinski <[email protected]>

* History processor replaces message history (pydantic#2324)

Co-authored-by: Marcelo Trylesinski <[email protected]>

* ci: split test suite (pydantic#2436)

Co-authored-by: Douwe Maan <[email protected]>

* ci: use the right install command (pydantic#2506)

* Update config.yaml (pydantic#2514)

* Skip testing flaky evals example (pydantic#2518)

* Fix error when parsing usage details for video without audio track in Google models (pydantic#2507)

* Make OpenAIResponsesModelSettings.openai_builtin_tools work again (pydantic#2520)

* Let Agent be run in a Temporal workflow by moving model requests, tool calls, and MCP to Temporal activities (pydantic#2225)

* Install only dev in CI (pydantic#2523)

* Improve CLAUDE.md (pydantic#2524)

* Add best practices regarding to coverage to CLAUDE.md (pydantic#2527)

* Add support for `"openai-responses"` model inference string (pydantic#2528)

Co-authored-by: Claude <[email protected]>

* docs: Confident AI (pydantic#2529)

* chore: mention what to do with the documentation when deprecating a class (pydantic#2530)

* chore: drop hyperlint (pydantic#2531)

* ci: improve matrix readability (pydantic#2532)

* Add pip to dev deps for PyCharm (pydantic#2533)

Co-authored-by: Marcelo Trylesinski <[email protected]>

* Add genai-prices to dev deps and a basic test (pydantic#2537)

* Add `--durations=100` to all pytest calls in CI (pydantic#2534)

* Cleanup snapshot in test_evaluate_async_logfire (pydantic#2538)

* Make some minor tweaks to the temporal docs (pydantic#2522)

Co-authored-by: Douwe Maan <[email protected]>

* Add new OpenAI GPT-5 models (pydantic#2503)

* Fix `FallbackModel` to respect each model's model settings (pydantic#2540)

* Add support for OpenAI verbosity parameter in Responses API (pydantic#2493)

Co-authored-by: Claude <[email protected]>
Co-authored-by: Douwe Maan <[email protected]>

* Add `UsageLimits.count_tokens_before_request` using Gemini `count_tokens` API (pydantic#2137)

Co-authored-by: Douwe Maan <[email protected]>

* chore: Fix uv.lock (pydantic#2546)

* Stop calling MCP server `get_tools` ahead of `agent run` span (pydantic#2545)

* Disable instrumentation by default in tests (pydantic#2535)

Co-authored-by: Marcelo Trylesinski <[email protected]>

* Only wrap necessary parts of type aliases in forward annotations (pydantic#2548)

* Remove anthropic-beta default header set in `AnthropicModel` (pydantic#2544)

Co-authored-by: Marcelo Trylesinski <[email protected]>

* docs: Clarify why AG-UI example links are on localhost (pydantic#2549)

* chore: Fix path to agent class in CLAUDE.md (pydantic#2550)

* Ignore leading whitespace when streaming from Qwen or DeepSeek (pydantic#2554)

* Ask model to try again if it produced a response without text or tool calls, only thinking (pydantic#2556)

Co-authored-by: Douwe Maan <[email protected]>

* chore: Improve Temporal test to check trace as tree instead of list (pydantic#2559)

* Fix: Forward max_uses parameter to Anthropic WebSearchTool (pydantic#2561)

* Let message history end on ModelResponse and execute pending tool calls (pydantic#2562)

* Fix type issues

* skip tests requiring API keys

* add `google-genai` dependency

* add other provider deps

* add pragma: no cover for untested logic

---------

Co-authored-by: akenar <[email protected]>
Co-authored-by: Tony Woland <[email protected]>
Co-authored-by: Douwe Maan <[email protected]>
Co-authored-by: Yi-Chen Lin <[email protected]>
Co-authored-by: José I. Escudero <[email protected]>
Co-authored-by: jescudero <[email protected]>
Co-authored-by: Marcelo Trylesinski <[email protected]>
Co-authored-by: William Easton <[email protected]>
Co-authored-by: David Montague <[email protected]>
Co-authored-by: Guillermo <[email protected]>
Co-authored-by: Hamza Farhan <[email protected]>
Co-authored-by: Mohamed Amine Zghal <[email protected]>
Co-authored-by: Yinon Ehrlich <[email protected]>
Co-authored-by: Matthew Brandman <[email protected]>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Douwe Maan <[email protected]>
Co-authored-by: Alex Enrique <[email protected]>
Co-authored-by: Jerry Yan <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Mayank <[email protected]>
Co-authored-by: Alex Hall <[email protected]>
Co-authored-by: Jerry Lin <[email protected]>
Co-authored-by: Raymond Xu <[email protected]>
Co-authored-by: kauabh <[email protected]>
Co-authored-by: Victorien <[email protected]>
Co-authored-by: Ethan Brooks <[email protected]>
Co-authored-by: eballesteros <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

KeyError: 'token_count' when using Gemini with VideoUrl
2 participants