-
Notifications
You must be signed in to change notification settings - Fork 48
Open
Description
Context
This issue originates from a discussion about creating a "Popular Models" documentation page in pydantic-ai
The goal is to auto-generate this page from genai-prices data rather than maintaining it manually.
Key feedback from the discussion:
- Samuel suggested adding fields to
genai-pricesso the document can be generated from that data - For maintenance: include date updated
- Need to list all full
provider:modelstrings and link cross-provider models
Research: Industry Standards for LLM Model Metadata
We researched how other companies/libraries handle LLM model metadata. There is no universal industry standard, but several de facto patterns have emerged.
Sources Investigated
- OpenRouter API - Most comprehensive aggregator schema
- LiteLLM model_prices_and_context_window.json - Widely adopted pricing database
- LiteLLM Add Model Pricing Docs
- AWS Bedrock GetFoundationModel API
- Hugging Face Model Cards
- Anthropic List Models API
- OpenAI Models API
Findings: Common Field Names Across Sources
| Concept | OpenRouter | LiteLLM | Anthropic | Bedrock | HuggingFace |
|---|---|---|---|---|---|
| Identifier | id |
key name | id |
modelId |
repo name |
| Display name | name |
- | display_name |
modelName |
- |
| Context length | context_length |
max_input_tokens |
- | - | - |
| Input modalities | input_modalities |
supports_vision etc. |
- | inputModalities |
- |
| Output modalities | output_modalities |
- | - | outputModalities |
- |
| Provider | in ID | litellm_provider |
- | providerName |
- |
| Deprecation | - | deprecation_date |
- | modelLifecycle |
new_version |
| Mode/Task | modality |
mode |
- | - | pipeline_tag |
Key Gaps in Existing Standards
- No universal model identity - Same model has different IDs across providers
- Pricing not in APIs - Most providers don't expose pricing via API
- Capabilities vary - No standard for declaring what features a model supports
Decision: Align with LiteLLM Conventions
LiteLLM has the most widely adopted schema and already covers most edge cases. We'll align field naming where possible.
Fields to Add to ModelInfo
| Field | Type | Description | LiteLLM Equivalent |
|---|---|---|---|
max_output_tokens |
int | None |
Maximum output tokens (separate from context) | max_output_tokens |
modalities |
Modalities | None |
Input/output modality support | supports_vision, supports_audio_input, etc. |
capabilities |
Capabilities | None |
Feature flags (function calling, streaming, etc.) | supports_function_calling, supports_streaming, etc. |
release_date |
date | str | None |
Model release date (YYYY-MM or YYYY-MM-DD) | - (we add this) |
deprecation_date |
date | None |
When model becomes deprecated | deprecation_date |
canonical_model |
str | None |
Reference to canonical model for cross-provider entries | Similar to HF base_model |
Example Schema
# In anthropic.yml (canonical source)
- id: claude-opus-4-5
name: Claude Opus 4.5
description: Premium model combining maximum intelligence with practical performance
context_window: 200000
max_output_tokens: 32000
modalities:
input: [text, image]
output: [text]
capabilities:
function_calling: true
streaming: true
prompt_caching: true
release_date: 2025-11
prices:
input_mtok: 5
output_mtok: 25
# In aws.yml (cross-provider reference)
- id: us.anthropic.claude-opus-4-5-20251101-v1:0
canonical_model: anthropic:claude-opus-4-5
match:
contains: claude-opus-4-5
prices:
input_mtok: 5
output_mtok: 25What NOT to Add
- Performance ratings (Cost/Speed/Intelligence) - Subjective, belongs in pydantic-ai docs as editorial content
- Tokenizer info - Too technical for this use case
- Supported regions - Provider-specific, not model metadata
Next Steps
- Add new fields to
ModelInfoinprices/src/prices/prices_types.py - Update JSON schema generation
- Add
modalitiesandcapabilitiesto popular models in provider YAML files (start with Anthropic, OpenAI, Google, xAI) - Add
release_dateto popular models - Add
canonical_modelreferences to cross-provider entries (AWS Bedrock, Google Vertex AI) - Update
data.jsonoutput to include new fields - Document the new fields in CLAUDE.md or a schema doc
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels