Unlock VideoUrl for OpenAI Models

[brianalexander]

Description

https://ai.pydantic.dev/api/models/openai/#pydantic_ai.models.openai.OpenAIModel

                elif isinstance(item, VideoUrl):  # pragma: no cover
                    raise NotImplementedError('VideoUrl is not supported for OpenAI.')

I cannot use video reasoning with vLLM because of this.

Using a custom override for now

elif isinstance(item, VideoUrl):  # Updated to handle VideoUrl
                    # First, check if it's a local file path or a URL
                    import os
                    from urllib.parse import urlparse
                    
                    parsed = urlparse(item.url)
                    if parsed.scheme in ('http', 'https', 'file', ''):  # URL or local path
                        actual_file_path = item.url
                        if parsed.scheme == 'file':
                            actual_file_path = parsed.path
                        elif parsed.scheme in ('', 'file') and os.path.exists(item.url):
                            actual_file_path = item.url
                            
                        if os.path.exists(actual_file_path):
                            # It's a local file, read and encode as base64
                            with open(actual_file_path, 'rb') as video_file:
                                video_base64 = base64.b64encode(video_file.read()).decode('utf-8')
                            
                            # Determine the video format from the file extension
                            _, ext = os.path.splitext(actual_file_path)
                            mime_type = f'video/{ext[1:]}' if ext else 'video/mp4'
                            
                            # Create a data URI with the video content
                            video_data_uri = f'data:{mime_type};base64,{video_base64}'
                            
                            # Add to content as video_url part (the correct format for video)
                            content.append({
                                'type': 'video_url',
                                'video_url': {'url': video_data_uri}
                            })
                        else:
                            # It's a URL, try to download and handle it
                            downloaded_item = await download_item(item, data_format='base64_uri', type_format='extension')
                            content.append({
                                'type': 'video_url',
                                'video_url': {'url': item.url}
                            })
                    else:
                        # It's a URL, try to download and handle it
                        downloaded_item = await download_item(item, data_format='base64_uri', type_format='extension')
                        content.append({
                            'type': 'video_url',
                            'video_url': {'url': item.url}
                        })
                else:
                    assert_never(item)

vLLM command

python -m vllm.entrypoints.openai.api_server   --model cpatonn/Qwen3-VL-8B-Instruct-AWQ-4bit   --trust-remote-code   --max-model-len 16384

Relevant links:

• https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_client_for_multimodal.py
• https://ai.pydantic.dev/api/models/openai/#pydantic_ai.models.openai.OpenAIResponsesModel
• https://ai.pydantic.dev/input/#user-side-download-vs-direct-file-url

Thanks!

References

No response