FEAT: Adding audio and tool support to chat completions #1311
Conversation
|
|
||
|
|
||
| @dataclass | ||
| class OpenAICompletionsAudioConfig: |
There was a problem hiding this comment.
optional NIT: do we want to consider renaming to OpenAIChatAudioConfig to correspond to our OpenAIChatTarget and make it clear that it's not OpenAICompletionTarget
There was a problem hiding this comment.
I disagree, this is neither "nit" nor optional 😆
|
|
||
| # Voices supported by OpenAI Chat Completions API audio output. | ||
| # See: https://platform.openai.com/docs/guides/text-to-speech#voice-options | ||
| CompletionsAudioVoice = Literal["alloy", "ash", "ballad", "coral", "echo", "sage", "shimmer", "verse", "marin", "cedar"] |
There was a problem hiding this comment.
curious why this isn't exactly the same as the list on the platform.openai.com webpage that's linked above? (missing fable, nova, onyx)
| extension=extension, | ||
| ) | ||
|
|
||
| if audio_format == "pcm16": |
There was a problem hiding this comment.
might be missing something, but is there a unit test for pcm16 specifically?
|
|
||
| # Skip audio for assistant messages - OpenAI only allows audio in user messages. | ||
| # For assistant responses, the transcript text piece should already be included. | ||
| if role == "assistant": |
There was a problem hiding this comment.
is assistant the only other option besides user ?
| if not pieces: | ||
| raise EmptyResponseException(message="Failed to extract any response content.") | ||
|
|
||
| return Message(message_pieces=pieces) |
There was a problem hiding this comment.
so right now you wouldn't be able to tell what is the transcript text and what is just text content, so hypothetically, there could be no transcipt but there could be text content and there would be no distinction. Do you see a distinction being useful (I'm not sure whether the content / value of text content vs transcript makes it obvious which is which so that being more explicit is unnecessary) ?
| content.append(entry) | ||
| elif message_piece.converted_value_data_type == "audio_path": | ||
| ext = DataTypeSerializer.get_extension(message_piece.converted_value) | ||
| if not ext or ext.lower() not in [".wav", ".mp3"]: |
There was a problem hiding this comment.
https://platform.openai.com/docs/guides/speech-to-text says "mp3, mp4, mpeg, mpga, m4a, wav, and webm" so is this just that pyrit + openai chat completions only supports .wav & .mp3 ? because then we should maybe more exact
4 participants
Title says it all! Supporting audio for gpt-audio and also tool calls.
Tests: