YoutubeAudioLoader and OpenAIWhisperParser - can they also return timestamps

[srinisubramanian]

The YoutubeAudioLoader and OpenAIWhisperParser together return the transcipt text for a Youtube video link.

Is there any way to also get the timestamps for the transcript? In order to best work with the video the timestamps are also needed

[dosubot[bot]]

🤖

Hello,

Yes, it is possible to modify the YoutubeAudioLoader and OpenAIWhisperParser to return timestamps along with the transcript text for a Youtube video link. The youtube_transcript_api package used in the YoutubeLoader and GoogleApiYoutubeLoader classes fetches the transcript in a list of dictionaries, where each dictionary represents a segment of the transcript and contains the text, start time, and duration.

However, in the current implementation, the start time and duration are not included in the final transcript. The line transcript = " ".join([t["text"].strip(" ") for t in transcript_pieces]) only extracts the text from each segment and joins them together.

To include the timestamps, you can modify this line to include the start time and duration. Here's an example of how you can do this:

transcript = [(t["start"], t["duration"], t["text"]) for t in transcript_pieces]

This will return a list of tuples, where each tuple contains the start time, duration, and text of a segment. If you want to format this in a different way, you can adjust the code accordingly.

Please note that this change will affect the output format of the load method in the YoutubeLoader and GoogleApiYoutubeLoader classes, so you may need to adjust the rest of your code to handle this new format.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/document_loaders/youtube.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[srinisubramanian]

@dosu-bot I am not using the transcript API but rather the AudioLoader which passes the audio to The WhisperAPI and recieves the transcript from Whisper. Where are the transcript pieces handled in this case?

[dosubot[bot]]

🤖

Hello,

Thank you for your question. Currently, the OpenAI Whisper API does not support returning timestamps along with the transcript text. Therefore, it is not possible to modify the OpenAIWhisperParser class to return timestamps along with the transcript text when using the Whisper API.

However, if OpenAI updates their Whisper API to support timestamps in the future, you would likely need to modify the lazy_parse method in the OpenAIWhisperParser class to include the timestamps in the Document object that it yields.

Here is a hypothetical example of how you might modify the lazy_parse method:

# Transcribe
print(f"Transcribing part {split_number+1}!")
attempts = 0
while attempts < 3:
    try:
        transcript = openai.Audio.transcribe("whisper-1", file_obj)
        break
    except Exception as e:
        attempts += 1
        print(f"Attempt {attempts} failed. Exception: {str(e)}")
        time.sleep(5)
else:
    print("Failed to transcribe after 3 attempts.")
    continue

for timestamp, text in transcript:
    yield Document(
        page_content=text,
        metadata={"source": blob.source, "chunk": split_number, "timestamp": timestamp},
    )

Please note that this is a hypothetical example and the actual implementation would depend on how the Whisper API returns the timestamps.

As for the YoutubeAudioLoader class, it is a subclass of the BlobLoader class and is designed to load YouTube URLs as audio files. The class has two methods: __init__ and yield_blobs. The __init__ method initializes the class with a list of URLs and a directory to save the downloaded audio files. The yield_blobs method yields audio blobs for each URL. It uses the yt_dlp package to download audio from a YouTube URL, saves the audio file in the specified directory, and then yields the written blobs.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/document_loaders/parsers/audio.py
• libs/langchain/langchain/document_loaders/blob_loaders/youtube_audio.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[srinisubramanian]

@dosu-bot. Looks like when Whisper is run locally, the segments have timestamps

[dosubot[bot]]

🤖

Hello,

Based on the current implementation of the OpenAIWhisperParser class in the LangChain framework, it seems that timestamps are not currently extracted and returned along with the transcript text. The Document object that is yielded by the OpenAIWhisperParser class only includes the transcribed text and some metadata such as the source of the blob and the chunk number.

However, if the Whisper API returns timestamps when run locally, you can modify the OpenAIWhisperParser class to include these timestamps in the Document object. Here's a rough idea of how you might do this:

class OpenAIWhisperParser(BaseParser):
    # ... existing code ...

    def parse(self, blob: Blob) -> Iterable[Document]:
        # ... existing code ...

        transcript = openai.Audio.transcribe("whisper-1", file_obj)

        yield Document(
            page_content=transcript.text,
            metadata={
                "source": blob.source,
                "chunk": split_number,
                "timestamps": transcript.timestamps,  # new line
            },
        )

In this code, transcript.timestamps is assumed to be the timestamps returned by the Whisper API. Please replace it with the actual attribute or method that returns the timestamps.

Please note that this is a rough idea and the actual implementation may vary depending on the structure of the response object returned by the openai.Audio.transcribe method when the Whisper API is run locally. Unfortunately, I couldn't find the exact structure of this response object in the provided context.

I hope this helps! If you have any further questions, please let me know.

Sources

• libs/langchain/langchain/chat_models/openai.py
• libs/langchain/langchain/document_loaders/blob_loaders/youtube_audio.py
• libs/langchain/langchain/document_loaders/parsers/audio.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.