Transcribe MP3 as Blob?

[dmb0058]

I can't find any documentation on how to do this but it looks like it should be possible from the source code.

I have an AWS Lambda component that receives an MP3 stream in the event body, base64 encoded, as part of a multipart request. I've extracted the mp3 data using AWS multipart parse() which returns the data in a Buffer, and verified that it's complete and valid. If I save it to an S3 bucket I can download and play it.

The langchain OpenAIWhisperAudio constructor looks like it can take a filePath or blob. I don't want to save the mp3 to a local file and then pass the file path if I can just send a blob, whatever that is. I tried

const { Blob } " require('node:buffer');

and then

let loader=new OpenAIWhisperAudio(new Blob(data, { type: 'audio/mp3' })); loader.load();

but while the object looks OK, the load() function returns an 'Unsupported file type: supported file types are mp3, mpeg, m4a, ...' error.

Any info on what kind of blob I need to pass?

[dosubot[bot]]

🤖

Hi @dmb0058, good to see you again!

From your description, it seems like you're on the right track. The OpenAIWhisperAudio constructor in the langchainjs framework does indeed accept a Blob, which in Node.js is represented as a Buffer.

Here's how you can convert your base64 encoded MP3 data to a Buffer and pass it to the constructor:

import { OpenAIWhisperAudio } from 'langchain/src/document_loaders/fs/openai_whisper_audio.ts';

// Assuming `event` is your AWS Lambda event containing the base64 encoded MP3
const base64MP3 = event.body; // replace with actual path to base64 MP3 data in your event

// Convert base64 string to Buffer
const mp3Buffer = Buffer.from(base64MP3, 'base64');

// Create a new instance of OpenAIWhisperAudio with the Buffer
const whisperAudio = new OpenAIWhisperAudio(mp3Buffer, {
  clientOptions: {
    // your OpenAI client options here
  }
});

// Now you can use the `whisperAudio` instance to interact with the OpenAI Whisper API

This should work because the OpenAIWhisperAudio constructor accepts a Blob, which in Node.js is represented as a Buffer. The Buffer.from method is used to convert the base64 encoded string to a Buffer. The Buffer is then passed to the OpenAIWhisperAudio constructor.

You can find more details about the OpenAIWhisperAudio constructor in the langchain/src/document_loaders/fs/openai_whisper_audio.ts file.

I hope this helps! Let me know if you have any other questions.

Sources

• langchain/src/document_loaders/fs/openai_whisper_audio.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot[bot]]

🤖

Based on the information you've provided and the context of the langchainjs framework, it seems like the OpenAIWhisperAudio constructor is expecting a Blob object or a file path as its first argument. The error you're encountering, "this.filePathOrBlob.arrayBuffer is not a function", suggests that the constructor is receiving an argument that doesn't have an arrayBuffer method, which is a method of Blob objects.

The Blob object you're creating from the Buffer should theoretically work, but it seems like there might be an issue with the Blob implementation in the environment you're running your code in. The Blob implementation in Node.js might differ from the one in browsers, which could be causing this issue.

Here's a potential solution using the blob package, which provides a browser-like Blob implementation for Node.js:

import Blob from 'blob';
import { Buffer } from 'buffer';

// Convert base64 string to Buffer
const mp3Buffer = Buffer.from(base64MP3, 'base64');

// Convert Buffer to Blob
const mp3Blob = new Blob([mp3Buffer], { type: 'audio/mpeg' });

// Create a new instance of OpenAIWhisperAudio with the Blob
const whisperAudio = new OpenAIWhisperAudio(mp3Blob, {
  clientOptions: {
    // your OpenAI client options here
  }
});

This code creates a Blob object from the Buffer using the blob package, and then passes that Blob to the OpenAIWhisperAudio constructor.

If this doesn't solve your issue, it would be helpful to know more about the environment you're running your code in. Are you using a specific version of Node.js or a specific AWS Lambda runtime? This could help identify any potential compatibility issues.

Sources

• langchain/src/document_loaders/fs/openai_whisper_audio.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dmb0058]

@dosu-bot No, that gives "Error: Blob is not a constructor"

[dosubot[bot]]

🤖

The error "Blob is not a constructor" is likely due to the fact that the Blob object is not available in a Node.js environment. Blob objects are a browser-based feature and are not natively supported in Node.js.

However, the good news is that the OpenAIWhisperAudio constructor in the langchainjs framework does accept Buffer objects. The constructor takes a parameter 'filePathOrBlob' which can be either a string or a Blob object. However, the 'parse' method of the class uses this parameter as a Buffer object. This indicates that the constructor can handle both file paths (as strings) and Blob objects, but ultimately treats them as Buffer objects for processing.

Here is the relevant code:

constructor(
  filePathOrBlob: string | Blob,
  fields?: {
    clientOptions?: ClientOptions;
  }
) {
  super(filePathOrBlob);
  this.openAIClient = new OpenAIClient(fields?.clientOptions);
}

protected async parse(
  raw: Buffer,
  metadata: Record<string, string>
): Promise<Document[]> {
  const fileName =
    metadata.source === "blob" ? metadata.blobType : metadata.source;
  const transcriptionResponse =
    await this.openAIClient.audio.transcriptions.create({
      file: await toFile(raw, fileName),
      model: MODEL_NAME,
    });
  const document = new Document({
    pageContent: transcriptionResponse.text,
    metadata,
  });
  return [document];
}

In the 'parse' method, 'raw' is used as a Buffer object, which is passed to the 'toFile' function. This 'raw' is the 'filePathOrBlob' that was passed to the constructor.

So, you can directly pass the Buffer object to the OpenAIWhisperAudio constructor. Here's how you can modify your code:

import { Buffer } from 'buffer';
import { OpenAIWhisperAudio } from 'langchainjs';

// Convert base64 string to Buffer
const mp3Buffer = Buffer.from(base64MP3, 'base64');

// Create a new instance of OpenAIWhisperAudio with the Buffer
const whisperAudio = new OpenAIWhisperAudio(mp3Buffer, {
  clientOptions: {
    // your OpenAI client options here
  }
});

This should resolve the issue you're facing. If you encounter any other problems, feel free to ask.

Sources

• examples/src/document_loaders/openai_whisper_audio.ts
• langchain/src/document_loaders/fs/openai_whisper_audio.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dmb0058]

@dosu-bot It is simply untrue that "... the OpenAIWhisperAudio constructor in the langchainjs framework does accept Buffer objects". The error returned from the OpenAIWhisperAudio constructor is "Error: this.filePathOrBlob.arrayBuffer is not a function"

[dosubot[bot]]

🤖

The error you're encountering is likely due to the fact that the OpenAIWhisperAudio constructor in the langchainjs framework expects a string or a Blob object as the first argument, not a Node.js Buffer object. Blob objects have a method called arrayBuffer(), which is probably what the constructor is trying to call, leading to the error message you're seeing.

To resolve this issue, you can convert your Buffer object to a Blob object before passing it to the OpenAIWhisperAudio constructor. However, Blob objects are not natively supported in Node.js, so you'll need to use a package like blob-polyfill or buffer-to-blob.

Here's an example of how you can do this using the buffer-to-blob package:

import bufferToBlob from 'buffer-to-blob';
import { OpenAIWhisperAudio } from 'langchain/src/document_loaders/fs/openai_whisper_audio.ts';

// Assuming `buffer` is your Buffer object
const blob = bufferToBlob(buffer);
const openAIWhisperAudio = new OpenAIWhisperAudio(blob);

This will convert your Buffer object to a Blob object, which can then be used with the OpenAIWhisperAudio constructor without causing the "this.filePathOrBlob.arrayBuffer is not a function" error.

As for the OpenAIWhisperAudio constructor, it has the following signature:

constructor(
  filePathOrBlob: string | Blob,
  fields?: {
    clientOptions?: ClientOptions;
  }
)

The filePathOrBlob parameter can be either a string or a Blob. If it's a string, it's expected to be the file path of the audio file. If it's a Blob, it's expected to be the audio data. The fields parameter is optional and, if provided, should be an object that may contain a clientOptions property, which is used to configure the OpenAI client.

You can find more information about this in the langchain/src/document_loaders/fs/openai_whisper_audio.ts file in the langchainjs repository.

Sources

• langchain/src/document_loaders/fs/openai_whisper_audio.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.