extracting Document from a buffer

[mieslep]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Allow document loaders (like PDFLoader) to be accept buffer inputs.

Motivation

Next.js apps can end up with buffer objects, for example, submitting a file via a form:

export async function POST(request: NextRequest) {
  const data = await request.formData();
  const file: File | null = data.get('file') as unknown as File;
  const bytes = await file.arrayBuffer();
  const buffer = Buffer.from(bytes);
    
  const ids = await saveToVectorCollection(buffer, undefined, {docName: "testing document"});
}

Calling (code snippet):

export async function saveToVectorCollection(buffer: Buffer, docId?: string, metadata?: Record<string, any>): Promise<string[]> {
    const tempDir = await mkdtemp(join(os.tmpdir(), `${astraConfig.collection}-`));
    const filePath = join(tempDir, 'uploadedFile');
    await writeFile(filePath, buffer);      
    const loader = new PDFLoader(filePath, {
        splitPages: false,
        });          
    const docs = await loader.load();

Because PDFLoader takes only a string | Blob , it can't handle a buffer so in this code, the buffer has been written to a temporary file.

Proposal (If applicable)

Either allow BufferLoader to accept a Node buffer object, or build a utility that can convert buffer to Blob. Note that as the BufferLoader sets source attribute, that might be something we'd want to send as metadata.

Not sure the right/best way to go on this, but it's kinda painful. But maybe my code example is not the "best" way to do it (though to my eyes, it's certainly fairly clean?)

[jacoblee93]

Definitely in favor of making this easier, but I think I remember looking into it and finding that not 100% of web environments support Buffer.

I like your idea of making a utility for translating it back and forth though!