Readdir is not a function

[carl-youngman]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import React from "react";
import ReactDOM from "react-dom/client";
import "./assets/scss/_main.scss";
import { DirectoryLoader } from "langchain/document_loaders/fs/directory";

import {
  JSONLoader,
  JSONLinesLoader,
} from "langchain/document_loaders/fs/json";
import { TextLoader } from "langchain/document_loaders/fs/text";
import { CSVLoader } from "langchain/document_loaders/fs/csv";

const SUPABASE_API_KEY = import.meta.env.VITE_SUPABASE_API_KEY;
const SUPABASE_URL = import.meta.env.VITE_SUPABASE_URL;
const OPEN_AI_API_KEY = import.meta.env.VITE_OPEN_AI_API_KEY;

// Assuming the data directory is located inside the src directory
const dataDirectory = "src/assets/data";

async function loadDataAndStore() {
  try {
    const loader = new DirectoryLoader(dataDirectory, {
      ".json": (path) => new JSONLoader(path, "/texts"),
      ".jsonl": (path) => new JSONLinesLoader(path, "/html"),
      ".txt": (path) => new TextLoader(path),
      ".csv": (path) => new CSVLoader(path, { separator: "," }),
    });

    const docs = await loader.load();

    console.log(docs);

  
  } catch (error) {
    console.log(error);
  }
}

loadDataAndStore();

ReactDOM.createRoot(document.getElementById("root")).render(
  <React.StrictMode></React.StrictMode>,
);

Description

I'm unable to return any data from any type of file using the DirectoryLoader. TypeError: readdir is not a function. Could someone point me in the write direction? Possibly something to do with being within a VITE project

System Info

langchain: v1.22.19
Windows
node: v18.15.0
yarn 1.22.19

[dosubot[bot]]

Hey there @carl-youngman! 🚀 I'm Dosu, your friendly neighborhood bot, here to help you squash bugs, answer your burning questions, and guide you on your journey to becoming a LangChain JS contributor. While we're waiting for a human maintainer, I'm here to lend you a hand. Let's tackle this together!

Based on the code snippet you've provided and the error message you're encountering, it seems like you're trying to use the DirectoryLoader class from the LangChain JS framework in a browser environment. The DirectoryLoader class is designed to work in a Node.js environment, as it relies on Node.js-specific modules (fs and path), which are not available in the browser environment. This is likely the cause of the TypeError: readdir is not a function error you're seeing.

The DirectoryLoader class tries to dynamically import these Node.js-specific modules in its imports method, which fails in a browser context. This is why you're seeing the error at the line where you're trying to instantiate the DirectoryLoader class.

Vite, the build tool you're using, does not polyfill Node.js core modules by default for browser targets. This means that it doesn't provide browser-compatible versions of Node.js-specific modules like fs and path.

To resolve this issue, you have a few options:

1. Use a Backend: If your application architecture allows it, you could move the functionality that requires DirectoryLoader to a backend service. This way, you can use Node.js on the server to handle file system operations and communicate with the frontend via API calls.

2. Mock or Shim: If you're using this functionality for development or testing purposes in the frontend, you could consider mocking or creating a shim for the fs and path modules. This involves creating a fake implementation that mimics the expected behavior but doesn't perform actual file system operations. However, this might not be practical for all use cases, especially if you need to perform real file system operations.

3. Alternative Libraries: You could look for browser-compatible libraries that can offer similar functionality without relying on Node.js-specific modules. For instance, if you're trying to load documents from a directory structure represented within the browser, you might use browser APIs like the File API for handling file inputs and manipulating files in a way that's compatible with your project's requirements.

4. Configuration Changes: If your use case absolutely requires these Node.js modules and you cannot move the logic to the backend, you might need to reconsider your choice of tools or frameworks for the frontend, or explore advanced configurations with Vite or alternative build tools that might offer better support for Node.js modules in the browser through polyfills or shims. However, this is generally not recommended due to the fundamental differences between browser and Node.js environments.

I hope this helps! If you have any further questions, feel free to ask.

For more information, you can refer to the source code of the DirectoryLoader class in the LangChain JS framework here.

Sources

• readdir is not a function at DirectoryLoader.load (directory.js:50:29) #324
• Import error when using in React-vite project
• langchain/src/document_loaders/fs/directory.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.