Is there a way to dynamically chat with RAG documents when using agents

[KeenanFernandes2000]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

This is how i setup my agent 


const llm = new ChatOpenAI({
    modelName: "gpt-4-0125-preview", //gpt-3.5-turbo-0125
    temperature: 0,
  });

  const tools = [
    // search api
    new SerpAPI(process.env.SERPAI_KEY, {
      location: "Dubai,United Arab Emirates",
      hl: "en",
      gl: "us",
    }),
    // word document generator
    new DynamicStructuredTool({
      name: "wordDocGen",
      description:
        "Tool to compile JSON data into a word document and returns a download link",
      schema: z.object({
        data: z.object({
          main: z.string(),
          body: z.array(z.object({ heading: z.string(), content: z.string() })),
        }),
      }),
      func: async function ({ data }) {
        // uncomment this to see how the agent parses the data
        // console.log(data);
        const content = fs.readFileSync(
          "./public/assets/templates/allTemplate.docm"
        );

        const zip = new PizZip(content);
        const doc = new Docxtemplater(zip, {
          linebreaks: true,
        });
        await doc.renderAsync(data);
        const buf = doc.getZip().generate({ type: "nodebuffer" });
        const fileName = `documentGen${uuidv4()}`;
        const outputFilePath = `./public/audio/${fileName}.docm`;
        fs.writeFileSync(outputFilePath, buf);

        const downloadLink = `https://${host}/automarketing/download/${fileName}.docm`;
        return downloadLink;
      },
    }),
    // webscraper
    new DynamicStructuredTool({
      name: "scraperTool",
      description: "Tool to scrape data from the url provided",
      schema: z.object({
        url: z.string().describe("URL of the website to scrape"),
      }),
      func: async function ({ url }) {
        const response = await axios.get(url, { timeout: 10000 });

        if (response.status === 200) {
          const $ = cheerio.load(response.data);
          const scrapedData: string[] = [];

          // Example: scraping all paragraph texts
          $("p").each((index, element) => {
            const paragraph = $(element).text().trim();
            if (paragraph) {
              scrapedData.push(paragraph);
            }
          });
          // Send scraped data as response
          const allData = scrapedData.join("\n");

          return allData;
        }
        return "No data could be scraped";
      },
    }),
    // automarketing
    new DynamicStructuredTool({
      name: "marketingTool",
      description:
        "Tool to generate a pdf with marketing content for businesses ONLY. If it successfully generates the pdf the tool will return a link to the pdf",
      schema: z.object({
        email: z.string().describe("The user's email"),
        question: z.string().describe("The name of the user's business"),
        question1: z
          .string()
          .describe("The products being sold by the business"),
      }),
      func: async function ({ email, question, question1 }): Promise<any> {
        let info = {
          stage: 1,
          email: email,
          question: question,
          question1: question1,
        };

        sseEmitter.emit(`${sessionID}:accept`);
        const res = await apiCall(sessionID, info, host);
        return res;
      },
    }),
    // image generator
    new DynamicStructuredTool({
      name: "imgGenTool",
      description: "Tool to generate images",
      schema: z.object({
        imgDesc: z.string().describe("Description of the image to generate"),
      }),
      func: async function ({ imgDesc }): Promise<any> {
        const tool = new DallEAPIWrapper({});

        const imageURL = await tool.invoke(imgDesc);

        return imageURL;
      },
    }),
  ];

  const prompt = await pull<ChatPromptTemplate>("thecodingbarista/superagent");

  const agent = await createOpenAIFunctionsAgent({
    llm,
    tools,
    prompt,
  });

  const agentExecutor = new AgentExecutor({
    agent,
    tools,
  });

  const agentWithChatHistory = new RunnableWithMessageHistory({
    runnable: agentExecutor,
    getMessageHistory: (_sessionId) =>
      new UpstashRedisChatMessageHistory({
        sessionId: `${_sessionId}-superAgent`,
        config: {
          url: process.env.REDIS_URL!,
          token: process.env.REDIS_TOKEN!,
        },
        sessionTTL: 300,
      }),
    inputMessagesKey: "input",
    outputMessagesKey: "output",
    historyMessagesKey: "chat_history",
  });

Description

I want to try to have a agent that has a bunch of tools and in those tools I want there to be a rag feature where the user can upload a file and chat with it. However through my experimenting I was only able to use rag if I initialized the document when starting up my node server so it can embed as well.

Is there a way to allow an agent to accept a document midway of a conversation with a user while keeping memory and access to all the other tools

System Info

Node + Express

[dosubot[bot]]

To dynamically chat with documents during a conversation with a user, while also maintaining access to other tools, you can leverage the AutoGPT class from the langchainjs framework. This approach allows for the integration of document handling tools (ReadFileTool and WriteFileTool) with language models, enabling complex interactions within a conversational context.

Here's a simplified example of how to set up the AutoGPT class with the necessary tools for document handling:

const autogpt = AutoGPT.fromLLMAndTools(
  new ChatOpenAI({ temperature: 0 }),
  [
    new ReadFileTool({ store: new InMemoryFileStore() }),
    new WriteFileTool({ store: new InMemoryFileStore() }),
    // Add other tools as needed
  ],
  {
    memory: new MemoryVectorStore(new OpenAIEmbeddings()).asRetriever(),
    // Additional configuration options
  },
);

In this setup:

• ChatOpenAI is initialized with your desired settings.
• ReadFileTool and WriteFileTool are included for document interactions.
• The memory store is configured to maintain conversation history and context.

The AutoGPT class does not directly manage document uploads mid-conversation. Instead, it processes inputs and actions, including those involving documents, as part of its conversational flow. The handling of documents is facilitated through the tools integrated into the AutoGPT instance.

For more detailed implementation, including methods in the ReadFileTool and WriteFileTool for document interactions, refer to the LangChain JS repository, specifically the langchain/src/experimental/autogpt/agent.ts and langchain/src/tools/fs.ts files.

This approach ensures that your agent can dynamically interact with uploaded documents without losing the conversational context or access to other functionalities.

Details

Sources

• langchain/src/agents/toolkits/conversational_retrieval/openai_functions.ts
• langchain/src/experimental/autogpt/agent.ts
• langchain/src/tools/fs.ts
• examples/src/agents/custom_agent.ts
• examples/src/agents/structured_chat.ts
• examples/src/agents/custom_llm_agent_chat.ts

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[KeenanFernandes2000]

@dosu given the example let's say the user and the autogpt have had a conversation and now the user has uploaded the file, how does this get added into the context window for autogpt to refer to

[dosubot[bot]]

To dynamically integrate a user-uploaded document into the AutoGPT conversation context, follow these steps:

1. Process the Uploaded Document: Convert the document to plain text. This is necessary for AutoGPT to understand and use the document's content in the conversation.

2. Update the Vector Store: Add the text of the uploaded document to the VectorStoreRetrieverInterface (referred to as memory). This is crucial for making the document's content retrievable during the conversation.

3. Incorporate Document Content into the Conversation: When generating prompts for AutoGPT, include the content from the uploaded document by retrieving it from the vector store. This ensures that AutoGPT can refer to the document's content in its responses.

4. Manage Token Limits: Be mindful of the total token count when including document content in prompts. Adjust the sendTokenLimit or the logic for document inclusion as necessary to stay within limits.

5. Dynamically Update Conversation Context: Each time a new document is uploaded, update the vector store with its content. This allows the newly uploaded document to be immediately included in the conversation context.

Here's an example of how you might add a document to the memory:

async function addDocumentToMemory(documentText: string, memory: VectorStoreRetrieverInterface) {
  const processedDocument = await preprocessDocument(documentText); // Convert document to a suitable format
  await memory.addDocument(processedDocument); // Add to vector store
}

// Invoke this function with the text of the uploaded document
addDocumentToMemory(userUploadedDocumentText, memoryInstance);

This method ensures that AutoGPT can dynamically refer to the content of user-uploaded documents during an ongoing conversation, enhancing the interaction by making it more relevant and context-aware.

Details

Sources

• langchain/src/experimental/autogpt/prompt.ts
• langchain/src/experimental/autogpt/output_parser.ts
• langchain-core/src/messages/system.ts
• langchain-core/src/vectorstores.ts

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[KeenanFernandes2000]

@dosu how do I get an instance of memory if I setup my autogpt like this

const autogpt = AutoGPT.fromLLMAndTools(
new ChatOpenAI({ temperature: 0 }),
[
new ReadFileTool({ store: new InMemoryFileStore() }),
new WriteFileTool({ store: new InMemoryFileStore() }),
// Add other tools as needed
],
{
memory: new MemoryVectorStore(new OpenAIEmbeddings()).asRetriever(),
// Additional configuration options
},
);

[dosubot[bot]]

To integrate a user-uploaded document into the AutoGPT conversation context after setting up your agent as described, you'll need to manage the document's content within the conversation's memory. Here's a concise guide to achieve this:

1. Initialize Memory Vector Store: Before starting your AutoGPT instance, create a MemoryVectorStore with OpenAIEmbeddings to handle the embeddings for the document's content. This store is essential for managing the vectorized representations of the documents and conversation snippets.

2. Configure Vector Store Retriever Memory: Use VectorStoreRetrieverMemory with your MemoryVectorStore instance. This configuration allows you to specify how many documents to return during retrieval and enables the use of metadata for filtering, which is crucial for dynamically updating the conversation context based on the user's uploaded document.

3. Save and Retrieve Conversation Context: Implement methods to save conversation snippets (input and output pairs) to the memory and to retrieve this information based on the current prompt. This dynamic retrieval is key to maintaining a coherent conversation flow that incorporates the content of the uploaded document.

4. Integrate with AutoGPT: Ensure that your AutoGPT setup includes the memory management steps above. This integration allows the language model to access and utilize relevant pieces of previous conversations and the uploaded document's content, enhancing the interaction quality.

By following these steps, you can dynamically integrate user-uploaded documents into the AutoGPT conversation context, allowing for a more enriched and context-aware interaction. This approach leverages the capabilities of MemoryVectorStore and VectorStoreRetrieverMemory within the LangChain framework, ensuring that your agent can refer to the content of user-uploaded documents during an ongoing conversation.

Details

Sources

• langchain/src/memory/vector_store.ts
• langchain/src/vectorstores/memory.ts
• examples/src/indexes/vector_stores/memory.ts
• examples/src/memory/vector_store.ts

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[KeenanFernandes2000]

@dosu

these examples add a document to the memory, is there a way to also attach message memory to the AutoGPT. Please provide a code if possible that adds documents dynamically and have persistent session memory.

[dosubot[bot]]

To dynamically integrate user-uploaded documents into the AutoGPT conversation context and ensure persistent session memory, you can follow a structured approach using MemoryVectorStore with OpenAIEmbeddings for document content management, alongside VectorStoreRetrieverMemory for session memory management. Here's a concise guide:

1. Initialize Memory Vector Store: Create an instance of MemoryVectorStore initialized with OpenAIEmbeddings to handle document embeddings. This allows for efficient storage and retrieval of document content.

   import { OpenAIEmbeddings } from "@langchain/openai";
   import { MemoryVectorStore } from "langchain/vectorstores/memory";
   
   const vectorStore = new MemoryVectorStore(new OpenAIEmbeddings());

2. Manage Session Memory: Utilize VectorStoreRetrieverMemory to manage both the conversation history and the content of user-uploaded documents. This ensures a dynamic and context-aware conversation flow.

   import { VectorStoreRetrieverMemory } from "langchain/memory";
   
   const memory = new VectorStoreRetrieverMemory({
     vectorStoreRetriever: vectorStore.asRetriever(1),
     memoryKey: "history",
   });

3. Saving and Retrieving Context: Implement methods to save conversation snippets and user-uploaded document content to the memory, and to retrieve this information based on the current prompt. This approach maintains a coherent conversation flow that incorporates the content of the uploaded document.

   // Saving conversation context
   await memory.saveContext(
     { input: "User's question or statement" },
     { output: "Agent's response" }
   );
   
   // Retrieving relevant documents and conversation history
   const context = await memory.loadMemoryVariables({ prompt: "User's current question" });

By following these steps, your AutoGPT-based agent can dynamically integrate user-uploaded documents into the conversation context with persistent session memory. This setup enhances the agent's ability to provide contextually relevant responses based on both the conversation history and the content of user-uploaded documents.

For detailed implementation, refer to the LangChain JS repository, specifically the files related to MemoryVectorStore and VectorStoreRetrieverMemory. This will give you a comprehensive understanding of how to manage document content and conversation memory effectively within your AutoGPT setup.

Details

Sources

• langchain/src/memory/vector_store.ts
• examples/src/indexes/vector_stores/memory.ts
• examples/src/indexes/vector_stores/memory_fromdocs.ts
• examples/src/memory/vector_store.ts

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[KeenanFernandes2000]

@dosu

Can the same logic be applied to regular agent using the code below as reference
const agent = await createOpenAIFunctionsAgent({
llm,
tools,
prompt,
});

const agentExecutor = new AgentExecutor({
agent,
tools,
});

const agentWithChatHistory = new RunnableWithMessageHistory({
runnable: agentExecutor,
getMessageHistory: (_sessionId) =>
new UpstashRedisChatMessageHistory({
sessionId: ${_sessionId}-superAgent,
config: {
url: process.env.REDIS_URL!,
token: process.env.REDIS_TOKEN!,
},
sessionTTL: 300,
}),
inputMessagesKey: "input",
outputMessagesKey: "output",
historyMessagesKey: "chat_history",
});

is there a way to add a retriever tool that dynamically changes the document if there is an uploaded document

[dosubot[bot]]

To dynamically integrate user-uploaded documents into your agent's conversation context and maintain persistent session memory, you can add a custom tool to your agent setup. This tool will handle the preparation, loading, and embedding of the document content, making it accessible for the conversation. Here's a streamlined approach:

1. Create a Document Preparation Tool: This tool will load the user-uploaded document, split it into manageable chunks, and index these chunks for efficient retrieval. You'll need to use classes like UnstructuredLoader for loading, RecursiveCharacterTextSplitter for splitting, and MemoryVectorStore with OpenAIEmbeddings for indexing.

   const documentPreparationTool = new DynamicStructuredTool({
     name: 'prepareDocument',
     description: 'Prepares a document for integration into the conversation context',
     func: async (filePath: string) => {
       // Implementation details here
       return { documentId: filePath };
     },
   });

2. Integrate the Tool into Your Agent: Add the new document preparation tool to your agent's tools array. This allows your agent to process uploaded documents dynamically during a conversation.

   const agent = await createOpenAIFunctionsAgent({
     llm,
     tools: [...tools, documentPreparationTool],
     prompt,
   });

3. Modify Agent Execution for Document Handling: Adjust your agent's execution logic to check for document uploads in the user input. Use the document preparation tool to process the uploaded document and integrate its content into the conversation context.

   class DocumentIntegratedAgentExecutor extends AgentExecutor {
     async execute(input: any, context: any) {
       if (input.documentUpload) {
         // Process the uploaded document
       }
       return super.execute(input, context);
     }
   }

By implementing these steps, your agent will be capable of dynamically incorporating content from user-uploaded documents into ongoing conversations. This setup enriches user interactions by making them more context-aware and relevant, leveraging the capabilities of your agent's existing tools and memory management.

Details

Sources

• langchain/src/agents/tests/structured_output_runnables.int.test.ts
• examples/src/agents/structured_chat.ts
• examples/src/guides/conversational_retrieval/agent.ts
• examples/src/use_cases/chatbots/quickstart.ts
• langchain/src/agents/toolkits/vectorstore/prompt.ts
• langchain/src/load/import_constants.ts
• langchain/src/document_loaders/fs/unstructured.ts
• examples/src/memory/redis.ts
• libs/langchain-redis/src/chat_histories.ts
• examples/src/memory/redis-sentinel.ts
• examples/src/memory/redis-advanced.ts
• examples/src/memory/upstash_redis_advanced.ts
• examples/src/agents/chat_convo_with_tracing_runnable.ts
• examples/src/agents/custom_agent.ts
• langchain/src/experimental/plan_and_execute/agent_executor.ts

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.