Pandas dataframe agent

[harsh3986]

Suppose I have fine tuned starcoder model. How can I use pandas dataframe agent using this local fine tuned model or any other open source model from hugging-face ?

[dosubot[bot]]

Answer generated by a 🤖

Answer

I understand that you have a fine-tuned Starcoder model and you're looking to use it with the pandas dataframe agent in LangChain. You're also interested in knowing how to use an open-source model from Hugging Face in the same context.

Here's how you can achieve this:

First, you'll need to import the model and use it when creating the agent. Here's a sample code snippet to illustrate this:

from langchain.agents import create_pandas_dataframe_agent
from langchain.agents.agent_types import AgentType
from langchain.schema.language_model import BaseLanguageModel
import pandas as pd

# Load your dataframe
df = pd.read_csv("titanic.csv")

# Load your fine-tuned model
# Replace BaseLanguageModel with the class used to load your model
model = BaseLanguageModel(model_path="path/to/your/model")

# Create the agent
agent = create_pandas_dataframe_agent(
    llm=model, 
    df=df, 
    verbose=True, 
    agent_type=AgentType.OPENAI_FUNCTIONS
)

# Run the agent
# Replace "how many rows are there?" with your question
agent.run("how many rows are there?")

In the above code, replace "path/to/your/model" with the actual path to your fine-tuned model. The BaseLanguageModel class is a placeholder representing the language model, you would need to replace it with the actual class or function used to load your model in LangChain.

If you're looking to use a Hugging Face model, you would need to replace BaseLanguageModel with the appropriate class for loading Hugging Face models. This is assuming that LangChain has support for Hugging Face models. If it doesn't, you might need to add this support yourself.

Also, LangChain has a create_csv_agent function that creates a pandas dataframe agent from a CSV file. If your data is in a CSV file, you can use this function to create your agent. The function takes a path to the CSV file as an argument and loads the data into a pandas dataframe. You can use this function in a similar way to the create_pandas_dataframe_agent function, but you would need to replace the dataframe with the path to your CSV file.

Please note that this is one potential solution based on the information provided. If you find this solution helpful and believe it could benefit others, I encourage you to make a pull request to update the LangChain documentation. This way, other users facing the same issue can also benefit from your experience.

Sources

• docs/extras/modules/agents/toolkits/pandas.ipynb
• tests/integration_tests/agent/test_pandas_agent.py
• langchain/agents/agent_toolkits/pandas/base.py
• langchain/agents/agent_toolkits/csv/base.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[harsh3986]

No, it won't work as Open AI functions only support ChatOpen AI models.
How can I use starcoder to parse the output into pandas agent of langchain ?

[dosubot[bot]]

Answer generated by a 🤖

Answer

I found related issues that might be helpful. I did my best to summarize the solutions, but I recommend looking at the related issues yourself.

Open Issues

Could not parse LLM output when using 'create_pandas_dataframe_agent' with open source models (any model other than OpenAI models)

The suggested solution is:

1. Verify the output of the Vicuna 13B model to ensure it's in the expected format. The error might be due to the model's output not aligning with what the agent expects.

2. If the output format is as expected, you might need to adjust the parsing logic to handle the specific output format of the Vicuna 13B model. This could involve modifying the agent's code to correctly interpret the model's output.

If these steps don't align with your experience or the issue persists, it is recommended to check if you're using the latest version of LangChain. Sometimes, issues can be resolved by simply updating to the latest version of the software.

ChatOpenai (gpt3-turbo) isn't compatible with create_pandas_dataframe_agent, create_csv_agent etc

This issue was referenced by:

• The Agent is incompatible with the GPT-3.5-turbo-16k model.

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.