Summary of text data extracted from sql tables

[yetaotjroc93]

Hi everyone, can anybody help me with the following task?

I'd like to extract text data from sql table, and then summarize it.

I'm thinking to use the existing sql chain or sql toolkit, and then use summary chain. However, I face two issues:

1. The returned text is too large.
2. The output from sql chain is str instead of page_content.

If I choose to get the sql query instead of the text data. I wonder how to write a tool to load the text data as page_content based on the sql query?

Thanks.

[dosubot[bot]]

Answer generated by a 🤖

Answer

I understand that you're trying to extract text data from an SQL table and summarize it using LangChain. You're considering using the SQLDatabaseChain or SQL toolkit, followed by the summary chain. However, you're encountering two issues: the returned text is too large, and the output from the SQL chain is a string instead of page_content. You're also considering getting the SQL query instead of the text data and are wondering how to load the text data as page_content based on the SQL query.

Here's how I would approach your problem:

1. To manage the size of the returned text, you can limit the number of rows returned by using the 'top_k' parameter when initializing the SQLDatabaseChain. For instance, SQLDatabaseChain.from_llm(llm, db, verbose=True, use_query_checker=True, top_k=3) will limit the output to the top 3 rows.

2. To convert the string output from the SQL chain to page_content, you can use Python's built-in functions or libraries.

As for loading the text data as page_content based on the SQL query, you can create a function that executes the SQL query, fetches the data, and then formats it as page_content. Here's a basic example:

from langchain import OpenAI, SQLDatabase, SQLDatabaseChain

db = SQLDatabase.from_uri("your_database_uri")
llm = OpenAI(temperature=0, verbose=True)
db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True, top_k=3)

def load_data_as_page_content(query):
    # Execute the query and fetch the data
    db_chain.run(query)
    data = db_chain.result

    # Convert the data to page_content format
    page_content = str(data)

    return page_content

Please replace "your_database_uri" with your actual database URI and query in load_data_as_page_content(query) with your actual SQL query.

This is a simplified example and you might need to adjust the code to fit your specific needs, such as handling database connections and exceptions.

For data-sensitive projects, you can specify return_direct=True in the SQLDatabaseChain initialization to directly return the output of the SQL query without any additional formatting. This prevents the LLM from seeing any contents within the database. Note, however, the LLM still has access to the database scheme (i.e. dialect, table and key names) by default.

This is one potential solution to your problem. If you find this solution helpful and believe it could benefit others, I encourage you to make a pull request to update the LangChain documentation. This way, other users facing similar issues can benefit from your experience.

Sources

• docs/snippets/modules/chains/popular/sqlite.mdx

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.