Fine tuning a text bison model on custom dataset

[Abe410]

Hi everyone

I currently have a pipeline using OpenAI where I pass information about my internal company database tables as a prompt, and then ask a user defined question, and then get an SQL query and a response.

As you might have guessed, this takes alot of tokens since I need to describe my tables in the prompt and costs alot.

I am trying to now fine tune a text-bison model by passing it training examples of the input text along with an appropriate output response. For the training, I can pass the same prompt as the OpenAI pipeline, where I describe my tables and then ask the model to generate a query.

But, the Vertex AI page on fine tuning says to use training examples which will be the same as you would get an input in production. This would mean that I also pass the whole table description in the production pipeline as well, and this is exactly what I am trying to avoid.

As an example:

For training:

Input: You have the following tables to gather data from:

Table 1 description
Table 2 description

Text: What is the price of Lockheed Martin stocks?

Output: The price of stock is X.

In the above example, the model knows the tables through the prompt and then finds the appropriate table for the text it was given.

But in a production environment, I want to give only the 'text', and not the table descriptions, since that would take up tokens and cost more, and that is what I am trying to avoid in the first place.

Any idea how to go about this or am I approaching the problem in the wrong way?

Thanks

[polong-lin]

Great question -- a couple of thoughts here:

1. As the cost is equal to [# of input/output characters in PaLM API] * [# of calls] -- you might try to explore times in which you can reduce the # of calls that are using long prompts unnecessarily?
2. Rather than including all the tables/descriptions in the prompt every time, could you perhaps leverage RAG to help you identify only the tables that are most relevant and only add those to your prompt before sending it to the LLM? This way, for example, for questions related to "stocks", it could retrieve only the relevant tables/table descriptions related to stocks, instead of all the tables available.
3. Are there opportunities to do analytics on the most common queries from your users so that you can convert them into direct hard-coded queries (made up of hard-coded queries), rather than have the queries be generated by the LLM? You can use Pub/Sub, for example, to help you capture your LLM requests & responses and log them into Cloud Storage or BigQuery. In this way, a question structured like "What is the price of {user_input} stocks?" could directly call a stored procedure (for example), instead of being sent to the LLM. To do so, you might need another layer that can route questions to either an LLM or specific hard-coded queries. This "hybrid" approach might also help to guardrail your system from generating queries that have innate uncertainty (they might be inaccurate or might not work) while still maintaining the flexibility to users of being able to "ask any question in freeform English".

[cah-jonathan-cachat01]

Using PaLM API in LangChain can also help differentiate between input questions that should be answered with SQL on BQ tables vs other analysis paths