Support custom datasets

[irar2]

Allow user provided custom datasets for response generation.

At the moment, the simulator has a short set of sentences from which it randomly chooses a response text. Support an option for user-provided custom dataset instead.

[pancak3]

Hi Ira, do you have more information/ideas about the arch of this module? For example, what format the dataset would be, whether to use a database, and/or what sizes the datasets would be, what libs might be used

I am thinking of:

Loaded dataset in mem

prompt_hash	generated_text
hashA	I am an AI assitant
hashB	You are definitely right
... ...	... ...
hashZ	Sure. Here is the code
nil	This is perfect completion text

When a request comes, the prompt is hashed, and the hash will be used to look for generated_text,

• If hits, the response is returned, controlled by the latencies.
• If not, randomly select one from the dataset
• the max_tokens or max_completion_tokens will be used to truncate the generated_text if necessary

The benefit is that users can convert existing real datasets and receive corresponding responses.

[irar2]

@pancak3 Hi, your idea looks really good to me.

Two comments:

1. For chat completions, we will need to create hashes for all the previous requests and responses, for example for
   {"role": "user", "content": req1},
   {"role": "assistant", "content": resp1},
   {"role": "user", "content": req2},
   {"role": "assistant", "content": resp2},
   ...
   the hash for resp2 will be hash(req1+resp1+req2).

2. In case of a miss and a random selection, when we have max_tokens, we would like to still support the current behavior, and randomly choose the length of the response, and then choose several responses from the dataset according to the selected length.

For the datasets format we should probably look at Hugging Face datasets.

[pancak3]

/assign