Multiple answer generation

[olinorwell]

Hi everyone - thanks for reading and any help that could be offered.

I've been working on code to run multiple generations of the same prompt. Specifically, the instruction type prompt that works well with the Alpaca model. I've based my code on the main.cpp file which I've spent a fair few days editing now.

Here's what I'm doing.

So for example my requirement might be: "List 3 European countries".

I know with these instruction style prompts you have the format:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:

List 3 European countries.

### Response:

I've got code that runs the above request multiple times, and it basically works. However, I'd like some hints on whether I'm on the right track or not!

Now, I have the tokens for that initial part of the prompt, and I am reusing them.

What I'm looping through is the 'eval()' each time.

Now, what I'd like to know is, can I save the state after evaluating that first part of the prompt (I've seen sample code discussed on various issues on this repo), then run the 'generative' part of the process multiple times. After getting an answer then revert to where I was after it had 'read' the final bit of the prompt.

I don't even need to save the state to a file, the key is just to be able to limit what I do every loop.

Also, when looping do I need to reset the model somehow? This ultimate goal of this tool is to be able to have a prompt such as "Write a one line description of a house in the countrisde", and set it to produce 100 samples of that generation. All 100 are independent and don't need to affect the others, they would of course be random and contain quite a bit of variation (I have the temperature setting up quite high anyway).

Right now, I'm struggle to know whether there is something obvious I'm missing, at the point of having my first 'result'. What do I need to do to get it ready for running the 'answer' part of the prompt again?

Many thanks in advance for any hints and tips. My plan is to share this project when I'm done.

[SlyEcho]

I think resetting n_past to where your prompt ends should be enough. Maybe also remove the last generated tokens from last_n_tokens, or not: it may help to generate different outputs.

Currently it's not possible to generate multiple outputs at the same time because the state is shared but you could do it in a loop.

EDIT: But I think it should be just possible to use ./main and a session file right now to achieve the same result without any code changes.

[olinorwell]

Thanks for your help! I'll implement that now.