How does the ability to remember past messages work with the API?

[super-15]

Hey, I was wondering how does it work for my own similar project. I tried putting the past messages in the new messages but that uses up a bunch of tokens.

[waylaidwanderer]

By default, it reserves up to 3k tokens for the conversation history. Any older messages that would exceed this limit are discarded.

[super-15]

Oh okay, thanks. Also how does the "typing" effect where it shows what it has generated so far work?

[waylaidwanderer]

Responses are generated by the AI one token at a time, so I'm just displaying it as it comes in. Same way ChatGPT works.

[super-15]

I tried it with JavaScript using the fetch function to make the api request, and it just sends the JSON when it finishes, not each token live.

[waylaidwanderer]

You need to set stream to true and read the body as a buffer IIRC.

[waylaidwanderer]

Look at my implementation in the demos folder.