feat: add chunking to ChangeTone, ContextWrite, ProofreadProvider, ReformulateProvider, TopicsProvider, and TranslateProvider

[lukasdotcom]

The implementation for TopicsProvider isn't that great, but is passable.
Also I added progress to all the chunked providers because it made debugging easier.

[julien-nc]

For some task types like proofread, the output can be weird as one chunk could be correct and another one incorrect. So one completion request might produce there are no spelling or grammar mistakes and the other one would list the mistakes, ending up in a confusing/contradicting answer.

Maybe the proofread prompt could be adjusted to something like "do not output anything if there is no mistake or correction suggestion". And in the end we assemble the results and if all the chunks were clean, we artificially set the response to there are no spelling or grammar mistakes. Wdyt?

[lukasdotcom]

For some task types like proofread, the output can be weird as one chunk could be correct and another one incorrect. So one completion request might produce there are no spelling or grammar mistakes and the other one would list the mistakes, ending up in a confusing/contradicting answer.

Maybe the proofread prompt could be adjusted to something like "do not output anything if there is no mistake or correction suggestion". And in the end we assemble the results and if all the chunks were clean, we artificially set the response to there are no spelling or grammar mistakes. Wdyt?

I did try to change the prompt to attempt to output nothing, but it really doesn't want to do that. It will either ignore that instruction or literally say 'empty string' or 'nothing' or it will output an empty json list. This happens with many different models. Also I don't think we can just set the response to there are no spelling and grammar mistakes when none of them give an output in case the user doesn't expect/speak english.

[marcelklehr]

We could use another LLM request to combine the outputs

[julien-nc]

We could use another LLM request to combine the outputs

@lukasdotcom That sounds safer indeed. This will hopefully get rid of the contradiction 😁

[kyteinsky]

For some task types like proofread, the output can be weird as one chunk could be correct and another one incorrect
We could use another LLM request to combine the outputs

one other method could be to use structured outputs and instruct the LLM to output changes in different JSON keys. Even smaller models are capable of that now.
for example for the summary task type, appending this text works nicely:

output the summary in the following JSON format:
{"summary": "string"}

the actual text can be wrapped in something like """ """

[lukasdotcom]

For some task types like proofread, the output can be weird as one chunk could be correct and another one incorrect
We could use another LLM request to combine the outputs

one other method could be to use structured outputs and instruct the LLM to output changes in different JSON keys. Even smaller models are capable of that now. for example for the summary task type, appending this text works nicely:

output the summary in the following JSON format:
{"summary": "string"}

the actual text can be wrapped in something like """ """

The advantage I see of @marcelklehr method is that it would also allow for duplicate recommendations to be removed. Like if the person misspelled the same word multiple times in different chunks, but the JSON could save an llm request.

[marcelklehr]

What would be the benefit of having the LLM wrap the output in json? We still need to combine potentially conflicting outputs somehow

[lukasdotcom]

What would be the benefit of having the LLM wrap the output in json? We still need to combine potentially conflicting outputs somehow

That llms will actually output an empty string.

[lukasdotcom]

What would be the benefit of having the LLM wrap the output in json? We still need to combine potentially conflicting outputs somehow

Implemented another LLM request when multiple chunks exist to merge all the feedback into one list. Also made the formatting nicer because a lot of the time the LLM spit out a numbered list and it looks weird when it counts 1, 2, 3, 1, 2...

[julien-nc]

Works nicely with gpt-3.5.

We should improve the system prompts to increase the chances the answer is given in the same language as the input text. Can be done in another issue/PR.