FIX/REFACTOR: FoldContent revamp #866
Conversation
We hit a snag with our hot topic gist strategy: the regex we used to split the content didn't work, so we cannot send the original post separately. This was important for letting the model focus on what's new in the topic. The algorithm doesn’t give us full control over how prompts are written, and figuring out how to format the content isn't straightforward. This means we're having to use more complicated workarounds, like regex. To tackle this, I'm suggesting we simplify the approach a bit. Let's focus on summarizing as much as we can upfront, then gradually add new content until there's nothing left to summarize. Also, the "extend" part is mostly for models with small context windows, which shouldn't pose a problem 99% of the time with the content volume we're dealing with.
romanrizzi
force-pushed
the
fold_revamp
branch
from
f711226 to
dacb7db
Compare
Given that 99% of the time we will only do a single pass, shouldn't we only support that to simplify code? Like if "topic fits in context"
summarize with all posts
else
summarize using best replies mode
end |
This brings up another point: with the average context window being much larger than it was a year ago, are we being overly cautious in how we choose posts to summarize? I think the answer is probably yes, and we could probably start sending the entire topic to the model. However, I don’t think we should completely drop the folding. It’s a good safety net for avoiding overwhelming the LLM with too much content. I wouldn't expect this approach to change much, especially now that we've separated the "what" and the "how" into different strategies. |
LLMM config gives us the exact context size, so we know if it fits or not. We should check, if the topic size is < 80% of the context window we should send it all. |
Fair enough. I'll follow-up on a different PR. |
We hit a snag with our hot topic gist strategy: the regex we used to split the content didn't work, so we cannot send the original post separately. This was important for letting the model focus on what's new in the topic.
The algorithm doesn’t give us full control over how prompts are written, and figuring out how to format the content isn't straightforward. This means we're having to use more complicated workarounds, like regex.
To tackle this, I'm suggesting we simplify the approach a bit. Let's focus on summarizing as much as we can upfront, then gradually add new content until there's nothing left to summarize.
Also, the "extend" part is mostly for models with small context windows, which shouldn't pose a problem 99% of the time with the content volume we're dealing with.