You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<supclassName="text-orange-500">1</sup>Northeastern University, <supclassName="text-orange-500">2</sup>Stanford University, <supclassName="text-orange-500">3</sup>West Virginia University
When we first started working with large language models, we noticed something puzzling. We saw that these models were incredibly capable. However, we all experienced a particular kind of frustration, illustrated perfectly by this example from{' '}
<li>How can we get satisfactory results if LLMs make assumptions about our preferences?</li>
841
-
<li>How reliable is it to consult AI on healthcare, legal, or financial decisions?</li>
842
-
<li>How much time and patience are we expected to waste just trying to get our point across?</li>
843
-
</ul>
844
-
845
-
<pclassName="mb-6 text-lg text-gray-500">
846
-
The problem runs deeper. We typically evaluate LLMs in <strong>simple, sanitized test environments</strong>—single-turn prompts with clear, unambiguous instructions. But is that how real communication works?
847
-
</p>
848
-
849
-
<pclassName="mb-6 text-lg text-gray-500">
850
-
In real life, solving meaningful problems requires <strong>collaboration, iteration, and contextual awareness</strong>. Moreover, if humans and LLMs are going to tackle groundbreaking problems together, AI systems can't just passively respond to human requests—they need to actively <strong>stimulate human creativity</strong> and guide the collaborative process.
The core idea behind CollabLLM is simple: in a multi-turn interaction, what matters most is not how good a single response is—but how it affects the rest of the conversation.
: Rachel and Joey are talking about dating strategies. Rachel asks a seemingly simple question:
884
-
<em>"So, where'd you grow up?"</em> Joey immediately mocks her—<em>"That's your move?"</em>—implying the question is naive.
885
-
But a few turns later, his tone changes. He's genuinely impressed: <em>"Wow!"</em>—because the question led him to open up and connect. The key insight? <strong>What matters isn't how a response is judged in the moment, but how it shapes the entire conversation.</strong>
886
-
</p>
887
-
888
-
<pclassName="mb-4 text-lg">
889
-
Now imagine a model that chooses to ask a clarifying question instead of giving a direct answer. Standard reinforcement learning from human feedback (RLHF) might penalize that—it didn't provide information right away. But if the question helps uncover useful context that improves the conversation downstream, shouldn't it be rewarded?
That's exactly what CollabLLM does. We define a new reward function that measures the <strong>causal effect</strong> of a model's response on the future trajectory of a conversation. We call this the <strong>Multiturn-aware Reward (MR)</strong>. It evaluates a single model action based on its longer-term impact—not just immediate helpfulness.
896
-
</p>
897
-
</div>
898
-
899
-
<pclassName="mb-4 text-lg">
900
-
<strong>Quiz:</strong> is asking a question always better than giving an answer? The answer is—not necessarily. It depends entirely on the objective.
901
-
In most real-world situations, repeatedly asking questions without making progress is inefficient, because the ultimate goal remains unmet.
902
-
But take the game <em>20 Questions</em> as an example—where the objective is to guess what someone is thinking by asking a limited number of yes/no questions.
903
-
In that case, asking questions is essential, and giving an answer too early would break the format and defeat the purpose of the game.
904
-
This is where Multiturn-aware Reward (MR) comes in: it allows the model to adapt its behavior based on the context, learning <em>when</em> to ask and <em>when</em> to answer—depending entirely on what the task requires.
905
-
</p>
906
-
907
-
<pclassName="mb-4 text-lg">
908
-
Now, going back to the <em>Friends</em> example with Rachel and Joey—how do we measure the value of Rachel's question over the course of a conversation? We need two components:<br/>
909
-
1) A <strong>user simulator</strong> to generate realistic follow-up responses (e.g., what Joey might say next), and<br/>
910
-
2) An <strong>evaluator</strong> to judge whether the interaction is successful—such as whether Joey becomes more romantically engaged.
911
-
</p>
912
-
913
-
<pclassName="mb-4 text-lg">
914
-
Fortunately, both parts are quite feasible. First, the model you're training—let's call it "Rachel"—serves as the policy model generating responses. To simulate realistic dialogue, we prompt another model to act as "Joey," a proxy for the user. While inspired by our earlier example, "Joey" can represent <strong>any user simulator</strong>: a shopper trying to order groceries, a student asking math questions, or a writer seeking feedback. Second, we define task-specific metrics to evaluate success. In the dating example, it might be emotional engagement; in writing, it could be clarity or persuasiveness; in a question-answering task, it might be factual correctness. These evaluation criteria can even be combined—it's entirely up to your application!
915
-
</p>
916
-
917
-
<pclassName="mb-4 text-lg">
918
-
With Multiturn-aware Reward in place, the goal becomes straightforward: train the policy model to maximize this reward. In doing so, the model learns to drive the conversation effectively toward the desired outcome—whether that's solving a task, clarifying a request, or building rapport.
After all, you don't need massive changes to build a collaborative model. Just a new way to define the objective—and a longer lens for measuring what matters in a conversation.
The applications of collaborative AI are vast and exciting. From working on document editing to solving complex scientific problems, CollabLLM opens up new possibilities for human-AI collaboration.
935
-
</p>
936
-
937
-
<pclassName="mb-6 text-gray-500 text-lg">
938
-
We've seen remarkable results in our initial testing, with collaborative LLMs outperforming non-collaboratively trained LLMs across various benchmarks. More importantly, users report a more efficient, engaging, and reliable interaction experience when working with the collaborative LLMs.
We're continuously refining our approach, exploring new collaboration patterns. Our goal is to democratize collaborative AI and enable anyone to build more effective AI-powered solutions.
Join us in building the future of collaborative AI. Check out our code, contribute to the project, and help us shape the next generation of AI systems that truly understand the power of working together.
0 commit comments