Skip to content
This repository was archived by the owner on Jul 22, 2025. It is now read-only.

Commit a71a1e2

Browse files
committed
DEV: Add summarization type to eval (#1138)
Adds `type: summarization` for topic summarization eval: discourse/discourse-ai-evals#4
1 parent e016be6 commit a71a1e2

File tree

1 file changed

+22
-0
lines changed

1 file changed

+22
-0
lines changed

evals/lib/eval.rb

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,8 @@ def run(llm:)
6060
prompt_call(llm, **args)
6161
when "edit_artifact"
6262
edit_artifact(llm, **args)
63+
when "summarization"
64+
summarization(llm, **args)
6365
end
6466

6567
if expected_output
@@ -312,4 +314,24 @@ def valid_javascript?(str)
312314
rescue StandardError
313315
false
314316
end
317+
318+
def summarization(llm, input:)
319+
topic =
320+
Topic.new(
321+
category: Category.last,
322+
title: "Eval topic for topic summarization",
323+
id: -99,
324+
user_id: Discourse.system_user.id,
325+
)
326+
Post.new(topic: topic, id: -99, user_id: Discourse.system_user.id, raw: input)
327+
328+
strategy =
329+
DiscourseAi::Summarization::FoldContent.new(
330+
llm.llm_proxy,
331+
DiscourseAi::Summarization::Strategies::TopicSummary.new(topic),
332+
)
333+
334+
summary = DiscourseAi::TopicSummarization.new(strategy, Discourse.system_user).summarize
335+
summary.summarized_text
336+
end
315337
end

0 commit comments

Comments
 (0)