FEATURE: automation triage using personas (#1126)

## LLM Persona Triage
- Allows automated responses to posts using AI personas
- Configurable to respond as regular posts or whispers
- Adds context-aware formatting for topics and private messages
- Provides special handling for topic metadata (title, category, tags)

## LLM Tool Triage
- Enables custom AI tools to process and respond to posts
- Tools can analyze post content and invoke personas when needed
- Zero-parameter tools can be used for automated workflows
- Not enabled in production yet

## Implementation Details
- Added new scriptable registration in discourse_automation/ directory
- Created core implementation in lib/automation/ modules
- Enhanced PromptMessagesBuilder with topic-style formatting
- Added helper methods for persona and tool selection in UI
- Extended AI Bot functionality to support whisper responses
- Added rate limiting to prevent abuse

## Other Changes
- Added comprehensive test coverage for both automation types
- Enhanced tool runner with LLM integration capabilities
- Improved error handling and logging

This feature allows forum admins to configure AI personas to automatically respond to posts based on custom criteria and leverage AI tools for more complex triage workflows.

Tool Triage has been disabled in production while we finalize details of new scripting capabilities.

Author: SamSaffron

config/locales/client.en.yml

@@ -90,6 +90,24 @@ en:
               label: "Top P"
               description: "Top P to use for the LLM, increase to increase randomness (leave empty to use model default)"
 
+        llm_tool_triage:
+          fields:
+            model:
+              label: "Model"
+              description: "The default language model used for triage"
+            tool:
+              label: "Tool"
+              description: "Tool to use for triage (tool must have no parameters defined)"
+
+
+        llm_persona_triage:
+          fields:
+            persona:
+              label: "Persona"
+              description: "AI Persona to use for triage (must have default LLM and User set)"
+            whisper:
+              label: "Reply as Whisper"
+              description: "Whether the persona's response should be a whisper"
         llm_triage:
           fields:
             system_prompt:

config/locales/server.en.yml

@@ -6,6 +6,12 @@ en:
         spam: "Flag as spam and hide post"
         spam_silence: "Flag as spam, hide post and silence user"
     scriptables:
+      llm_tool_triage:
+        title: Triage posts using AI Tool
+        description: "Triage posts using custom logic in an AI tool"
+      llm_persona_triage:
+        title: Triage posts using AI Persona
+        description: "Respond to posts using a specific AI persona"
       llm_triage:
         title: Triage posts using AI
         description: "Triage posts using a large language model"

discourse_automation/llm_persona_triage.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+if defined?(DiscourseAutomation)
+  DiscourseAutomation::Scriptable.add("llm_persona_triage") do
+    version 1
+    run_in_background
+
+    triggerables %i[post_created_edited]
+
+    field :persona,
+          component: :choices,
+          required: true,
+          extra: {
+            content: DiscourseAi::Automation.available_persona_choices,
+          }
+    field :whisper, component: :boolean
+
+    script do |context, fields|
+      post = context["post"]
+      next if post&.user&.bot?
+
+      persona_id = fields["persona"]["value"]
+      whisper = fields["whisper"]["value"]
+
+      begin
+        RateLimiter.new(
+          Discourse.system_user,
+          "llm_persona_triage_#{post.id}",
+          SiteSetting.ai_automation_max_triage_per_post_per_minute,
+          1.minute,
+        ).performed!
+
+        RateLimiter.new(
+          Discourse.system_user,
+          "llm_persona_triage",
+          SiteSetting.ai_automation_max_triage_per_minute,
+          1.minute,
+        ).performed!
+
+        DiscourseAi::Automation::LlmPersonaTriage.handle(
+          post: post,
+          persona_id: persona_id,
+          whisper: whisper,
+          automation: self.automation,
+        )
+      rescue => e
+        Discourse.warn_exception(
+          e,
+          message: "llm_persona_triage: skipped triage on post #{post.id}",
+        )
+        raise e if Rails.env.tests?
+      end
+    end
+  end
+end

discourse_automation/llm_tool_triage.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+# TODO: this is still highly experimental and subject to a lot of change
+# leaving it off in production for now Sam
+if defined?(DiscourseAutomation) && !Rails.env.production?
+  DiscourseAutomation::Scriptable.add("llm_tool_triage") do
+    version 1
+    run_in_background
+
+    triggerables %i[post_created_edited]
+
+    field :tool,
+          component: :choices,
+          required: true,
+          extra: {
+            content: DiscourseAi::Automation.available_custom_tools,
+          }
+
+    script do |context, fields|
+      tool_id = fields["tool"]["value"]
+      post = context["post"]
+      return if post&.user&.bot?
+
+      begin
+        RateLimiter.new(
+          Discourse.system_user,
+          "llm_tool_triage_#{post.id}",
+          SiteSetting.ai_automation_max_triage_per_post_per_minute,
+          1.minute,
+        ).performed!
+
+        RateLimiter.new(
+          Discourse.system_user,
+          "llm_tool_triage",
+          SiteSetting.ai_automation_max_triage_per_minute,
+          1.minute,
+        ).performed!
+
+        DiscourseAi::Automation::LlmToolTriage.handle(
+          post: post,
+          tool_id: tool_id,
+          automation: self.automation,
+        )
+      rescue => e
+        Discourse.warn_exception(e, message: "llm_tool_triage: skipped triage on post #{post.id}")
+      end
+    end
+  end
+end

lib/ai_bot/playground.rb

@@ -170,7 +170,7 @@ def update_playground_with(post)
         schedule_bot_reply(post) if can_attach?(post)
       end
 
-      def conversation_context(post)
+      def conversation_context(post, style: nil)
         # Pay attention to the `post_number <= ?` here.
         # We want to inject the last post as context because they are translated differently.
 
@@ -205,6 +205,7 @@ def conversation_context(post)
             )
 
         builder = DiscourseAi::Completions::PromptMessagesBuilder.new
+        builder.topic = post.topic
 
         context.reverse_each do |raw, username, custom_prompt, upload_ids|
           custom_prompt_translation =
@@ -245,7 +246,7 @@ def conversation_context(post)
           end
         end
 
-        builder.to_a
+        builder.to_a(style: style || (post.topic.private_message? ? :bot : :topic))
       end
 
       def title_playground(post, user)
@@ -418,7 +419,7 @@ def get_context(participants:, conversation_context:, user:, skip_tool_details:
         result
       end
 
-      def reply_to(post, custom_instructions: nil, &blk)
+      def reply_to(post, custom_instructions: nil, whisper: nil, context_style: nil, &blk)
         # this is a multithreading issue
         # post custom prompt is needed and it may not
         # be properly loaded, ensure it is loaded
@@ -428,12 +429,18 @@ def reply_to(post, custom_instructions: nil, &blk)
         post_streamer = nil
 
         post_type =
-          post.post_type == Post.types[:whisper] ? Post.types[:whisper] : Post.types[:regular]
+          (
+            if (whisper || post.post_type == Post.types[:whisper])
+              Post.types[:whisper]
+            else
+              Post.types[:regular]
+            end
+          )
 
         context =
           get_context(
             participants: post.topic.allowed_users.map(&:username).join(", "),
-            conversation_context: conversation_context(post),
+            conversation_context: conversation_context(post, style: context_style),
             user: post.user,
           )
         context[:post_id] = post.id