Skip to content
This repository was archived by the owner on Jul 22, 2025. It is now read-only.

Commit 5b6d39a

Browse files
authored
FEATURE: flexible image handling within messages (#1214)
* DEV: refactor bot internals This introduces a proper object for bot context, this makes it simpler to improve context management as we go cause we have a nice object to work with Starts refactoring allowing for a single message to have multiple uploads throughout * transplant method to message builder * chipping away at inline uploads * image support is improved but not fully fixed yet partially working in anthropic, still got quite a few dialects to go * open ai and claude are now working * Gemini is now working as well * fix nova * more dialects... * fix ollama * fix specs * update artifact fixed * more tests * spam scanner * pass more specs * bunch of specs improved * more bug fixes. * all the rest of the tests are working * improve tests coverage and ensure custom tools are aware of new context object * tests are working, but we need more tests * resolve merge conflict * new preamble and expanded specs on ai tool * remove concept of "standalone tools" This is no longer needed, we can set custom raw, tool details are injected into tool calls
1 parent f3e78f0 commit 5b6d39a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+1380
-722
lines changed

app/controllers/discourse_ai/admin/ai_tools_controller.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ def test
5555
# we need an llm so we have a tokenizer
5656
# but will do without if none is available
5757
llm = LlmModel.first&.to_llm
58-
runner = @ai_tool.runner(parameters, llm: llm, bot_user: current_user, context: {})
58+
runner = @ai_tool.runner(parameters, llm: llm, bot_user: current_user)
5959
result = runner.invoke
6060

6161
if result.is_a?(Hash) && result[:error]

app/jobs/regular/stream_discover_reply.rb

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,13 @@ def execute(args)
3030

3131
base = { query: query, model_used: llm_model.display_name }
3232

33-
bot.reply(
34-
{ conversation_context: [{ type: :user, content: query }], skip_tool_details: true },
35-
) do |partial|
33+
context =
34+
DiscourseAi::AiBot::BotContext.new(
35+
messages: [{ type: :user, content: query }],
36+
skip_tool_details: true,
37+
)
38+
39+
bot.reply(context) do |partial|
3640
streamed_reply << partial
3741

3842
# Throttle updates.

app/models/ai_tool.rb

Lines changed: 161 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ def function_call_name
3535
tool_name.presence || name
3636
end
3737

38-
def runner(parameters, llm:, bot_user:, context: {})
38+
def runner(parameters, llm:, bot_user:, context: nil)
3939
DiscourseAi::AiBot::ToolRunner.new(
4040
parameters: parameters,
4141
llm: llm,
@@ -59,86 +59,166 @@ def regenerate_rag_fragments
5959

6060
def self.preamble
6161
<<~JS
62-
/**
63-
* Tool API Quick Reference
64-
*
65-
* Entry Functions
66-
*
67-
* invoke(parameters): Main function. Receives parameters (Object). Must return a JSON-serializable value.
68-
* Example:
69-
* function invoke(parameters) { return "result"; }
70-
*
71-
* details(): Optional. Returns a string describing the tool.
72-
* Example:
73-
* function details() { return "Tool description."; }
74-
*
75-
* Provided Objects
76-
*
77-
* 1. http
78-
* http.get(url, options?): Performs an HTTP GET request.
79-
* Parameters:
80-
* url (string): The request URL.
81-
* options (Object, optional):
82-
* headers (Object): Request headers.
83-
* Returns:
84-
* { status: number, body: string }
85-
*
86-
* http.post(url, options?): Performs an HTTP POST request.
87-
* Parameters:
88-
* url (string): The request URL.
89-
* options (Object, optional):
90-
* headers (Object): Request headers.
91-
* body (string): Request body.
92-
* Returns:
93-
* { status: number, body: string }
94-
*
95-
* (also available: http.put, http.patch, http.delete)
96-
*
97-
* Note: Max 20 HTTP requests per execution.
98-
*
99-
* 2. llm
100-
* llm.truncate(text, length): Truncates text to a specified token length.
101-
* Parameters:
102-
* text (string): Text to truncate.
103-
* length (number): Max tokens.
104-
* Returns:
105-
* Truncated string.
106-
*
107-
* 3. index
108-
* index.search(query, options?): Searches indexed documents.
109-
* Parameters:
110-
* query (string): Search query.
111-
* options (Object, optional):
112-
* filenames (Array): Limit search to specific files.
113-
* limit (number): Max fragments (up to 200).
114-
* Returns:
115-
* Array of { fragment: string, metadata: string }
116-
*
117-
* 4. upload
118-
* upload.create(filename, base_64_content): Uploads a file.
119-
* Parameters:
120-
* filename (string): Name of the file.
121-
* base_64_content (string): Base64 encoded file content.
122-
* Returns:
123-
* { id: number, short_url: string }
124-
*
125-
* 5. chain
126-
* chain.setCustomRaw(raw): Sets the body of the post and exist chain.
127-
* Parameters:
128-
* raw (string): raw content to add to post.
129-
*
130-
* Constraints
131-
*
132-
* Execution Time: ≤ 2000ms
133-
* Memory: ≤ 10MB
134-
* HTTP Requests: ≤ 20 per execution
135-
* Exceeding limits will result in errors or termination.
136-
*
137-
* Security
138-
*
139-
* Sandboxed Environment: No access to system or global objects.
140-
* No File System Access: Cannot read or write files.
141-
*/
62+
/**
63+
* Tool API Quick Reference
64+
*
65+
* Entry Functions
66+
*
67+
* invoke(parameters): Main function. Receives parameters defined in the tool's signature (Object).
68+
* Must return a JSON-serializable value (e.g., string, number, object, array).
69+
* Example:
70+
* function invoke(parameters) { return { result: "Data processed", input: parameters.query }; }
71+
*
72+
* details(): Optional function. Returns a string (can include basic HTML) describing
73+
* the tool's action after invocation, often using data from the invocation.
74+
* This is displayed in the chat interface.
75+
* Example:
76+
* let lastUrl;
77+
* function invoke(parameters) {
78+
* lastUrl = parameters.url;
79+
* // ... perform action ...
80+
* return { success: true, content: "..." };
81+
* }
82+
* function details() {
83+
* return `Browsed: <a href="${lastUrl}">${lastUrl}</a>`;
84+
* }
85+
*
86+
* Provided Objects & Functions
87+
*
88+
* 1. http
89+
* Performs HTTP requests. Max 20 requests per execution.
90+
*
91+
* http.get(url, options?): Performs GET request.
92+
* Parameters:
93+
* url (string): The request URL.
94+
* options (Object, optional):
95+
* headers (Object): Request headers (e.g., { "Authorization": "Bearer key" }).
96+
* Returns: { status: number, body: string }
97+
*
98+
* http.post(url, options?): Performs POST request.
99+
* Parameters:
100+
* url (string): The request URL.
101+
* options (Object, optional):
102+
* headers (Object): Request headers.
103+
* body (string | Object): Request body. If an object, it's stringified as JSON.
104+
* Returns: { status: number, body: string }
105+
*
106+
* http.put(url, options?): Performs PUT request (similar to POST).
107+
* http.patch(url, options?): Performs PATCH request (similar to POST).
108+
* http.delete(url, options?): Performs DELETE request (similar to GET/POST).
109+
*
110+
* 2. llm
111+
* Interacts with the Language Model.
112+
*
113+
* llm.truncate(text, length): Truncates text to a specified token length based on the configured LLM's tokenizer.
114+
* Parameters:
115+
* text (string): Text to truncate.
116+
* length (number): Maximum number of tokens.
117+
* Returns: string (truncated text)
118+
*
119+
* llm.generate(prompt): Generates text using the configured LLM associated with the tool runner.
120+
* Parameters:
121+
* prompt (string | Object): The prompt. Can be a simple string or an object
122+
* like { messages: [{ type: "system", content: "..." }, { type: "user", content: "..." }] }.
123+
* Returns: string (generated text)
124+
*
125+
* 3. index
126+
* Searches attached RAG (Retrieval-Augmented Generation) documents linked to this tool.
127+
*
128+
* index.search(query, options?): Searches indexed document fragments.
129+
* Parameters:
130+
* query (string): The search query used for semantic search.
131+
* options (Object, optional):
132+
* filenames (Array<string>): Filter search to fragments from specific uploaded filenames.
133+
* limit (number): Maximum number of fragments to return (default: 10, max: 200).
134+
* Returns: Array<{ fragment: string, metadata: string | null }> - Ordered by relevance.
135+
*
136+
* 4. upload
137+
* Handles file uploads within Discourse.
138+
*
139+
* upload.create(filename, base_64_content): Uploads a file created by the tool, making it available in Discourse.
140+
* Parameters:
141+
* filename (string): The desired name for the file (basename is used for security).
142+
* base_64_content (string): Base64 encoded content of the file.
143+
* Returns: { id: number, url: string, short_url: string } - Details of the created upload record.
144+
*
145+
* 5. chain
146+
* Controls the execution flow.
147+
*
148+
* chain.setCustomRaw(raw): Sets the final raw content of the bot's post and immediately
149+
* stops the tool execution chain. Useful for tools that directly
150+
* generate the full response content (e.g., image generation tools attaching the image markdown).
151+
* Parameters:
152+
* raw (string): The raw Markdown content for the post.
153+
* Returns: void
154+
*
155+
* 6. discourse
156+
* Interacts with Discourse specific features. Access is generally performed as the SystemUser.
157+
*
158+
* discourse.search(params): Performs a Discourse search.
159+
* Parameters:
160+
* params (Object): Search parameters (e.g., { search_query: "keyword", with_private: true, max_results: 10 }).
161+
* `with_private: true` searches across all posts visible to the SystemUser. `result_style: 'detailed'` is used by default.
162+
* Returns: Object (Discourse search results structure, includes posts, topics, users etc.)
163+
*
164+
* discourse.getPost(post_id): Retrieves details for a specific post.
165+
* Parameters:
166+
* post_id (number): The ID of the post.
167+
* Returns: Object (Post details including `raw`, nested `topic` object with ListableTopicSerializer structure) or null if not found/accessible.
168+
*
169+
* discourse.getTopic(topic_id): Retrieves details for a specific topic.
170+
* Parameters:
171+
* topic_id (number): The ID of the topic.
172+
* Returns: Object (Topic details using ListableTopicSerializer structure) or null if not found/accessible.
173+
*
174+
* discourse.getUser(user_id_or_username): Retrieves details for a specific user.
175+
* Parameters:
176+
* user_id_or_username (number | string): The ID or username of the user.
177+
* Returns: Object (User details using UserSerializer structure) or null if not found.
178+
*
179+
* discourse.getPersona(name): Gets an object representing another AI Persona configured on the site.
180+
* Parameters:
181+
* name (string): The name of the target persona.
182+
* Returns: Object { respondTo: function(params) } or null if persona not found.
183+
* respondTo(params): Instructs the target persona to generate a response within the current context (e.g., replying to the same post or chat message).
184+
* Parameters:
185+
* params (Object, optional): { instructions: string, whisper: boolean }
186+
* Returns: { success: boolean, post_id?: number, post_number?: number, message_id?: number } or { error: string }
187+
*
188+
* discourse.createChatMessage(params): Creates a new message in a Discourse Chat channel.
189+
* Parameters:
190+
* params (Object): { channel_name: string, username: string, message: string }
191+
* `channel_name` can be the channel name or slug.
192+
* `username` specifies the user who should appear as the sender. The user must exist.
193+
* The sending user must have permission to post in the channel.
194+
* Returns: { success: boolean, message_id?: number, message?: string, created_at?: string } or { error: string }
195+
*
196+
* 7. context
197+
* An object containing information about the environment where the tool is being run.
198+
* Available properties depend on the invocation context, but may include:
199+
* post_id (number): ID of the post triggering the tool (if in a Post context).
200+
* topic_id (number): ID of the topic (if in a Post context).
201+
* private_message (boolean): Whether the context is a private message (in Post context).
202+
* message_id (number): ID of the chat message triggering the tool (if in Chat context).
203+
* channel_id (number): ID of the chat channel (if in Chat context).
204+
* user (Object): Details of the user invoking the tool/persona (structure may vary, often null or SystemUser details unless explicitly passed).
205+
* participants (string): Comma-separated list of usernames in a PM (if applicable).
206+
* // ... other potential context-specific properties added by the calling environment.
207+
*
208+
* Constraints
209+
*
210+
* Execution Time: ≤ 2000ms (default timeout in milliseconds) - This timer *pauses* during external HTTP requests or LLM calls initiated via `http.*` or `llm.generate`, but applies to the script's own processing time.
211+
* Memory: ≤ 10MB (V8 heap limit)
212+
* Stack Depth: ≤ 20 (Marshal stack depth limit for Ruby interop)
213+
* HTTP Requests: ≤ 20 per execution
214+
* Exceeding limits will result in errors or termination (e.g., timeout error, out-of-memory error, TooManyRequestsError).
215+
*
216+
* Security
217+
*
218+
* Sandboxed Environment: The script runs in a restricted V8 JavaScript environment (via MiniRacer).
219+
* No direct access to browser or environment, browser globals (like `window` or `document`), or the host system's file system.
220+
* Network requests are proxied through the Discourse backend, not made directly from the sandbox.
221+
*/
142222
JS
143223
end
144224

lib/ai_bot/bot.rb

Lines changed: 10 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -75,32 +75,35 @@ def get_updated_title(conversation_context, post, user)
7575
def force_tool_if_needed(prompt, context)
7676
return if prompt.tool_choice == :none
7777

78-
context[:chosen_tools] ||= []
78+
context.chosen_tools ||= []
7979
forced_tools = persona.force_tool_use.map { |tool| tool.name }
80-
force_tool = forced_tools.find { |name| !context[:chosen_tools].include?(name) }
80+
force_tool = forced_tools.find { |name| !context.chosen_tools.include?(name) }
8181

8282
if force_tool && persona.forced_tool_count > 0
8383
user_turns = prompt.messages.select { |m| m[:type] == :user }.length
8484
force_tool = false if user_turns > persona.forced_tool_count
8585
end
8686

8787
if force_tool
88-
context[:chosen_tools] << force_tool
88+
context.chosen_tools << force_tool
8989
prompt.tool_choice = force_tool
9090
else
9191
prompt.tool_choice = nil
9292
end
9393
end
9494

9595
def reply(context, &update_blk)
96+
unless context.is_a?(BotContext)
97+
raise ArgumentError, "context must be an instance of BotContext"
98+
end
9699
llm = DiscourseAi::Completions::Llm.proxy(model)
97100
prompt = persona.craft_prompt(context, llm: llm)
98101

99102
total_completions = 0
100103
ongoing_chain = true
101104
raw_context = []
102105

103-
user = context[:user]
106+
user = context.user
104107

105108
llm_kwargs = { user: user }
106109
llm_kwargs[:temperature] = persona.temperature if persona.temperature
@@ -277,27 +280,15 @@ def process_tool(
277280
name: tool.name,
278281
}
279282

280-
if tool.standalone?
281-
standalone_context =
282-
context.dup.merge(
283-
conversation_context: [
284-
context[:conversation_context].last,
285-
tool_call_message,
286-
tool_message,
287-
],
288-
)
289-
prompt = persona.craft_prompt(standalone_context)
290-
else
291-
prompt.push(**tool_call_message)
292-
prompt.push(**tool_message)
293-
end
283+
prompt.push(**tool_call_message)
284+
prompt.push(**tool_message)
294285

295286
raw_context << [tool_call_message[:content], tool_call_id, "tool_call", tool.name]
296287
raw_context << [invocation_result_json, tool_call_id, "tool", tool.name]
297288
end
298289

299290
def invoke_tool(tool, llm, cancel, context, &update_blk)
300-
show_placeholder = !context[:skip_tool_details] && !tool.class.allow_partial_tool_calls?
291+
show_placeholder = !context.skip_tool_details && !tool.class.allow_partial_tool_calls?
301292

302293
update_blk.call("", cancel, build_placeholder(tool.summary, "")) if show_placeholder
303294

0 commit comments

Comments
 (0)