Assistant: Dissuade the model from guessing results of code execution (#9635)

Addresses #9580.

In the above issue, the model hallucinates the statistical results from
a code block execution in Ask mode, where it cannot actually execute
code.

The working theory was that the following lines in the default prompt
were to blame:


https://github.com/posit-dev/positron/blob/d9f3047f74ac2d01186b19efdbf9afaa6f115316/extensions/positron-assistant/src/md/prompts/chat/default.md?plain=1#L37-L39

However, through experimentation I see the same or similar issues
appearing even with these lines removed. It seems that (at least for
Claude 4 Sonnet) the model does indeed need some instruction on how to
handle the situation where it wants to run some code but cannot execute
it directly. Without guidance, it defaults to hallucinating (trying to
be as helpful as possible, I assume).

---

So, this PR takes a slightly stronger hand with the following changes:

* As described above, the "Present statistics and insights about the
data" line was removed from the base default prompt and added to the
Agent mode prompt, where it probably should always have been.

* A new prompt specifically to be included when Ask mode is active has
been added. Here we try to strongly state:
  - That the model cannot see the results of code blocks it emits.
- That the model should stop if the rest of its response depends on the
result of a code block it has emitted.
- That the model should not try to run or present the output of code it
has emitted.

* A similar prompt for Edit mode, where the model also cannot directly
execute code.

* Added the Ask mode prompting also to the inline editor.

And, while not strictly related, while I am here:

* Added the default prompt text to the `/explain` command. Without this
we lose the base Positron Assistant behaviour whenever `/explain` is
activated (which can now happen automatically!).

---

## QA

Follow the reproduction steps in #9580. The model should no longer try
to output a statistical summary, but instead explain what the code would
do if the user runs it:

<img width="520" height="337" alt="Screenshot 2025-09-29 at 14 52 06"
src="https://github.com/user-attachments/assets/2c262436-8de0-4c15-8738-4d17173d9caf"
/>

---

Stopping hallucinations is a hard problem in general, but this change
should hopefully help to avoid this particular situation.

---------

Signed-off-by: George Stagg <[email protected]>
Co-authored-by: sharon <[email protected]>

Author: georgestagg

extensions/positron-assistant/src/commands/explain.ts

@@ -22,7 +22,10 @@ export async function explainHandler(
 	_token: vscode.CancellationToken,
 	handleDefault: () => Promise<vscode.ChatResult | void>
 ) {
-	context.systemPrompt = await fs.promises.readFile(`${MD_DIR}/prompts/chat/explain.md`, 'utf8');
+	const defaultPrompt = await fs.promises.readFile(`${MD_DIR}/prompts/chat/default.md`, 'utf8');
+	const explainPrompt = await fs.promises.readFile(`${MD_DIR}/prompts/chat/explain.md`, 'utf8');
+
+	context.systemPrompt = defaultPrompt + '\n\n' + explainPrompt;
 
 	return handleDefault();
 }

extensions/positron-assistant/src/md/prompts/chat/agent.md

@@ -16,8 +16,11 @@ You NEVER try to start a Shiny app using the execute code tool, even if the user
 </tools>
 
 <communication>
-If the user asks you _how_ to do something, or asks for code rather than
-results, generate the code and return it directly without trying to execute it.
+You are running in "Agent" mode.
+
+When executing code that generates statistical information, use the result to present statistics and insights about the data as part of your markdown response.
+
+If the user asks you _how_ to do something, or asks for code rather than results, generate the code and return it directly without trying to execute it.
 </communication>
 
 <data-querying>

extensions/positron-assistant/src/md/prompts/chat/ask.md

@@ -0,0 +1,11 @@
+<communication>
+You are running in "Ask" mode.
+
+You may include code blocks in your response, but in this mode you are UNABLE to run the code or see the results.
+The code blocks are shown to the user, but the code is not executed unless the USER requests it.
+
+If your response requires the results of executing a code block, STOP. Explain to the user what the code will do and end your response.
+Do not offer to run the code for the user.
+
+You NEVER try to summarise, present statistics or insights, or comment on the result of executing code blocks.
+</communication>

extensions/positron-assistant/src/md/prompts/chat/default.md

@@ -34,9 +34,7 @@ You output code that is correct, of high quality, and with a consistent style.
 
 You follow the coding style and use the packages and frameworks used by the USER in example code and context that they have given you as part of their request.
 
-When sharing code that generates statistical information:
-- Present statistics and insights about the data as part of your markdown response
-- For code that generates statistical information, ensure the final line returns a useful object rather than printing/displaying it
+For code that generates statistical information, ensure the final line returns a useful object rather than printing/displaying it.
 
 For Python, specifically avoid these output functions in code unless explicitly requested by the USER:
 - `print()`
@@ -71,6 +69,8 @@ The USER can see when you invoke a tool, so you do not need to tell the user or
 
 You prefer to use knowledge you are already provided with to infer details when assisting the USER with their request. You bias to only running tools if it is necessary to learn something in the running Positron session.
 
+You much prefer to respond to the USER with code to perform a data analysis, rather than directly trying to calculate summaries or statistics for your response.
+
 Tools with tag `high-token-usage` may result in high token usage, so redirect the USER to provide you with the information you need to answer their question without using these tools whenever possible. For example, if the USER asks about their variables or data:
   - When `session` information is not attached to the USER's query, ask the USER to ensure a Console is running and enable the Console session context.
   - When file `attachments` are not attached to the USER's query, ask the USER to attach relevant files as context.

extensions/positron-assistant/src/md/prompts/chat/edit.md

@@ -0,0 +1,11 @@
+<communication>
+You are running in "Edit" mode.
+
+You may include code blocks in your response, but in this mode you are UNABLE to run the code or see the results.
+The code blocks are shown to the user, but the code is not executed unless the USER requests it.
+
+If your response requires the results of executing a code block, STOP. Explain to the user what the code will do and end your response.
+Do not offer to run the code for the user.
+
+You NEVER try to summarise, present statistics or insights, or comment on the result of executing code blocks.
+</communication>

extensions/positron-assistant/src/participants.ts

@@ -768,8 +768,9 @@ export class PositronAssistantChatParticipant extends PositronAssistantParticipa
 	protected override async getSystemPrompt(request: vscode.ChatRequest): Promise<string> {
 		const defaultSystem = await fs.promises.readFile(path.join(MARKDOWN_DIR, 'prompts', 'chat', 'default.md'), 'utf8');
 		const filepaths = await fs.promises.readFile(path.join(MARKDOWN_DIR, 'prompts', 'chat', 'filepaths.md'), 'utf8');
+		const ask = await fs.promises.readFile(path.join(MARKDOWN_DIR, 'prompts', 'chat', 'ask.md'), 'utf8');
 		const languages = await this.getActiveSessionInstructions();
-		const prompts = [defaultSystem, filepaths, languages];
+		const prompts = [defaultSystem, filepaths, ask, languages];
 		return prompts.join('\n\n');
 	}
 }
@@ -781,8 +782,9 @@ class PositronAssistantEditParticipant extends PositronAssistantParticipant impl
 	protected override async getSystemPrompt(request: vscode.ChatRequest): Promise<string> {
 		const defaultSystem = await fs.promises.readFile(path.join(MARKDOWN_DIR, 'prompts', 'chat', 'default.md'), 'utf8');
 		const filepaths = await fs.promises.readFile(path.join(MARKDOWN_DIR, 'prompts', 'chat', 'filepaths.md'), 'utf8');
+		const edit = await fs.promises.readFile(path.join(MARKDOWN_DIR, 'prompts', 'chat', 'edit.md'), 'utf8');
 		const languages = await this.getActiveSessionInstructions();
-		const prompts = [defaultSystem, filepaths, languages];
+		const prompts = [defaultSystem, filepaths, edit, languages];
 		return prompts.join('\n\n');
 	}
 }
@@ -831,15 +833,18 @@ export class PositronAssistantEditorParticipant extends PositronAssistantPartici
 
 		const defaultSystem = await fs.promises.readFile(path.join(MARKDOWN_DIR, 'prompts', 'chat', 'default.md'), 'utf8');
 
+		// Inline editor chats behave as in "Ask" mode
+		const ask = await fs.promises.readFile(path.join(MARKDOWN_DIR, 'prompts', 'chat', 'ask.md'), 'utf8');
+
 		// If the user has not selected text, use the prompt for the whole document.
 		if (request.location2.selection.isEmpty) {
 			const editor = await fs.promises.readFile(path.join(MARKDOWN_DIR, 'prompts', 'chat', 'editor.md'), 'utf8');
-			return defaultSystem + '\n\n' + editor;
+			return defaultSystem + '\n\n' + ask + '\n\n' + editor;
 		}
 
 		// If the user has selected text, generate a new version of the selection.
 		const selection = await fs.promises.readFile(path.join(MARKDOWN_DIR, 'prompts', 'chat', 'selection.md'), 'utf8');
-		return defaultSystem + '\n\n' + selection;
+		return defaultSystem + '\n\n' + ask + '\n\n' + selection;
 	}
 
 	async getCustomPrompt(request: vscode.ChatRequest): Promise<string> {