feat(human-relay): add clipboard monitoring for AI responses

[NyxJae]

• Add automatic detection of AI responses copied from browser
• Implement configurable monitoring interval (100-2000ms)
• Add duplicate response detection with warning alerts
• Update UI to show monitoring status and warnings
• Add new configuration options in settings

Context

This PR enhances the Human Relay feature by adding clipboard monitoring capabilities. Currently, users have to manually copy AI responses from web interfaces and paste them into the dialog box, which adds friction to the workflow. This enhancement aims to simplify this process by automatically detecting when the user copies an AI response, reducing the number of manual steps required during a Human Relay interaction.

Implementation

Module-level state management: Added variables to track previous responses and prompt text at the module level in human-relay.ts to maintain state between calls.
Text normalization: Implemented utility functions (normalizeText and isTextEqual) to reliably compare text content while ignoring whitespace differences.
Clipboard monitoring: Added an interval-based polling mechanism to check for clipboard changes. When a change is detected that isn't the original prompt or a previously detected response, it automatically submits the content as the AI's response.
Duplicate detection: Added logic to detect when a user accidentally copies the previous AI response instead of responding to the current prompt, showing a warning in such cases.
UI enhancements: Updated the dialog component to show monitoring status and warnings when needed.
Configuration: Added settings to enable/disable the feature and adjust the polling interval.

The implementation follows a non-intrusive approach - the original functionality remains intact, with the monitoring being an optional enhancement that users can enable.

Screenshots

before	after

How to Test

Get in Touch

---

Important

Enhances Human Relay with clipboard monitoring for AI responses, configurable settings, and UI updates for monitoring status and duplicate detection.

• Behavior:
  • Adds clipboard monitoring for AI responses in human-relay.ts with configurable intervals (100-2000ms).
  • Detects duplicate responses and shows warnings in HumanRelayDialog.tsx.
  • Updates UI in App.tsx and HumanRelayDialog.tsx to show monitoring status and warnings.
• Configuration:
  • Adds humanRelayMonitorClipboard and humanRelayMonitorInterval options in api.ts and globalState.ts.
  • Updates ApiOptions.tsx to include new settings for clipboard monitoring.
• Messages:
  • Adds closeHumanRelayDialog and showDuplicateResponseAlert message types in WebviewMessage.ts.

^{This description was created by for 9697f6f. It will automatically update as commits are pushed.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 7b3bfca

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ellipsis-dev]

Ensure unit/integration tests cover the new clipboard monitoring functionality, including duplicate response detection.

[mrubens]

This is cool tech, but I'm pretty unsure about the privacy implications of this. Is it just copying anything that's in the user's clipboard and sending it to the LLM without confirmation? That sounds risky to me.

[NyxJae]

@mrubens
Understanding Your Concerns

In simple terms, the act of sending clipboard content to the LLM is always a manual action by the user, and it is never sent automatically without user confirmation. However, it might indeed be necessary to add an explanation in the interface for the user.

In detail, the overall process when the auto-paste and send feature is enabled is as follows:

1. Roo prepares the prompt to be sent to the LLM.
2. A pop-up prompts the user and copies the prompt to the user's clipboard (automatically, transparent to the user. At this point, Roo starts monitoring the clipboard).
3. The user manually pastes the content to the web LLM (this is never automatic under any circumstances).
4. The LLM outputs a response, and the user manually copies it (the content sent to the LLM and the LLM's output are fully visible to the user on the web page, and copying the LLM's response is never automatic under any circumstances).
5. Roo detects a change in the clipboard, stops monitoring, and automatically pastes and sends the content to Roo (automatically, but it does not send the content to the LLM. However, it checks to ensure the content is a new AI response, not the current prompt or a previous AI response, to avoid cases of user mis-copying).
6. Roo prepares the next prompt.

The overall process does not involve automatically sending content to the LLM, and even all interactions with the LLM are fully visible to the user. If there are any concerns, it could only be that Roo automatically monitors the latest item on the user's clipboard, which might raise privacy concerns for some users. In that case, we could add a note stating, "Roo only monitors the latest item on the user's clipboard and does not send any clipboard content anywhere." Additionally, we can clarify that the code is open-source and can be reviewed by users.

[mrubens]

@mrubens Understanding Your Concerns

In simple terms, the act of sending clipboard content to the LLM is always a manual action by the user, and it is never sent automatically without user confirmation. However, it might indeed be necessary to add an explanation in the interface for the user.

In detail, the overall process when the auto-paste and send feature is enabled is as follows:

1. Roo prepares the prompt to be sent to the LLM.
2. A pop-up prompts the user and copies the prompt to the user's clipboard (automatically, transparent to the user. At this point, Roo starts monitoring the clipboard).
3. The user manually pastes the content to the web LLM (this is never automatic under any circumstances).
4. The LLM outputs a response, and the user manually copies it (the content sent to the LLM and the LLM's output are fully visible to the user on the web page, and copying the LLM's response is never automatic under any circumstances).
5. Roo detects a change in the clipboard, stops monitoring, and automatically pastes and sends the content to Roo (automatically, but it does not send the content to the LLM. However, it checks to ensure the content is a new AI response, not the current prompt or a previous AI response, to avoid cases of user mis-copying).
6. Roo prepares the next prompt.

The overall process does not involve automatically sending content to the LLM, and even all interactions with the LLM are fully visible to the user. If there are any concerns, it could only be that Roo automatically monitors the latest item on the user's clipboard, which might raise privacy concerns for some users. In that case, we could add a note stating, "Roo only monitors the latest item on the user's clipboard and does not send any clipboard content anywhere." Additionally, we can clarify that the code is open-source and can be reviewed by users.

Here’s the situation I’m thinking about:

1. You send a message with human relay, which gets copied to your clipboard
2. You go to your web LLM of choice
3. You realize you’re logged out
4. You open 1password and copy your password
5. That password gets sent to Roo

Is that what would happen, or am I misunderstanding? Overall I guess I’m just not sure if pasting the copied text is a large enough inconvenience to justify, but let me know if you disagree.

[NyxJae]

Overall, I guess I'm just not sure if pasting the copied text is a large enough inconvenience to justify, but let me know if you disagree.

Oh! Yes, there will be such a problem

[arthurauffray]

2. You go to your web LLM of choice
3. You realize you’re logged out
4. You open 1password and copy your password
5. That password gets sent to Roo

I see the implication this has, but considering:

1. roo is running locally
2. roo will throw a tool error + the user can see the last chat message
3. the user has enabled auto copy in settings, which explains this risk
4. 2fa is quite prominent, plus you would need an email/username if the password got into roo and then leaked out somehow

When checking if the clipboard has the new web LLM response, do some sort of check to confirm its an expected response with roo syntax. (Don't responses from LLM with roo always have XML tags?)

To me, it's worth the risk; I wouldn't mind full automation with the web LLM.

[NyxJae]

The clipboard switch has been moved to the dialog window. If users temporarily need to copy sensitive information, they can freely enable or disable clipboard monitoring.
Updates have been made to the review of copied content. AI responses that conform to the RooCode format must include at least one XML tag (for now, there is no requirement for a pair of XML tags, as it should be sufficient to filter out accidental copies like general passwords).
Additionally, the classification display for prompts about incorrectly copied content has been improved.

[snippins]

It might be better to add a shortcut to pull the clipboard from the chat. I was using aider copy-paste for a bit and the auto-copy is actually quite cumbersome in practice. It is better to hit copy in the web chat + some shortcut to pull it into vscode. If there is no way to do global shorcut, I think copy on web chat + alt-tab + one more shortcut to pull it into the chat would be convenient enough I think.

[NyxJae]

i8n, internationalization humanRelay

[NyxJae]

Hi @mrubens,
What are your thoughts on the current status of this PR?
It has been synced with the latest updates—would you consider merging it into the main branch?
If you have any suggestions or changes needed, please let me know.
Thanks for your review!

[hannesrudolph]

@NyxJae I am concerned this PR will get us on the shit lists of companies like Anthropic.

[NyxJae]

@NyxJae I am concerned this PR will get us on the shit lists of companies like Anthropic.我担心这个 PR 会让我们进入像 Anthropic 这样的公司的黑名单。

Hello, I think this PR complies with the vendor’s various terms of service, and I initially referred to the details outlined at https://aider.chat/docs/usage/copypaste.html#terms-of-service. Of course, if there are any concerns, I respect your perspective. Thank you for reviewing it~

[arthurauffray]

Vouching my continued support for this feature! When I want to use o1 pro without paying $150in,$600out this would greatly improve the UX for me. (especially when reading a bunch of files/other repetitive actions)

I understand your concern in terms of the naughty lists, but what would anthropic/others do?

[hannesrudolph]

As per our conversation @NyxJae I am closing this. Thank you for your excellent work here and sorry we are not merging it. We really do appreciate you and your contributions.