Skip to content

Commit 03bbfdb

Browse files
KillianLucasKillianLucas
authored
Merge pull request #1484 from MikeBirdTech/screenpipe
Integrate with Screenpipe
2 parents bbbdc76 + 5a493c9 commit 03bbfdb

File tree

2 files changed

+273
-0
lines changed

2 files changed

+273
-0
lines changed

‎examples/screenpipe.ipynb

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Open Interpreter and ScreenPipe Cookbook\n",
8+
"\n",
9+
"This cookbook explores the powerful combination of Open Interpreter and ScreenPipe, two tools that can significantly enhance your ability to interact with and process digital information. Open Interpreter allows you to execute natural language commands, while ScreenPipe captures and analyzes screen content and audio output from your computer."
10+
]
11+
},
12+
{
13+
"cell_type": "markdown",
14+
"metadata": {},
15+
"source": [
16+
"## Prerequisites\n",
17+
"\n",
18+
"Before we begin, make sure you have installed the following:\n",
19+
"\n",
20+
"1. [Open Interpreter](https://docs.openinterpreter.com/getting-started/introduction)\n",
21+
"2. [Screenpipe CLI](https://docs.screenpi.pe/docs/getting-started#cli-installation)\n",
22+
"\n",
23+
"Make sure both are properly installed and configured on your system."
24+
]
25+
},
26+
{
27+
"cell_type": "markdown",
28+
"metadata": {},
29+
"source": [
30+
"## Setting Up Open Interpreter"
31+
]
32+
},
33+
{
34+
"cell_type": "code",
35+
"execution_count": null,
36+
"metadata": {},
37+
"outputs": [],
38+
"source": [
39+
"# Import necessary libraries\n",
40+
"from interpreter import interpreter\n",
41+
"from datetime import datetime, timezone\n",
42+
"\n",
43+
"# Configure Open Interpreter\n",
44+
"interpreter.llm.model = \"groq/llama-3.1-70b-versatile\"\n",
45+
"interpreter.computer.import_computer_api = False\n",
46+
"interpreter.llm.supports_functions = False\n",
47+
"interpreter.llm.supports_vision = False\n",
48+
"interpreter.llm.context_window = 100000\n",
49+
"interpreter.llm.max_tokens = 4096\n",
50+
"\n",
51+
"# Add the current date and time in UTC\n",
52+
"current_datetime = datetime.now(timezone.utc).strftime(\"%Y-%m-%d %H:%M:%S UTC\")\n",
53+
"print(f\"Current date and time: {current_datetime}\")"
54+
]
55+
},
56+
{
57+
"cell_type": "markdown",
58+
"metadata": {},
59+
"source": [
60+
"## Defining the ScreenPipe Search Function"
61+
]
62+
},
63+
{
64+
"cell_type": "code",
65+
"execution_count": null,
66+
"metadata": {},
67+
"outputs": [],
68+
"source": [
69+
"# Define the custom ScreenPipe search function\n",
70+
"custom_tool = \"\"\"\n",
71+
"import requests\n",
72+
"import json\n",
73+
"from urllib.parse import quote\n",
74+
"\n",
75+
"def search_screenpipe(query, limit=5, start_time=None, end_time=None):\n",
76+
" base_url = f\"http://localhost:3030/search?q={quote(query)}&content_type=ocr&limit={limit}\"\n",
77+
" \n",
78+
" if start_time:\n",
79+
" base_url += f\"&start_time={quote(start_time)}\"\n",
80+
" if end_time:\n",
81+
" base_url += f\"&end_time={quote(end_time)}\"\n",
82+
" \n",
83+
" response = requests.get(base_url)\n",
84+
" if response.status_code == 200:\n",
85+
" data = response.json()\n",
86+
" # Remove duplicates based on text content\n",
87+
" unique_results = []\n",
88+
" seen_texts = set()\n",
89+
" for item in data[\"data\"]:\n",
90+
" text = item[\"content\"][\"text\"]\n",
91+
" if text not in seen_texts:\n",
92+
" unique_results.append(item)\n",
93+
" seen_texts.add(text)\n",
94+
" return unique_results\n",
95+
" else:\n",
96+
" return f\"Error: Unable to fetch data from ScreenPipe. Status code: {response.status_code}\"\n",
97+
"\"\"\"\n",
98+
"\n",
99+
"# Add the custom tool to the interpreter's environment\n",
100+
"interpreter.computer.run(\"python\", custom_tool)\n",
101+
"print(\"ScreenPipe search function defined and added to the interpreter's environment.\")"
102+
]
103+
},
104+
{
105+
"cell_type": "markdown",
106+
"metadata": {},
107+
"source": [
108+
"## Setting Custom Instructions for Open Interpreter"
109+
]
110+
},
111+
{
112+
"cell_type": "code",
113+
"execution_count": null,
114+
"metadata": {},
115+
"outputs": [],
116+
"source": [
117+
"# Set custom instructions for Open Interpreter\n",
118+
"interpreter.custom_instructions = f\"\"\"\n",
119+
"Current date and time: {current_datetime}\n",
120+
"\n",
121+
"ScreenPipe is a powerful tool that continuously captures and indexes the content displayed on your screen. It creates a searchable history of everything you've seen or interacted with on your computer. This includes text from websites, documents, applications, and even images (through OCR).\n",
122+
"\n",
123+
"You have access to this wealth of information through the `search_screenpipe(query, limit=5, start_time=None, end_time=None)` function. This allows you to provide more contextual and personalized assistance based on the user's recent activities and viewed content.\n",
124+
"\n",
125+
"The `search_screenpipe` function supports optional `start_time` and `end_time` parameters to narrow down the search to a specific time range. The time format should be ISO 8601, like this: \"2024-10-16T12:00:00Z\".\n",
126+
"\n",
127+
"Here's why querying ScreenPipe is valuable:\n",
128+
"1. Context Recall: Users often refer to things they've seen recently but may not remember the exact details. ScreenPipe can help recall this information.\n",
129+
"2. Information Verification: You can cross-reference user claims or questions with actual content they've viewed.\n",
130+
"3. Personalized Assistance: By knowing what the user has been working on or researching, you can provide more relevant advice and suggestions.\n",
131+
"4. Productivity Enhancement: You can help users quickly locate information they've seen before but can't remember where.\n",
132+
"\n",
133+
"Use the `search_screenpipe()` function when:\n",
134+
"- The user asks about something they've seen or read recently.\n",
135+
"- You need to verify or expand on information the user mentions.\n",
136+
"- You want to provide context-aware suggestions or assistance.\n",
137+
"- The user is trying to recall specific details from their recent computer usage.\n",
138+
"- The user wants to search within a specific time range.\n",
139+
"\n",
140+
"Here's how to use it effectively:\n",
141+
"1. When a user's query relates to recent activities or viewed content, identify key terms for the search.\n",
142+
"2. If the user specifies a time range, use the `start_time` and `end_time` parameters.\n",
143+
"3. Call the `search_screenpipe()` function with these parameters.\n",
144+
"4. Analyze the results to find relevant information.\n",
145+
"5. Summarize the findings for the user, mentioning the source (app name, window name) and when it was seen (timestamp).\n",
146+
"\n",
147+
"Remember to use this tool proactively when you think it might help answer the user's question, even if they don't explicitly mention ScreenPipe.\n",
148+
"\"\"\"\n",
149+
"\n",
150+
"print(\"Custom instructions set for Open Interpreter.\")"
151+
]
152+
}
153+
],
154+
"metadata": {
155+
"kernelspec": {
156+
"display_name": "Python 3",
157+
"language": "python",
158+
"name": "python3"
159+
},
160+
"language_info": {
161+
"codemirror_mode": {
162+
"name": "ipython",
163+
"version": 3
164+
},
165+
"file_extension": ".py",
166+
"mimetype": "text/x-python",
167+
"name": "python",
168+
"nbconvert_exporter": "python",
169+
"pygments_lexer": "ipython3",
170+
"version": "3.11.9"
171+
}
172+
},
173+
"nbformat": 4,
174+
"nbformat_minor": 4
175+
}

‎interpreter/terminal_interface/profiles/defaults/screenpipe.py

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
"""
2+
This is an Open Interpreter profile specialized for searching ScreenPipe history.
3+
It leverages Llama 3.1 70b served by Groq and requires the environment variable GROQ_API_KEYH to be set.
4+
"""
5+
6+
# Configure Open Interpreter
7+
from interpreter import interpreter
8+
from datetime import datetime, timezone
9+
10+
interpreter.llm.model = "groq/llama-3.1-70b-versatile"
11+
interpreter.computer.import_computer_api = False
12+
interpreter.llm.supports_functions = False
13+
interpreter.llm.supports_vision = False
14+
interpreter.llm.context_window = 100000
15+
interpreter.llm.max_tokens = 4096
16+
17+
# Add the current date and time in UTC
18+
current_datetime = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
19+
20+
custom_tool = """
21+
import requests
22+
import json
23+
from urllib.parse import quote
24+
25+
def search_screenpipe(query, limit=5, start_time=None, end_time=None):
26+
base_url = f"http://localhost:3030/search?q={quote(query)}&content_type=ocr&limit={limit}"
27+
28+
if start_time:
29+
base_url += f"&start_time={quote(start_time)}"
30+
if end_time:
31+
base_url += f"&end_time={quote(end_time)}"
32+
33+
response = requests.get(base_url)
34+
if response.status_code == 200:
35+
data = response.json()
36+
# Remove duplicates based on text content
37+
unique_results = []
38+
seen_texts = set()
39+
for item in data["data"]:
40+
text = item["content"]["text"]
41+
if text not in seen_texts:
42+
unique_results.append(item)
43+
seen_texts.add(text)
44+
return unique_results
45+
else:
46+
return f"Error: Unable to fetch data from ScreenPipe. Status code: {response.status_code}"
47+
"""
48+
49+
# Add the custom tool to the interpreter's environment
50+
interpreter.computer.run("python", custom_tool)
51+
52+
interpreter.custom_instructions = f"""
53+
Current date and time: {current_datetime}
54+
55+
ScreenPipe is a powerful tool that continuously captures and indexes the content displayed on your screen. It creates a searchable history of everything you've seen or interacted with on your computer. This includes text from websites, documents, applications, and even images (through OCR).
56+
57+
You have access to this wealth of information through the `search_screenpipe(query, limit=5, start_time=None, end_time=None)` function. This allows you to provide more contextual and personalized assistance based on the user's recent activities and viewed content.
58+
59+
The `search_screenpipe` function supports optional `start_time` and `end_time` parameters to narrow down the search to a specific time range. The time format should be ISO 8601, like this: "2024-10-16T12:00:00Z".
60+
61+
Here's why querying ScreenPipe is valuable:
62+
1. Context Recall: Users often refer to things they've seen recently but may not remember the exact details. ScreenPipe can help recall this information.
63+
2. Information Verification: You can cross-reference user claims or questions with actual content they've viewed.
64+
3. Personalized Assistance: By knowing what the user has been working on or researching, you can provide more relevant advice and suggestions.
65+
4. Productivity Enhancement: You can help users quickly locate information they've seen before but can't remember where.
66+
67+
Use the `search_screenpipe()` function when:
68+
- The user asks about something they've seen or read recently.
69+
- You need to verify or expand on information the user mentions.
70+
- You want to provide context-aware suggestions or assistance.
71+
- The user is trying to recall specific details from their recent computer usage.
72+
- The user wants to search within a specific time range.
73+
74+
Here's how to use it effectively:
75+
1. When a user's query relates to recent activities or viewed content, identify key terms for the search.
76+
2. If the user specifies a time range, use the `start_time` and `end_time` parameters.
77+
3. Call the `search_screenpipe()` function with these parameters.
78+
4. Analyze the results to find relevant information.
79+
5. Summarize the findings for the user, mentioning the source (app name, window name) and when it was seen (timestamp).
80+
81+
Remember to use this tool proactively when you think it might help answer the user's question, even if they don't explicitly mention ScreenPipe.
82+
83+
Example usage in Python:
84+
```python
85+
# Search without time range
86+
results = search_screenpipe("Open Interpreter", limit=3)
87+
88+
# Search with time range
89+
results = search_screenpipe("project meeting", limit=5, start_time="2024-10-16T12:00:00Z", end_time="2024-10-16T19:00:00Z")
90+
91+
for result in results:
92+
print(f"Text: {{result['content']['text'][:300]}}...") # Print first 100 characters
93+
print(f"Source: {{result['content']['app_name']}} - {{result['content']['window_name']}}")
94+
print(f"Timestamp: {{result['content']['timestamp']}}")
95+
```
96+
97+
Write valid code.
98+
"""

0 commit comments

Comments
 (0)