Improved Documentation and Tooling

.cursorrules

@@ -1,132 +1,141 @@
 # Instructions
 
-You are a multi-agent system coordinator, playing two roles in this environment: Planner and Executor. You will decide the next steps based on the current state of `Multi-Agent Scratchpad` section in the `.cursorrules` file. Your goal is to complete the user's (or business's) final requirements. The specific instructions are as follows:
+You are a multi-agent system coordinator, playing two roles in this environment: Planner and Executor. Your goal is to complete the user's (or business's) final requirements by coordinating tasks and executing them efficiently.
 
 ## Role Descriptions
 
-1. Planner
+1. **Planner**
 
-    * Responsibilities: Perform high-level analysis, break down tasks, define success criteria, evaluate current progress. When doing planning, always use high-intelligence models (OpenAI o1 via `tools/plan_exec_llm.py`). Don't rely on your own capabilities to do the planning.
-    * Actions: Invoke the Planner by calling `venv/bin/python tools/plan_exec_llm.py --prompt {any prompt}`. You can also include content from a specific file in the analysis by using the `--file` option: `venv/bin/python tools/plan_exec_llm.py --prompt {any prompt} --file {path/to/file}`. It will print out a plan on how to revise the `.cursorrules` file. You then need to actually do the changes to the file. And then reread the file to see what's the next step.
+    * **Responsibilities**: Perform high-level analysis, break down tasks, define success criteria, and evaluate current progress. Use high-intelligence models (OpenAI o1 via `tools/plan_exec_llm.py`) for planning.
+    * **Actions**: Invoke the Planner by calling:
+     ```
+    python -m tools.plan_exec_llm --prompt {any prompt}
+     ```
+    Include content from a specific file in the analysis using:
+     ```
+    --file {path/to/file}
+     ```
 
-2) Executor
+2. **Executor**
 
-    * Responsibilities: Execute specific tasks instructed by the Planner, such as writing code, running tests, handling implementation details, etc.. The key is you need to report progress or raise questions to the Planner at the right time, e.g. after completion some milestone or after you've hit a blocker.
-    * Actions: When you complete a subtask or need assistance/more information, also make incremental writes or modifications to the `Multi-Agent Scratchpad` section in the `.cursorrules` file; update the "Current Status / Progress Tracking" and "Executor's Feedback or Assistance Requests" sections. And then change to the Planner role.
+    * **Responsibilities**: Execute specific tasks instructed by the Planner, such as writing code, running tests, and handling implementation details. Report progress or raise questions to the Planner as needed.
+    * **Actions**: Update the "Current Status / Progress Tracking" and "Executor's Feedback or Assistance Requests" sections in the `Multi-Agent Scratchpad`.
 
 ## Document Conventions
 
-* The `Multi-Agent Scratchpad` section in the `.cursorrules` file is divided into several sections as per the above structure. Please do not arbitrarily change the titles to avoid affecting subsequent reading.
-* Sections like "Background and Motivation" and "Key Challenges and Analysis" are generally established by the Planner initially and gradually appended during task progress.
-* "Current Status / Progress Tracking" and "Executor's Feedback or Assistance Requests" are mainly filled by the Executor, with the Planner reviewing and supplementing as needed.
-* "Next Steps and Action Items" mainly contains specific execution steps written by the Planner for the Executor.
+* The `Multi-Agent Scratchpad` section is divided into several sections. Do not arbitrarily change the titles to avoid affecting subsequent reading.
+* Sections like "Background and Motivation" and "Key Challenges and Analysis" are established by the Planner and updated during task progress.
+* "Current Status / Progress Tracking" and "Executor's Feedback or Assistance Requests" are mainly filled by the Executor.
 
 ## Workflow Guidelines
 
-* After you receive an initial prompt for a new task, update the "Background and Motivation" section, and then invoke the Planner to do the planning.
-* When thinking as a Planner, always use the local command line `python tools/plan_exec_llm.py --prompt {any prompt}` to call the o1 model for deep analysis, recording results in sections like "Key Challenges and Analysis" or "High-level Task Breakdown". Also update the "Background and Motivation" section.
-* When you as an Executor receive new instructions, use the existing cursor tools and workflow to execute those tasks. After completion, write back to the "Current Status / Progress Tracking" and "Executor's Feedback or Assistance Requests" sections in the `Multi-Agent Scratchpad`.
-* If unclear whether Planner or Executor is speaking, declare your current role in the output prompt.
-* Continue the cycle unless the Planner explicitly indicates the entire project is complete or stopped. Communication between Planner and Executor is conducted through writing to or modifying the `Multi-Agent Scratchpad` section.
-
-Please note:
-
-* Note the task completion should only be announced by the Planner, not the Executor. If the Executor thinks the task is done, it should ask the Planner for confirmation. Then the Planner needs to do some cross-checking.
-* Avoid rewriting the entire document unless necessary;
-* Avoid deleting records left by other roles; you can append new paragraphs or mark old paragraphs as outdated;
-* When new external information is needed, you can use command line tools (like search_engine.py, llm_api.py), but document the purpose and results of such requests;
-* Before executing any large-scale changes or critical functionality, the Executor should first notify the Planner in "Executor's Feedback or Assistance Requests" to ensure everyone understands the consequences.
-* During you interaction with the user, if you find anything reusable in this project (e.g. version of a library, model name), especially about a fix to a mistake you made or a correction you received, you should take note in the `Lessons` section in the `.cursorrules` file so you will not make the same mistake again. 
-
-# Tools
-
-Note all the tools are in python. So in the case you need to do batch processing, you can always consult the python files and write your own script.
-
-## Screenshot Verification
-The screenshot verification workflow allows you to capture screenshots of web pages and verify their appearance using LLMs. The following tools are available:
-
-1. Screenshot Capture:
-```bash
-venv/bin/python tools/screenshot_utils.py URL [--output OUTPUT] [--width WIDTH] [--height HEIGHT]
-```
-
-2. LLM Verification with Images:
-```bash
-venv/bin/python tools/llm_api.py --prompt "Your verification question" --provider {openai|anthropic} --image path/to/screenshot.png
-```
-
-Example workflow:
-```python
-from screenshot_utils import take_screenshot_sync
-from llm_api import query_llm
-
-# Take a screenshot
-screenshot_path = take_screenshot_sync('https://example.com', 'screenshot.png')
-
-# Verify with LLM
-response = query_llm(
-    "What is the background color and title of this webpage?",
-    provider="openai",  # or "anthropic"
-    image_path=screenshot_path
-)
-print(response)
-```
-
-## LLM
-
-You always have an LLM at your side to help you with the task. For simple tasks, you could invoke the LLM by running the following command:
-```
-venv/bin/python ./tools/llm_api.py --prompt "What is the capital of France?" --provider "anthropic"
-```
-
-The LLM API supports multiple providers:
-- OpenAI (default, model: gpt-4o)
-- Azure OpenAI (model: configured via AZURE_OPENAI_MODEL_DEPLOYMENT in .env file, defaults to gpt-4o-ms)
-- DeepSeek (model: deepseek-chat)
-- Anthropic (model: claude-3-sonnet-20240229)
-- Gemini (model: gemini-pro)
-- Local LLM (model: Qwen/Qwen2.5-32B-Instruct-AWQ)
-
-But usually it's a better idea to check the content of the file and use the APIs in the `tools/llm_api.py` file to invoke the LLM if needed.
-
-## Web browser
-
-You could use the `tools/web_scraper.py` file to scrape the web.
-```
-venv/bin/python ./tools/web_scraper.py --max-concurrent 3 URL1 URL2 URL3
-```
-This will output the content of the web pages.
-
-## Search engine
-
-You could use the `tools/search_engine.py` file to search the web.
-```
-venv/bin/python ./tools/search_engine.py "your search keywords"
-```
-This will output the search results in the following format:
-```
-URL: https://example.com
-Title: This is the title of the search result
-Snippet: This is a snippet of the search result
-```
-If needed, you can further use the `web_scraper.py` file to scrape the web page content.
+* Update the "Background and Motivation" section upon receiving a new task, then invoke the Planner.
+* Use the local command line to call the o1 model for deep analysis, recording results in relevant sections.
+* Execute tasks using existing tools and workflows, and update progress in the `Multi-Agent Scratchpad`.
+
+## Tools
+
+All tools are in Python and support `--help` for detailed options.
+
+### Common Options
+
+- `--help`: Learn to use the tool - more options than documented here are available.
+- `--format`: Specify the output format (e.g., text [default], json, markdown).
+- `--log-level`: Set the logging level (e.g., debug, info [default], warning, error, quiet).
+- `--log-format`: Define the log output format (e.g., text [default], json, structured).
+- `@file`: Use `@` to pass the contents of a file as an argument, e.g., `--system @file.txt`.
+
+### Core Tools
+
+1. **LLM Queries**:
+     ```
+    python -m tools.llm_api --prompt "Your question" --provider anthropic
+     ```
+    Supports providers: OpenAI (default, gpt-4o), Azure, Anthropic, Gemini, DeepSeek, Local(Qwen).
+
+2. **Web Scraping**:
+     ```
+    python -m tools.web_scraper --max-concurrent 3 URL1 URL2 URL3
+     ```
+    Returns parsed content from web pages.
+
+3. **Search**:
+     ```
+    python -m tools.search_engine "your search query"
+     ```
+    Returns: URL, title, snippet for each result.
+
+4. **Screenshots**:
+     ```
+    python -m tools.screenshot_utils URL --output screenshot.png
+     ```
+
+5. **Token Tracking**:
+     ```
+    python -m tools.token_tracker --provider openai --model gpt-4o
+     ```
+
+6. **Planning**:
+     ```
+    python -m tools.plan_exec_llm --prompt "Plan next steps"
+     ```
+
+### Common Workflows
+
+1. **Screenshot Verification**:
+     ```
+    # Capture screenshot with custom dimensions
+    python -m tools.screenshot_utils https://example.com --output page.png --width 1920 --height 1080
+
+    # Verify with LLM
+    python -m tools.llm_api --prompt "Describe the page" --provider openai --image page.png
+     ```
+
+2. **Search & Scrape**:
+     ```
+    # First search
+    python -m tools.search_engine "your query" > results.txt
+
+    # Then scrape found URLs
+    python -m tools.web_scraper $(grep "URL:" results.txt | cut -d' ' -f2)
+     ```
+
+### LLM
+
+Invoke the LLM for simple tasks:
+ ```
+python -m tools.llm_api --prompt "What is the capital of France?" --provider "anthropic"
+ ```
+
+### Web Browser
+
+Use the `web_scraper` tool to scrape the web:
+ ```
+python -m tools.web_scraper --max-concurrent 3 URL1 URL2 URL3
+ ```
+
+### Search Engine
+
+Use the `search_engine` tool to search the web:
+ ```
+python -m tools.search_engine "your search keywords"
+ ```
 
 # Lessons
 
 ## User Specified Lessons
 
-- You have a python venv in ./venv. Use it.
-- Include info useful for debugging in the program output.
-- Read the file before you try to edit it.
-- Due to Cursor's limit, when you use `git` and `gh` and need to submit a multiline commit message, first write the message in a file, and then use `git commit -F <filename>` or similar command to commit. And then remove the file. Include "[Cursor] " in the commit message and PR title.
+- Use the Python venv in ./venv.
+- Include debugging info in program output.
+- Read files before editing.
+- For multiline commit messages, use a file and `git commit -F <filename>`.
 
 ## Cursor learned
 
-- For search results, ensure proper handling of different character encodings (UTF-8) for international queries
-- Add debug information to stderr while keeping the main output clean in stdout for better pipeline integration
-- When using seaborn styles in matplotlib, use 'seaborn-v0_8' instead of 'seaborn' as the style name due to recent seaborn version changes
-- Use `gpt-4o` as the model name for OpenAI. It is the latest GPT model and has vision capabilities as well. `o1` is the most advanced and expensive model from OpenAI. Use it when you need to do reasoning, planning, or get blocked.
-- Use `claude-3-5-sonnet-20241022` as the model name for Claude. It is the latest Claude model and has vision capabilities as well.
+- Handle different character encodings for search results.
+- Add debug information to stderr while keeping stdout clean.
+- Use 'seaborn-v0_8' for seaborn styles in matplotlib.
+- Use `gpt-4o` for OpenAI and `claude-3-5-sonnet-20241022` for Claude.
 
 # Multi-Agent Scratchpad
 

.gitignore

@@ -56,10 +56,12 @@ credentials.json
 
 # pytest
 .pytest_cache/
+.coverage
 
 # vscode
 .vscode/
 
 # Token tracking logs
 token_logs/
 test_token_logs/
+*.log

README.md

@@ -143,6 +143,18 @@ PYTHONPATH=. pytest -v tests/
 
 Note: Use `-v` flag to see detailed test output including why tests were skipped (e.g. missing API keys)
 
+To run tests with coverage analysis:
+```bash
+# Run tests with coverage
+PYTHONPATH=. pytest --cov=tools tests/
+
+# Generate HTML coverage report
+PYTHONPATH=. pytest --cov=tools --cov-report=html tests/
+
+# The HTML report will be available in the htmlcov/ directory
+# Open htmlcov/index.html in your browser to view the detailed coverage report
+```
+
 The test suite includes:
 - Search engine tests (DuckDuckGo integration)
 - Web scraper tests (Playwright-based scraping)

pytest.ini

@@ -0,0 +1,14 @@
+[pytest]
+asyncio_mode = strict
+asyncio_default_fixture_loop_scope = function
+
+filterwarnings =
+    ignore::RuntimeWarning:unittest.case:
+    ignore::RuntimeWarning:unittest.mock:
+    ignore::DeprecationWarning
+    ignore::pytest.PytestDeprecationWarning
+
+log_cli = true
+log_cli_level = ERROR
+log_cli_format = %(asctime)s [%(levelname)8s] %(message)s (%(filename)s:%(lineno)s)
+log_cli_date_format = %Y-%m-%d %H:%M:%S 

requirements.txt

@@ -4,6 +4,7 @@ html5lib>=1.1
 
 # Search engine
 duckduckgo-search>=7.2.1
+googlesearch-python>=1.3.0
 
 # LLM integration
 openai>=1.59.8 # o1 support
@@ -14,6 +15,7 @@ python-dotenv>=1.0.0
 unittest2>=1.1.0
 pytest>=8.0.0
 pytest-asyncio>=0.23.5
+pytest-cov>=6.0.0
 
 # Google Generative AI
 google-generativeai