Enhance tool descriptions and update documentation for semantic and regex searches

Author: JegernOUTT

refact-agent/engine/README.md

@@ -64,7 +64,8 @@ Installable by the end user:
 - [x] Code completion with RAG
 - [x] Chat with tool usage
 - [x] definition() references() tools
-- [x] vecdb search() with scope
+- [x] vecdb search() with scope (semantic search)
+- [x] regex_search() with scope (pattern matching)
 - [x] @file @tree @web @definition @references @search mentions in chat
 - [x] locate() uses test-time compute to find good project cross-section
 - [x] Latest gpt-4o gpt-4o-mini

refact-agent/engine/src/agentic/compress_trajectory.rs

@@ -31,7 +31,7 @@ OR     (1) goal/thinking/coding/outcome (2) string according to the guidelines
 Example:
 [
 ["goal", "Rename my_function1 to my_function2"],
-["thinking", "There are definition(), search() and locate() tools, all can be used to find my_function1, system prompt says I need to start with locate()."],
+["thinking", "There are definition(), search(), regex_search() and locate() tools, all can be used to find my_function1, system prompt says I need to start with locate()."],
 ["locate(problem_statement=\"Rename my_function1 to my_function2\")", "The file my_script.py (1337 lines) has my_function1 on line 42."],
 ["thinking", "I can rewrite my_function1 inside my_script.py, so I'll do that."],
 ["update_textdoc(path=\"my_script\", old_str=\"...\", replacement=\"...\", multiple=false)", "The output of update_textdoc() has 15 lines_add and 15 lines_remove, confirming the operation."],

refact-agent/engine/src/tools/tool_locate_search.rs

@@ -27,15 +27,19 @@ MORE_TOCHANGE = likely to change as well, as a consequence of completing the tas
 USAGE = code that uses the things the task description is about
 SIMILAR = code that might provide an example of how to write similar things
 
-Your job is to use search() calls and summarize the results.
+Your job is to use search() and regex_search() calls and summarize the results.
 
 Some good ideas:
 
-search("MyClass1")                  -- if MyClass1 mentioned in the task, for each symbol
-search("log message 1 mentioned")   -- when the task has log messages, for each message
-search("    def f():\n        print(\"the example function!\")")   -- look for the code piece mentioned in the task
+search("MyClass1")                  -- if MyClass1 mentioned in the task, for semantic search of each symbol
+search("log message 1 mentioned")   -- when the task has log messages, for semantic search of each message
+search("    def f():\n        print(\"the example function!\")")   -- look for semantically similar code to the piece mentioned in the task
 search("imaginary_call(imaginary_arguments)\nmore_calls()\n")      -- you can imagine what kind of code you need to find
 
+regex_search("MyClass1")            -- if you need to find exact occurrences of a class name
+regex_search("(?i)error.*not found") -- if you need to find specific error patterns
+regex_search("function\\s+name\\s*\\(")  -- if you need to find function declarations with specific patterns
+
 Call any of those that make sense in parallel. Make at least two calls in parallel, pay special attention that at least one
 search() call should not have a restrictive scope, because you are running the risk of getting no results at all.
 "###;
@@ -191,13 +195,13 @@ async fn find_relevant_files_with_search(
     let mut msgs = vec![];
     msgs.push(ChatMessage::new("system".to_string(), LS_SYSTEM_PROMPT.to_string()));
     msgs.push(ChatMessage::new("user".to_string(), user_query.to_string()));
-    msgs.push(ChatMessage::new("cd_instruction".to_string(), "Look at user query above. Follow the system prompt. Run several search() calls in parallel.".to_string()));
+    msgs.push(ChatMessage::new("cd_instruction".to_string(), "Look at user query above. Follow the system prompt. Run several search() and regex_search() calls in parallel.".to_string()));
 
     let result = subchat(
         ccx.clone(),
         subchat_params.subchat_model.as_str(),
         msgs,
-        vec!["search".to_string()],
+        vec!["search".to_string(), "regex_search".to_string()],
         1,
         subchat_params.subchat_max_new_tokens,
         LS_WRAP_UP,

refact-agent/engine/src/tools/tool_relevant_files.rs

@@ -186,9 +186,10 @@ GOTODEF = call definition("xxx", skeleton=true) in parallel for symbols either v
 or symbols you can guess; don't call definition() for symbols from standard libraries, only symbols
 within the project are indexed.
 
-VECDBSEARCH = call up to five search() in parallel, some good ideas on what to look for: symbols
+VECDBSEARCH = call up to five search() or regex_search() in parallel. For semantic search, use search() with symbols
 mentioned in the task, one call for each symbol, strings mentioned, or write imaginary code that does the
-thing to fix search("    def f():\n        print(\"the example function!\")")
+thing to fix search("    def f():\n        print(\"the example function!\")"). For exact pattern matching, use
+regex_search() with regular expressions to find specific patterns like regex_search("function\\s+name\\s*\\(")
 
 You'll receive additional instructions that start with 💿. Those are not coming from the user, they are programmed to help you operate
 well between chat restarts and they are always in English. Answer in the language the user prefers.
@@ -236,8 +237,10 @@ Examples:
 definition("my_method1")
 definition("MyClass2")
 references("my_method2")
-search("    def f():\n        print(\"the example function!\")")
+search("    def f():\n        print(\"the example function!\")")  # semantic search for similar code
 search("    my_object->tricky_call(with, weird, parameters)")
+regex_search("my_method1")  # exact pattern matching
+regex_search("(?i)error.*not found")  # case-insensitive pattern matching
 
 Limits on the number of calls are pretty liberal, 10 definitions, 5 references and 3 searches is a reasonable request.
 

refact-agent/engine/src/tools/tools_description.rs

@@ -175,11 +175,11 @@ pub async fn tools_merged_and_filtered(
 const BUILT_IN_TOOLS: &str = r####"
 tools:
   - name: "search"
-    description: "Find similar pieces of code or text using vector database"
+    description: "Find semantically similar pieces of code or text using vector database (semantic search)"
     parameters:
       - name: "query"
         type: "string"
-        description: "Single line, paragraph or code sample to search for similar content."
+        description: "Single line, paragraph or code sample to search for semantically similar content."
       - name: "scope"
         type: "string"
         description: "'workspace' to search all files in workspace, 'dir/subdir/' to search in files within a directory, 'dir/file.ext' to search in a single file."
@@ -461,7 +461,7 @@ tools:
       - "language_slash_framework"
       
   - name: "regex_search"
-    description: "Search for patterns in files using regular expressions"
+    description: "Search for exact text patterns in files using regular expressions (pattern matching)"
     parameters:
       - name: "pattern"
         type: "string"

refact-agent/engine/src/yaml_configs/customization_compiled_in.yaml

@@ -46,7 +46,7 @@ PROMPT_EXPLORATION_TOOLS: |
   **Determine if the question is related to the current project**:
    - **If yes**:
        - Explain your plan briefly before calling any tools
-       - Gather the necessary context using `tree()`, `cat()`, `search()`, `definition()`, `references()` and other tool calls, or follow the user’s instructions.
+       - Gather the necessary context using `tree()`, `cat()`, `search()` (semantic search), `regex_search()` (pattern matching), `definition()`, `references()` and other tool calls, or follow the user’s instructions.
        - Ask clarifying questions if needed, making as many iterations as necessary to refine the context.
        - After gathering context, propose required project changes.
        - Then use `*_textdoc()` tools to make changes.
@@ -108,7 +108,7 @@ PROMPT_THINKING_AGENT: |
      - **Objective**: Expand your view of the project so no relevant information is overlooked.
      - Use `tree()` to explore the project structure.
      - Use `locate()` With the Full Problem Statement
-     - Use all other tools such as `search()`, `cat()`, `definition()`, etc. to collect every piece of relevant context.
+     - Use all other tools such as `search()` (semantic search), `regex_search()` (pattern matching), `cat()`, `definition()`, etc. to collect every piece of relevant context.
      - Open all files that might be indirectly referenced by the code.
   2. Plan Thoroughly With `think()`
      - **Objective**: Develop a precise plan before making any changes.