feat: Edit with self correcting patches (#441)

The problem: Line editing is unreliable, full writes work but can have
side effects and devour tokens.

Main challenge is that LLMs suck at counting reliably. This solution
removes the need for counting and builds on the assumption that there is
plenty of patch data an LLM trained on.

But, likely the numbers are still close, which helps us deal with any
ambiguity in the patch. Bonus is that with this we can support multiple
edits in a file with a single LLM call and minimal tokens.

How it works:
* First parse the patch into a loosy format we can work with
* Iterate over the lines in the target file, if there is a match with a
hunk, create a candidate
* If that line matches with the next line for any current candidate,
keep the candidate, otherwise ditch
* Finally, we can recalculate the source and dest lines and ranges, and
re-render the patch.

The algorithm supports ambiguity (multiple hunks matching). Still needs
to be dealt with.

TODO:
- [x] Deal with ambiguity
- [x] More tests for multiple hunks
- [x] Test with diffy if the regenerated patches are correct
- [x] Add it all to the tool
- [x] Run some evals

If the evals pass we'll keep it in. After we run a set of SWE, if we see
no issues, as far as I'm concerned we can make it the default (and write
a cool blog article).

Aside, it should also be possible to Cow/borrow most of the inner values
and have this be near zero copy. The way it will be used however it's no
where near a bottleneck. Cool exercise though.

Author: timonv

Cargo.lock

@@ -96,6 +96,7 @@ update-informer = { version = "1.2.0", features = [
   "ureq",
   "rustls-tls",
 ], default-features = false }
+diffy = "0.4.2"
 
 # Something is still pulling in libssl, this is a quickfix and should be investigated
 [target.'cfg(linux)'.dependencies]
@@ -146,6 +147,7 @@ evaluations = []
 
 
 [patch.crates-io]
+# diffy = { git = "https://github.com/timonv/diffy", branch = "fix/debug-wrong-line" }
 # arrow = { version = "=53.2.0", optional = false }
 # arrow-arith = { version = "=53.2.0", optional = false }
 # swiftide = { path = "../swiftide/swiftide" }

Cargo.toml

@@ -230,13 +230,15 @@ Additionally, kwaak provides a number of slash commands, `/help` will show all a
 
 ### How does it work?
 
-On initial boot up, Kwaak will index your codebase. This can take a while, depending on the size. Once indexing has been completed once, subsequent startups will be faster. Indexes are stored with [duckdb](https://duckdb.org), and indexing is cached with [redb](https://github.com/cberner/redb).
+On initial boot up, Kwaak will index your codebase. This can take a while, depending on the size. Once indexing has been completed once, subsequent startups will be faster. Indexes are stored with [duckdb](https://duckdb.org). Kwaak uses the index to provide context to the agents.
 
 Kwaak provides a chat interface similar to other LLM chat applications. You can type messages to the agent, and the agent will try to accomplish the task and respond.
 
 When starting a chat, the code of the current branch is copied into an on-the-fly created docker container. This container is then used to run the code and execute the commands.
 
-After each chat completion, kwaak will lint, commit, and push the code to the remote repository if any code changes have been made. Kwaak can also create a pull request. Pull requests include an issue link to #48. This helps us identify the success rate of the agents, and also enforces transparency for code reviewers.
+After each chat completion, kwaak can lint, commit, and push the code to the remote repository if any code changes have been made. Kwaak can also create a pull request. Pull requests include an issue link to #48. This helps us identify the success rate of the agents, and also enforces transparency for code reviewers. This behaviour is fully configurable.
+
+Kwaak uses patch based editing by default. This means that only the changed lines are sent to the agent. This is more efficient. If you experience issues, try changing the edit mode to `whole` or `line`.
 
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
 
@@ -376,7 +378,7 @@ max_elapsed_time_sec = 120
 #### Other configuration
 
 - **`agent_custom_constraints`**: Additional constraints / instructions for the agent.
-  These are passes to the agent in the system prompt and are rendered in a list. If you
+  These are passes to the agent in the system prompt. If you
   intend to use more complicated instructions, consider adding a file to read in the
   repository instead.
 - **`cache_dir`, `log_dir`**: Directories for cache and logs. Defaults are within your system's cache directory.
@@ -386,7 +388,7 @@ max_elapsed_time_sec = 120
 - **`otel_enabled`**: Enables OpenTelemetry tracing if set and respects all the standard OpenTelemetry environment variables.
 - **`tool_executor`**: Defaults to `docker`. Can also be `local`. We **HIGHLY** recommend using `docker` for security reasons unless you are running in a secure environment.
 - **`tavily_api_key`**: Enables the agent to use [tavily](https://tavily.com) for web search. Their entry-level plan is free. (we are not affiliated)
-- **`agent_edit_mode`**: Defaults to `whole` (write full files at the time). If you experience issues with (very) large files, you can experiment with `line` edits.
+- **`agent_edit_mode`**: Defaults to `patch`. Other options are `whole` and `line`. If you experience issues, try changing the edit mode. `whole` will always write the full file. This consumes more tokens and can have side effects.
 - **`git.auto_push_remote`**: Enabled by default if a github key is present. Automatically pushes to the remote repository after each chat completion. You can disable this by setting it to `false`.
 - **`git.auto_commit_disabled`**: Opt-out of automatic commits after each chat completion.
 - **`tools`**: A list of tool names to enable or disable.

README.md

@@ -213,6 +213,13 @@ pub fn build_system_prompt(repository: &Repository) -> Result<Prompt> {
         ].into_iter().map(Into::into));
     }
 
+    if repository.config().agent_edit_mode.is_patch() {
+        constraints.extend([
+            "Prefer editing files with `patch_file` over `write_file`".into(),
+            "If `patch_file` continues to be troublesome, defer to `write_file` instead".into(),
+        ]);
+    }
+
     if repository.config().endless_mode {
         constraints
             .push("You cannot ask for feedback and have to try to complete the given task".into());

src/agent/agents/coding.rs

@@ -51,6 +51,7 @@ impl ConversationSummarizer {
         }
     }
 
+    #[must_use]
     pub fn summarize_hook(self) -> impl AfterEachFn {
         move |agent| {
             let llm = self.llm.clone();

src/agent/conversation_summarizer.rs

@@ -1,6 +1,6 @@
 pub mod agents;
 mod commit_and_push;
-mod conversation_summarizer;
+pub mod conversation_summarizer;
 pub mod env_setup;
 pub mod running_agent;
 pub mod session;

src/agent/mod.rs

@@ -402,6 +402,10 @@ pub fn available_tools(
             tools.push(tools::replace_lines());
             tools.push(tools::add_lines());
         }
+        AgentEditMode::Patch => {
+            tools.push(tools::read_file_with_line_numbers());
+            tools.push(tools::patch_file());
+        }
     }
 
     // gitHub-related tools

src/agent/session.rs

@@ -1,7 +1,9 @@
 mod delegate_agent;
+mod patch_file;
 mod replace_lines;
 
 pub use delegate_agent::DelegateAgent;
+pub use patch_file::patch_file;
 pub use replace_lines::replace_lines;
 
 use std::sync::Arc;