get_dependency agent tool

[mcelrath]

Create an agent tool like get_dependency that will get the full source code for an actual dependency in a project. For instance consider a Rust project and running cargo build -v -- it tells you the relevant file:

error[E0599]: no function or associated item named `xx` found for struct `libp2p::libp2p_noise::Config` in the current scope
   --> relay_test_client_destination/src/main.rs:40:37
    |
40  |     let noise_config = NoiseConfig::xx(identity.clone())
    |                                     ^^ function or associated item not found in `Config`
    |
note: if you're trying to build a new `libp2p::libp2p_noise::Config`, consider using `libp2p::libp2p_noise::Config::new` which returns `Result<libp2p::libp2p_noise::Config, libp2p::libp2p_noise::Error>`
   --> /home/mcelrath/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/libp2p-noise-0.46.0/src/lib.rs:101:5
    |
101 |     pub fn new(identity: &identity::Keypair) -> Result<Self, Error> {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For more information about this error, try `rustc --explain E0599`.
error: could not compile `relay_test_client_2` (bin "relay_test_client_2") due to 1 previous error

where in this case we want to parse /home/mcelrath/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/libp2p-noise-0.46.0/src/lib.rs and feed it to the LLM before asking it to fix the bug. Without this, the LLM will hallucinate the functions and parameters involved.

There are many ways this tool could work. It could respond with a class structure, function signature, or the code of a function. (I want to reduce context here, so it would be better to use tree-sitter to find the relevant parts of the file so as to not have to give the LLM the entire file)

This is language-specific and will require different branches/functions for each language. Python, go, js generally have the source being referenced somewhere on disk. C/C++ may not and may be using an interface .h file and a static or dynamically linked library.

There's a way to do this (specifically for the problem of solving compiler errors) where the dependencies are automatically provided. This ticket is to provide a tool that an AI can choose to use or not in the course of its reasoning.

Compilers always know exactly which code they're looking at, because of Makefiles, Cargo.toml, requirements.txt, pyproject,toml, etc. The goal here is to obtain exactly the same thing the compiler is seeing in a tool, using the compiler's own tools.

[mcelrath]

The way I envision this being used as a tool (instead of compiler error solving which probably only requires one trip through the LLM) is that during the planning stage (architect mode) the LLM might suggest adding a dependency to requirements.txt/Cargo.toml. Before letting it finish it's plan, we really should fetch the new dependency and give it some info about it.

Likewise if the plan requires using a new function or data structure not currently being used by the existing code, we should fetch the required source files before letting the LLM finish it's plan.

This would be a more "agentic" form of "architect" mode that would require several trips through the LLM during the planning stage:

1. Formulate draft plan
2. Identify new functions, data structures, packages used in the draft plan
3. Fetch relevant files, possibly update requirements.txt/cargo.toml and let the compiler/build system fetch them
4. Provide the files as context and the draft plan. Ask the LLM to refine the plan
5. Repeat 2-4 until there's nothing new
6. Finalize plan. Identify any outstanding issues or unknowns (figure out what to do if the LLM expressed uncertainty here...)
7. Pass the plan and context to the code editor LLM (which may or may not be the same LLM used for planning)