Add tests for toolchain functionality and enhance Python tool setup with UV integration

- Introduced `toolchain_tests.rs` for validating Python tool setup, installation, and directory structure.
- Enhanced Python tool package installation by integrating `uv` for performance; implemented fallback to `pip`.
- Improved error handling, logging, and installation path validation in Python tool setup.
- Updated hook execution to parse and manage commands with arguments more robustly.
- Extended documentation and planning with new tasks and options for Python environment setup.

Author: jensenbox

docs/plan.md

@@ -13,8 +13,10 @@ Tasks:
 - [x] We need a fundamental change in the way we run hooks. The configuration file should be used to create the context for each task we want to run and the context should contain the entry we are needing to run as a process. We need to be able to specify hooks that are to be ran from the built-in hooks versus hooks that are external. We could potentially run them in the same process or in a separate process. If we choose to run them in a separate process then we need to ensure that there is a cli command that the entry can point to. This will allow us to run the hooks in parallel and also allow us to run hooks in a different process than the main application. Should help debugging specific hooks.
 - [x] Make sure that the file in `docs/.pre-commit-config.yaml` can be used and converted to the rustyhook configuration file format. Importantly it should set up its own python environment and install the required packages without needing the system to have anything installed.
 - [x] Instead of having a separate binary called "rustyhooks-hooks" I think we should continue to use our primary "rustyhook" binary and have a sub-command that can be used to run the hooks. This will allow us to run the hooks in parallel and also allow us to run the hooks in a different process than the main application. Should help debugging specific hooks.
-- [ ] replace the use of pip with uv - potentially link to the source code so as to get the benefit without needing a local binary. This should help speed up the installation of a python interpreter as well as improve the performance ot package installation. We will likely be able to remove a substantial amount of the code we have in our source code for installing python interpreters and installing packages.
+- [ ] Use either https://gregoryszorc.com/docs/python-build-standalone/main/ or https://pyoxidizer.readthedocs.io/en/latest/pyoxy.html to download the specified python version and install it locally.
+- [ ] Have uv use the downloaded python interpreter that is specified in the .python-version file. use uv for package installation. We will likely be able to remove a substantial amount of the code we have in our source code for installing python interpreters and installing packages.
 - [ ] emulate all known pre-commit hooks for use with native execution - all known hooks at https://github.com/pre-commit/pre-commit-hooks
+- [ ] Break up the implementations of the hooks into separate files and create a test suite for each file.
 - [ ] create tests for each hook and complete coverage
 - [ ] replace the npm and nvm install process with fnm if it makes sense to do so
 - [ ] expand all built in hooks to include all known hooks from https://pre-commit.com/hooks.html

src/runner/hook_context.rs

@@ -148,10 +148,27 @@ impl HookContext {
     pub fn run_in_separate_process(&self) -> Result<(), HookContextError> {
         println!("Running hook {} in separate process", self.id);
 
+        // Parse the entry to separate the command from any arguments
+        let parts: Vec<&str> = self.entry.split_whitespace().collect();
+        if parts.is_empty() {
+            return Err(HookContextError::ProcessError(format!(
+                "Empty entry for hook {}", self.id
+            )));
+        }
+
+        // The first part is the command, the rest are arguments
+        let command_name = parts[0];
+        let command_args = &parts[1..];
+
         // Create a command to run the hook
-        let mut command = Command::new(&self.entry);
+        let mut command = Command::new(command_name);
+
+        // Add any arguments from the entry
+        for arg in command_args {
+            command.arg(arg);
+        }
 
-        // Add arguments
+        // Add arguments from the hook configuration
         for arg in &self.args {
             command.arg(arg);
         }

src/toolchains/python.rs

@@ -302,27 +302,158 @@ impl PythonTool {
         Ok(())
     }
 
-    /// Install packages in the virtualenv
+    /// Install packages in the virtualenv using pip (traditional approach)
     fn install_packages(&self, ctx: &SetupContext) -> Result<(), ToolError> {
+        // Find the python executable in the virtualenv
+        let python = if cfg!(windows) {
+            ctx.install_dir.join("Scripts").join("python.exe")
+        } else {
+            ctx.install_dir.join("bin").join("python")
+        };
+
         // Find the pip executable in the virtualenv
         let pip = if cfg!(windows) {
             ctx.install_dir.join("Scripts").join("pip.exe")
         } else {
             ctx.install_dir.join("bin").join("pip")
         };
 
-        // Install each package
-        for package in &self.packages {
+        // Check if the python executable exists in the virtualenv
+        if !python.exists() {
+            return Err(ToolError::ExecutionError(
+                format!("Python executable not found at {:?}", python),
+            ));
+        }
+
+        log::debug!("Python executable found at {:?}", python);
+
+        // Install all packages at once for better performance
+        if !self.packages.is_empty() {
+            // First, try to install uv directly using pip
             let status = Command::new(&pip)
                 .arg("install")
-                .arg(package)
+                .arg("uv")
                 .status()
-                .map_err(|e| ToolError::ExecutionError(format!("Failed to install {}: {}", package, e)))?;
+                .map_err(|e| ToolError::ExecutionError(format!("Failed to install uv: {}", e)))?;
 
             if !status.success() {
-                return Err(ToolError::ExecutionError(
-                    format!("Failed to install {}", package),
-                ));
+                log::warn!("Failed to install uv, falling back to regular pip for package installation");
+
+                // If uv installation fails, fall back to regular pip for package installation
+                let mut cmd = Command::new(&pip);
+                cmd.arg("install");
+
+                // Add all packages as arguments
+                for package in &self.packages {
+                    cmd.arg(package);
+                }
+
+                log::debug!("Running pip command: {:?}", cmd);
+
+                let output = cmd.output()
+                    .map_err(|e| ToolError::ExecutionError(format!("Failed to install packages with pip: {}", e)))?;
+
+                if !output.status.success() {
+                    let stderr = String::from_utf8_lossy(&output.stderr);
+                    let stdout = String::from_utf8_lossy(&output.stdout);
+                    log::error!("pip stderr: {}", stderr);
+                    log::error!("pip stdout: {}", stdout);
+                    return Err(ToolError::ExecutionError(
+                        format!("Failed to install packages with pip: {}", stderr),
+                    ));
+                }
+
+                log::debug!("Successfully installed packages with pip");
+            } else {
+                // If uv installation succeeds, use it to install packages
+                let uv = if cfg!(windows) {
+                    ctx.install_dir.join("Scripts").join("uv.exe")
+                } else {
+                    ctx.install_dir.join("bin").join("uv")
+                };
+
+                // Check if the uv executable exists
+                if !uv.exists() {
+                    log::warn!("uv executable not found at {:?}, falling back to regular pip", uv);
+
+                    // If uv is not found, fall back to regular pip
+                    let mut cmd = Command::new(&pip);
+                    cmd.arg("install");
+
+                    // Add all packages as arguments
+                    for package in &self.packages {
+                        cmd.arg(package);
+                    }
+
+                    log::debug!("Running pip command: {:?}", cmd);
+
+                    let output = cmd.output()
+                        .map_err(|e| ToolError::ExecutionError(format!("Failed to install packages with pip: {}", e)))?;
+
+                    if !output.status.success() {
+                        let stderr = String::from_utf8_lossy(&output.stderr);
+                        let stdout = String::from_utf8_lossy(&output.stdout);
+                        log::error!("pip stderr: {}", stderr);
+                        log::error!("pip stdout: {}", stdout);
+                        return Err(ToolError::ExecutionError(
+                            format!("Failed to install packages with pip: {}", stderr),
+                        ));
+                    }
+
+                    log::debug!("Successfully installed packages with pip");
+                } else {
+                    // Use uv to install packages
+                    let mut cmd = Command::new(&uv);
+                    cmd.arg("pip")
+                        .arg("install");
+
+                    // Add all packages as arguments
+                    for package in &self.packages {
+                        cmd.arg(package);
+                    }
+
+                    log::debug!("Running uv command: {:?}", cmd);
+
+                    let output = cmd.output()
+                        .map_err(|e| ToolError::ExecutionError(format!("Failed to install packages with uv: {}", e)))?;
+
+                    if !output.status.success() {
+                        let stderr = String::from_utf8_lossy(&output.stderr);
+                        let stdout = String::from_utf8_lossy(&output.stdout);
+                        log::error!("uv stderr: {}", stderr);
+                        log::error!("uv stdout: {}", stdout);
+
+                        log::warn!("Failed to install packages with uv, falling back to regular pip");
+
+                        // If uv fails, fall back to regular pip
+                        let mut cmd = Command::new(&pip);
+                        cmd.arg("install");
+
+                        // Add all packages as arguments
+                        for package in &self.packages {
+                            cmd.arg(package);
+                        }
+
+                        log::debug!("Running pip command: {:?}", cmd);
+
+                        let output = cmd.output()
+                            .map_err(|e| ToolError::ExecutionError(format!("Failed to install packages with pip: {}", e)))?;
+
+                        if !output.status.success() {
+                            let stderr = String::from_utf8_lossy(&output.stderr);
+                            let stdout = String::from_utf8_lossy(&output.stdout);
+                            log::error!("pip stderr: {}", stderr);
+                            log::error!("pip stdout: {}", stdout);
+                            return Err(ToolError::ExecutionError(
+                                format!("Failed to install packages with pip: {}", stderr),
+                            ));
+                        }
+
+                        log::debug!("Successfully installed packages with pip");
+                    } else {
+                        log::debug!("Successfully installed packages with uv");
+                    }
+                }
             }
         }
 
@@ -399,14 +530,28 @@ impl Tool for PythonTool {
     }
 
     fn is_installed(&self) -> bool {
+        // For Python tools, we need to check if the Python executable and the tool executable exist
+        let python_path = if cfg!(windows) {
+            self.install_dir.join("Scripts").join("python.exe")
+        } else {
+            self.install_dir.join("bin").join("python")
+        };
+
         // Check if the tool executable exists in the virtualenv
         let tool_path = if cfg!(windows) {
             self.install_dir.join("Scripts").join(format!("{}.exe", self.name))
         } else {
             self.install_dir.join("bin").join(&self.name)
         };
 
-        tool_path.exists()
+        // Log the paths for debugging
+        log::debug!("Checking if Python tool is installed:");
+        log::debug!("  Python path: {:?}, exists: {}", python_path, python_path.exists());
+        log::debug!("  Tool path: {:?}, exists: {}", tool_path, tool_path.exists());
+
+        // For Python tools, we consider them installed if both the Python executable
+        // and the tool executable exist
+        python_path.exists() && tool_path.exists()
     }
 
     fn install_dir(&self) -> &PathBuf {

tests/toolchain_tests.rs

@@ -0,0 +1,107 @@
+//! Tests for toolchain functionality
+
+use rustyhook::toolchains::{PythonTool, Tool, SetupContext};
+
+#[test]
+fn test_python_tool_with_uv() {
+    // Create a temporary directory for the test
+    let temp_dir = tempfile::tempdir().unwrap();
+    let cache_dir = temp_dir.path().join(".rustyhook").join("cache");
+
+    // Create a Python tool with a test package
+    let python_tool = PythonTool::new("pytest", "1.0.0", vec!["pytest".to_string()]);
+
+    // Get the installation directory from the Python tool
+    let install_dir = python_tool.install_dir().clone();
+
+    // Create the cache directory
+    std::fs::create_dir_all(&cache_dir).unwrap();
+
+    // Create a setup context
+    let ctx = SetupContext {
+        cache_dir: cache_dir.clone(),
+        install_dir: install_dir.clone(),
+        force: false,
+        version: Some("1.0.0".to_string()),
+    };
+
+    // Set up the Python tool (this will install uv and use it to install pytest)
+    let result = python_tool.setup(&ctx);
+
+    // Check that the setup was successful
+    assert!(result.is_ok(), "Failed to set up Python tool: {:?}", result);
+
+    // Check the installation directory structure
+    println!("Installation directory: {:?}", install_dir);
+    if install_dir.exists() {
+        println!("Installation directory exists");
+
+        // List the contents of the installation directory
+        if let Ok(entries) = std::fs::read_dir(&install_dir) {
+            println!("Contents of installation directory:");
+            for entry in entries {
+                if let Ok(entry) = entry {
+                    println!("  {:?}", entry.path());
+                }
+            }
+        } else {
+            println!("Failed to read installation directory");
+        }
+
+        // Check bin directory
+        let bin_dir = if cfg!(windows) {
+            install_dir.join("Scripts")
+        } else {
+            install_dir.join("bin")
+        };
+
+        if bin_dir.exists() {
+            println!("Bin directory exists: {:?}", bin_dir);
+
+            // List the contents of the bin directory
+            if let Ok(entries) = std::fs::read_dir(&bin_dir) {
+                println!("Contents of bin directory:");
+                for entry in entries {
+                    if let Ok(entry) = entry {
+                        println!("  {:?}", entry.path());
+                    }
+                }
+            } else {
+                println!("Failed to read bin directory");
+            }
+        } else {
+            println!("Bin directory does not exist: {:?}", bin_dir);
+        }
+    } else {
+        println!("Installation directory does not exist");
+    }
+
+    // Check that the Python tool is installed
+    println!("Checking if Python tool is installed...");
+    let is_installed = python_tool.is_installed();
+    println!("Python tool is installed: {}", is_installed);
+
+    // Get the tool path that is being checked
+    let tool_path = if cfg!(windows) {
+        install_dir.join("Scripts").join(format!("{}.exe", python_tool.name()))
+    } else {
+        install_dir.join("bin").join(python_tool.name())
+    };
+    println!("Tool path being checked: {:?}", tool_path);
+    println!("Tool path exists: {}", tool_path.exists());
+
+    // Check that the pytest package is installed
+    let pytest_path = if cfg!(windows) {
+        install_dir.join("Scripts").join("pytest.exe")
+    } else {
+        install_dir.join("bin").join("pytest")
+    };
+    println!("Pytest path: {:?}", pytest_path);
+    println!("Pytest path exists: {}", pytest_path.exists());
+
+    // Assert that the Python tool is installed
+    assert!(is_installed, "Python tool is not installed");
+
+    // Assert that the pytest package is installed
+    assert!(pytest_path.exists(), "pytest package is not installed");
+}