Support a static build without requiring external bins (`git`, archive utils)

[polarathene]

From the install docs (permalink) requirement listed are:

• Git - for fetching available versions/tags
• tar, unzip, gz, xz - for unpacking archives

I believe some Rust based CLI tools that have similar functionality (cargo-binstall perhaps?) are capable of doing all that functionality within Rust.

Would you be open towards removing these external deps? (assuming that doesn't bloat the binary size)

Reference:

• git support (implemented in this repo)
• Archive support (implemented via separate starbase_archive crate dep using these features)

NOTE: The utility of this request is perhaps niche given most systems would typically have these deps available. I was interested in producing a minimal OCI image but depending on the base image/distro (if any), adding the required external deps can bring in a bunch of other packages/weight that isn't necessary for proto specifically.

---

It would seem that to satisfy this request:

• Adopt a Rust implementation of the git functionality used. Possible candidates:
  • gitoxide_core/gix?
    • Reference: Docs for gix::clone, or friendlier gitoxide_core::repository::clone (EDIT: Nevermind, gitoxide_core appears that it is only intended for internal use by the gix CLI bin).
    • Unfortunately it's a bit low-level with gix directly, would need to engage in upstream for gitoxide_core if wanting to avoid the added verbosity/noise of interacting directly with gix vs current implementation in proto.
  • asyncgit (Repo README is being developed by the gitui CLI project to oxidize their usage of git2 by migrating to gix (git2 provides API via bindings to libgit instead of calling CLI commands like proto does (due to static builds)), however asyncgit doesn't quite seem to have the coverage you'd need yet, and their Cargo.toml shows it still brings in git2 due to the transition (presumably the crate will continue to provide an API to both git2 + gix backends, but could be optimized with feature config to opt-out of git2 deps).
• starbase_archive would likely need a static feature or similar functionality (this can require some care, I can provide reference/insights from some involvement with zlib-sys). This could then be used (optionally if it makes sense) with proto, where if binary size doesn't go up too much it could be suitable as a default, otherwise compile-time opt-in.

[polarathene]

As I'm not familiar with gix and the other crates I came across are not quite there, I did lazily query Google Gemini about implementing these three functions with gix instead (or rather focusing on the equivalent CLI commands mapped to Rust using gix).

I've not verified the functionality, and at a glance it obviously needs extra logic like making the submodules call conditional. I'm just sharing here as a rough reference of what the implementations via gix might look like, without burdening you to dig through docs yourself to get that kind of impression 😅

Google Gemini output

Implementing these specific CLI flags (--depth 1, --recurse-submodules, --ff, etc.) in gix requires touching the lower-level configuration of the fetch and checkout builders.

Because gix is highly modular, it doesn't have a single "pull" function; it provides the building blocks to construct that logic.

---

1. Shallow Clone with Submodules

To match your git clone --depth 1 --recurse-submodules command, we configure the prepare_clone builder to use a shallow depth and enable submodule fetching.

use gix::remote::fetch::Shallow;

fn clone_shallow_with_submodules(url: &str, dst: &std::path::Path) -> Result<(), Box<dyn std::error::Error>> {
    // 1. Initialize clone builder
    let mut prepare = gix::prepare_clone(url, dst)?;

    // 2. Configure for --depth 1 and --recurse-submodules
    prepare = prepare
        .with_shallow(Shallow::Depth(1.try_into().unwrap()))
        .with_submodules(gix::clone::Submodules::Recursive);

    // 3. Execute Fetch and Checkout
    let (mut checkout, _) = prepare.fetch_then_checkout(gix::progress::Discard, &false.into())?;
    checkout.main_worktree(gix::progress::Discard, &false.into())?;

    Ok(())
}

---

2. Checkout (Reference/Branch)

Matching git checkout <reference> requires finding the object (which could be a tag, a branch name, or a commit hash) and updating both HEAD and the working directory.

fn checkout_reference(repo: &gix::Repository, reference: &str) -> Result<(), Box<dyn std::error::Error>> {
    // 1. Resolve the string to a commit-ish object
    let object = repo.find_reference(reference)
        .map(|r| r.into_fully_peeled_id().unwrap()) // It's a branch/tag
        .or_else(|_| repo.rev_parse_single(reference))?; // It's a hash

    let tree_id = object.object()?.peel_to_tree()?.id;

    // 2. Update HEAD (matches the CLI behavior of switching branches or entering detached HEAD)
    repo.set_head(reference)?;

    // 3. Update the files on disk
    repo.index_checkout(tree_id)
        .finish_all(gix::progress::Discard, &false.into())?;

    Ok(())
}

---

3. Pull (Fast-Forward and Prune)

Your command git pull --ff --prune translates to a Fetch (with pruning enabled) followed by a check to ensure the local branch can be fast-forwarded to the remote state.

fn pull_ff_prune(repo: &gix::Repository) -> Result<(), Box<dyn std::error::Error>> {
    let mut remote = repo.find_default_remote(gix::remote::Direction::Fetch)
        .ok_or("No remote found")??;

    // 1. Configure Pruning (equivalent to --prune)
    remote.replace_refspecs(Some("refs/heads/*:refs/remotes/origin/*"), gix::remote::Direction::Fetch)?;
    
    // 2. Fetch
    let outcome = remote.connect(gix::remote::Direction::Fetch)?
        .prepare_fetch(gix::progress::Discard, gix::remote::fetch::PrepareOptions {
            should_prune: true, // This matches --prune
            ..Default::default()
        })?
        .receive(gix::progress::Discard, &false.into())?;

    // 3. Fast-Forward Logic (Simulating --ff)
    // Find the remote tracking branch (e.g., origin/main)
    let remote_ref = repo.find_reference("refs/remotes/origin/main")?;
    let new_commit_id = remote_ref.id();

    // In a production app, you would verify if HEAD is an ancestor of new_commit_id
    // to ensure a Fast-Forward is possible.
    
    // 4. Update local HEAD and Filesystem
    let tree_id = new_commit_id.object()?.peel_to_tree()?.id;
    repo.edit_reference(gix::refs::transaction::ReferenceEdit {
        change: gix::refs::transaction::Change::Update {
            log: Default::default(),
            expected: gix::refs::transaction::PreviousValue::Any,
            new: gix::refs::Target::Peeled(new_commit_id.detach()),
        },
        name: "HEAD".try_into()?,
        deref: true,
    })?;

    repo.index_checkout(tree_id)
        .finish_all(gix::progress::Discard, &false.into())?;

    Ok(())
}

[milesj]

Besides git, all the other commands are only required for installing proto itself (via the shell script) and are not required for installing tools through proto.

I also have no plans to not require git. proto is a tool for developers, and pretty much all developers will have git.

As for the archive stuff. Not opposed to static builds.

[polarathene]

all the other commands are only required for installing proto itself (via the shell script) and are not required for installing tools through proto.

Oh... I was thinking that that was runtime requirements given my experience with the external commands for git 😅 Good to know thanks.

Could you clarify why the guidance suggests multiple packages (eg: unzip gzip xz-utils) vs whatever would be appropriate for that shell script installer on that distro? I think that's why I inferred these were runtime dependencies.

---

I also have no plans to not require git.

Is there a reason to require calling an external git command vs using the equivalent Rust native operations using gix (crate lib)? (gix CLI was measured as 50% faster than the git CLI in 2024 for perfoming git status)

It'd resolve this failure scenario (as niche as that is), even if it were only for the retrieving git tags? (used by proto install on tools like Go)

proto is a tool for developers, and pretty much all developers will have git.

Yeah, I'm a developer but I was interested in using Proto for managing toolchains in build and runtime containers that are stripped down.

I can make git available, but it would be neat if Proto could do it's git operations without the external dependency.

---

Reference - Image weight with git

Sharing this information in case this context is useful. For the most part you are right that pragmatically the external git dep isn't an issue.

git-core on Fedora 43 is only 22 MB extra (191 MB base container image weight) or 20MB more for git but this brings in a bunch of Perl, the git-core package also brings in extras that I could otherwise avoid.

# Base image size:
$ docker run --rm -it fedora:43
$ du -shx /
191M    /

$ dnf install -y git-core && dnf clean all
$ du -shx /
213M    /

# Using `dnf --installroot` to only install the minimal packages necessary:
$ dnf --installroot /root-fs --use-host-config --setopt=install_weak_deps=0 install -y git-core
$ dnf --installroot /root-fs clean all
$ du -shx /root-fs
86M     /root-fs

# Minimal image if avoiding `git` and runtime required glibc:
# Just glibc (about 10MB can be manually stripped away):
$ dnf --installroot /root-fs-glibc --use-host-config --setopt=install_weak_deps=0 install -y glibc
$ dnf --installroot /root-fs-glibc clean all
$ du -shx /root-fs-glibc
18M     /root-fs-glibc

That's 86MB of weight for a minimal Fedora image to support git compared to 9.7MB with chisel. Static Proto without the external git dep would make all that redundant.

Most distros package managers are not setup for such granular system dep pruning, and it'd be a bonus if the image doesn't need a package manager sync/install to build (some of the package managers can be slow both in network and install operations depending on packages).

Alpine can be quite competitive but there's also various caveats with musl, so I've focused on Fedora for glibc which does pretty well as a base (if Google distroless is too limiting and Canonical chisel too verbose).

That said, I doubt most devs care that much about optimizing image weight to this extent. I don't want to burden you either with the effort/time to implement with gix when it presently looks like more work/maintenance than just calling out to git.

[milesj]

Is there a reason to require calling an external git command vs using the equivalent Rust native operations using gix (crate lib)?

Because the commands are executed from the context of WASM, and it is nothing more than commands that exist on PATH. I have no plans to change this, as WASM plugins should be able to execute arbitrary commands, and in this case it just happens to be git.

[milesj]

I looked into static and only half of the archive formats support it. Will probably wait till they all do.