Submodule of Git Dependency Hanging in Git CLI Mode

[baxterjo]

Problem

When adding a dependency with a git reference, if that dependency has a submodule in it, cargo will hang when trying to download the submodule. If the dependency is added as a bare crates.io style dependency then everything works fine.

This only happens with CARGO_NET_GIT_FETCH_WITH_CLI=true is applied.

I have ruled out my git config being the issue here by cloning the repository down and issuing the exact same command that cargo issues (found by running carbo build --verbose) and the command executed with no problems.

The git config is the issue see comment below

Normally I would just turn off CLI mode, but my company uses git dependencies with SSH addresses for private crates, so this is not an option.

Since this is a net operation I have included a PCAP with this issue that contains only traffic to / from github (140.82.116.4)
cargo_build_with_git_cli.pcapng.zip

Steps

Minimum Cargo.toml for repro:

[package]
name = "test-submodule-dep"
version = "0.1.0"
edition = "2024"

[dependencies]
rdkafka = { git = "https://github.com/fede1024/rust-rdkafka.git" } # This repository uses a submodule for its `*-sys` dependency.

1. CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build --verbose verbose is not required
2. cargo will hang when trying to fetch the submodule

Possible Solution(s)

No response

Notes

I have tried messing with some of the unstable features concerning git and cargo. Although I think they all get overridden when using git in CLI mode.

Version

cargo 1.88.0 (873a06493 2025-05-10)
release: 1.88.0
commit-hash: 873a0649350c486caf67be772828a4f36bb4734c
commit-date: 2025-05-10
host: aarch64-apple-darwin
libgit2: 1.9.0 (sys:0.20.0 vendored)
libcurl: 8.7.1 (sys:0.4.80+curl-8.12.1 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 3.4.1 11 Feb 2025
os: Mac OS 15.5.0 [64-bit]

git version 2.50.1

[baxterjo]

I just swapped over to my linux machine and it works here:

cargo 1.88.0 (873a06493 2025-05-10)
release: 1.88.0
commit-hash: 873a0649350c486caf67be772828a4f36bb4734c
commit-date: 2025-05-10
host: x86_64-unknown-linux-gnu
libgit2: 1.9.0 (sys:0.20.0 vendored)
libcurl: 8.12.1-DEV (sys:0.4.80+curl-8.12.1 vendored ssl:OpenSSL/3.4.1)
ssl: OpenSSL 3.4.1 11 Feb 2025
os: Ubuntu 24.4.0 (noble) [64-bit]

git version 2.43.0

Next week I will look into rolling back my Mac's version of git to to see if that fixes anything.

[baxterjo]

Next week I will look into rolling back my Mac's version of git to to see if that fixes anything.

Just tried this, no help.

[baxterjo]

I made some better progress on this. It looks like the culprit is git fsmonitor. After turning this off in my gitconfig everything worked as expected. I have been using this setting for several months with no issue, but it looks like there was recently some activity on git that changed things (release notes).

I am not yet closing this issue as I'm still not sure who is responsible for the fix, could it be that cargo implemented a bug workaround that no longer works with git? Or is is that the specific CLI command that cargo issues is no longer compatible with the fsmonitor option?

Or is it that git did not actually implement the FS monitor fix that they thought?

Here is the full command for reference:

cd "$USER/.cargo/git/checkouts/rust-rdkafka-a76ba4b4cae1aa5d/a8d089d/rdkafka-sys/librdkafka/.git/" \
&& env -u GIT_ALTERNATE_OBJECT_DIRECTORIES -u GIT_INDEX_FILE -u GIT_OBJECT_DIRECTORY -u GIT_WORK_TREE GIT_DIR="$USER/.cargo/git/checkouts/rust-rdkafka-a76ba4b4cae1aa5d/a8d089d/rdkafka-sys/librdkafka/.git/" \
"git" "fetch" "--no-tags" "--verbose" "--force" "--update-head-ok" "https://github.com/edenhill/librdkafka" "+c024ac13daf98667de2b8724986e97f489644c15:refs/commit/c024ac13daf98667de2b8724986e97f489644c15"

Also, the reason my linux machine worked is that there is no fs monitor option for linux and that config is ignored by git on linux machines.

[kevindong]

I just ran into a similar issue around cargo hanging around updating a git submodule. As the reporter helpfully found out, removing this fsmonitor option from my global .gitconfig file fixes it:

# ~/.gitconfig: the culprit; remove to stop Cargo from hanging
[core]
    fsmonitor = true

[weihanglo]

What was the git CLI version that failed? And if you bump to the latest version (2.50.1 atm, or 2.47.1 that fixes the macOS race cond), would the hanging disappear?

[baxterjo]

What was the git CLI version that failed? And if you bump to the latest version (2.50.1 atm, or 2.47.1 that fixes the macOS race cond), would the hanging disappear?

git CLI versions 2.50.1, 2.49.0, AND 2.43.0 all failed on MacOS with core.fsmonitor = true in my .gitconfig. I have the exact same config on my linux machine and I think the only reason it worked is because there is no support for fsmonitor on linux. (docs)

[kevindong]

I'm on the homebrew/macOS version of git:

$ which git
/opt/homebrew/bin/git
$ git --version
git version 2.50.1

It seems to be working for me right now, but it definitely wasn't working yesterday. However, I did do some brew upgrade yesterday after I had removed the option from my config so maybe something else changed along the way?

edit: a coworker ran into the same issue today and raised that maybe the reason why things got fixed up for me is because Git has cached the submodule such that it's not running into the bug anymore (at least until the next time the dependency gets bumped up).

[baxterjo]

edit: a coworker ran into the same issue today and raised that maybe the reason why things got fixed up for me is because Git has cached the submodule such that it's not running into the bug anymore (at least until the next time the dependency gets bumped up).

I actually think the caching is happening in ~/.cargo/registry. I had to wipe that directory many times to keep reproducing the bug.