git2 errors when using zig + pyo3

[konstin]

Cloning a repo with git2, the rust wrapper around libgit2, fails, but only when using zig and pyo3. Both using the default linker with pyo3 and using a binary linked with zig works.

Minimized example:

[package]
name = "git_fail"
version = "0.1.0"
edition = "2021"

[lib]
name = "git_fail"
crate-type = ["cdylib", "rlib"]

[dependencies]
git2 = { version = "0.14.4", features = ["vendored-openssl", "vendored-libgit2"] }
libz-sys = { version = "1.1.8", features = ["static"] }
pyo3 = { version = "0.16.5", optional = true, features = ["abi3-py37", "extension-module"] }


[features]
default = []
python-bindings = ["pyo3"]

use git2::Repository;

#[cfg_attr(feature = "python-bindings", pyo3::pyfunction)]
pub fn run() {
    Repository::clone("https://github.com/sokrypton/ColabFold", "repodir").unwrap();
}

#[cfg(feature = "python-bindings")]
#[pyo3::pymodule]
fn git_fail(_py: pyo3::Python<'_>, m: &pyo3::types::PyModule) -> pyo3::PyResult<()> {
    m.add_function(pyo3::wrap_pyfunction!(run, m)?)?;
    Ok(())
}

use git_fail::run;

fn main() {
    println!("Hello, world!");
    run()
}

• binary with zig: rm -rf repodir && cargo zigbuild && target/debug/gitter && ls repodir/: works
• pyo3 without zig: rm -rf repodir && maturin build --features python-bindings --bindings pyo3 && pip install --force-reinstall target/wheels/git_fail-0.1.0-cp37-abi3-manylinux_2_28_x86_64.whl && python -c 'from git_fail import run; run()' && ls repodir: works
• pyo3 with zig: : fails with Error { code: -1, klass: 2, message: "failed to stat 'repodir/.git': " }.

I'd like to try without the vendored/static, but then zig cc (unlike gcc) doesn't find the headers anymore, so i'd also appreciate any hints how to get zig cc to use the system headers.

Any idea how to even debug this?

[davidhewitt]

Can you use rust-gdb (or similar) to step through and see if you can spot any differences in execution paths? Afraid I personally don't have experience with zig, although @messense might know how to configure zig cc.

[messense]

I'd like to try without the vendored/static, but then zig cc (unlike gcc) doesn't find the headers anymore, so i'd also appreciate any hints how to get zig cc to use the system headers.

Does CFLAGS="-Ixxx" work?

[messense]

Try LDFLAGS='-L/usr/lib64 -L/usr/lib'

[konstin]

LDFLAGS unfortunately didn't work, but you lead me in the right direction: CFLAGS='-I/usr/lib/x86_64-linux-gnu -I/usr/lib64 -I/usr/lib' maturin build --zig --features python-bindings --bindings pyo3 -- -L/usr/lib64 -L/usr/lib -L/usr/lib/x86_64-linux-gnu/ compiles. I've also found out why zig doesn't find them by itself, given that libz.so is in /usr/lib/x86_64-linux-gnu/:

$ python -m ziglang cc -print-search-dirs
programs: =/usr/bin:/home/konsti/monotrail/.venv-b/lib/python3.8/site-packages/ziglang:/usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../x86_64-linux-gnu/bin
libraries: =/home/konsti/monotrail/.venv-b/lib/python3.8/site-packages/lib/clang/13.0.1:/usr/bin/../lib/gcc/x86_64-linux-gnu/10:/usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../lib64:/lib/x86_64-linux-gnu:/lib/../lib64:/usr/lib/x86_64-linux-gnu:/usr/lib/../lib64:/lib:/usr/lib
$ python -m ziglang cc -target native-linux-gnu.2.12 -print-search-dirs
programs: =/usr/bin:/home/konsti/monotrail/.venv-b/lib/python3.8/site-packages/ziglang
libraries: =/home/konsti/monotrail/.venv-b/lib/python3.8/site-packages/lib/clang/13.0.1:/lib/../lib64:/usr/lib/../lib64:/lib:/usr/lib

Unfortunately that only leads me deeper down the rabbit whole:

$ gdb --args python -c 'from git_fail import run; run()'
[...]
Program received signal SIGILL, Illegal instruction.
0x00007ffff72aea7f in sha1_compression_states (ihv=ihv@entry=0xacc118, m=<optimized out>, W=W@entry=0xacc1b0, states=states@entry=0xacc430) at libgit2/src/util/hash/sha1/sha1dc/sha1.c:798
798     }
(gdb) bt
#0  0x00007ffff72aea7f in sha1_compression_states (ihv=ihv@entry=0xacc118, m=<optimized out>, W=W@entry=0xacc1b0, states=states@entry=0xacc430)
    at libgit2/src/util/hash/sha1/sha1dc/sha1.c:798
#1  0x00007ffff72aee4b in sha1_process (ctx=ctx@entry=0xacc110, block=block@entry=0xae0fb5) at libgit2/src/util/hash/sha1/sha1dc/sha1.c:1725
#2  0x00007ffff72aebff in SHA1DCUpdate (ctx=0xacc110, 
    buf=0xae0fb5 "\337*\336R\n\022J\023\071!p\n\361TI%\205\022\062ɒLM\225L\207\004\246$M\261\236iThpK+\016\246\222\244\022Y\232`(\303LK\024U҉L\022\210\323$Æ\206\061&\232\254*S\001\367\374\\\267`\215[\316*\260\347\224U4\317i\v\336\370\231\226ߴ\373\v?\362\022\263\342\065\251˟@\322T8ՠ\244\352\340\005\032\020\nOZ2ΟV\207\361EO\300[U\267\264)Ə\234\361sO\376c˛\274c9\370\361[\026r\226>\330:[\260_:\276y\b\003\364\315\005 \200[g%\226iZ3\323\334Y\273\025\213\231e\314\002\313\325\317_\223͠\264˛i\352\244[\232."..., len=8119)
    at libgit2/src/util/hash/sha1/sha1dc/sha1.c:1843
#3  0x00007ffff72ac4c4 in git_hash_sha1_update (ctx=0xacc118, data=0xae0fb5, len=11321776) at libgit2/src/util/hash/sha1/collisiondetect.c:35
#4  0x00007ffff725feb4 in git_hash_update (ctx=0xacc118, ctx@entry=0xacc110, data=0xae0fb5, data@entry=0xae0f80, len=11321776, len@entry=8172) at libgit2/src/util/hash.c:60
#5  0x00007ffff7214794 in hash_partially (idx=idx@entry=0xac91d0, 
    data=data@entry=0xae0f80 "Z\231,x\234u\221M\217\242@\030\204\357\374\212\276Ovh@@\222\231̀\264\250 *\212\070ܺ\241\301\036\371\022\032\024\177\375\272\263ɞv\353\370$\225z\337*\336R\n\022J\023\071!p\n\361TI%\205\022\062ɒLM\225L\207\004\246$M\261\236iThpK+\016\246\222\244\022Y\232`(\303LK\024U҉L\022\210\323$Æ\206\061&\232\254*S\001\367\374\\\267`\215[\316*\260\347\224U4\317i\v\336\370\231\226ߴ\373\v?\362\022\263\342\065\251˟@\322T8ՠ\244\352\340\005\032\020\nOZ2ΟV\207\361EO\300[U\267\264)Ə\234\361sO\376c˛\274c9\370", <incomplete sequence \361>..., 
    size=size@entry=8192) at libgit2/src/libgit2/indexer.c:592
#6  0x00007ffff7213aa4 in git_indexer_append (idx=0xac91d0, data=0xae0f80, size=8192, stats=0xab24a8) at libgit2/src/libgit2/indexer.c:825
#7  0x00007ffff7222051 in pack_backend__writepack_append (_writepack=<optimized out>, data=0xae0fb5, size=11321776, stats=0xacc430) at libgit2/src/libgit2/odb_pack.c:681
#8  0x00007ffff7288b90 in git_smart__download_pack (transport=0xab2ab0, repo=<optimized out>, stats=0xab24a8) at libgit2/src/libgit2/transports/smart_protocol.c:599
#9  0x00007ffff7246baf in git_fetch_download_pack (remote=<optimized out>, remote@entry=0xab23b0) at libgit2/src/libgit2/fetch.c:196
#10 0x00007ffff7242ccc in git_remote__download (remote=<optimized out>, remote@entry=0xab23b0, refspecs=<optimized out>, opts=opts@entry=0x7fffffffb690) at libgit2/src/libgit2/remote.c:1309
[...]

I have no clue why this is now failing

[messense]

Could it be related to this https://github.com/ziglang/zig/wiki/FAQ#why-do-i-get-illegal-instruction-when-using-with-zig-cc-to-build-c-code ?

[messense]

I think it's exactly this issue since libgit2 suppressed ubsan for sha1_compression_states: https://github.com/libgit2/libgit2/blob/3847522e86e9c65be674f1372cefefdbfbe9ba2b/script/sanitizers.supp

[konstin]

wow your great, i'd never found that, thanks!

I finally tracked it down:

CFLAGS='-fno-sanitize=undefined -idirafter /usr/include -idirafter /usr/include/x86_64-linux-gnu/' build --zig --features python-bindings --bindings pyo3 -- -L/usr/lib/x86_64-linux-gnu

(the -idirafter instead of -I is crucial) for compiling with git2 = { version = "0.14.4" } in Cargo.toml.

by starting python -c through gdb after setting a breakpoint in the libgit2 error handler function i found that the problem was the lstat return value, which has already been reportred in ziglang/zig#11878. So not a zlib problem but rather a bug in zig's glibc code.

Thanks again for the debugging help 🙏