importers: high cpu usage during git clone phase

[helio-frota]

not 100% sure but I think we can use perf + hotspot to indentify who is causing this cpu high usage

• enable the osv-github importer
• edit Cargo.toml adding this content at the end of the file

[profile.release]
debug = true

• ofc build with --release
• run the importer and then perf record -F99 --call-graph dwarf -p PID_here, collect that until the cpu decrease
• sudo dnf install hotspot or download and run https://github.com/KDAB/hotspot/releases appImage using/opening the generated perf.data file
• see what happened

[helio-frota]

git2 is causing this mess according to perf and hotspot:

[helio-frota]

I played a bit with those global git2 options and unfortunatelly I found no differences when watching the cpu usage

Changes:

➜  trustify git:(main) ✗ git diff
diff --git a/modules/importer/src/runner/common/walker/git.rs b/modules/importer/src/runner/common/walker/git.rs
index 0934c948..0cd76355 100644
--- a/modules/importer/src/runner/common/walker/git.rs
+++ b/modules/importer/src/runner/common/walker/git.rs
@@ -492,7 +492,11 @@ mod test {
 
     #[test_log::test(tokio::test)]
     async fn test_walker() -> Result<(), anyhow::Error> {
-        const SOURCE: &str = "https://github.com/RConsortium/r-advisory-database";
+        git2::opts::enable_caching(false);
+        git2::opts::strict_hash_verification(false);
+        git2::opts::strict_object_creation(false);
+        // const SOURCE: &str = "https://github.com/RConsortium/r-advisory-database";
+        const SOURCE: &str = "https://github.com/github/advisory-database";
         let path = PathBuf::from(format!(
             "{}target/test.data/test_walker.git",
             env!("CARGO_WORKSPACE_ROOT")

Run the test and take a look on the CPU usage

➜  trustify git:(main) ✗ cargo nextest run test_walker
   Compiling trustify-module-importer v0.4.0-beta.1 (/home/heliofrota/Desktop/tc/trustify/modules/importer)
   Compiling trustify-server v0.4.0-beta.1 (/home/heliofrota/Desktop/tc/trustify/server)
   Compiling xtask v0.4.0-beta.1 (/home/heliofrota/Desktop/tc/trustify/xtask)
   Compiling trustify-trustd v0.4.0-beta.1 (/home/heliofrota/Desktop/tc/trustify/trustd)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7.85s
────────────
 Nextest run ID a941fd1b-b3ce-40c6-b24a-a27439586106 with nextest profile: default
    Starting 2 tests across 36 binaries (670 tests skipped)
        PASS [   1.231s] trustify-module-importer runner::common::walker::git::test::test_walker_fail_escape
        SLOW [> 60.000s] trustify-module-importer runner::common::walker::git::test::test_walker
        SLOW [>120.000s] trustify-module-importer runner::common::walker::git::test::test_walker
        SLOW [>180.000s] trustify-module-importer runner::common::walker::git::test::test_walker
        SLOW [>240.000s] trustify-module-importer runner::common::walker::git::test::test_walker

[helio-frota]

feel free to try https://github.com/helio-frota/git2BBQ?tab=readme-ov-file#git2bbq 👍