Can't build project - too many open files when linking

[alexg-axis]

Environment

macOS 13.4.1 running Colima (with a VM with 4GBs of RAM and 4 cores).

Colima is started like so:

colima start --cpu 4 --memory 4 --disk 50 --ssh-agent --mount-type 9p

Issue

When I run a build for a package I get some too many open files errors.

[cross] warning: using newer rustc `1.71.0 (8ede3aae2 2023-07-12)` for the target. Current active rustc on the host is `rustc 1.66.1 (90743e729 2023-01-10)`.
 > Update with `rustup update`
   Compiling proc-macro2 v1.0.66
   Compiling libc v0.2.147
   Compiling futures-core v0.3.28
   Compiling memchr v2.5.0
   Compiling crossbeam-utils v0.8.16
   Compiling typenum v1.16.0
   Compiling concurrent-queue v2.2.0
   Compiling quote v1.0.31
   Compiling serde v1.0.173
   Compiling syn v2.0.26
   Compiling syn v1.0.109
   Compiling futures-lite v1.13.0
   Compiling zvariant_utils v1.0.1
   Compiling generic-array v0.14.7
   Compiling log v0.4.19
   Compiling serde_derive v1.0.173
   Compiling aho-corasick v1.0.2
   Compiling slab v0.4.8
   Compiling getrandom v0.2.10
   Compiling enumflags2_derive v0.7.7
   Compiling regex-automata v0.3.3
   Compiling rand_core v0.6.4
   Compiling zvariant_derive v3.15.0
   Compiling memoffset v0.7.1
   Compiling io-lifetimes v1.0.11
   Compiling crypto-common v0.1.6
   Compiling block-buffer v0.10.4
   Compiling enumflags2 v0.7.7
   Compiling async-channel v1.9.0
   Compiling rustix v0.37.23
   Compiling blocking v1.3.1
   Compiling rand_chacha v0.3.1
   Compiling digest v0.10.7
   Compiling regex v1.9.1
   Compiling nix v0.26.2
   Compiling polling v2.8.0
   Compiling futures-task v0.3.28
   Compiling tracing-attributes v0.1.26
   Compiling cpufeatures v0.2.9
   Compiling socket2 v0.4.9
   Compiling futures-util v0.3.28
   Compiling async-io v1.13.0
error: failed to build archive: Too many open files

If I try to build over and over again, eventually I make it through the list of dependencies. Sometimes during linking I get the following error.

error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/rust/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/rust/bin" VSLANG="1033" "cc" "-Wl,--version-script=/tmp/rustcLbjQzk/list" "-Wl,--no-undefined-version" "-m64" "/tmp/rustcLbjQzk/symbols.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.0.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.1.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.10.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.11.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.12.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.13.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.14.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.15.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.2.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.3.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.4.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.5.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.6.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.7.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.8.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.zbus_macros.4d5365996cfde4ad-cgu.9.rcgu.o" "/target/debug/deps/zbus_macros-8f0739efa95401d4.2okvke8ap3lakttq.rcgu.rmeta" "/target/debug/deps/zbus_macros-8f0739efa95401d4.1ud4bh0rakb366jo.rcgu.o" "-Wl,--as-needed" "-L" "/target/debug/deps" "-L" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/target/debug/deps/libproc_macro_crate-d4fccfd0f987af81.rlib" "/target/debug/deps/libtoml_edit-d584be3d2c3da8b0.rlib" "/target/debug/deps/libindexmap-8338fbc2737c4bda.rlib" "/target/debug/deps/libequivalent-7e80bb81dfd33ddf.rlib" "/target/debug/deps/libhashbrown-0e42cec4e8b8b4ec.rlib" "/target/debug/deps/libwinnow-2af52d857bf79fc3.rlib" "/target/debug/deps/libtoml_datetime-ca124dd8d89a882e.rlib" "/target/debug/deps/libonce_cell-09af8552c39183b1.rlib" "/target/debug/deps/libregex-5cdc19f620172a7a.rlib" "/target/debug/deps/libregex_automata-d0763bbdaf5b629a.rlib" "/target/debug/deps/libaho_corasick-dc7c6829f729899b.rlib" "/target/debug/deps/libmemchr-3b8f6239a0ca5820.rlib" "/target/debug/deps/libregex_syntax-2d1444f1fd034bb5.rlib" "/target/debug/deps/libzvariant_utils-7b13dc8d6deff401.rlib" "/target/debug/deps/libsyn-348ca7a4bcf213a1.rlib" "/target/debug/deps/libquote-b037bd75812b1633.rlib" "/target/debug/deps/libproc_macro2-38f21555d1aeaf4e.rlib" "/target/debug/deps/libunicode_ident-90d5eb3412520722.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libproc_macro-7ede36ba902b7649.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-0d91c78a7710ed2e.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-9bfeb974ba4dc4e7.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-7dd2d47ddb5fff81.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-18d5ce8e8a320b85.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-69d2770595dc6161.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-b156a020cc470e38.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-84091f15e468b1ee.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-fbde6acb28b510ca.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-e9de5d8ee4e7a3dd.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-9023252e4f119830.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-fcbb01769e88af40.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-f69e84994e245fea.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-27b55c02caca49ea.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-579b26075cbe9eca.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-54c30397d4b33d3b.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-65fb576691133eee.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-3fff6412017c0b89.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-fd3918c72578db43.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-2a597573799b576f.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/target/debug/deps/libzbus_macros-8f0739efa95401d4.so" "-Wl,--gc-sections" "-shared" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: /usr/bin/ld: cannot find /rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-54c30397d4b33d3b.rlib: Too many open files
          /usr/bin/ld: cannot find /rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-65fb576691133eee.rlib: Too many open files
          /usr/bin/ld: cannot find /rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-3fff6412017c0b89.rlib: Too many open files
          /usr/bin/ld: cannot find /rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-fd3918c72578db43.rlib: Too many open files
          /usr/bin/ld: cannot find /rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-2a597573799b576f.rlib: Too many open files
          collect2: error: ld returned 1 exit status
          

error: could not compile `zbus_macros` (lib) due to previous error
warning: build failed, waiting for other jobs to finish...

Note that in the example above the error is for zbus_macros, but it seems to be mostly random which crate causes the issue.

One error I can't get around in one project is the final linking step of the binary. It always fails with an error just like the one above, but for the main package.

Importantly, this issue arose after the project reached a certain size (just a few KLOC, but with larger dependencies such as zbus, image, imageproc etc.). I've been able to use cross many times with the current setup.

Things I've tried

• I've tried to increase the file limit, but I discovered cross already seems to set it to the maximum value (1M)
• I've tried to increase the file limit on the host (which I'm guessing has nothing to do with this)
• I've used lsof to print open file descriptors for any related process during compilation (it's like 7k for the hole container)
• I've tried to correlate the issue with software bugs in the toolchain. I did find that ld of binutils 2.30 and 2.31 seem to have some issues at times 1 2. But the version used in the cross container I have is 2.26.1, which doesn't seem to be affected.
• I've tried to increase the file limit in colima using the security limits file, but it doesn't seem to affect anything
• Increase the file limit in colima using /proc/sys/fs/file-max (by editing the /etc/sysctl.conf file)

[Emilgardis]

Does changing ulimit change anything when done in the container?

try ulimit -n 1024000

# Cross.toml
[build]
pre-build = ["ulimit -n 1024000"]

[alexg-axis]

No, it doesn't seem to make any difference.

[alexg-axis]

I've found a solution. As I'd ruled out just about everything else, I started to blame the virtual file system in use by colima. Sure enough, when I moved away from using 9p and started to use virtiofs instead in a vz type VM, things went better.

I had to first remove my current colima instance.

colima delete

Then, I started a new instance.

colima start --cpu 4 --memory 4 --disk 50 --ssh-agent --mount-type virtiofs --vm-type=vz

[NobodyXu]

P.S. Perhaps you can try orbstack?
I recently discovered this on twitter and it seems to be quite good, with an emphasis on performance.

Comparison against colima looks quite good: https://docs.orbstack.dev/compare/colima