bundle gem --ext=rust does not install well as a gem out of the box

[gl-yziquel]

Hi.

Describe the problem as clearly as you can

I've been trying to scaffold mixed rust/ruby code with magnus. So I tried the bundle gem --ext=rust command. And tweaked it a bit to make it work, but very minimally. Here is the result:

https://github.com/gl-yziquel/skeleton_rust_gem.git

Tagged: https://github.com/gl-yziquel/skeleton_rust_gem/releases/tag/gh-rubygems-7693

I added a Justfile to automate some tasks.

Bottom line: just build works fine and creates a gem in pkg/. However, just bug-1 and just bug-2 show that the scaffolded code is not designed, out of the box, to get the gem installed. I believe this is a shortcoming of the current scaffolding of rust gems.

mini-me@virtucon ~/h/s/skeleton_rust_gem (master)> bundle exec rake install
cd tmp/x86_64-linux-gnu/skeleton_rust_gem/3.1.2
/usr/bin/ruby3.1 -I. ../../../../ext/skeleton_rust_gem/extconf.rb
checking for x86_64-linux-gnu-gcc... yes
checking for x86_64-linux-gnu-g++... yes
checking for x86_64-linux-gnu-gcc-ar... yes
cd -
cd tmp/x86_64-linux-gnu/skeleton_rust_gem/3.1.2
/usr/bin/gmake
generating /home/mini-me/home/stove/skeleton_rust_gem/target/release/libskeleton_rust_gem.so (release)
cargo rustc  --manifest-path /home/mini-me/home/stove/skeleton_rust_gem/ext/skeleton_rust_gem/Cargo.toml --target-dir /home/mini-me/home/stove/skeleton_rust_gem/target --lib --profile release -- -C linker=x86_64-linux-gnu-gcc -L native=/usr/lib -C link-arg=-Wl,-Bsymbolic-functions -C link-arg=-Wl,-z,relro -C link-arg=-Wl,-z,now -C link-arg=-lm
    Finished release [optimized] target(s) in 0.09s
cd -
/usr/bin/gmake install sitearchdir=../../../../lib/skeleton_rust_gem sitelibdir=../../../../lib/skeleton_rust_gem target_prefix=
generating /home/mini-me/home/stove/skeleton_rust_gem/target/release/libskeleton_rust_gem.so (release)
cargo rustc  --manifest-path /home/mini-me/home/stove/skeleton_rust_gem/ext/skeleton_rust_gem/Cargo.toml --target-dir /home/mini-me/home/stove/skeleton_rust_gem/target --lib --profile release -- -C linker=x86_64-linux-gnu-gcc -L native=/usr/lib -C link-arg=-Wl,-Bsymbolic-functions -C link-arg=-Wl,-z,relro -C link-arg=-Wl,-z,now -C link-arg=-lm
    Finished release [optimized] target(s) in 0.09s
installing skeleton_rust_gem.so to ../../../../lib/skeleton_rust_gem
/usr/bin/install -c -m 0755 skeleton_rust_gem.so ../../../../lib/skeleton_rust_gem
cp tmp/x86_64-linux-gnu/skeleton_rust_gem/3.1.2/skeleton_rust_gem.so tmp/x86_64-linux-gnu/stage/lib/skeleton_rust_gem/skeleton_rust_gem.so
skeleton_rust_gem 0.1.0 built to pkg/skeleton_rust_gem-0.1.0.gem.
rake aborted!
Running `gem install /home/mini-me/home/stove/skeleton_rust_gem/pkg/skeleton_rust_gem-0.1.0.gem` failed with the following output:

ERROR:  Error installing /home/mini-me/home/stove/skeleton_rust_gem/pkg/skeleton_rust_gem-0.1.0.gem:
        ERROR: Failed to build gem native extension.

    current directory: /home/mini-me/.local/share/gem/ruby/3.1.0/gems/skeleton_rust_gem-0.1.0/ext/skeleton_rust_gem
cargo rustc --target-dir /home/mini-me/.local/share/gem/ruby/3.1.0/extensions/x86_64-linux/3.1.0/skeleton_rust_gem-0.1.0 --manifest-path /home/mini-me/.local/share/gem/ruby/3.1.0/gems/skeleton_rust_gem-0.1.0/ext/skeleton_rust_gem/Cargo.toml --lib --release --locked -- -C linker\=x86_64-linux-gnu-gcc -L native\=/usr/lib -C link_arg\=-Wl,-Bsymbolic-functions -C link_arg\=-Wl,-z,relro -C link_arg\=-Wl,-z,now -l m -C debuginfo\=1
    Updating crates.io index
error: the lock file /home/mini-me/.local/share/gem/ruby/3.1.0/gems/skeleton_rust_gem-0.1.0/Cargo.lock needs to be updated but --locked was passed to prevent this
If you want to try to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.

cargo failed, exit code 101

Gem files will remain installed in /home/mini-me/.local/share/gem/ruby/3.1.0/gems/skeleton_rust_gem-0.1.0 for inspection.
Results logged to /home/mini-me/.local/share/gem/ruby/3.1.0/extensions/x86_64-linux/3.1.0/skeleton_rust_gem-0.1.0/gem_make.out

/home/mini-me/.local/share/gem/ruby/3.1.0/gems/bundler-2.5.6/lib/bundler/gem_helper.rb:210:in `sh'
/home/mini-me/.local/share/gem/ruby/3.1.0/gems/bundler-2.5.6/lib/bundler/gem_helper.rb:101:in `install_gem'
/home/mini-me/.local/share/gem/ruby/3.1.0/gems/bundler-2.5.6/lib/bundler/gem_helper.rb:57:in `block in install'
/home/mini-me/.local/share/gem/ruby/3.1.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => install
(See full trace by running task with --trace)
mini-me@virtucon ~/h/s/skeleton_rust_gem (master) [1]> cd pkg
mini-me@virtucon ~/h/s/s/pkg (master)> gem install skeleton_rust_gem-0.1.0.gem
Building native extensions. This could take a while...
ERROR:  Error installing skeleton_rust_gem-0.1.0.gem:
        ERROR: Failed to build gem native extension.

    current directory: /home/mini-me/.local/share/gem/ruby/3.1.0/gems/skeleton_rust_gem-0.1.0/ext/skeleton_rust_gem
cargo rustc --target-dir /home/mini-me/.local/share/gem/ruby/3.1.0/extensions/x86_64-linux/3.1.0/skeleton_rust_gem-0.1.0 --manifest-path /home/mini-me/.local/share/gem/ruby/3.1.0/gems/skeleton_rust_gem-0.1.0/ext/skeleton_rust_gem/Cargo.toml --lib --release --locked -- -C linker\=x86_64-linux-gnu-gcc -L native\=/usr/lib -C link_arg\=-Wl,-Bsymbolic-functions -C link_arg\=-Wl,-z,relro -C link_arg\=-Wl,-z,now -l m -C debuginfo\=1
    Updating crates.io index
error: the lock file /home/mini-me/.local/share/gem/ruby/3.1.0/gems/skeleton_rust_gem-0.1.0/Cargo.lock needs to be updated but --locked was passed to prevent this
If you want to try to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.

cargo failed, exit code 101

Gem files will remain installed in /home/mini-me/.local/share/gem/ruby/3.1.0/gems/skeleton_rust_gem-0.1.0 for inspection.
Results logged to /home/mini-me/.local/share/gem/ruby/3.1.0/extensions/x86_64-linux/3.1.0/skeleton_rust_gem-0.1.0/gem_make.out
mini-me@virtucon ~/h/s/s/pkg (master) [1]>

Ideally, I'd expect the scaffolding of such a rust extension gem to be installable in some way. Currently, it does not seem to be the case.

Did you try upgrading rubygems & bundler?

No. not yet. On my todo list.

[gl-yziquel]

This fix of the scaffolded code tends to fix just bug-2, i.e. installation by cd pkg && gem install *.gem:

diff --git a/skeleton_rust_gem.gemspec b/skeleton_rust_gem.gemspec
index 3bc794d..ee4396d 100644
--- a/skeleton_rust_gem.gemspec
+++ b/skeleton_rust_gem.gemspec
@@ -32,7 +32,7 @@ Gem::Specification.new do |spec|
   spec.bindir = "exe"
   spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
-  spec.extensions = ["ext/skeleton_rust_gem/Cargo.toml"]
+  spec.extensions = ["ext/skeleton_rust_gem/extconf.rb"]
 
   # Uncomment to register a new dependency of your gem
   # spec.add_dependency "example-gem", "~> 1.0"

Is that the template that is being scaffolded ?

bundler/lib/bundler/templates/newgem/newgem.gemspec.tt

[gl-yziquel]

#7695

[gl-yziquel]

After the above fix, we have the following:

• cd pkg && gem install *.gem works.
• rake install works.
• bundle exec rake install does not work.

Follow-ups:

• Installation with bundle exec rake install not working oxidize-rb/rb-sys#380
• Duplicate shared libraries in gem store oxidize-rb/rb-sys#382

bundle exec rake install
cd tmp/x86_64-linux-gnu/skeleton_rust_gem/3.1.2
/usr/bin/ruby3.1 -I. ../../../../ext/skeleton_rust_gem/extconf.rb
checking for x86_64-linux-gnu-gcc... yes
checking for x86_64-linux-gnu-g++... yes
checking for x86_64-linux-gnu-gcc-ar... yes
cd -
cd tmp/x86_64-linux-gnu/skeleton_rust_gem/3.1.2
/usr/bin/gmake
generating /home/mini-me/home/stove/skeleton_rust_gem/target/release/libskeleton_rust_gem.so (release)
cargo rustc  --manifest-path /home/mini-me/home/stove/skeleton_rust_gem/ext/skeleton_rust_gem/Cargo.toml --target-dir /home/mini-me/home/stove/skeleton_rust_gem/target --lib --profile release -- -C linker=x86_64-linux-gnu-gcc -L native=/usr/lib -C link-arg=-Wl,-Bsymbolic-functions -C link-arg=-Wl,-z,relro -C link-arg=-Wl,-z,now -C link-arg=-lm
    Finished release [optimized] target(s) in 0.08s
cd -
/usr/bin/gmake install sitearchdir=../../../../lib/skeleton_rust_gem sitelibdir=../../../../lib/skeleton_rust_gem target_prefix=
generating /home/mini-me/home/stove/skeleton_rust_gem/target/release/libskeleton_rust_gem.so (release)
cargo rustc  --manifest-path /home/mini-me/home/stove/skeleton_rust_gem/ext/skeleton_rust_gem/Cargo.toml --target-dir /home/mini-me/home/stove/skeleton_rust_gem/target --lib --profile release -- -C linker=x86_64-linux-gnu-gcc -L native=/usr/lib -C link-arg=-Wl,-Bsymbolic-functions -C link-arg=-Wl,-z,relro -C link-arg=-Wl,-z,now -C link-arg=-lm
    Finished release [optimized] target(s) in 0.04s
installing skeleton_rust_gem.so to ../../../../lib/skeleton_rust_gem
/usr/bin/install -c -m 0755 skeleton_rust_gem.so ../../../../lib/skeleton_rust_gem
cp tmp/x86_64-linux-gnu/skeleton_rust_gem/3.1.2/skeleton_rust_gem.so tmp/x86_64-linux-gnu/stage/lib/skeleton_rust_gem/skeleton_rust_gem.so
skeleton_rust_gem 0.1.0 built to pkg/skeleton_rust_gem-0.1.0.gem.
rake aborted!
Running `gem install /home/mini-me/home/stove/skeleton_rust_gem/pkg/skeleton_rust_gem-0.1.0.gem` failed with the following output:

ERROR:  Error installing /home/mini-me/home/stove/skeleton_rust_gem/pkg/skeleton_rust_gem-0.1.0.gem:
        ERROR: Failed to build gem native extension.

    No such file or directory @ dir_s_mkdir - /home/mini-me/.local/share/gem/ruby/3.1.0/gems/skeleton_rust_gem-0.1.0/ext/skeleton_rust_gem/.gem.20240529-749888-8ra4qo

Gem files will remain installed in /home/mini-me/.local/share/gem/ruby/3.1.0/gems/skeleton_rust_gem-0.1.0 for inspection.
Results logged to /home/mini-me/.local/share/gem/ruby/3.1.0/extensions/x86_64-linux/3.1.0/skeleton_rust_gem-0.1.0/gem_make.out

/home/mini-me/.local/share/gem/ruby/3.1.0/gems/bundler-2.5.6/lib/bundler/gem_helper.rb:210:in `sh'
/home/mini-me/.local/share/gem/ruby/3.1.0/gems/bundler-2.5.6/lib/bundler/gem_helper.rb:101:in `install_gem'
/home/mini-me/.local/share/gem/ruby/3.1.0/gems/bundler-2.5.6/lib/bundler/gem_helper.rb:57:in `block in install'
/home/mini-me/.local/share/gem/ruby/3.1.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => install
(See full trace by running task with --trace)
error: Recipe `bug-1` failed on line 31 with exit code 1

Moreover, when it works, the shared library is deployed in duplicate (in ext/skeleton_rust_gem/skeleton_rust_gem.so and in lib/skeleton_rust_gem/skeleton_rust_gem.so) in the gem store:

mini-me@virtucon ~/.l/s/g/r/3/g/skeleton_rust_gem-0.1.0> tree
.
├── Cargo.toml
├── ext
│   └── skeleton_rust_gem
│       ├── Cargo.toml
│       ├── extconf.rb
│       ├── Makefile
│       ├── skeleton_rust_gem.so
│       └── src
│           └── lib.rs
├── lib
│   ├── skeleton_rust_gem
│   │   ├── skeleton_rust_gem.so
│   │   └── version.rb
│   └── skeleton_rust_gem.rb
├── Rakefile
├── README.md
└── sig
    └── skeleton_rust_gem.rbs

7 directories, 12 files

In triplicate, in fact, as it is also deployed in the extensions hierarchy of the gem store:

mini-me@virtucon ~/.l/s/g/r/3/e/x/3/skeleton_rust_gem-0.1.0> tree
.
├── gem.build_complete
├── gem_make.out
├── mkmf.log
└── skeleton_rust_gem
    └── skeleton_rust_gem.so

2 directories, 4 files

[gl-yziquel]

Moreover, the rake-compiler should probably be declared as a development dependency in the gemspec file, not in the Gemfile.