Update --ext=rust to support compiling the native extension from source

[karreiro]

What was the end-user or developer problem that led to this PR?

I needed to apply some tiny changes to the skeleton generated by the --ext=rust option to compile my native extension from source.

What is your fix for the problem, implemented in this PR?

When gem consumers install a gem with a native extension and their platform is not included the pre-compiled bundles, they need to compile it locally and rely on the extconf.rb file.

Referencing the extconf.rb file in the gemspec is required to make that work. We also need to include rb_sys as a dependency because the Rust extconf.rb depends on it.

Here's a demo of this change in action :)

Screen.Recording.2024-04-25.at.21.22.36.mov

Closes #7693.

[welcome]

Thanks for opening a pull request and helping make RubyGems and Bundler better! Someone from the RubyGems team will take a look at your pull request shortly and leave any feedback. Please make sure that your pull request has tests for any changes or added functionality.

We use GitHub Actions to test and make sure your change works functionally and uses acceptable conventions, you can review the current progress of GitHub Actions in the PR status window below.

If you have any questions or concerns that you wish to ask, feel free to leave a comment in this PR or join our #rubygems or #bundler channel on Slack.

For more information about contributing to the RubyGems project feel free to review our CONTRIBUTING guide

[karreiro]

👋 Hey @simi, thanks for chiming in on this PR. I've noticed the branch was outdated with the master, so I just updated it.

I'd appreciate your review when you have a moment :)

[simi]

Hello @karreiro and thanks for your PR. Can you please explain how to reproduce your errors? I just tried in latest bundler to create empty skeleton gem, pack it into gem and install with no problems. I have did following.

$ gem list | grep rb_sys  # ensure rb_sys is not installed already
$ bundle gem --ext=rust ryba > /dev/null # create empty gem
$ cd ryba # move into gem directory
$ cargo c # resolve the Cargo.lock somehow
$ git add Cargo.lock # add lock to the final gem, since gem files are collected from git by default
$ gem build --force # build gem, ignore TODO in gemspec
$ gem install ryba-0.1.0.gem # install local gem

[karreiro]

Hello @simi, thank you for reviewing this PR :)

Setup

To reproduce the error, let's create 3 ryba gems:

ryba1

• Run bundle gem --ext=rust ryba1 > /dev/null
• Run cd ryba1
• Run cargo c
• Run git add Cargo.lock
• Run gem build --force

ryba2

• Run bundle gem --ext=rust ryba2 > /dev/null
• Run cd ryba2
• Run cargo c
• Run git add Cargo.lock
• Update the ryba2.gemspec file to use spec.extensions = ["ext/ryba2/extconf.rb"] instead of spec.extensions = ["ext/ryba2/Cargo.toml"]
• Run gem build --force

ryba3

• Run bundle gem --ext=rust ryba3 > /dev/null
• Run cd ryba3
• Run cargo c
• Run git add Cargo.lock
• Update the ryba3.gemspec file to use spec.extensions = ["ext/ryba3/extconf.rb"] instead of spec.extensions = ["ext/ryba3/Cargo.toml"]
• Update the ryba3.gemspec file to include spec.add_dependency "rb_sys", "~> 0.9.91"
• Run gem build --force

Results

Now, we have 3 .gem files:

• ryba1-0.1.0.gem
• ryba2-0.1.0.gem
• ryba3-0.1.0.gem

Let's move to a machine without Rust installed now.

ryba1 (result)

When we run gem install ryba1-0.1.0.gem, we get an error like this:

Building native extensions. This could take a while...
ERROR:  Error installing ryba1-0.1.0.gem:
        ERROR: Failed to build gem native extension.

    cargo metadata failed No such file or directory - cargo

Gem files will remain installed in /opt/rubies/ruby-3.1.4/lib/ruby/gems/3.1.0/gems/ryba-0.1.0 for inspection.
Results logged to /opt/rubies/ruby-3.1.4/lib/ruby/gems/3.1.0/extensions/x86_64-linux/3.1.0/ryba-0.1.0/gem_make.out

ryba2 (result)

When we run gem install ryba2-0.1.0.gem, we get an error like this:

Building native extensions. This could take a while...
ERROR:  Error installing ryba2-0.1.0.gem:
        ERROR: Failed to build gem native extension.

    current directory: /opt/rubies/ruby-3.1.4/lib/ruby/gems/3.1.0/gems/ryba2-0.1.0/ext/ryba2
/opt/rubies/ruby-3.1.4/bin/ruby -I /opt/rubies/ruby-3.1.4/lib/ruby/site_ruby/3.1.0 extconf.rb
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/opt/rubies/ruby-3.1.4/bin/$(RUBY_BASE_NAME)
<internal:/opt/rubies/ruby-3.1.4/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:88:in `require': cannot load such file -- rb_sys/mkmf (LoadError)
        from <internal:/opt/rubies/ruby-3.1.4/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:88:in `require'
        from extconf.rb:4:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /opt/rubies/ruby-3.1.4/lib/ruby/gems/3.1.0/gems/ryba2-0.1.0 for inspection.
Results logged to /opt/rubies/ruby-3.1.4/lib/ruby/gems/3.1.0/extensions/x86_64-linux/3.1.0/ryba2-0.1.0/gem_make.out

ryba3 (result)

When we run gem install ryba3-0.1.0.gem, it's going to work 🎉

[karreiro]

I just rebased this branch :)

[deivid-rodriguez]

Is this fixing the same problem as #7695, closing #7693? I'm surprised that RubyGems will automatically install rust if not present when finding a rust extension.

Not sure how I feel about that. I guess it's handy although I'd never expect gem install to install a programming language... 😅

[karreiro]

Is this fixing the same problem as #7695, closing #7693? I'm surprised that RubyGems will automatically install rust if not present when finding a rust extension.

Yes, it addresses the same issue as #7695 and closes #7693.

However, this PR goes a bit further by moving rb_sys to the gemspec, as we alreqady have this line in the extconf-rust.rb.tt file:

require "rb_sys/mkmf"

Not sure how I feel about that. I guess it's handy although I'd never expect gem install to install a programming language... 😅

heh, I share the same feeling... the extconf.rb file is quite powerful 😅

We could possibly propose making that opt-in on rb_sys. Still, the main goal of this PR is to ensure that the gem's scaffold works end-to-end with all dependencies in place, so new developers may successfully compile their gems from source with the extconf.rb file.

[deivid-rodriguez]

However, this PR goes a bit further by moving rb_sys to the gemspec, as we alreqady have this line in the extconf-rust.rb.tt file.

Cool, yeah, that makes sense. If rb_sys is required by the extconf.rb file, and that file is run at installation time, then rb_sys should be a runtime dependency 👍.

We could possibly propose making that opt-in on rb_sys. Still, the main goal of this PR is to ensure that the gem's scaffold works end-to-end with all dependencies in place, so new developers may successfully compile their gems from source with the extconf.rb file.

Yep, seems something separate and should not affect this PR. Still surprising though!

[deivid-rodriguez]

This looks good to me, but I'll let @simi give final approval!

[deivid-rodriguez]

@simi Does this look good to you? I think it makes sense and it has been proposed independently 3 times (here, #7695 and #7608).

[simi]

@deivid-rodriguez yes, I'll test and merge.

[deivid-rodriguez]

@simi I did test this myself this month and it worked fine, I think it'd be fine to merge if you don't have too much time to do the testing yourself.

@ianks Since you proposed a slightly overlapping PR at #7608 (and you're rb-sys author after all :)), would like to give this a quick look? We can them move on to #7608 after merging this.

[simi]

@deivid-rodriguez I'll test soon 🙏, initially looks good.

[deivid-rodriguez]

Ok, thank you @simi 😃.