Road to RubyGems 4

[deivid-rodriguez]

We'll be releasing RubyGems 4 this year.

This is a placeholder ticket to hold discussions about which breaking changes and big features we intend to ship.

I'll update this list as we go:

• Set Gem.install_extension_in_lib to false by default.

[deivid-rodriguez]

Today I've been researching about how to disable Gem.install_extension_in_lib in a graceful manner. and it seems pretty hard.

If we make this change, every gem that requires an extension relatively from lib code is going to break. ox is one such gem, but I'm sure there are many others. Of course, we can try to identify and fix in advance all affected gems, but old versions will of course stay broken.

The only idea that comes to mind is to not make the change but start by, while still installing an extension to lib, install a dummy wrapper .so file that warns when loaded and then loads the real .so file installed in the proper arch-specific location. That way we'll be able to let affected gems know in advance that they need to change their code.

Also, the require_relative case is just one particular case that will break, but I suspect there will be others, too.

[deivid-rodriguez]

If the wrapper idea is overkill or too complicated, and we find a good strategy to find affected gems, we could propose fixes to affected gems, and then wait some time (1 or 2 years, maybe), to make sure the community has moved on to the fixed versions.

[deivid-rodriguez]

For starters I proposed a fix to ox here: ohler55/ox#383. It'd be nice to find more such gems.

[jonbarlo]

@deivid-rodriguez hello good morning i have worked on the suggestions you mentioned #8928, install_extension_in_lib will create wrapper files in /lib this PRt verifies that the wrapper files are created correctly and that the path to the extension is correct it also verifies that the deprecation warning is shown (everything it can be removed when ffi-compiler is updated to use the new path) i also understand what the ffi-compiler is doing... the magic happens in this file: lib/ffi-compiler/loader.rb i think over there we could add code for trying the new extension dir first, then fallback to the legacy folder

[jonbarlo]

@deivid-rodriguez in the meantime i am going to keep investigating the dummy native extensions approach as well as will check OX, i am going to also try to find a list with the patterns and gems that could break, i think we should look for patterns like:

• require 'lib/extension'
• Fiddle.dlopen('lib/so')
• C code expects lib/so
• Dir.glob('lib/*.so')

And i totally agreed this one is complex, check the .rb wrapper file we adding in this PR hopefully this is a good first step

[jonbarlo]

i have raised this PR for the ffi-compiler changes as well

[deivid-rodriguez]

Researched a bit more today and found more gems that will break:

• https://github.com/gverger/cbc-wrapper/blob/c3622e362c2bf85542c4f06967527670c75a4c0c/lib/cbc-wrapper.rb#L4 (not using ffi-compiler).
• Most gems using FFI::Compiler::Loader.find, like https://github.com/pbhogan/scrypt/blob/2491ad3c9865ebdb76abfec41414994b339203e8/lib/scrypt/scrypt_ext.rb#L11. This could be fixed in ffi-compiler but gems using old versions of it will still fail to load.
• Gems using ffi-compiler, but not FFI::Compiler::Loader.find directly, like https://github.com/schuyler/levenshtein/blob/9b80b9a08b7536f2a048ab77398c71a8ad1696c3/lib/levenshtein.rb#L8-L10.

What's probably worse and suggests that many other gems will be broken, I found that the best guide out there to learn about extensions also ships a sample gem that will break. See flavorjones/ruby-c-extensions-explained#44.

Overall, this seems super complicated to do smoothly. The wrapper idea should work fine, but it seems a lot of work, and I don't think it's a good idea to break this abruptly.