Add support for pkg-config

[baarde]

When pkg-config is installed and finds the libraries, use those rather than relying on Homebrew or bundling OpenSSL.

Fixes #2544

---

A few comments about package managers

• I've tried hard not to break things for Homebrew. When the libraries and pkg-config have been installed using Homebrew, pkg-config finds the libraries in /opt/homebrew/Cellar/. However, those have their version number in the path, so I decided to ignore any library in */Cellar/* to avoid breaking Ruby when running brew upgrade.
• When the libraries and pkg-config have been installed using different package managers, whoever comes first in the search path wins.
• A package manager that distributes ruby-build should have pkg-config as a dependency as well as all required and optional libraries (openssl, readline, libffi, gmp) to make sure all the linked libraries come from the same source.

Alternatives considered

Skip --with-xyz-dir=

pkg-config is used internally by Ruby build scripts. So my initial idea was to skip the --with-xyz-dir= option when a library is discovered by pkg-config and let the build scripts do their job. However:

• That doesn't work for gmp.
• That may not work for other libraries with older Rubies.
• Having the --with-xyz-dir= options appear in the output is helpful to make sure that the correct libraries are used.
• "Explicit is better than implicit." 😉

Do nothing

My goal was to write a simple fix to have ruby-build work well with MacPorts (and other package managers). But doing so without breaking things turned out a little more complex than anticipated. Maybe it's just not worth it.

[halostatue]

@headius @eregon It would be really nice to have ruby-build work better with MacPorts by using pkg-config as this PR enables. I stopped trusting Homebrew for dev build dependencies a long time ago.

I think that there may be similar configuration changes required for TruffleRuby because it requires that I override the libyaml prefix for all runs.

[mislav]

Hi, thank you so much for the contribution, and sorry for the late review.

I generally agree that we should support discovering openssl from pkg-config if available instead of blindly downloading and compiling a new OpenSSL instance, but for the sake of a smaller diff, I would strongly prefer if the initial code change was scoped to just that.

Also, since Ruby's build system already supports discovering libraries from pkg-config, I would avoid trying to replicate that functionality too much in ruby-build. I know you suggested that explicitly passing --with-*-dir=... flags is good for visibility, but we also want to let Ruby's build system have a chance to do the right thing. We only added support for discovering Homebrew-installed libraries in ruby-build because Ruby doesn't have any capabilities to do that on its own.

So here's an idea for a minimal fix: instead of all the changes in this branch, how about scoping the fix to just needs_openssl so that it returns false if a compatible openssl version was found with pkg-config? Something like this:

  local system_version
  if has_broken_mac_openssl; then
    system_version="$(pkg-config --modversion openssl 2>/dev/null || true)"
  else
    system_version="$(system_openssl_version)"
  fi

The logic behind this would be: if on a Mac with a broken system OpenSSL version (the default for all macOS systems), treat the pkgconfig-discovered version as the "system" version and exit if that satisfies the requirement. Would this be enough to support MacPorts?

[halostatue]

I'm not on the computer where I was looking at this, but if you look at the linked MacPorts PR, there's a few libraries which are available via MacPorts that aren't found by ruby-build without explicit configuration.

I've seen a problem where ruby-build appears to favour Homebrew-discovered libraries over equivalent (better, IMO) installations from MacPorts. As I'm using mise mostly these days, this affects me since it is based on ruby-build, but I'm also a maintainer of the Port.

[mislav]

if you look at the linked MacPorts PR, there's a few libraries which are available via MacPorts that aren't found by ruby-build without explicit configuration.

I've taken a look at the linked PR, but I don't understand which are these libraries you are referring to. Are you saying that there can be MacPorts-installed libraries such as libyaml that are discoverable using pkg-config, but Ruby itself won't discover them unless an explicit --with-*-dir= was passed to its configure step?

I've seen a problem where ruby-build appears to favour Homebrew-discovered libraries

Ah, I think I understand this part. If Homebrew-discovered libraries were found by ruby-build, then indeed it will automatically configure Ruby by having --with-*-dir= point to Homebrew libraries, overriding Ruby build system otherwise discovering those libraries from elsewhere on the system. Out project assumes that if you have brew in your PATH and libraries installed via Homebrew, you likely want to use those libraries. Is that a wrong assumption?

We could potentially avoid doing the Homebrew discovery step if the same libraries were found using pkg-config. However, in that case I still wouldn't pass the paths to those libraries using --with-*-dir=, and instead just pass nothing and let Ruby figure it out.

Or, we could add an environment variable that would prevent brew from ever being consulted while looking up libraries, which could help users like you who have Homebrew installed but don't necessarily want to link any programs to it.

[halostatue]

Yes. I have both homebrew and MacPorts installed and want libraries used from MacPorts and never homebrew.

Homebrew is an unreliable source for build libraries because they do rolling updates. MacPorts ensures that they are usable in the same way that Debian does. (I use Homebrew primarily for cask management and some tools which aren't yet available from MacPorts or mise; sometimes this drags in dependencies that then cause me to have broken Rubies.)

My comment (macports/macports-ports#28748 (comment)) which triggered this PR from @baarde is specifically because:

1. If brew libraries are installed, they are used. This is incorrect when MacPorts is installed and the library can be found by pkg-config.
2. If the brew libraries are not installed, there is no search in MacPorts and ruby-build falls back to bundled OpenSSL, requires an explicit libyaml declaration, and skips GMP, both of which are installed by MacPorts.

IMO, homebrew should probably be the last choice for ruby-build when there are other options.

[eregon]

1. If brew libraries are installed, they are used. This is incorrect when MacPorts is installed and the library can be found by pkg-config.

I think this is subjective. In that case I think using either the library from Homebrew or MacPorts (or in general from one of the package managers where that package is installed) is correct. I'm pretty sure many people prefer Homebrew to MacPorts, and some might have both installed (though that's clearly a recipe for troubles).

2. If the brew libraries are not installed, there is no search in MacPorts and ruby-build falls back to bundled OpenSSL, requires an explicit libyaml declaration, and skips GMP, both of which are installed by MacPorts.

That I understand would be better, to consult pkg-config.
(BTW how does that work for MacPorts, where does it install the pkg-config executable and how does that end up in PATH?)

However, I have found that using pkg-config is unreliable and problematic in several cases:

• There is a pure-Ruby gem named pkg-config with an executable of the same name which has slightly different semantics. In my experience, it just breaks stuff and so calling to pkg-config if that is installed breaks things.
• Not considering the gem, there are multiple implementations of pkg-config (e.g. pkgconf on Fedora & related) and the semantics do not always match, see for example explore alternatives to setting LIBRARY_PATH flavorjones/mini_portile#118
• Only the first pkg-config in PATH will be tried (AFAIK), but it might not be the right one. There could be like 4 of them or so (gem, system, brew, macports).
• There is well-known issue with older versions of the openssl gem where one needed to set both --with-openssl-dir and PKG_CONFIG_PATH otherwise you'd end up with mixing headers from some openssl version and library from some other openssl version, this got fixed in ignore pkgconfig when openssl-dir option is specified ruby/openssl#486. So at least in that case it's not fine to just rely on pkg-config and auto-detection.
• As you said, pkg-config might give too precise paths like you said it does for Homebrew, that's counter-productive.

For these reasons I'm wary to use pkg-config for anything, at least I would use it as the last thing to try, after trying well-known package managers like Homebrew & MacPorts.
It's also possible that pkg-config config files in installed packages are wrong or missing and so pkg-config just returns incorrect data then.

I think that there may be similar configuration changes required for TruffleRuby because it requires that I override the libyaml prefix for all runs.

TruffleRuby explicitly looks for Homebrew and MacPorts, documented here
https://github.com/oracle/truffleruby/blob/master/doc/user/installing-libyaml.md
https://github.com/oracle/truffleruby/blob/master/doc/user/installing-libssl.md
and the search is done in
https://github.com/oracle/truffleruby/blob/master/lib/truffle/truffle/libyaml-prefix.rb
https://github.com/oracle/truffleruby/blob/master/lib/truffle/truffle/openssl-prefix.rb

It should work, as it tries MacPorts if it doesn't find it in Homebrew.
TruffleRuby tries Homebrew first, because that's what we can test because it's what we have in CI and BTW also I think what most Rubyists use.

---

IMO these complications mostly stem from the fact macOS doesn't have a proper package manager where libraries are just found without extra config. On Linux it tends to just work because there is typically one system package manager and stuff just ends up in /usr.
Homebrew for darwin-amd64 installs to /usr/local and that just works, but Homebrew for darwin-aarch64 installs to /opt/homebrew and that's not found by default and needs to be explicit, and worse than that macOS doesn't allow adding to DYLD_LIBRARY_PATH etc :( (see Homebrew/brew#13481)
I suppose in theory pkg-config could maybe be a solution for this, but for the reasons above I don't think it's reliable enough and also plenty of packages don't ship a config for pkg-config, or simply the system doesn't have pkg-config installed, so I guess it would be of little help e.g. for the ffi gem to find libraries.

---

I like @mislav's suggestion in #2547 (review).
For cases of having both the package installed in Homebrew and MacPorts I think it's reasonable for those users preferring MacPorts to explicitly specify it with --with-openssl-dir, etc. We can't satisfy both cases anyway, some people would prefer Homebrew in that scenario.

[halostatue]

I don't agree with your assessment that developers would prefer homebrew over MacPorts of both are installed—as I said earlier the problem is that Homebrew tends to move the files around when they decide that they're no longer where they think they should be.

MacPorts (default install) is always in /opt/local and nowhere else and doesn't move files around at a whim the way that Homebrew does. If libraries are found in both places, the stable one should be preferred.

I'm not on my home computer to be able to assess the rest of what you said at this point, so this comment might be updated.

[mislav]

Hi, thank you for the contribution and for simplifying the diff like I asked! I merged this after making some tweaks of my own, namely:

1. Homebrew's Cellar directory is now matched precisely against paths reported by $(brew --prefix)/Cellar and $(brew --repository)/Cellar. Any other path with "Cellar" being one of its segments is not considered related to Homebrew.
2. I've fixed some tests that were failing due to test stubs that were outdated or incomplete. (The stubs approach in our test suite is a little messy and brittle, so apologies if that was hard to pin down when you were working on your contribution.)

[eregon]

$prefix_prefix -> $brew_prefix probably?

[mislav]

I do not understand?

[halostatue]

Suggested change

	if [[ -n $prefix_prefix && ( $prefix == "$brew_prefix"/Cellar/* || \
	if [[ -n $brew_prefix && ( $prefix == "$brew_prefix"/Cellar/* || \

[eregon]

Ah, copy-paste mistake, I'll update my comment above for posterity.
But yes, what @halostatue showed (I tried to find the suggestion thing but wasn't available in the commit/diff view for some reason).

[mislav]

Ah, thanks both! 🤦