Support installation of .dylib, .dll, and .so file extensions directly without requiring a build

[godruoyi]

Description

Thank you for creating such a nice tool that allows us to avoid using PECL any more.

I would like to ask if there are any plans to support the direct installation of extension files like .dylib, .dll, and .so, without the need for building them when we run pie install . It would be simpler since we can build the extension in another language and publish the artifacts to GitHub, so that we can install them directly without having to build them again. I would be interested in supporting this feature if it is feasible.

[asgrim]

There is upsides and downsides to this approach:

First up regarding Linux/Mac (.so) binaries: Packages providing binaries would become responsible for building binaries for every architecture/system, which is a huge undertaking requiring massive CI pipelines to produce all the necessary permutations of the binaries for different platforms; consider that you have arm64, amd64, x86, as common ones, but there are also many other less common architectures. Additionally, you may get dependent library compatibility issues; for example glibc vs musl, or even just dependent libraries that may have existed on the CI builder, but don't actually exist on the end user's machine. I really do think this maintenance burden on extension maintainers is too much; it might be okay for some, but I certainly would not want the responsibility of maintaining that! This is the reason PIE (and PECL before it) compiles binaries on the end user's system; to ensure everything is compatible for their system.

Secondly, regarding Windows (.dll) binaries: PIE compatible extensions on Windows must already provide binaries. This has historically always been precompiled on PECL. This used to be done by a "Windows PECL build server"; once an extension release was uploaded to the PECL site, a build job to produce the Windows binaries was run. This is a MUCH simpler feat than the Linux ecosystem, due to the enormously fewer architecture and dependency issues. For PIE, we take a similar approach, but the responsibility for publishing Windows binaries moves to the extension maintainers, instead of PECL website. Thankfully, at least on GitHub, this process is very simple thanks to @shivammathur 's excellent work on the php/php-windows-builder GitHub action.

I am not sure how this can be workable or practical for Linux/Mac systems, unfortunately.

[remicollet]

I can't imagine how this can be possible

There are so many distributions, versions, php versions, installation modes, providers, and architectures, I can only imagine how the matrix will be huge.

Only for Fedora/RHEL, each release commonly means >100 builds (for my repo)

P.S. to summarize my thinking

• People building PHP from sources should be able to build any extension from sources (using pecl, pie...)

• People using a binary installation of PHP should find the binary of wanted extensions from the same provider (ex: using the distribution package manager)

[Xuanwo]

Hi, maybe we can follow python's approach that provides pre-built binaries as best efforts and fallback to build locally if not found.

In this way, we can build extensions for most platforms like {linux,mac,windows} * {x86_64, aarch64}.

[godruoyi]

Thank you, @asgrim and @remicollet for your reply, but I still think this approach is feasible, here are my options:

1. PIE is a tool designed to install PHP extensions. While it primarily facilitates installation from source, we can also add functionality that allows users to install PHP extensions from binary driectly(like github releases). This way, we don't need to rely on extension maintainers providing binaries for all platforms or versions. Instead, if the user explicitly specifies that want to install from binary without build from course, PIE can attempt to download it first directly.

2. As @Xuanwo mentioned, extension maintainers typically only publish binaries for mainstream operating systems and versions. If a specific OS or version isn't supported by prebuilt binaries, we can fall back on building the extension from source as needed.

Implements detail may like:

• extension maintainers publish pre-built binaries to github like
  • opeadal-0.1.0-php8.1-maxosx_10_12. dylib
  • opeadal-0.1.0-php8.2-maxosx_10_12. dylib
  • opeadal-0.1.0-php8.2-manylinux_2_28.so
• pie install opendal/opeadal:0.1.0 --binary
• match pre-built binary base on local os(linux,mac,etc) and php version
• if found, install it directly, if not, back to build from source.

what do you guys think, pls Let me know your thoughts!

[remicollet]

opeadal-0.1.0-php8.2-manylinux_2_28.so

Can't work, too many dependencies on too many different library versions (libc which you probably refer in "2_28" part is not enough)
Generic linux build is a fantasy

[godruoyi]

@remicollet do you have more context why PyPI can works fine when user want install smme package from pre-built binaries directly without build, you can see there are many pre-buiild binaries https://pypi.org/project/opendal/#files.

[remicollet]

@remicollet do you have more context why PyPI can works fine when user want install smme package from pre-built binaries directly without build, you can see there are many pre-buiild binaries https://pypi.org/project/opendal/#files.

As I already said "too many dependencies on too many different library versions"
Can work with bundling everything, but this sounds terrible.

As I also already said "People using a binary installation of PHP should find the binary of wanted extensions from the same provider "

[Radiergummi]

I don't think this topic should be left just like this. The amount of repeated builds of the same source code in modern container environments is horrendous, and wastes incredible amounts of compute for the same result.

PHP extensions don't need to be built for Itanium processors with obscure libc anymore; following the Pareto principle and building for AMD64/ARM64, the currently actively supported PHP versions, MacOS/Windows/the top three Linux distributions leaves you with 2x2x5 combinations per extension.

That is the approach pypi follows, and wheels can be downloaded almost always in favor to building fresh.
This is more responsible, both economically and ecologically.