Splitting this package in managable chunks

[hmaarrfk]

Comment:

This package currently requires more than 16 builds to be build manually to ensure that it completes in time on the CIs.

Step 1: No more git clone

rgommers identified that one portion of the build process that takes time is cloning the repository. In my experience, cloning the 1.5GB repo can take up to 10 min on my powerful local machine, but I feel like it can take much longer on the CIs.

To avoid cloning, we will have to list out all the submodule manually, or make the conda-forge installable dependencies.

I mostly got this working using a recursive script which should help us keep it maintained: #109

Option 1: Split off Dependencies:

Dependency	linux	mac	win	GPU Aware	PR	system deps
pybind11				no	https://github.com/conda-forge/pybind11-feedstock	USE_SYSTEM_PYBIND11
cub				no	https://github.com/conda-forge/cub-feedstock
eigen				no	https://github.com/conda-forge/eigen-feedstock	USE_SYSTEM_EIGEN_INSTALL
googletest				no	will not package
benchmark				no	https://github.com/conda-forge/benchmark-feedstock
protobuf				no	https://github.com/conda-forge/libprotobuf-feedstock
ios-cmake					not needed since we don't target ios
NNPACK	yes	yes		no	conda-forge/staged-recipes#19103
gloo	yes	yes		yes	conda-forge/staged-recipes#19103	USE_SYSTEM_GLOO
pthreadpool	yes	yes		no	conda-forge/staged-recipes#19103	USE_SYSTEM_PTHREADPOOL
FXdiv	yes	yes		header	conda-forge/staged-recipes#19103	USE_SYSTEM_FXDIV
FP16	yes	yes		header	conda-forge/staged-recipes#19103	USE_SYSTEM_FP16
psimd	yes	yes		header	conda-forge/staged-recipes#19103	USE_SYSTEM_PSIMD
zstd	yes	yes	yes	no	https://github.com/conda-forge/zstd-feedstock
cpuinfo	yes	yes	no	no	conda-forge/staged-recipes#19103	USE_SYSTEM_CPUINFO
python-enum				no	https://github.com/conda-forge/enum34-feedstock
python-peachpy	yes	yes	yes	no	conda-forge/staged-recipes#19103
python-six	yes	yes	yes	no	https://github.com/conda-forge/six-feedstock
onnx				no	https://github.com/conda-forge/onnx-feedstock	USE_SYSTEM_ONNX
onnx-tensorrt				only
sleef				no	https://github.com/conda-forge/sleef-feedstock	USE_SYSTEM_SLEEF
ideep
oneapisrc
nccl					https://github.com/conda-forge/nccl-feedstock
gemmlowp
QNNPACK	yes	yes			conda-forge/staged-recipes#19103
neon2sse
fbgemm				yes
foxi
tbb					https://github.com/conda-forge/tbb-feedstock	USE_SYSTEM_TBB (deprecated)
fbjni
XNNPACK	yes	yes			conda-forge/staged-recipes#19103	USE_SYSTEM_XNNPACK
fmt					https://github.com/conda-forge/fmt-feedstock
tensorpipe				yes
cudnn_frontend
kineto
pocketfft
breakpad
flatbuffers	yes	yes	yes	no	https://github.com/conda-forge/flatbuffers-feedstock
clog	static	static			conda-forge/staged-recipes#19103

• clog seems to be a pretty low level library that is assisted by compile time flags. I think it is best if we don't package that one as a library. It seems like it will require some serious consideration in terms of performance if we do. They typically the full source in the repository. The only problematic thing, is that each package attempts to install the static library into the library path.
• QNNPACK has a build option to allow a special provision for CAFFE2's implementation of pthreadpool
  • It seems to be problematic with pthreadpool on OSX.
• QNNPACK likely has two different implementations, the one they vendored in ATen, and the one they vendored in third_party.
• NNPACK has two different backens, one generated by python it seems, but for some reason fp16.py cannot be found, the other with psimd.

Option 2 - step 1: Build a libpytorch package or something

By setting BUILD_PYTHON=OFF in #112 we then end up with the following libraries in lib and include:

Dependency	linux	mac	win	GPU Aware	PR
libasmjit	yes	yes			conda-forge/staged-recipes#19103
libc10	yes	yes			conda-forge/staged-recipes#19103
libfbgemm	yes	yes		yes	conda-forge/staged-recipes#19103
libgloo	yes	yes		yes
libkineto	yes			yes	conda-forge/staged-recipes#19103
libnnpack	yes			???	conda-forge/staged-recipes#19103
libpytorch_qnnpack	yes	yes			conda-forge/staged-recipes#19103
libqnnpack	yes	yes			conda-forge/staged-recipes#19103
libtensorpipe				yes
libtorch
libtorch_cpu
libtorch_global_deps
Header only
ATen
c10d
caffe2
libnop	yes	yes			conda-forge/staged-recipes#19103

Option 2 - step 2: Depend on new ATen/libpytorch package

Compilation time progress

platform	python	cuda	main	tar gh-109	system deps
linux 64	3.7	no	1h57m	1h54m
linux 64	3.8	no	2h0m	1h51m
linux 64	3.9	no	2h31m	2h2m
linux 64	3.10	no	2h26m	2h7m
linux 64	3.7	11.2	6h+ (3933/4242 309 remaining)	6h+
linux 64	3.8	11.2	6h+ (3897/4242 345 remaning)	6h+
linux 64	3.9	11.2	6h+ (3924/4242 318 remaining)	6h+	6h+1656/1969 313 remaining
linux 64	3.10	11.2	6h+ (3962/4242 280 remaining)	6h+
osx-64	3.7		2h42m	2h39m
osx-64	3.8		3h28m	2h52m
osx-64	3.9		2h40m	2h42m
osx-64	3.10		3h2m	2h42m
osx-arm-64	3.8		1h51	1h37m
osx-arm-64	3.9		2h20m	2h10m
osx-arm-64	3.10		4h25m	2h1m

There are approximately:

• 3600 files to compile for cmake for the CPU builds with the standard build process
• 1600-1800 files to compile when using system dependencies: WIP: Use more system libs #111

[rgommers]

To avoid cloning, we will have to list out all the submodule manually, or make the conda-forge installable dependencies.

Cloning with --depth 1 seems preferable to separately building as dependencies. Separate dependencies/feedstocks/packages is a lot of overhead and noise for something that isn't usable by anything other than this feedstock.

The script in gh-109 looks interesting. Should work like that I guess; I just forgot why using --depth doesn't work? Seems like a lacking feature in git itself if it doesn't allow a shallow clone.

[hmaarrfk]

I think the problem is that conda first clones the main branch with depth 1, then cannot switch to an older tag like version v1.11.0 because it didn't clone it.

It also didn't play well with caching.

It is somewhat of a job to unbundle but i guess I find it worthwhile if it means we can release this more easily. I'm hoping i can patch things in a way that is acceptable upstream.

[hmaarrfk]

I remember what conda tried to do:

• it tried to clone a bare repo locally.
• then uses that as a cache to clone the sources for the build.

That's not super valuable for CI workflows, but makes it hard to do a shallow clone. At the time I couldn't think of a solution to propose upstream to conda build.

[rgommers]

It is somewhat of a job to unbundle but i guess I find it worthwhile if it means we can release this more easily. I'm hoping i can patch things in a way that is acceptable upstream.

Makes sense. I have no problem with unbundling provided it doesn't change the sources that are built. The kind of unbundling Linux distros do, like "hey this project is pinned to version X of , but we insist on using our own version Y" is much more problematic, because you then build a combo of sources that is not tested at all in upstream CI, and may be plain buggy.

[hmaarrfk]

We kinda do that with along of c dependencies don't we (not as much in pytorc)

My hope is that i can split off onnx and ATen in versions that match pytorch.

[hmaarrfk]

You can follow somewhat of a first pass at step 2 here conda-forge/staged-recipes#19103 (comment)

There are quite a few header only, and other libraries, that get downloaded on the fly using custom cmake code.

That isn't really fun. So even what I did in gh-109 isn't really complete, in terms of not downloading during bulid.

[h-vetinari]

That isn't really fun. So even what I did in gh-109 isn't really complete, in terms of not downloading during bulid.

That list of submodules is insane. Reminds me of what I quipped in #76 when I first came across that:

Pytorch has [...] under third_party/ (side note: holy maccaroni, that folder is a packager's nightmare 😅).

Seems even that was underestimating the extent of the issue. Unsurprisingly, I really dislike this "we vendor specific commits of open source projects" development model - it's a very "my castle" approach.

On the other hand, I see where it is coming from, with C/C++'s complete lack of standardised tooling around distribution.

[h-vetinari]

But I don't get so many things in that list, especially mature projects. Why vendor six? tbb? fmt? pybind11? The list goes on.

All in all, I fully support ripping this apart one by one (hopefully even in ways that would be palatable upstream), but I get Ralf's point about not diverging from what's actually being tested - though I'd be fine to caveat that based on an actually conceivable risk of breakage (e.g. if there are no functional changes between the vendored commit and a released version in a given submodule)

[hmaarrfk]

On the other hand, I see where it is coming from, with C/C++'s complete lack of standardised tooling around distribution.

Right. This is likely what the original creators were grappling with. They decided to either use git submodules in certain projects, or cmake code to download things they needed. Bazel does the same.

But I don't get so many things in that list, especially mature projects. Why vendor six? tbb? fmt? pybind11? The list goes on.

The issue occurs on who is in charge of the support. pytorch (and facebook) cannot force six or tbb to push a fix if their users (other developers at facebook) find a problem. Eventually, one user will have an issue. Because they have the developer resources, they decide to take on the responsibility of maintaining it for their package.

When pip was the only option, you were beholden to the creator of the original package on pypi pleading them to support a feature you need (i've been there many time, and in a sense, we are there with our packaging asks for pytorch).

[h-vetinari]

The issue occurs on who is in charge of the support. pytorch (and facebook) cannot force six or tbb to push a fix if their users (other developers at facebook) find a problem. Eventually, one user will have an issue. Because they have the developer resources, they decide to take on the responsibility of maintaining it for their package.

Sure, but what's missing IMO is closing the loop to a released version with the bugfix afterwards.

[hmaarrfk]

Sure, but what's missing IMO is closing the loop to a released version with the bugfix afterwards.

Its pretty hard to make a business case as to why you should spend a few hours, and likely more time, submitting a fix upstream after you have fixed things for your users.

Anyway, i'm just going through listing things that need to be done. There are a few big packages that we might be able to take advantage from.