[BUG] Race condition due to reuse of build folder when building extensions in parallel

[Flamefire]

setuptools version

setuptools=70.0.0

Python version

Python 3.12

OS

RHEL 9

Additional environment information

No response

Description

Having multiple extensions in ext_modules being compiled in parallel with pip install --config-setting='--build-option=build_ext' --config-setting='--build-option=-j24' ... causes issues because they share the same build folder set at

setuptools/setuptools/_distutils/command/build.py

Lines 111 to 112 in 5a13876

	if self.build_temp is None:
	self.build_temp = os.path.join(self.build_base, 'temp' + plat_specifier)

So when they write anything into that folder they may overwrite or delete files used by other extensions

Expected behavior

Building extensions in parallel has the same result as building serially

How to Reproduce

• Add multiple extensions to ext_modules that use e.g. the same source files but compile them e.g. with different flags.
• Install with the parallel flag set

This was observed with DeepSpeed so you could try installing that from source with "enough" cores but it might be too much to setup

Output

Output varies and it might succeed depending on the amount of parallel actions going on and the available CPU cores. So it may sometimes succeed or fail seemingly randomly

[abravalheri]

I believe this is a duplicate of #3119

[Flamefire]

This is a separate issue, which someone has brought up there though: #3119 (comment)

The issue over there seems to be running 2 builds in parallel, while this one is about setuptools itself running extension builds in parallel while the package is built serially/alone.

@Yay295 has a nice small example over there:

setup(
    ext_modules=[
        Extension("A", ["a.c", "common.c"]),
        Extension("B", ["b.c", "common.c"])
    ]
)

That issue over there might be (partially?) resolved though:

Try to perform parallel builds on it (eg: building wheels for different Python versions, for C extensions).

The build dir encodes some platform specifics which usually includes the Python version:

setuptools/setuptools/_distutils/command/build.py

Line 112 in 5a13876

self.build_temp = os.path.join(self.build_base, 'temp' + plat_specifier)

&

setuptools/setuptools/_distutils/command/build.py

Line 80 in 5a13876

plat_specifier = f".{self.plat_name}-{sys.implementation.cache_tag}"

(through sys.implementation.cache_tag)

[abravalheri]

OK, I see, thanks. So the TL;DR is that 3119 is about multiple wheels in parallel. This issue is about a single wheel with multiple pieces in parallel.

I am marking this issue with help wanted so that members of the community can submit PRs targeting it. Contributors are welcome.

One thing that requires attention though is that changes in the setuptools/_distutils directory need to be addressed in the pypa/distutils repository.

[Flamefire]

A rough idea I have is replacing this line:

setuptools/setuptools/_distutils/command/build.py

Line 112 in 5a13876

self.build_temp = os.path.join(self.build_base, 'temp' + plat_specifier)

with tempfile.mkdtemp to get a unique folder. Not sure if this would break anything, e.g. if anyone relies on the exact name of the folder, but that would be bad practice anyway.
Coming up with a test might be harder.

[abravalheri]

e.g. if anyone relies on the exact name of the folder, but that would be bad practice anyway.

In practice you find all sorts of assumptions of 3rd-party code depending on setuptools 😅

Maybe one approach would be to go one directory deeper using mkdtemp, but only when the number of parallel builds is > 1? (that would be a "soft opt-in" instead of requiring a full new opt-in flag - not sure if this approach is "gentle" enough though).

Thank you very much!

[Flamefire]

That could work. And it would likely need to be done further below in build_extension

setuptools/setuptools/_distutils/command/build_ext.py

Line 520 in 5a13876

def build_extension(self, ext) -> None:

Because all extensions seem to share the build_temp as the build class isn't per extension but the "collection".
Upside is that at that point parallel is available, downside that subclasses of build_ext do not automatically get that. Or maybe some custom code for build wherever it passes build_temp to the build_ext commands to create those subdirs there. That could even be deterministic e.g. using the extension-name and adding a number if required.

But I'm not familiar enough with how that all works together, so someone else would need to look into that.