Accepted macOS tags don't match canonical list.

[freakboy3742]

Describe the bug

The list of architecture tags accepted for macOS includes:

warehouse/warehouse/forklift/legacy.py

Lines 135 to 147 in 88bc28b

	_macosx_arches = {
	"ppc",
	"ppc64",
	"i386",
	"x86_64",
	"arm64",
	"intel",
	"fat",
	"fat32",
	"fat64",
	"universal",
	"universal2",
	}

However, the official list of tags generated by CPython are:
https://github.com/python/cpython/blob/cdcacec79f7a216c3c988baa4dc31ce4e76c97ac/Lib/_osx_support.py#L546-L562

This means warehouse accepts fat32 (which isn't defined), but doesn't accept fat3 (which is).

Expected behavior

fat3 wheels should be accepted for upload.

fat32 wheels should be rejected.

To Reproduce

Attempt to upload a fat3 macOS wheel.

My Platform

macOS.

Additional context

The code fix is trivial - it's almost certainly a typo misreading fat3 as fat32. However, it's not clear if there's any additional fixes required on the backend - if there are any fat32 wheels that have been successfully uploaded.

[freakboy3742]

Related - utils/wheel.py doesn't provide any annotations for combined wheels other than universal2. Adding them isn't a problem... but they're also of almost entirely historical significance.

[woodruffw]

I could be wrong, but I think that's actually a typo in CPython, rather than in Warehouse 😅 -- fat64 presumably refers to a 64-bit "fat" Mach-O header, which exists not for 64-bit arch support but because the original 32-bit "fat" Mach-O header wasn't big enough to store multiple large Mach-O slices.

You can see this in fat_arch and fat_arch_64 within Apple's cctools/libmacho, e.g.:

https://github.com/apple-oss-distributions/cctools/blob/3406a8e0f9ec28862967217797fe2b9a7b3d10ed/include/mach-o/fat.h#L54-L87

(Out of curiosity, what kind of build/toolchain produced a 32-bit "fat" Mach-O for you? I'm pretty surprised this wasn't hit before!)

[di]

For context, these architectures don't seem to be being used:

warehouse=> select count(*) from file_registry where filename ilike '%_fat.whl';
 count
-------
    14
(1 row)

warehouse=> select count(*) from file_registry where filename ilike '%_fat32.whl';
 count
-------
     0
(1 row)

warehouse=> select count(*) from file_registry where filename ilike '%_fat64.whl';
 count
-------
     0
(1 row)

There are some eggs which include them though:

warehouse=> select count(*) from file_registry where filename ilike '%_fat.egg';
 count
-------
   129
(1 row)

warehouse=> select count(*) from file_registry where filename ilike '%_fat32.egg';
 count
-------
     0
(1 row)

warehouse=> select count(*) from file_registry where filename ilike '%_fat64.egg';
 count
-------
   103
(1 row)

[woodruffw]

Yeah, that makes a lot of sense to me -- a "fat" Mach-O is a multi-arch binary that needs to be parsed to figure out which architectures it supports, so having a fat*-tagged wheel doesn't help pip or other tools actually decide whether or not to install/resolve that wheel.

From a quick look at my own native wheels, it looks like compressed tag sets suffix work correctly, e.g.: pyrage-1.2.3-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl. That wheel de facto contains one or more "fat" Mach-Os, since that's the only way it could work on the incompatible architectures specified in the tag set.

[freakboy3742]

I could be wrong, but I think that's actually a typo in CPython, rather than in Warehouse 😅 -- fat64 presumably refers to a 64-bit "fat" Mach-O header, which exists not for 64-bit arch support but because the original 32-bit "fat" Mach-O header wasn't big enough to store multiple large Mach-O slices.

I don't think it's referring to the header - it's referring to the slices in the combined binary. fat3 contains an x86-64 slice, and matches the "3-way" configuration option to --with-universal-archs. fat has only 32-bit slices.

(Out of curiosity, what kind of build/toolchain produced a 32-bit "fat" Mach-O for you? I'm pretty surprised this wasn't hit before!)

Me personally? I haven't produce a fat binary. I've been formally documenting the macOS tag format, and trying to explain what universal2 described opened a can of worms :-)

For context, these architectures don't seem to be being used:

The fat32 usage numbers are consistent with Warehouse not accepting a format that Python produces (and vice versa). The worst case would have been if there was a fat32 wheel/egg... and then trying to work out what it was :-)

[woodruffw]

I don't think it's referring to the header - it's referring to the slices in the combined binary. fat3 contains an x86-64 slice, and matches the "3-way" configuration option to --with-universal-archs. fat has only 32-bit slices.

Oh, you're right -- I misread that fat was also present. Weird!

So, if I understand correctly, a wheel tagged with fat3 would be considered suitable for resolution on a macOS host that has a machine arch of i386, ppc, or x86-64?

[freakboy3742]

So, if I understand correctly, a wheel tagged with fat3 would be considered suitable for resolution on a macOS host that has a machine arch of i386, ppc, or x86-64?

That's correct; and universal (without the 2) would be ppc, ppc64, i386 and x86-64.

For completeness, we clearly we need to add a parallel-universal target that includes ppc, ppc64, i386 and x86-64 and arm64 😜 (EDITOR: no, we do not)