Mac Catalyst Support

[johnzhou721]

Refs #7 and #5, with #7 being the messed-up version of this patch. This is still draft. Part of beeware/Python-Apple-support#117

[johnzhou721]

OK so command I'm testing locally w/ now (the cflags and ldflags don't worry they are my own directories w/ unpacked deps packages

export PATH=./MacCatalyst/Resources/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
./configure --host=arm64-apple-ios14.0-macabi --build=arm64-apple-darwin --enable-framework --with-build-python=/Library/Frameworks/Python.framework/Versions/3.14/bin/python3.14 CFLAGS="-I/Users/johnzhou/cpython-macabi-deps/include" LDFLAGS="-L/Users/johnzhou/cpython-macabi-deps/lib" LIBLZMA_LIBS="-llzma" BZIP2_LIBS="-lbz2" LIBMPDEC_LIBS="-lmpdec" LIBFFI_LIBS="-lffi" --with-openssl=/Users/johnzhou/cpython-macabi-deps/ && make -j16 && make install

This is just for my own benefit.

[johnzhou721]

A way more involved set of changes is required; Mac Catalyst is versioned.

[johnzhou721]

Need logic to get the MACOS version for this -- currently we have ios deployment target:

<key>LSMinimumSystemVersion</key>
	<string>10.0</string>

[johnzhou721]

OK just destroyed the iOS testbed -- Mac Catalyst has a sort of different process to do stuff (not using simulators, diff ways to codesign etc) and I'm not that familiar and will investigate later.

[johnzhou721]

More stuff also needs to be modified w.r.t. versioned structure (but embedded only) for Mac Catalyst.

[johnzhou721]

Just pushed 2 nonworking commits on the versioned structure for mac catalyst but I have absolutely no clue right now... so I need to dig in a bit

Also apologize for the large amounts of small comments.

[freakboy3742]

This branch has now diverged pretty far from the current state of 3.14-patched; I've got all these changes collapsed into a single commit if you'd like me to force-push an update to this branch.

In my experimentation, the build completes, but testing isn't a simple process. The testbed shouldn't be needed, as there's no need to start a simulator... but python.exe doesn't work either, because of RPATH-based framework linking issues.

[johnzhou721]

Framework structure is sort of complicated right now, there's probably no need for the unix tools (unless someone wants a repl interface to experiment w/ stuff on MacCatalyst, which is doable), and it has to be relocatable (I mean technically you could have a Catalyst Python install but my gut feeling is that it's not going to be that useful), so we need to come up with something. My latest commit on that PR has singled out the Mac Catalyst case in the framework install targets section of the configure.ac. Go ahead and force push, but please reply with a commit hash of the latest commit I have for backup. Definitely need to sort out the RPATH stuff. Anyway this is further away from merging than I thought when I first pulled the pr from @asavva-2016.

…

On Thu, Apr 24, 2025 at 11:23 PM Russell Keith-Magee < ***@***.***> wrote: *freakboy3742* left a comment (freakboy3742/cpython#8) <#8 (comment)> This branch has now diverged pretty far from the current state of 3.14-patched; I've got all these changes collapsed into a single commit if you'd like me to force-push an update to this branch. In my experimentation, the build *completes*, but testing isn't a simple process. The testbed shouldn't be needed, as there's no need to start a simulator... but python.exe doesn't work either, because of RPATH-based framework linking issues. — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BRODAIZOBPCWLUC6NSO76B323G2EJAVCNFSM6AAAAAB3XTYCZCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMRZGM2DAMZSGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[johnzhou721]

OK just finished force-pushing -- added some new changes.

[johnzhou721]

Right now we have

ln -fs "../../../Python" "iOS/Frameworks/arm64-iphoneos-macabi/Python.framework/Versions/3.14/lib/python3.14/config-3.14-arm64-iphoneos-macabi/libpython3.14.a"
ln: iOS/Frameworks/arm64-iphoneos-macabi/Python.framework/Versions/3.14/lib/python3.14/config-3.14-arm64-iphoneos-macabi/libpython3.14.a: No such file or directory
make: *** [frameworkinstallmaclib] Error 1

[johnzhou721]

OK that is fixed and now this builds!

rpath needs to be fixed definitely -- but I think we should start the review process.

Remarks:

1. The plist generated as of 2025 now uses the macOS version instead of the iOS version in the minimum deployment target for MacCatalyst, we need to find a mapping of these values and hardcode them -- https://stackoverflow.com/questions/63581114/mac-catalyst-version may be helpful but I'll save the polishing until later -- we need the embedding done first.
2. When we run Python.exe now in the terminal, what is failing is NOT the rpath (although it is problematic, see later) but some stuff during startup using _ios_support and somehow getting a none somewhere -- also ios platform isn't really accomodated for terminal stuff I guess so this might require changes, but just for testing purposes -- but I'd rather have an ``embedded''/testbed test because then we test how it actually works in an application and whether any of the embedding process is broken.
3. rpath issue. Right now it referenced the path of the framework but within the build directory, which is problematic. Will investigate later.
4. Testing -- To [provide a testbed] or not to [provide a testbed], that is the question. My inclination is yes -- see two items abover; however we shall figure out how to put stuff in the bundle etc.

Overall I'm ending my day now but will make this ready for review so you might leave some additional remarks, even though it'll probably take at least 1-2 weeks on my end to get this merged, because I have school mon-fri and (just like you, sorry for not respecting that and the repeated pings) spend time w/ family (aka. mom dad and sibling(s) and those only) on sat and sun, so I might only have 1h at worst per day.

[johnzhou721]

Oh I'm really sorry but forgot to add

5. Platform identitification -- how can we know we're on Catalyst? Any stuff to take care of regarding iOS vs Catalyst?

Also that said I'll be waking up very early tomorrow so like if you have urgent questions you can ask 6:20 pm in Perth, Australia (it's 5:20am here, but I have some stuff at 5:30am which I won't disclose).

[freakboy3742]

OK that is fixed and now this builds!

rpath needs to be fixed definitely -- but I think we should start the review process.

So - it's very difficult to review something that doesn't actually do anything yet. I can pick on cosmetic stuff (the indentation in the configure.ac is the biggest thing that stands out); but unless I can start an interpreter or run the test suite... it doesn't work yet. There's nothing to meaningfully review.

This PR doesn't contain a testbed project; and as I said yesterday, when I run python.exe from any location, or with any DYLD_LIBRARY_PATH set, I get RPATH errors... so if you're getting further than that, you'll need to reveal your testing strategy.

[johnzhou721]

@freakboy3742 Thanks for organizing everything in the latest PR but afraid would have to revert some changes in the configure.ac -- anyway is a separate testbed acceptable? Or would you prefer it to be one? The latter would require some stuff and the glue scripts to be re-written.

[johnzhou721]

Will bundle a testbed soon but in the meantime a humongous amount of tests are failing on my end.

[freakboy3742]

@freakboy3742 Thanks for organizing everything in the latest PR but afraid would have to revert some changes in the configure.ac -- anyway is a separate testbed acceptable? Or would you prefer it to be one? The latter would require some stuff and the glue scripts to be re-written.

I'm not clear why you've reverted the configure.ac changes though. Why does MacCatalyst require a versioned framework? A versioned framework is required on macOS because it's installed in /Library; that won't be the case here.

As for the testbed; I've been considering whether the iOS, tvOS, watchOS, visionOS and MacCatalyst directories need to be merged into a single Apple folder so that they can share a common testbed project and script, with multiple output targets. We don't need to solve that problem right now, but it's something that probably needs to be done sooner rather than later. For right now, I'm OK with a duplicated testbed project.

[freakboy3742]

Also - as there have been changes to the visionOS patch, this PR now needs to be updated; when you do, you need to use a rebase, not a merge commit. The "patch tree" approach to managing the 3.14-patched branch works isn't compatible with merge commits.

[johnzhou721]

I'm not clear why you've reverted the configure.ac changes though. Why does MacCatalyst require a versioned framework? A versioned framework is required on macOS because it's installed in /Library; that won't be the case here.

macOS distribution requires versioned framework AFAIK; see, for example, the Frameworks folder of LyX, a WYSIWYM LaTeX editor, and things like QtCore.framework is versioned:

[johnzhou721]

@freakboy3742 Done

[johnzhou721]

There's no way AFAIK to obtain the macOS version from the Mac Catalyst verison, the former of which is actually requried in our plist, so I provided the correct value for the default version and added --with-catalyst-macos-version to pass in the macOS version.

[johnzhou721]

I just killed the MacCatalyst testbed and merged with iOS testbed. Let me know what you think -- b/c this is usually what they do when they build actual apps.

[freakboy3742]

I'm not clear why you've reverted the configure.ac changes though. Why does MacCatalyst require a versioned framework? A versioned framework is required on macOS because it's installed in /Library; that won't be the case here.

macOS distribution requires versioned framework AFAIK; see, for example, the Frameworks folder of LyX, a WYSIWYM LaTeX editor, and things like QtCore.framework is versioned:

I'm not arguing that you can't use a versioned framework - I'm asking if we must use a versioned framework.

I just killed the MacCatalyst testbed and merged with iOS testbed. Let me know what you think -- b/c this is usually what they do when they build actual apps.

Some notes inline; keep in mind that one of the primary goals here is that this is a patch on top of cpython, so changes must be kept to a minimium. Even "minor reorganisations" are a problem because they won't merge/rebase cleanly.

There's no way AFAIK to obtain the macOS version from the Mac Catalyst verison, the former of which is actually requried in our plist, so I provided the correct value for the default version and added --with-catalyst-macos-version to pass in the macOS version.

Why is the macOS version needed? macOS version matters when it's a gateway for features... but if you're on Catalyst, the macOS version isn't the gateway being used - it's the iOS version.

[johnzhou721]

Sorry for the late response. Life has been crazy

The multiprocessing thing may be because that stuff assume the current stuff is a Python interpreter so it forks the iOSTestbed program instead of the python program which does nothing... and the forked process isn't debugged or tested. I'm not sure about this though, this is purely speculative.

Edits

EDIT: I might be right. Might be an embedded mode problem.

#include <Python/Python.h>

int main(int argc, char *argv[]) {
    Py_Initialize();

    PyRun_SimpleString(
        "import multiprocessing\n"
        "def worker():\n"
        "    print('Worker running')\n"
        "\n"
        "if __name__ == '__main__':\n"
        "    p = multiprocessing.Process(target=worker)\n"
        "    p.start()\n"
        "    p.join()\n"
    );

    Py_Finalize();
    return 0;
}

will hang and when I tried interrupting, there's an infinite scroll of errors, and when I force it to close, it showed that 1110 ish processes has been created. Yikes!

Since we're currently using embedded mode only, it'd make sense to skip this so we don't open this can of worms.

@freakboy3742 Any problems with my reproducer? Compiling using Python 3.14 framework.

I'll mark multiprocessing as done for now.

EDIT 2 -- I am right, see https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_executable, by default sys.executable is used, we can't provide a Python interpreter, and sys.executable is actually the embedder. So this is resolved -- no multiprocessing in embed mode.

EDIT 3 -- But if you're allowed to invoke other binaries in Mac Catalyst, we should definitely sort out the thing about rpath and install it into the framework, sign the binary tool, and have the program use that binary. Should this be done? It's sort of complicated.

[johnzhou721]

My test_os modification might be suspect to https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use but I think it's sufficient here since we're not invoking any more system things that might be doing those stuff.

IMO we don't need to worry about multiprocessing right now since the macOS stuff doesn't have a bundled interpreter anyways.

[johnzhou721]

mimetypes reproducer: test_httplib test_mimetypes

[johnzhou721]

test.test_httplib.HTTPSTest.test_local_bad_hostname test.test_mimetypes.CommandLineTest.test_invocation_error reproduces!

[johnzhou721]

Nevermind... python#132266 seems to be solve this and is in Beta 1, so I'll cherry-pick f5a7037 into here to get a clean testbed run, but this will be unnessacary once we update the patch to Beta 1.

[johnzhou721]

Will rerequest review after the tests pass and I make a few final cleanups.

When I started this I thought it was just going to be some cherry-picking and addressing comments and merging, but this went from 6 to 29 pages for the patch length...

EDIT

Nevermind... another fail... I hate order dependent tests

[johnzhou721]

Turns out it was caused by me, myself. https://en.wikipedia.org/wiki/Memory_leak#Pseudocode is pretty similar, not sure if this is it, will retest

[johnzhou721]

Yes!!! For the first time in forever, I got a clean testbed run.

Requesting a review in GitHub again, but the existing things I have not resolved are still pending reply.

Thank you for your extremely nontrivial assistance on this patch.

[freakboy3742]

FYI - I likely won't get a chance to look at this for a couple of days; PyCon US sprints for the next couple of days will keep me fairly well occupied.

[johnzhou721]

@freakboy3742 Don't believe the force push is correct... will need to drop the tvOS, watchOS + visionOS commits.

EDIT or is this intentional to combine into 1 commit?

EDIT nevermind it's me who need to rebase... misread branch name, sorry

[johnzhou721]

Done. Really sorry for accientally misreading the branch name in the last comment.

[johnzhou721]

Aplogies for missing the latest force push.

[johnzhou721]

FInal pieces of cleanup.

[johnzhou721]

FYI: I've cleaned up some of the things I flagged and also cleaned up a bunch of comments.

[johnzhou721]

Hi @freakboy3742 , been digging through my list of old PRs and found this -- could another look be taken at this?

Current state of this PR should be passing the testbed; most changes have been straightforward except for the guarded file descriptor modification on line 2513 of test_os.py -- fstat fails for guarded FDs making Python think it's open to use, so for mac catalyst (not sure if relevant on other apple platforms?) the test keeps opening new FDs (since they're sequential with the exception of some of those system-guarded ones) on a file until they become non-sequential and uses the sequential range opened to test closerange.

Thanks!

[freakboy3742]

Apologies for the delay on reviewing this. I've been mostly deferring looking into this (and #11) because I knew that python#138176 was going to be on my todo list, and it's going to be a lot easier to land other Apple changes once that change is in place.

Actually getting to the point of writing that PR took a lot longer than I was hoping...

So - my current intention is to wait until that PR has landed, and then integrate these changes into that new directory (and build/testbed) structure.

[johnzhou721]

@freakboy3742 No worries, more of a mistake on my part flooding in new PRs when I haven't completely old ones.

I won't have time to integrate testbed into new directory structure though, so if your schedule's also pretty tight we either omit the testbed part or close this PR. Both options are fine with me.