fix(local_runtime): Improve local_runtime usability in macos / windows

[laramiel]

local_runtime fails to handle many variations of python install on windows and MacOS, such as:

• LDLIBRARY on MacOS may refer to a file under PYTHONFRAMEWORKPREFIX, not LIBDIR
• LDLIBRARY on Windows refers to pythonXY.dll, not the linkable pythonXY.lib
• LIBDIR may not be correctly set on Windows.
• On windows, interpreter_path needs to be normalized. Other paths also require this.
• SHLIB_SUFFIX does not indicate the correct suffix.

For examples, see

See: https://docs.python.org/3/extending/windows.html

In order to resolve this the shared library resolution has been moved into get_local_runtime_info.py, which now does the following:

• Constructs a list of paths to search based on LIBDIR, LIBPL, PYTHONFRAMEWORKPREFIX, and the executable directory.
• Constructs a list of libraries to search based on INSTSONAME, LDLIBRARY, pythonXY.lib, etc.
• Checks to see which files exist, partitioning the result into a list of "dynamic_libraries" and "static_libraries"

On windows and macos, since SHLIB_SUFFIX does not always indicate the filenames needed searching, this has been removed from local_runtime_repo_setup and replaced with an explicit file.

On windows the interpreter_path and other search paths are now normalized ( \ converted to /).

Additional logging added to local_runtime_repo.

Example get_local_runtime_info.py outputs:

rules_python:local_runtime_repo(@@local_python3) INFO: GetPythonInfo result:
  INSTSONAME: Python.framework/Versions/3.12/Python
  LDLIBRARY: Python.framework/Versions/3.12/Python
  LIBDIR: /Library/Frameworks/Python.framework/Versions/3.12/lib
  MULTIARCH: darwin
  PY3LIBRARY: <empty string>
  PYTHONFRAMEWORKPREFIX: /Library/Frameworks
  base_executable: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12
  implementation_name: cpython
  include: /Library/Frameworks/Python.framework/Versions/3.12/include/python3.12
  major: 3
  micro: 10
  minor: 12

Here we see the MacOS framework issue, namely LDLIBRARY is not under LIBDIR, but directly under frameworks.
The new code adds PYTHONFRAMEWORKPREFIX to the search path, so the library is found.

rules_python:local_runtime_repo(@@local_python3)  INFO: GetPythonInfo result:
  INSTSONAME: None
  LDLIBRARY: python313.dll
  LIBDIR: T:\build_temp\home\python_4379bdec28bdff81a567a01c9b8cf10e3856c8c966e4fe53945bedea6338b416\tools\libs
  MULTIARCH: None
  PY3LIBRARY: python313.lib
  base_executable: T:\build_temp\home\python_4379bdec28bdff81a567a01c9b8cf10e3856c8c966e4fe53945bedea6338b416\tools\python.exe
  implementation_name: cpython
  include: T:\build_temp\home\python_4379bdec28bdff81a567a01c9b8cf10e3856c8c966e4fe53945bedea6338b416\tools\Include
  major: 3
  micro: 1
  minor: 13

Here we see that under windows the python313.dll is reported; python313.lib is added via PY3LIBRARY.

Fixes #3055

Three more notes on this CL; they may be areas for future improvements.

• On MacOS, when linking against the system python, we may want to use "-Wl,-framework,Python". However this CL does not detect / change those linker options.

• It would be nice to have tests for a minimal C python extension and minimal C python embedded use for the local_runtime as well as the hermetic runtimes; this is similar to tests/cc/current_cc_py_libs

• On some systems it appears that numpy includes may be symlinked into the python include directories, which could lead to ambiguity when trying to use other numpy headers. Consider filtering them.

[laramiel]

After integrating this some more with tensorstore, I've reworked the library lookup to handle manylinux and other linux installs with no dynamic libraries exposed.

[aignas]

Consider also referencing the issue in the changelog.

This PR is very exciting, but I gave it a quick glance only since it is reasonably big.

I would love to ask the local_python example also be made to work with bzlmod - all of the improvements should work there too.

[rickeylev]

Oddly, I can edit and commit files via the ui, but can't push to laramiel/main on the command line. Is there something special setup with your repo or branch? I was going to move the example over into local toolchains, but it was a bit hard to refactor that via the ui alone.

[rickeylev]

The integration tests for local toolchains are in tests/integration/local_toolchains. It should already have workspace and bzlmod stuff setup. I think the only thing to do is remove the target_compatible_with constraint that makes it only run on linux/mac.

[laramiel]

I don't see which target_compatible_with constraint affects tests/integration Can you drop a reference?

[rickeylev]

I was mistaken. It looks like we only run tests/integration on Linux, not Mac or Windows. And it only uses bzlmod.

I've setup workspace in that directory and deleted these files.

Enabling it in CI might be tricky because I'm not sure we have enough CI slots left. I've added the jobs configs; lets see how it goes.

[laramiel]

Oddly, I can edit and commit files via the ui, but can't push to laramiel/main on the command line. Is there something special setup with your repo or branch? I was going to move the example over into local toolchains, but it was a bit hard to refactor that via the ui alone.

I wouldn't expect so; I forked using the web ui and haven't changed anything.

[rickeylev]

This is mostly LGTM, just need to remove the example as shown and update tests/integration/local_toolchains instead.

[rickeylev]

FWIW, I think srcs can be a mix of static and shared libraries. cc_library will sort them out.

[rickeylev]

Oddly, I can edit and commit files via the ui, but can't push to laramiel/main on the command line. Is there something special setup with your repo or branch? I was going to move the example over into local toolchains, but it was a bit hard to refactor that via the ui alone.

I wouldn't expect so; I forked using the web ui and haven't changed anything.

Weird 🤷 . Maybe because your fork's branch is the main branch? sometimes those have additional protections.