feature request: seamless import of Basilisp libraries in Python with implicit bootstrapping

[ikappaki]

Hi,

I'd like to request support for importing Basilisp libraries directly into Python applications like any standard Python library, without requiring explicit Basilisp bootstrapping or the user needing to know the library is written in Basilisp.

The goal is to enable writing Python packages in Basilisp that can be seamlessly imported into Python code, with Basilisp being implicitly bootstrapped as needed.

The Bootstrapping section of the Basilisp docs mentions basilisp.main.bootstrap and basilisp.main.init as bootstrapping methods, but it's unclear what happens when bootstrapping occurs multiple times, for example, if two Basilisp-based libraries each bootstrap themselves in their __init__.py.

The basilisp.main module also lists basilisp.main.bootstrap_python, but it's still unclear whether calling it multiple times is safe, or if the runtime will even be available without re-importing the module.

From reviewing the code, basilisp.main.init seems the most promising option

basilisp/src/basilisp/main.py

Lines 15 to 30 in 0fead05

	def init(opts: Optional[CompilerOpts] = None) -> None:
	"""
	Initialize the runtime environment for Basilisp code evaluation.
	Basilisp only needs to be initialized once per Python VM invocation. Subsequent
	imports of Basilisp namespaces will work using Python's standard ``import``
	statement and :external:py:func:`importlib.import_module` function.
	If you want to execute a Basilisp file which is stored in a well-formed package
	or module structure, you probably want to use :py:func:`bootstrap`.
	"""
	logconfig.configure_root_logger()
	runtime.init_ns_var()
	runtime.bootstrap_core(opts if opts is not None else compiler_opts())
	importer.hook_imports()
	importlib.import_module("basilisp.core")

Possibly in combination with a runtime check, such as whether BasilispImporter is already present in sys.meta_path, as done here:

basilisp/src/basilisp/importer.py

Lines 424 to 432 in 0fead05

	def hook_imports() -> None:
	"""Hook into Python's import machinery with a custom Basilisp code
	importer.
	Once this is called, Basilisp code may be called from within Python code
	using standard `import module.submodule` syntax."""
	if any(isinstance(o, BasilispImporter) for o in sys.meta_path):
	return
	sys.meta_path.insert(0, BasilispImporter())

Could you please advise of whether any of the existing bootstrapping methods can safely be used in this way (i.e., called multiple times at runtime) or suggest what changes might be needed to support this use case?

thanks

[chrisrink10]

This is definitely something that I am interested in exploring. Do you have a more concrete example for what specifically you're looking to be able to do? Like how would you be executing this code?

[ikappaki]

I've published a simple example Basilisp library at https://github.com/ikappaki/basilex-python-lib to demonstrate this use case.

The package bootstraps Basilisp in its __init__.py using basilisp.main.init:

https://github.com/ikappaki/basilex-python-lib/blob/374844e060959600e4f954c667be942428f8bbe7/src/basilex_python_lib/__init__.py#L1-L3

It defines a Basilisp function in basilex-python-lib.utils namespace:

https://github.com/ikappaki/basilex-python-lib/blob/374844e060959600e4f954c667be942428f8bbe7/src/basilex_python_lib/utils.lpy#L1-L4

This function can be imported and used from a clean Python environment like any standard Python function:

$ python3 -m venv tmp1 
$ source tmp1/bin/activate
(tmp1)  $ pip install git+https://github.com/ikappaki/basilex-python-lib.git
Collecting git+https://github.com/ikappaki/basilex-python-lib.git
...

(tmp1)  $ python
Python 3.13.0 (v3.13.0:60403a5409f, Oct  7 2024, 00:37:40) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from basilex_python_lib import utils
>>> utils.hello("there")
:hello there

Note that in this pristine environment, with only the Basilisp-based package installed, there’s no need to manually bootstrap Basilisp or even be aware that the function was written in Basilisp. The library behaves just like any regular Python package.

The open question is whether using basilisp.main.init in __init__.py is the correct approach. If another Basilisp based library were imported the same way, it would call init again, yet the behavior of multiple calls isn't clearly documented or defined.

thanks

[chrisrink10]

That's helpful to clarify the question. I think using basilisp.main.init in the way you're using it is my recommended approach, but to your point I do believe that there is currently nothing guarding against repeated invocations across other libraries.

Those repeated invocations of init() would likely cause basilisp.core to be reloaded on each call, which seems like it could be problematic.

I think for this ticket, we could add some synchronization within basilisp.main.init to short-circuit and return early if the runtime is already initialized.

We could also modify the documentation to include this pattern in the Bootstrapping section of the Getting Started page.

[ikappaki]

I think for this ticket, we could add some synchronization within basilisp.main.init to short-circuit and return early if the runtime is already initialized.

We could also modify the documentation to include this pattern in the Bootstrapping section of the Getting Started page.

Right, thanks! I'll stick with the if any(isinstance(o, BasilispImporter) for o in sys.meta_path): return guard as a workaround (mentioned earlier) while you consider the proper solution.

__init__.py

import sys

from basilisp.main import init
from basilisp.importer import BasilispImporter

if any(isinstance(o, BasilispImporter) for o in sys.meta_path):
    print("pass")
else:    
    init()

[amano-kenji]

Because python 3.13 has experimental free threading support, you might want to use a thread lock in basilisp.main.init.

Python is trying to free itself from global interpreter lock.

Perhaps, basilisp can have bootstrapped atom that's thread safe? If you use a thread lock, an atom may be unnecessary.

[amano-kenji]

We could also modify the documentation to include this pattern in the Bootstrapping section of the Getting Started page.

If the pull request documented this pattern, it would be perfect.