Skip to content

Commit f7da5a2

Browse files
authored
Add a CLI subcommand for bootstrapping the Python installation (#789)
Fixes #790 Fixes #791
1 parent 5f5ebe4 commit f7da5a2

File tree

13 files changed

+615
-31
lines changed

13 files changed

+615
-31
lines changed

CHANGELOG.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99
* Added support for passing through `:tag` metadata to the generated Python AST (#354)
1010
* Added support for calling symbols as functions on maps and sets (#775)
1111
* Added support for passing command line arguments to Basilisp (#779)
12-
* Added support for autocompleting names in the `python/` pseudo-namespace for Python builtins at the REPL (#787)
12+
* Added support for autocompleting names in the `python/` pseudo-namespace for Python builtins at the REPL (#787)
13+
* Added a subcommand for bootstrapping the Python installation with Basilisp (#790)
14+
* Added support for executing Basilisp namespaces directly via `basilisp run` and by `python -m` (#791)
1315

1416
### Changed
1517
* Optimize calls to Python's `operator` module into their corresponding native operators (#754)

Makefile

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,17 @@ format:
88
@poetry run sh -c 'isort . && black .'
99

1010

11+
.PHONY: check
12+
check:
13+
@rm -f .coverage*
14+
@TOX_SKIP_ENV='pypy3|safety|coverage' poetry run tox run-parallel -p auto
15+
16+
17+
.PHONY: lint
18+
lint:
19+
@poetry run tox run-parallel -m lint
20+
21+
1122
.PHONY: repl
1223
repl:
1324
@BASILISP_USE_DEV_LOGGER=true poetry run basilisp repl
@@ -16,7 +27,12 @@ repl:
1627
.PHONY: test
1728
test:
1829
@rm -f .coverage*
19-
@TOX_SKIP_ENV='pypy3|safety|coverage' poetry run tox run-parallel -p 4
30+
@TOX_SKIP_ENV='pypy3' poetry run tox run-parallel -m test
31+
32+
33+
.PHONY: type-check
34+
type-check:
35+
@poetry run tox run-parallel -m mypy
2036

2137

2238
lispcore.py:

docs/cli.rst

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,47 @@ You can run Basilisp code from a string or by directly naming a file with the CL
7171
7272
basilisp run path/to/some/file.lpy
7373
74+
Any arguments passed to ``basilisp run`` beyond the name of the file or the code string will be bound to the var :lpy:var:`*command-line-args*` as a vector of strings.
75+
If no arguments are provided, ``*command-line-args*`` will be ``nil``.
76+
77+
.. code-block:: bash
78+
79+
$ basilisp run -c '(println *command-line-args*)' 1 2 3
80+
[1 2 3]
81+
$ basilisp run -c '(println *command-line-args*)'
82+
nil
83+
84+
.. _run_basilisp_applications:
85+
86+
Run Basilisp as an Application
87+
------------------------------
88+
89+
Python applications don't have nearly as many constraints on their entrypoints as do Java applications.
90+
Nevertheless, developers may have a clear entrypoint in mind when designing their application code.
91+
In such cases, it may be desirable to take advantage of the computed Python ``sys.path`` to invoke your entrypoint.
92+
To do so, you can use the ``basilisp run -n`` flag to invoke an namespace directly:
93+
94+
.. code-block:: bash
95+
96+
basilisp run -n package.core
97+
98+
When invoking your Basilisp code via namespace name, the specified namespace name will be bound to the var :lpy:var:`*main-ns*` as a symbol.
99+
This allows you to gate code which should only be executed when this namespace is executed as an entrypoint, but would otherwise allow you to ``require`` the namespace normally.
100+
101+
.. code-block:: clojure
102+
103+
(when (= *main-ns* 'package.core)
104+
(start-app))
105+
106+
This approximates the Python idiom of gating execution on import using ``if __name__ == "__main__":``.
107+
108+
This variant of ``basilisp run`` also permits users to provide command line arguments bound to :lpy:var:`*command-line-args*` as described above.
109+
110+
.. note::
111+
112+
Only ``basilisp run -n`` binds the value of :lpy:var:`*main-ns*`.
113+
In all other cases, it will be ``nil``.
114+
74115
.. _run_basilisp_tests:
75116

76117
Run Basilisp Tests
@@ -82,4 +123,22 @@ If you installed the `PyTest <https://docs.pytest.org/en/7.0.x/>`_ extra, you ca
82123
83124
basilisp test
84125
85-
Because Basilisp defers all testing logic to PyTest, you can use any standard PyTest arguments and flags from this entrypoint.
126+
Because Basilisp defers all testing logic to PyTest, you can use any standard PyTest arguments and flags from this entrypoint.
127+
128+
.. _bootstrap_cli_command:
129+
130+
Bootstrap Python Installation
131+
-----------------------------
132+
133+
For some installations, it may be desirable to have Basilisp readily importable whenever the Python interpreter is started.
134+
You can enable that as described in :ref:`bootstrapping`:
135+
136+
.. code-block:: bash
137+
138+
basilisp bootstrap
139+
140+
If you would like to remove the bootstrapped Basilisp from your installation, you can remove it:
141+
142+
.. code-block:: bash
143+
144+
basilisp bootstrap --uninstall

docs/contributing.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,15 @@ All three steps can be performed across all supported versions of CPython using
8484

8585
.. code-block:: bash
8686
87+
make check
88+
89+
Likewise, individual steps can be run across all supported verions using their respective targets:
90+
91+
.. code-block::
92+
93+
make lint
8794
make test
95+
make type-check
8896
8997
To run a more targeted CI check directly from within the Poetry shell, developers can use ``tox`` commands directly.
9098
For instance, to run only the tests for ``basilisp.io`` on Python 3.12, you could use the following command:

docs/gettingstarted.rst

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -132,4 +132,39 @@ For systems where the shebang line allows arguments, you can use ``#!/usr/bin/en
132132
.. code-block:: clojure
133133
134134
#!/usr/bin/env basilisp-run
135-
(println "Hello world!")
135+
(println "Hello world!")
136+
137+
Finally, Basilisp has a command line option to bootstrap your Python installation such that Basilisp will already be importable whenever Python is started.
138+
This takes advantage of the ``.pth`` file feature supported by the `site <https://docs.python.org/3/library/site.html>`_ package.
139+
Specifically, any file with a ``.pth`` extension located in any of the known ``site-packages`` directories will be read at startup and, if any line of such a file starts with ``import``, it is executed.
140+
141+
.. code-block:: bash
142+
143+
$ basilisp bootstrap
144+
Your Python installation has been bootstrapped! You can undo this at any time with with `basilisp bootstrap --uninstall`.
145+
$ python
146+
Python 3.12.1 (main, Jan 3 2024, 10:01:43) [GCC 11.4.0] on linux
147+
Type "help", "copyright", "credits" or "license" for more information.
148+
>>> import importlib; importlib.import_module("basilisp.core")
149+
<module 'basilisp.core' (/home/chris/Projects/basilisp/src/basilisp/core.lpy)>
150+
151+
This method also enables you to directly execute Basilisp scripts as Python modules using ``python -m {namespace}``.
152+
Basilisp namespaces run as a Python module directly via ``python -m`` are resolved within the context of the current ``sys.path`` of the active Python interpreter.
153+
154+
.. code-block:: bash
155+
156+
basilisp bootstrap # if you haven't already done so
157+
SITEPACKAGES="$(python -c 'import site; print(site.getsitepackages()[0])')" echo '(println "Hi!")' >> "$SITEPACKAGES/somescript.lpy"
158+
python -m somescript
159+
160+
.. note::
161+
162+
Most modern Python packaging tools do not permit arbitrary code to be installed during package installation, so this step must be performed manually.
163+
It only needs to be run once per Python installation or virtualenv.
164+
165+
.. warning::
166+
167+
Code in ``.pth`` files is executed each time the Python interpreter is started.
168+
The Python ``site`` documentation warns that "[i]ts impact should thus be kept to a minimum".
169+
Bootstrapping Basilisp can take as long as 30 seconds (or perhaps longer, though typically much shorter on modern systems) on the first run due to needing to compile :lpy:ns:`basilisp.core` to Python bytecode.
170+
Subsequent startups should be considerable faster unless users have taken any measures to disable :ref:`namespace_caching`.

src/basilisp/cli.py

Lines changed: 127 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import io
44
import os
55
import sys
6+
import textwrap
67
import traceback
78
import types
89
from pathlib import Path
@@ -51,6 +52,14 @@ def eval_file(filename: str, ctx: compiler.CompilerContext, ns: runtime.Namespac
5152
return eval_str(f'(load-file "{filename}")', ctx, ns, eof=object())
5253

5354

55+
def eval_namespace(
56+
namespace: str, ctx: compiler.CompilerContext, ns: runtime.Namespace
57+
):
58+
"""Evaluate a file with the given name into a Python module AST node."""
59+
path = "/" + "/".join(namespace.split("."))
60+
return eval_str(f'(load "{path}")', ctx, ns, eof=object())
61+
62+
5463
def bootstrap_repl(ctx: compiler.CompilerContext, which_ns: str) -> types.ModuleType:
5564
"""Bootstrap the REPL with a few useful vars and returned the bootstrapped
5665
module so it's functions can be used by the REPL command."""
@@ -242,6 +251,71 @@ def _wrapped_subcommand(subparsers: "argparse._SubParsersAction"):
242251
return _wrap_add_subcommand
243252

244253

254+
def bootstrap_basilisp_installation(_, args: argparse.Namespace) -> None:
255+
if args.quiet:
256+
print_ = lambda v: v
257+
else:
258+
print_ = print
259+
260+
if args.uninstall:
261+
if not (
262+
removed := basilisp.unbootstrap_python(site_packages=args.site_packages)
263+
):
264+
print_("No Basilisp bootstrap files were found.")
265+
else:
266+
for file in removed:
267+
print_(f"Removed '{file}'")
268+
else:
269+
basilisp.bootstrap_python(site_packages=args.site_packages)
270+
print_(
271+
"Your Python installation has been bootstrapped! You can undo this at any "
272+
"time with with `basilisp bootstrap --uninstall`."
273+
)
274+
275+
276+
@_subcommand(
277+
"bootstrap",
278+
help="bootstrap the Python installation to allow importing Basilisp namespaces",
279+
description=textwrap.dedent(
280+
"""Bootstrap the Python installation to allow importing Basilisp namespaces"
281+
without requiring an additional bootstrapping step.
282+
283+
Python installations are bootstrapped by installing a `basilispbootstrap.pth`
284+
file in your `site-packages` directory. Python installations execute `*.pth`
285+
files found at startup.
286+
287+
Bootstrapping your Python installation in this way can help avoid needing to
288+
perform manual bootstrapping from Python code within your application.
289+
290+
On the first startup, Basilisp will compile `basilisp.core` to byte code
291+
which could take up to 30 seconds in some cases depending on your system and
292+
which version of Python you are using. Subsequent startups should be
293+
considerably faster so long as you allow Basilisp to cache bytecode for
294+
namespaces."""
295+
),
296+
handler=bootstrap_basilisp_installation,
297+
)
298+
def _add_bootstrap_subcommand(parser: argparse.ArgumentParser) -> None:
299+
parser.add_argument(
300+
"--uninstall",
301+
action="store_true",
302+
help="if true, remove any `.pth` files installed by Basilisp in all site-packages directories",
303+
)
304+
parser.add_argument(
305+
"-q",
306+
"--quiet",
307+
action="store_true",
308+
help="if true, do not print out any",
309+
)
310+
# Allow specifying the "site-packages" directories via CLI argument for testing.
311+
# Not intended to be used by end users.
312+
parser.add_argument(
313+
"--site-packages",
314+
action="append",
315+
help=argparse.SUPPRESS,
316+
)
317+
318+
245319
def nrepl_server(
246320
_,
247321
args: argparse.Namespace,
@@ -389,9 +463,19 @@ def _add_repl_subcommand(parser: argparse.ArgumentParser) -> None:
389463

390464

391465
def run(
392-
_,
466+
parser: argparse.ArgumentParser,
393467
args: argparse.Namespace,
394468
):
469+
target = args.file_or_ns_or_code
470+
if args.load_namespace:
471+
if args.in_ns is not None:
472+
parser.error(
473+
"argument --in-ns: not allowed with argument -n/--load-namespace"
474+
)
475+
in_ns = target
476+
else:
477+
in_ns = target if args.in_ns is not None else runtime.REPL_DEFAULT_NS
478+
395479
opts = compiler.compiler_opts(
396480
warn_on_shadowed_name=args.warn_on_shadowed_name,
397481
warn_on_shadowed_var=args.warn_on_shadowed_var,
@@ -403,19 +487,15 @@ def run(
403487
ctx = compiler.CompilerContext(
404488
filename=CLI_INPUT_FILE_PATH
405489
if args.code
406-
else (
407-
STDIN_INPUT_FILE_PATH
408-
if args.file_or_code == STDIN_FILE_NAME
409-
else args.file_or_code
410-
),
490+
else (STDIN_INPUT_FILE_PATH if target == STDIN_FILE_NAME else target),
411491
opts=opts,
412492
)
413493
eof = object()
414494

415495
core_ns = runtime.Namespace.get(runtime.CORE_NS_SYM)
416496
assert core_ns is not None
417497

418-
with runtime.ns_bindings(args.in_ns) as ns:
498+
with runtime.ns_bindings(in_ns) as ns:
419499
ns.refer_all(core_ns)
420500

421501
if args.args:
@@ -424,32 +504,62 @@ def run(
424504
cli_args_var.bind_root(vec.vector(args.args))
425505

426506
if args.code:
427-
eval_str(args.file_or_code, ctx, ns, eof)
428-
elif args.file_or_code == STDIN_FILE_NAME:
507+
eval_str(target, ctx, ns, eof)
508+
elif args.load_namespace:
509+
# Set the requested namespace as the *main-ns*
510+
main_ns_var = core_ns.find(sym.symbol(runtime.MAIN_NS_VAR_NAME))
511+
assert main_ns_var is not None
512+
main_ns_var.bind_root(sym.symbol(target))
513+
514+
eval_namespace(target, ctx, ns)
515+
elif target == STDIN_FILE_NAME:
429516
eval_stream(io.TextIOWrapper(sys.stdin.buffer, encoding="utf-8"), ctx, ns)
430517
else:
431-
eval_file(args.file_or_code, ctx, ns)
518+
eval_file(target, ctx, ns)
432519

433520

434521
@_subcommand(
435522
"run",
436-
help="run a Basilisp script or code",
437-
description="Run a Basilisp script or a line of code, if it is provided.",
523+
help="run a Basilisp script or code or namespace",
524+
description=textwrap.dedent(
525+
"""Run a Basilisp script or a line of code or load a Basilisp namespace.
526+
527+
If `-c` is provided, execute the line of code as given. If `-n` is given,
528+
interpret `file_or_ns_or_code` as a fully qualified Basilisp namespace
529+
relative to `sys.path`. Otherwise, execute the file as a script relative to
530+
the current working directory.
531+
532+
`*main-ns*` will be set to the value provided for `-n`. In all other cases,
533+
it will be `nil`."""
534+
),
438535
handler=run,
439536
)
440537
def _add_run_subcommand(parser: argparse.ArgumentParser):
441538
parser.add_argument(
442-
"file_or_code",
443-
help="file path to a Basilisp file or, if -c is provided, a string of Basilisp code",
539+
"file_or_ns_or_code",
540+
help=(
541+
"file path to a Basilisp file, a string of Basilisp code, or a fully "
542+
"qualified Basilisp namespace name"
543+
),
444544
)
445-
parser.add_argument(
545+
546+
grp = parser.add_mutually_exclusive_group()
547+
grp.add_argument(
446548
"-c",
447549
"--code",
448550
action="store_true",
449551
help="if provided, treat argument as a string of code",
450552
)
553+
grp.add_argument(
554+
"-n",
555+
"--load-namespace",
556+
action="store_true",
557+
help="if provided, treat argument as the name of a namespace",
558+
)
559+
451560
parser.add_argument(
452-
"--in-ns", default=runtime.REPL_DEFAULT_NS, help="namespace to use for the code"
561+
"--in-ns",
562+
help="namespace to use for the code (default: basilisp.user); ignored when `-n` is used",
453563
)
454564
parser.add_argument(
455565
"args",
@@ -516,6 +626,7 @@ def invoke_cli(args: Optional[Sequence[str]] = None) -> None:
516626
)
517627

518628
subparsers = parser.add_subparsers(help="sub-commands")
629+
_add_bootstrap_subcommand(subparsers)
519630
_add_nrepl_server_subcommand(subparsers)
520631
_add_repl_subcommand(subparsers)
521632
_add_run_subcommand(subparsers)

0 commit comments

Comments
 (0)