Apply :tag metadata to the Python AST (#680)

Fixes #354

Author: chrisrink10

CHANGELOG.md

@@ -5,6 +5,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+### Added
+ * Added support for passing through `:tag` metadata to the generated Python AST (#354)
 
 ### Changed
  * Optimize calls to Python's `operator` module into their corresponding native operators (#754)

docs/differencesfromclojure.rst

@@ -166,8 +166,9 @@ Host interoperability features generally match those of Clojure.
 Type Hinting
 ^^^^^^^^^^^^
 
-Type hints may be applied anywhere they are supported in Clojure, but the compiler does not currently use them.
-Support for attaching type hints to the Python AST is tracked in `#354 <https://github.com/basilisp-lang/basilisp/issues/354>`_\.
+Type hints may be applied anywhere they are supported in Clojure (as the ``:tag`` metadata key), but the compiler does not currently use them for any purpose.
+Tags provided for ``def`` names, function arguments and return values, and :lpy:form:`let` locals will be applied to the resulting Python AST by the compiler wherever possible.
+Particularly in the case of function arguments and return values, these tags maybe introspected from the Python `inspect <https://docs.python.org/3/library/inspect.html>`_ module.
 There is no need for type hints anywhere in Basilisp right now, however.
 
 .. _compilation_differences:

docs/gettingstarted.rst

@@ -131,5 +131,5 @@ For systems where the shebang line allows arguments, you can use ``#!/usr/bin/en
 
 .. code-block:: clojure
 
-   #!/usr/bin/env basilisp
+   #!/usr/bin/env basilisp-run
    (println "Hello world!")

docs/pyinterop.rst

@@ -211,3 +211,37 @@ As you can see in the example above, this strategy fits neatly with the existing
 The ``:collect`` strategy is a better accompaniment to functions with positional arguments.
 With this strategy, Python keyword arguments are converted into a Basilisp map with de-munged keyword arguments and passed as the final positional argument of the function.
 You can use map destructuring on this final positional argument, just as you would with the map in the ``:apply`` case above.
+
+Type Hinting
+------------
+
+Basilisp supports passing type hints through to the underlying generated Python using type hints by applying the ``:tag`` metadata to certain syntax elements.
+
+In Clojure, these tags are type declarations for certain primitive types.
+In Clojurescript, tags are type *hints* and they are only necessary in extremely limited circumstances to help the compiler.
+In Basilisp, tags are not used by the compiler at all.
+Instead, tags applied to function arguments and return values in Basilisp are applied to the underlying Python objects and are introspectable at runtime using the Python `inspect <https://docs.python.org/3/library/inspect.html>`_ standard library module.
+
+Type hints may be applied to :lpy:form:`def` names, function arguments and return values, and :lpy:form:`let` local forms.
+
+.. code-block:: clojure
+
+   (def ^python/str s "a string")
+
+   (defn upper
+     ^python/str [^python/str s]
+     (.upper s))
+
+   (let [^python/int i 64]
+     (* i 2))
+
+.. note::
+
+   The reader applies ``:tag`` :ref:`metadata` automatically for symbols following the ``^`` symbol, but users may manually apply ``:tag`` metadata containing any valid expression.
+   Python permits any valid expression in a variable annotation, so Basilisp likewise allows any valid expression.
+
+.. warning::
+
+   Due to the complexity of supporting multi-arity functions in Python, only return annotations are preserved on the arity dispatch function.
+   Return annotations are combined as by ``typing.Union``, so ``typing.Union[str, str] == str``.
+   The annotations for individual arity arguments are preserved in their compiled form, but they are challenging to access programmatically.

src/basilisp/contrib/pytest/testrunner.py

@@ -44,7 +44,7 @@ def __init__(self, message: str, data: lmap.PersistentMap) -> None:
 
     def __repr__(self):
         return (
-            "basilisp.contrib.pytest..testrunner.TestFailuresInfo"
+            "basilisp.contrib.pytest.testrunner.TestFailuresInfo"
             f"({self._msg}, {lrepr(self._data)})"
         )
 

src/basilisp/core.lpy

@@ -5565,8 +5565,9 @@
       {:first f
        :rest  r})"
   [body]
-  (let [args (first body)
-        body (rest body)
+  (let [args         (first body)
+        arg-vec-meta (meta args)
+        body         (rest body)
 
         arg-groups (split-with (partial not= '&) args)
         args       (first arg-groups)
@@ -5582,9 +5583,11 @@
                            (mapcat destructure-binding)))
 
         defs     (map destructure-def args)
-        arg-vec  (vec (concat
-                       (map :name defs)
-                       (map :name rest-defs)))
+        arg-vec  (with-meta
+                   (vec (concat
+                         (map :name defs)
+                         (map :name rest-defs)))
+                   arg-vec-meta)
         bindings (->> defs
                       (filter #(not= :symbol (:type %)))
                       (mapcat destructure-binding)

src/basilisp/lang/compiler/analyzer.py

@@ -68,6 +68,7 @@
     SYM_PRIVATE_META_KEY,
     SYM_PROPERTY_META_KEY,
     SYM_STATICMETHOD_META_KEY,
+    SYM_TAG_META_KEY,
     VAR_IS_PROTOCOL_META_KEY,
     SpecialForm,
 )
@@ -623,8 +624,8 @@ def has_meta_prop(o: Union[IMeta, Var]) -> bool:
 
 
 def _meta_getter(meta_kw: kw.Keyword) -> MetaGetter:
-    """Return a function which checks an object with metadata for a boolean
-    value by meta_kw."""
+    """Return a function which checks an object with metadata for a value by
+    meta_kw."""
 
     def get_meta_prop(o: Union[IMeta, Var]) -> Any:
         return Maybe(o.meta).map(lambda m: m.val_at(meta_kw, None)).value
@@ -641,6 +642,7 @@ def get_meta_prop(o: Union[IMeta, Var]) -> Any:
 _is_macro = _bool_meta_getter(SYM_MACRO_META_KEY)
 _is_no_inline = _bool_meta_getter(SYM_NO_INLINE_META_KEY)
 _inline_meta = _meta_getter(SYM_INLINE_META_KW)
+_tag_meta = _meta_getter(SYM_TAG_META_KEY)
 
 
 def _loc(form: Union[LispForm, ISeq]) -> Optional[Tuple[int, int, int, int]]:
@@ -785,6 +787,12 @@ def _call_args_ast(
         return args, kwargs
 
 
+def _tag_ast(form: Optional[LispForm], ctx: AnalyzerContext) -> Optional[Node]:
+    if form is None:
+        return None
+    return _analyze_form(form, ctx)
+
+
 def _with_meta(gen_node):
     """Wraps the node generated by gen_node in a :with-meta AST node if the
     original form has meta.
@@ -884,6 +892,8 @@ def _def_ast(  # pylint: disable=too-many-locals,too-many-statements
             f"def names must be symbols, not {type(name)}", form=name
         )
 
+    tag_ast = _tag_ast(_tag_meta(name), ctx)
+
     init_idx: Optional[int]
     children: vec.PersistentVector[kw.Keyword]
     if nelems == 2:
@@ -970,17 +980,26 @@ def _def_ast(  # pylint: disable=too-many-locals,too-many-statements
         with ctx.expr_pos():
             init = _analyze_form(runtime.nth(form, init_idx), ctx)
 
-        # Attach the automatically generated inline function (if one exists) to the Var
-        # and def metadata. We do not need to do this for user-provided inline
-        # functions (for which `init.inline_fn` will be None) since those should
-        # already be attached to the meta.
-        if isinstance(init, Fn) and init.inline_fn is not None:
-            assert isinstance(var.meta.val_at(SYM_INLINE_META_KW), bool), (  # type: ignore[union-attr]
-                "Cannot have a user-generated inline function and an automatically "
-                "generated inline function"
-            )
-            var.meta.assoc(SYM_INLINE_META_KW, init.inline_fn)  # type: ignore[union-attr]
-            def_meta = def_meta.assoc(SYM_INLINE_META_KW, init.inline_fn.form)  # type: ignore[union-attr]
+        if isinstance(init, Fn):
+            # Attach the automatically generated inline function (if one exists) to the
+            # Var and def metadata. We do not need to do this for user-provided inline
+            # functions (for which `init.inline_fn` will be None) since those should
+            # already be attached to the meta.
+            if init.inline_fn is not None:
+                assert isinstance(var.meta.val_at(SYM_INLINE_META_KW), bool), (  # type: ignore[union-attr]
+                    "Cannot have a user-generated inline function and an automatically "
+                    "generated inline function"
+                )
+                var.meta.assoc(SYM_INLINE_META_KW, init.inline_fn)  # type: ignore[union-attr]
+                def_meta = def_meta.assoc(SYM_INLINE_META_KW, init.inline_fn.form)  # type: ignore[union-attr]
+
+            if tag_ast is not None and any(
+                arity.tag is not None for arity in init.arities
+            ):
+                raise AnalyzerException(
+                    "def'ed Var :tag conflicts with defined function :tag",
+                    form=form,
+                )
     else:
         init = None
 
@@ -992,6 +1011,7 @@ def _def_ast(  # pylint: disable=too-many-locals,too-many-statements
         doc=doc,
         children=children,
         env=def_node_env,
+        tag=tag_ast,
     )
 
     # We still have to compile the meta here down to Python source code, so
@@ -1800,6 +1820,7 @@ def _deftype_ast(  # pylint: disable=too-many-locals
                 local=LocalType.FIELD,
                 is_assignable=is_mutable,
                 env=ctx.get_node_env(),
+                tag=_tag_ast(_tag_meta(field), ctx),
                 init=analyze_form(ctx, field_default)
                 if field_default is not __DEFTYPE_DEFAULT_SENTINEL
                 else None,
@@ -1852,6 +1873,7 @@ def __fn_method_ast(  # pylint: disable=too-many-locals
             raise AnalyzerException(
                 "function arity arguments must be a vector", form=params
             )
+        return_tag = _tag_ast(_tag_meta(params), ctx)
 
         has_vargs, vargs_idx = False, 0
         param_nodes = []
@@ -1870,6 +1892,7 @@ def __fn_method_ast(  # pylint: disable=too-many-locals
                 form=s,
                 name=s.name,
                 local=LocalType.ARG,
+                tag=_tag_ast(_tag_meta(s), ctx),
                 arg_id=i,
                 is_variadic=False,
                 env=ctx.get_node_env(),
@@ -1880,7 +1903,6 @@ def __fn_method_ast(  # pylint: disable=too-many-locals
         if has_vargs:
             try:
                 vargs_sym = params[vargs_idx + 1]
-
                 if not isinstance(vargs_sym, sym.Symbol):
                     raise AnalyzerException(
                         "function rest parameter name must be a symbol", form=vargs_sym
@@ -1890,6 +1912,7 @@ def __fn_method_ast(  # pylint: disable=too-many-locals
                     form=vargs_sym,
                     name=vargs_sym.name,
                     local=LocalType.ARG,
+                    tag=_tag_ast(_tag_meta(vargs_sym), ctx),
                     arg_id=vargs_idx + 1,
                     is_variadic=True,
                     env=ctx.get_node_env(),
@@ -1911,6 +1934,7 @@ def __fn_method_ast(  # pylint: disable=too-many-locals
                 form=form,
                 loop_id=fn_loop_id,
                 params=vec.vector(param_nodes),
+                tag=return_tag,
                 is_variadic=has_vargs,
                 fixed_arity=len(param_nodes) - int(has_vargs),
                 body=Do(
@@ -2472,6 +2496,7 @@ def _let_ast(form: ISeq, ctx: AnalyzerContext) -> Let:
                 form=name,
                 name=name.name,
                 local=LocalType.LET,
+                tag=_tag_ast(_tag_meta(name), ctx),
                 init=_analyze_form(value, ctx),
                 children=vec.v(INIT),
                 env=ctx.get_node_env(),

src/basilisp/lang/compiler/constants.py

@@ -49,6 +49,7 @@ class SpecialForm:
 SYM_NO_WARN_WHEN_UNUSED_META_KEY = kw.keyword("no-warn-when-unused")
 SYM_REDEF_META_KEY = kw.keyword("redef")
 SYM_STATICMETHOD_META_KEY = kw.keyword("staticmethod")
+SYM_TAG_META_KEY = kw.keyword("tag")
 
 ARGLISTS_KW = kw.keyword("arglists")
 COL_KW = kw.keyword("col")