Add custom data readers #924 (#979)

Fixes #924 

Hello,

I managed to get the custom data readers feature implemented. Let me
know if you have any question or would like to see any changes.

Core Changes:

- Added custom data readers, from files on the path as well as entry
points.
- Entry point groups can be customized or disabled with cli arguments
- Added `*default-data-reader-fn*`
- Changed the reader to allow any tag values like in Clojure, but
restrict tags from loaded data readers to only qualified symbols.
- Updated documentation with new features

Other Changes:

- Changed load functions to return the result of the last form evaluated
as in Clojure. I was touching this function anyway and noticed the
discrepancy.
- Added `basilisp.core/read-seq`; it allows other namespaces to stop
depending on `basilisp.lang.reader` directly.
- Added `basilisp.contrib.pytest.fixtures` namespace with a
`pytest.MonkeyPatch` test fixture to support testing this feature.
- Added cli arguments to some subcommands that I noticed were missing.

---------

Co-authored-by: Chris Rink <[email protected]>

Author: mitch-kyle

CHANGELOG.md

@@ -7,6 +7,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 ### Added
  * Added the `CollReduce` and `KVReduce` protocols in `basilisp.core.protocols` and implemented `reduce` in terms of those protocols (#927)
+ * Added support for custom data readers (#924)
+ * Added `*default-data-reader-fn*` (#924)
  * Added `basilisp.pprint/print-table` function (#983)
  * Added `basilisp.core/read-all` function (#986)
 

docs/reader.rst

@@ -465,6 +465,32 @@ Python literals use the matching syntax to the corresponding Python data type, w
 * ``#py {}`` produces a Python `dict <https://docs.python.org/3/library/stdtypes.html#dict>`_ type.
 * ``#py #{}`` produces a Python `set <https://docs.python.org/3/library/stdtypes.html#set>`_ type.
 
+.. _custom_data_readers:
+
+Custom Data Readers
+^^^^^^^^^^^^^^^^^^^
+
+`Like Clojure <https://clojure.org/reference/reader#tagged_literals>`_ , data readers can be changed by binding  :lpy:var:`*data-readers*`.
+
+When Basilisp starts it can load data readers from multiple sources.
+
+It will search in :external:py:attr:`sys.path` for files named ``data_readers.lpy`` or else ``data_readers.cljc``; each which must contain a mapping of qualified symbol tags to qualified symbols of function vars.
+
+.. code-block:: clojure
+    {my/tag my.namespace/tag-handler}
+
+It will also search for any :external:py:class:`importlib.metadata.EntryPoint` in the group ``basilisp_data_readers`` group.
+Entry points must refer to a map of data readers.
+This can be disabled by setting the ``BASILISP_USE_DATA_READERS_ENTRY_POINT`` environment variable to ``false``.
+
+.. _default_data_reader_fn:
+
+Default Data Reader Function
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+By default, an exception will be raised if the reader encounters a tag that it doesn't have a data reader for.
+This can be customised by binding :lpy:var:`*default-data-readers-fn*`.
+It should be a function which is a function that takes two arguments, the tag symbol, and the value form.
+
 .. _reader_special_chars:
 
 Special Characters
@@ -592,4 +618,4 @@ Platform Reader Features
 Basilisp includes a specialized reader feature based on the current platform (Linux, MacOS, Windows, etc.).
 There exist cases where it may be required to use different APIs based on which platform is currently in use, so having a reader conditional to detect the current platform can simplify the development process across multiple platforms.
 The reader conditional name is always a keyword containing the lowercase version of the platform name as reported by ``platform.system()``.
-For example, if ``platform.system()`` returns the Python string ``"Windows"``, the platform specific reader conditional would be ``:windows``.
+For example, if ``platform.system()`` returns the Python string ``"Windows"``, the platform specific reader conditional would be ``:windows``.

src/basilisp/cli.py

@@ -250,6 +250,29 @@ def _add_debug_arg_group(parser: argparse.ArgumentParser) -> None:
     )
 
 
+def _add_runtime_arg_group(parser: argparse.ArgumentParser) -> None:
+    group = parser.add_argument_group(
+        "runtime arguments",
+        description=(
+            "The runtime arguments below affect reader and execution time features."
+        ),
+    )
+    group.add_argument(
+        "--data-readers-entry-points",
+        action=_set_envvar_action(
+            "BASILISP_USE_DATA_READERS_ENTRY_POINT", parent=argparse._StoreAction
+        ),
+        nargs="?",
+        const=_to_bool(os.getenv("BASILISP_USE_DATA_READERS_ENTRY_POINT", "true")),
+        type=_to_bool,
+        help=(
+            "If true, Load data readers from importlib entry points in the "
+            '"basilisp_data_readers" group. (env: '
+            "BASILISP_USE_DATA_READERS_ENTRY_POINT; default: true)"
+        ),
+    )
+
+
 Handler = Callable[[argparse.ArgumentParser, argparse.Namespace], None]
 
 
@@ -384,6 +407,8 @@ def _add_nrepl_server_subcommand(parser: argparse.ArgumentParser) -> None:
         default=".nrepl-port",
         help='the file path where the server port number is output to, defaults to ".nrepl-port".',
     )
+    _add_runtime_arg_group(parser)
+    _add_debug_arg_group(parser)
 
 
 def repl(
@@ -478,6 +503,7 @@ def _add_repl_subcommand(parser: argparse.ArgumentParser) -> None:
         help="default namespace to use for the REPL",
     )
     _add_compiler_arg_group(parser)
+    _add_runtime_arg_group(parser)
     _add_debug_arg_group(parser)
 
 
@@ -588,6 +614,7 @@ def _add_run_subcommand(parser: argparse.ArgumentParser) -> None:
         help="command line args made accessible to the script as basilisp.core/*command-line-args*",
     )
     _add_compiler_arg_group(parser)
+    _add_runtime_arg_group(parser)
     _add_debug_arg_group(parser)
 
 
@@ -612,6 +639,8 @@ def test(
 )
 def _add_test_subcommand(parser: argparse.ArgumentParser) -> None:
     parser.add_argument("args", nargs=-1)
+    _add_runtime_arg_group(parser)
+    _add_debug_arg_group(parser)
 
 
 def version(_, __) -> None:

src/basilisp/core.lpy

@@ -4400,9 +4400,9 @@
   [fmt & args]
   (print (apply format fmt args)))
 
-;;;;;;;;;;;;;;;;;;;;
-;; REPL Utilities ;;
-;;;;;;;;;;;;;;;;;;;;
+;;;;;;;;;;;;;;;;;;;;;;;;;
+;; Reading and Loading ;;
+;;;;;;;;;;;;;;;;;;;;;;;;;
 
 (def
   ^{:doc "The default data readers used in reader macros. Overriding or
@@ -4428,6 +4428,14 @@
   *resolver*
   basilisp.lang.runtime/resolve-alias)
 
+(def ^{:doc "When no data reader is found for a tag and ``*default-data-reader-fn*``
+            is non-``nil``, it will be called with two arguments, the tag and the value.
+            If ``*default-data-reader-fn*`` is ``nil`` (the default), raise a
+            ``basilisp.lang.compiler.SyntaxError``."
+       :dynamic true}
+  *default-data-reader-fn*
+  nil)
+
 (defn- read-iterator
   [opts x]
   (let [read (:read opts basilisp.lang.reader/read)]
@@ -4457,8 +4465,9 @@
   "Read the next form from the ``stream``\\. If no stream is specified, uses the value
   currently bound to :lpy:var:`*in*`.
 
-  Callers may bind a map of readers to :lpy:var:`*data-readers*` to customize the data
-  readers used reading this string.
+  Callers may bind a map of readers to :lpy:var:`*data-readers*` or a default data
+  reader function to :lpy:var::`*default-data-reader-fn*` to customize the data
+  readers used reading this string
 
   The stream must satisfy the interface of :external:py:class:`io.TextIOBase`\\, but
   does not require any pushback capabilities. The default
@@ -4477,8 +4486,9 @@
    "Eagerly read all forms from the ``stream``\\. If no stream is specified, uses the
   value currently bound to :lpy:var:`*in*`.
 
-  Callers may bind a map of readers to :lpy:var:`*data-readers*` to customize
-  the data readers used reading this string
+  Callers may bind a map of readers to :lpy:var:`*data-readers*` or a default data
+  reader function to :lpy:var::`*default-data-reader-fn*` to customize the data
+  readers used reading this string
 
   The stream must satisfy the interface of :external:py:class:`io.TextIOBase`\\, but
   does not require any pushback capabilities. The default
@@ -7024,6 +7034,94 @@
     (.write writer content)
     nil))
 
+;;;;;;;;;;;;;;;;;;;;;;;;;
+;; Custom Data Readers ;;
+;;;;;;;;;;;;;;;;;;;;;;;;;
+
+(defmulti ^:private make-custom-data-readers
+  (fn [obj metadata] (type obj)))
+
+(defmethod make-custom-data-readers :default
+  [obj mta]
+  (throw (ex-info "Not a valid data-reader map" (assoc mta :object obj))))
+
+(defmethod make-custom-data-readers basilisp.lang.interfaces/IPersistentMap
+  [mappings mta]
+  (reduce (fn [m [k v]]
+            (let [v' (if (qualified-symbol? v)
+                       (intern (create-ns (symbol (namespace v)))
+                               (symbol (name v)))
+                       v)]
+              (cond
+                (not (qualified-symbol? k))
+                (throw
+                 (ex-info "Invalid tag in data-readers. Expected qualified symbol."
+                          (merge mta {:form k})))
+
+                (not (ifn? v'))
+                (throw (ex-info "Invalid reader function in data-readers"
+                                (merge mta {:form v})))
+
+                :else
+                (assoc m (with-meta k mta) v'))))
+          mappings
+          mappings))
+
+(defmethod make-custom-data-readers importlib.metadata/EntryPoint
+  [entry-point mta]
+  (make-custom-data-readers (.load entry-point)
+                            (assoc mta
+                                   :basilisp.entry-point/name (.-name entry-point)
+                                   :basilisp.entry-point/group (.-group entry-point))))
+
+(defmethod make-custom-data-readers pathlib/Path
+  [file mta]
+  (make-custom-data-readers
+   (with-open [rdr (basilisp.io/reader file)]
+     (read (if (.endswith (name file) "cljc")
+             {:eof nil :read-cond :allow}
+             {:eof nil})
+           rdr))
+   (assoc mta :file (str file))))
+
+(defn- data-readers-entry-points []
+  (when (#{"true" "t" "1" "yes" "y"} (.lower
+                                      (os/getenv
+                                       "BASILISP_USE_DATA_READERS_ENTRY_POINT"
+                                       "true")))
+    (#?@(:lpy39-  [get (.entry_points importlib/metadata)]
+         :lpy310+ [.entry_points importlib/metadata ** :group])
+     "basilisp_data_readers")))
+
+(defn- data-readers-files []
+  (->> sys/path
+       (mapcat file-seq)
+       (filter (comp #{"data_readers.lpy" "data_readers.cljc"} name))
+       (group-by #(.-parent %))
+       vals
+       ;; Only load one data readers file per directory and prefer
+       ;; `data_readers.lpy` to `data_readers.cljc`
+       (map #(first (sort-by name > %)))))
+
+(defn- load-data-readers []
+  (alter-var-root
+   #'*data-readers*
+   (fn [mappings additional-mappings]
+     (reduce (fn [m [k v]]
+               (if (not= (get m k v) v)
+                 (throw (ex-info "Conflicting data-reader mapping"
+                                 (merge (meta k) {:conflict k, :mappings m})))
+                 (assoc m k v)))
+             mappings
+             additional-mappings))
+   ;; Can't use `read` when altering `*data-readers*` so do reads ahead of time
+   (->> (concat (data-readers-files)
+                (data-readers-entry-points))
+        (mapcat #(make-custom-data-readers % nil))
+        doall)))
+
+(load-data-readers)
+
 ;;;;;;;;;;;;;;;;;
 ;; Transducers ;;
 ;;;;;;;;;;;;;;;;;