Support a wider array of integer literals (#832)

Fixes #769

Author: chrisrink10

.github/workflows/run-tests.yml

@@ -102,17 +102,17 @@ jobs:
         with:
           python-version: ${{ matrix.version }}
       - name: Cache dependencies
-        id: cache-deps
+        id: min-deps-test-cache-deps
         uses: actions/cache@v3
         with:
           path: |
             .tox
             ~/.cache/pip
             ~/.cache/pypoetry
             ~/.local/share/pypoetry
-          key: mdt-${{ runner.os }}-python-${{ matrix.version }}-poetry-${{ hashFiles('pyproject.toml', 'tox.ini') }}
+          key: min-deps-test-${{ runner.os }}-python-${{ matrix.version }}-poetry-${{ hashFiles('pyproject.toml', 'tox.ini') }}
       - name: Install Poetry
-        if: steps.cache-deps.outputs.cache-hit != 'true'
+        if: steps.min-deps-test-cache-deps.outputs.cache-hit != 'true'
         run: curl -sSL https://install.python-poetry.org | python3 -
       - name: Install Tox
         run: |

CHANGELOG.md

@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
  * Added support for executing Basilisp namespaces directly via `basilisp run` and by `python -m` (#791)
  * Added the `memoize` core fn (#812)
  * Added support for `thrown-with-msg?` assertions to `basilisp.test/is` (#831)
+ * Added support for reading scientific notation literals, octal and hex integer literals, and arbitrary base (2-36) integer literals (#769)
 
 ### Changed
  * Optimize calls to Python's `operator` module into their corresponding native operators (#754)

docs/reader.rst

@@ -21,6 +21,8 @@ Numeric Literals
 
 The Basilisp reader reads a wide range of numeric literals.
 
+.. _integer_numbers:
+
 Integers
 ^^^^^^^^
 
@@ -35,11 +37,22 @@ Integers
     basilisp.user=> (python/type 1N)
     <class 'int'>
 
-Integers are represented using numeric ``0-9`` and may be prefixed with any number of negative signs ``-``.
-The resulting integer will have the correct sign after resolving all of the supplied ``-`` signs.
+Integers are represented using numeric ``0-9`` and may be prefixed with a single negative sign ``-``.
 For interoperability support with Clojure, Basilisp integers may also be declared with the ``N`` suffix, like ``1N``.
 In Clojure, this syntax signals a ``BigInteger``, but Python's default ``int`` type supports arbitrary precision by default so there is no difference between ``1`` and ``1N`` in Basilisp.
 
+Integer literals may be specified in arbitrary bases between 2 and 36 by using the syntax ``[base]r[value]``.
+For example, in base 2 ``2r1001``, base 12 ``12r918a32``, and base 36 ``36r81jdk3kdp``.
+Arbitrary base literals do not distinguish between upper and lower case characters, so ``p`` and ``P`` are the same for bases which support ``P`` as a digit.
+Arbitrary base literals do not support the ``N`` suffix because ``N`` is a valid digit for some bases.
+
+For common bases such as octal and hex, there is a custom syntax.
+Octal literals can be specified with a ``0`` prefix; for example, the octal literal ``0777`` corresponds to the base 10 integer 511.
+Hex literals can be specified with a ``0x`` prefix; for example, the hex literal ``0xFACE`` corresponds to the base 10 integer 64206.
+Both octal and hex literals support the ``N`` suffix and it is treated the same as with base 10 integers.
+
+.. _floating_point_numbers:
+
 Floating Point
 ^^^^^^^^^^^^^^
 
@@ -55,11 +68,30 @@ Floating Point
    <class 'decimal.Decimal'>
 
 Floating point values are represented using ``0-9`` and a trailing decimal value, separated by a ``.`` character.
-Like integers, floating point values may be prefixed with an arbitrary number of negative signs ``-`` and the final read value will have the correct sign after resolving the negations.
+Like integers, floating point values may be prefixed with a single negative sign ``-``.
 By default floating point values are represented by Python's ``float`` type, which does **not** support arbitrary precision by default.
 Like in Clojure, floating point literals may be specified with a single ``M`` suffix to specify an arbitrary-precision floating point value.
 In Basilisp, a floating point number declared with a trailing ``M`` will return Python's `Decimal <https://docs.python.org/3/library/decimal.html>`_ type, which supports arbitrary floating point arithmetic.
 
+.. _scientific_notation:
+
+Scientific Notation
+^^^^^^^^^^^^^^^^^^^
+
+::
+
+   basilisp.user=> 2e6
+   2000000
+   basilisp.user=> 3.14e-1
+   0.31400000000000006
+
+Basilisp supports scientific notation using the ``e`` syntax common to many programming languages.
+The significand (the number to the left of the ``e`` ) may be an integer or floating point and may be prefixed with a single negative sign ``-``.
+The exponent (the number to the right of the ``e`` ) must be an integer and may be prefixed with a single negative sign ``-``.
+The resulting value will be either an integer or float depending on the type of the significand.
+
+.. _complex_numbers:
+
 Complex
 ^^^^^^^
 
@@ -76,7 +108,24 @@ Complex
 
 Basilisp includes support for complex literals to match the Python VM hosts it.
 Complex literals may be specified as integer or floating point values with a ``J`` suffix.
-Like integers and floats, complex values may be prefixed with an arbitrary number of negative signs ``-`` and the final read value will have the correct sign after resolving the negations.
+Like integers and floats, complex values may be prefixed with a single negative sign ``-``.
+
+.. _ratios:
+
+Ratios
+^^^^^^
+
+::
+
+   basilisp.user=> 22/7
+   22/7
+   basilisp.user=> -3/8
+   -3/8
+
+Basilisp includes support for ratios.
+Ratios are represented as the division of 2 integers which cannot be reduced to an integer.
+As with integers and floats, the numerator of a ratio may be prefixed with a single negative sign ``-`` -- a negative sign may not appear in the denominator.
+In Basilisp, ratios are backed by Python's `Fraction <https://docs.python.org/3/library/fractions.html>`_ type, which is highly interoperable with other Python numeric types.
 
 .. _strings:
 
@@ -101,6 +150,7 @@ String literals are always read with the UTF-8 encoding.
 String literals may contain the following escape sequences: ``\\``, ``\a``, ``\b``, ``\f``, ``\n``, ``\r``, ``\t``, ``\v``.
 Their meanings match the equivalent escape sequences supported in `Python string literals <https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals>`_\.
 
+.. _byte_strings
 
 Byte Strings
 ------------

src/basilisp/contrib/sphinx/autodoc.py

@@ -147,7 +147,8 @@ def filter_members(
         self, members: List[ObjectMember], want_all: bool
     ) -> List[Tuple[str, Any, bool]]:
         filtered = []
-        for name, val in members:
+        for member in members:
+            name, val = member.__name__, member.object
             assert isinstance(val, runtime.Var)
             if self.options.exclude_members and name in self.options.exclude_members:
                 continue
@@ -384,7 +385,8 @@ def filter_members(
         self, members: List[ObjectMember], want_all: bool
     ) -> List[Tuple[str, Any, bool]]:
         filtered = []
-        for name, val in members:
+        for member in members:
+            name, val = member.__name__, member.object
             assert isinstance(val, runtime.Var)
             if val.meta is not None:
                 if val.meta.val_at(_PRIVATE_KW):

src/basilisp/edn.lpy

@@ -88,7 +88,7 @@
   basilisp.lang.reader/ns-name-chars)
 
 (def ^:private num-chars
-  basilisp.lang.reader/num-chars)
+  #"[0-9]")
 
 (def ^:private unicode-char
   basilisp.lang.reader/unicode-char)

src/basilisp/lang/reader.py

@@ -67,7 +67,15 @@
 ns_name_chars = re.compile(r"\w|-|\+|\*|\?|/|\=|\\|!|&|%|>|<|\$|\.")
 alphanumeric_chars = re.compile(r"\w")
 begin_num_chars = re.compile(r"[0-9\-]")
-num_chars = re.compile("[0-9]")
+maybe_num_chars = re.compile(r"[0-9A-Za-z/\.]")
+integer_literal = re.compile(r"(-?(?:\d|[1-9]\d+))N?")
+float_literal = re.compile(r"(-?(?:\d|[1-9]\d+)(?:\.\d*)?)M?")
+complex_literal = re.compile(r"-?(\d+(?:\.\d*)?)J")
+arbitrary_base_literal = re.compile(r"-?(\d{1,2})r([0-9A-Za-z]+)")
+octal_literal = re.compile("-?0([0-7]+)N?")
+hex_literal = re.compile("-?0[Xx]([0-9A-Fa-f]+)N?")
+ratio_literal = re.compile(r"(-?\d+)/(\d+)")
+scientific_notation_literal = re.compile(r"-?(\d+(?:\.\d*)?)[Ee](-?\d+)")
 whitespace_chars = re.compile(r"[\s,]")
 newline_chars = re.compile("(\r\n|\r|\n)")
 fn_macro_args = re.compile("(%)(&|[0-9])?")
@@ -746,18 +754,13 @@ def _read_namespaced_map(ctx: ReaderContext) -> lmap.PersistentMap:
 MaybeNumber = Union[complex, decimal.Decimal, float, Fraction, int, MaybeSymbol]
 
 
-def _read_num(  # noqa: C901  # pylint: disable=too-many-statements
+def _read_num(  # noqa: C901  # pylint: disable=too-many-locals,too-many-statements
     ctx: ReaderContext,
 ) -> MaybeNumber:
     """Return a numeric (complex, Decimal, float, int, Fraction) from the input stream."""
     chars: List[str] = []
     reader = ctx.reader
 
-    is_complex = False
-    is_decimal = False
-    is_float = False
-    is_integer = False
-    is_ratio = False
     while True:
         token = reader.peek()
         if token == "-":
@@ -774,68 +777,62 @@ def _read_num(  # noqa: C901  # pylint: disable=too-many-statements
                 return _read_sym(ctx)
             chars.append(token)
             continue
-        elif token == ".":
-            if is_float:
-                raise ctx.syntax_error(
-                    "Found extra '.' in float; expected decimal portion"
-                )
-            is_float = True
-        elif token == "J":
-            if is_complex:
-                raise ctx.syntax_error("Found extra 'J' suffix in complex literal")
-            is_complex = True
-        elif token == "M":
-            if is_decimal:
-                raise ctx.syntax_error("Found extra 'M' suffix in decimal literal")
-            is_decimal = True
-        elif token == "N":
-            if is_integer:
-                raise ctx.syntax_error("Found extra 'N' suffix in integer literal")
-            is_integer = True
-        elif token == "/":
-            if is_ratio:
-                raise ctx.syntax_error("Found extra '/' in ratio literal")
-            is_ratio = True
-        elif not num_chars.match(token):
+        elif not maybe_num_chars.match(token):
             break
         reader.next_token()
         chars.append(token)
 
-    assert len(chars) > 0, "Must have at least one digit in integer or float"
+    assert len(chars) > 0, "Must have at least one digit in number"
 
     s = "".join(chars)
-    if (
-        sum(
-            [
-                is_complex and is_decimal,
-                is_complex and is_integer,
-                is_complex and is_ratio,
-                is_decimal or is_float,
-                is_integer,
-                is_ratio,
-            ]
-        )
-        > 1
-    ):
-        raise ctx.syntax_error(f"Invalid number format: {s}")
+    neg = s.startswith("-")
 
-    if is_complex:
-        imaginary = float(s[:-1]) if is_float else int(s[:-1])
-        return complex(0, imaginary)
-    elif is_decimal:
+    if (match := integer_literal.fullmatch(s)) is not None:
+        return int(match.group(1))
+    elif (match := float_literal.fullmatch(s)) is not None:
+        if s.endswith("M"):
+            try:
+                return decimal.Decimal(match.group(1))
+            except decimal.InvalidOperation:  # pragma: no cover
+                raise ctx.syntax_error(f"Invalid number format: {s}") from None
+        else:
+            return float(match.group(1))
+    elif (match := octal_literal.fullmatch(s)) is not None:
+        v = int(match.group(1), base=8)
+        return -v if neg else v
+    elif (match := hex_literal.fullmatch(s)) is not None:
+        v = int(match.group(1), base=16)
+        return -v if neg else v
+    elif (match := ratio_literal.fullmatch(s)) is not None:
+        num, denominator = match.groups()
+        if (numerator := int(num)) == 0:
+            return 0
         try:
-            return decimal.Decimal(s[:-1])
-        except decimal.InvalidOperation:
-            raise ctx.syntax_error(f"Invalid number format: {s}") from None
-    elif is_float:
-        return float(s)
-    elif is_ratio:
-        assert "/" in s, "Ratio must contain one '/' character"
-        num, denominator = s.split("/")
-        return Fraction(numerator=int(num), denominator=int(denominator))
-    elif is_integer:
-        return int(s[:-1])
-    return int(s)
+            return Fraction(numerator=numerator, denominator=int(denominator))
+        except ZeroDivisionError as e:
+            raise ctx.syntax_error(f"Invalid ratio format: {s}") from e
+    elif (match := scientific_notation_literal.fullmatch(s)) is not None:
+        sig = float(m) if "." in (m := match.group(1)) else int(m)
+        exp = int(match.group(2))
+        res = sig * (10**exp)
+        return -res if neg else res
+    elif (match := arbitrary_base_literal.fullmatch(s)) is not None:
+        base = int(match.group(1))
+        if not 2 <= base <= 36:
+            raise ctx.syntax_error(
+                f"Invalid base {base} for integer literal {s}: must be between 2 and 36"
+            )
+        try:
+            v = int(match.group(2), base=base)
+        except ValueError as e:
+            raise ctx.syntax_error(f"Invalid number format: {s}") from e
+        else:
+            return -v if neg else v
+    elif (match := complex_literal.fullmatch(s)) is not None:
+        imaginary_raw = match.group(1)
+        imaginary = float(imaginary_raw) if "." in imaginary_raw else int(imaginary_raw)
+        return complex(0, -imaginary if neg else imaginary)
+    raise ctx.syntax_error(f"Invalid number format: {s}")
 
 
 _STR_ESCAPE_CHARS = {