Generic types and 3.4 compatibility (#16)

* Add generic type support.

- Refactors the annotation support for test_attrs
- Refactors errors and types into their own modules.

* Mistake in changes to config.yml

* Fix broken import for typeddict.

* Make sure poetry >= 1.00

* Make SoftMod handle a SyntaxError

* Make sure poetry is the release version.

* Fix some syntax issues for python3.4

* Prepare for 2.2.0 release

* Add tests for generic types, improve supporting functions.

- Correct some issues with get_generic_origin.

* Ran black.

* Ensure get_origin works with non-parameterized generic types.

* Make tests skip if `slots=True` doesn't work with Generic base.

* get_origin falls back to the original type

* Add workaround for change in behavior of typ.__parameters__

* Improved testing advice.

- Remove black and dephell as dev dependencies.
- Add dephell config to easily generate docker
- Require a newer version of attrs.

* Tidying.

* Cleanup

Author: bsamuel-ui

.circleci/config.yml

@@ -1,67 +1,76 @@
 version: 2.1
 jobs:
-  test-35:
+  test-34:
     docker:
-      - image: circleci/python:3.5
+      - image: circleci/python:3.4
     environment:
       &std_env
       TERM: xterm
       LANG: en_US.UTF-8
+      PIP_DISABLE_PIP_VERSION_CHECK: 1
     working_directory: ~/json-syntax
     steps:
+      &steps34
       - checkout
       - run:
           name: Set up virtualenv
           command: |
-            pip install poetry
-            poetry install
+            pip install --user 'poetry>=1'
+            python -m poetry install
 
       - run:
           name: Run tests
           command: |
-            poetry run pytest tests/
+            python -m poetry run pytest tests/
 
       - store_artifacts:  # If a property test fails, this contains the example that failed.
           path: ".hypothesis"
           destination: ".hypothesis"
+  test-35:
+    docker:
+      - image: circleci/python:3.5
+    environment: *std_env
+    steps: *steps34
+    working_directory: ~/json-syntax
   test-36:
     docker:
       - image: circleci/python:3.6
     environment: *std_env
+    working_directory: ~/json-syntax
     steps:
-      &std_steps
+      &steps36
       - checkout
       - run:
           name: Set up virtualenv
           command: |
-            pip install poetry
-            poetry install
+            pip install --user 'poetry>=1'
+            python -m poetry install
 
       - run:
           name: Run tests
           command: |
-            poetry run pytest --doctest-modules json_syntax/ tests/
+            python -m poetry run pytest --doctest-modules json_syntax/ tests/
 
       - store_artifacts:  # If a property test fails, this contains the example that failed.
           path: ".hypothesis"
           destination: ".hypothesis"
-    working_directory: ~/json-syntax
   test-37:
     docker:
       - image: circleci/python:3.7
     environment: *std_env
-    steps: *std_steps
+    steps: *steps36
     working_directory: ~/json-syntax
   test-38:
     docker:
       - image: circleci/python:3.8
     environment: *std_env
-    steps: *std_steps
+    steps: *steps36
     working_directory: ~/json-syntax
 
 workflows:
   test:
     jobs:
+      - test-34
       - test-35
       - test-36
       - test-37

.gitignore

@@ -9,3 +9,4 @@
 setup.py
 requirements.txt
 .tox/
+README.rst

README.md

@@ -168,8 +168,8 @@ Thus we have:
  * `dict` and `Dict[K, V]`
 
 Tuple is a special case. In Python, they're often used to mean "frozenlist", so
-`Tuple[E, ...]` (the `...` is [the Ellipsis object][ellipsis]) indicates all elements have the type
-`E`.
+`Tuple[E, ...]` (the `...` is [the Ellipsis object][ellipsis]) indicates all elements have
+the type `E`.
 
 They're also used to represent an unnamed record. In this case, you can use
 `Tuple[A, B, C, D]` or however many types. It's generally better to use a `dataclass`.
@@ -180,6 +180,24 @@ The standard rules don't support:
  2. Using type variables.
  3. Any kind of callable, coroutine, file handle, etc.
 
+#### Support for deriving from Generic
+
+There is experimental support for deriving from `typing.Generic`. An `attrs` or `dataclass`
+may declare itself a generic class. If another class invokes it as `YourGeneric[Param,
+Param]`, those `Param` types will be substituted into the fields during encoding. This is
+useful to construct parameterized container types. Example:
+
+    @attr.s(auto_attribs=True)
+    class Wrapper(Generic[T, M]):
+        body: T
+        count: int
+        messages: List[M]
+
+    @attr.s(auto_attribs=True)
+    class Message:
+        first: Wrapper[str, str]
+        second: Wrapper[Dict[str, str], int]
+
 #### Unions
 
 A union type lets you present alternate types that the converters will attempt in
@@ -347,28 +365,19 @@ This package is maintained via the [poetry][] tool. Some useful commands:
 
  1. Setup: `poetry install`
  2. Run tests: `poetry run pytest tests/`
- 3. Reformat: `poetry run black json_syntax/ tests/`
-
-### Setting up tox
-
-You'll want pyenv, then install the pythons:
-
-    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
-    pyenv install --list | egrep '^ *3\.[4567]|^ *pypy3.5'
-    # figure out what versions you want
-    for v in 3.4.9 3.5.10 ...; do
-       pyenv install $v
-       PYENV_VERSION=$v python get-pip.py
-    done
+ 3. Reformat: `black json_syntax/ tests/`
+ 4. Generate setup.py: `dephell deps convert -e setup`
+ 5. Generate requirements.txt: `dephell deps convert -e req`
 
-Once you install `tox` in your preferred python, running it is just `tox`. (Note: this is
-largely redundant as the build is configured to all the different pythons on Circle.)
+### Running tests via docker
 
-### Contributor roll call
+The environments for 3.4 through 3.9 are in `pyproject.toml`, so just run:
 
-* @bsamuel-ui -- Ben Samuel
-* @dschep
-* @rugheid
+    dephell deps convert -e req  # Create requirements.txt
+    dephell docker run -e test34 pip install -r requirements.txt
+    dephell docker run -e test34 pytest tests/
+    dephell docker shell -e test34 pytest tests/
+    dephell docker destroy -e test34
 
 ### Notes
 

json_syntax/__init__.py

@@ -1,6 +1,6 @@
 """
-The JSON syntax library is a combinatorial parser / generator library for managing conversion of Python objects to and
-from common JSON types.
+The JSON syntax library is a combinatorial parser / generator library for managing
+conversion of Python objects to and from common JSON types.
 
 It's not strictly limited to JSON, but that's the major use case.
 """
@@ -39,9 +39,11 @@ def std_ruleset(
     cache=None,
 ):
     """
-    Constructs a RuleSet with the provided rules. The arguments here are to make it easy to override.
+    Constructs a RuleSet with the provided rules. The arguments here are to make it easy to
+    override.
 
-    For example, to replace ``decimals`` with ``decimals_as_str`` just call ``std_ruleset(decimals=decimals_as_str)``
+    For example, to replace ``decimals`` with ``decimals_as_str`` just call
+    ``std_ruleset(decimals=decimals_as_str)``
     """
     return custom(
         enums,
@@ -59,5 +61,5 @@ def std_ruleset(
         stringify_keys,
         unions,
         *extras,
-        cache=cache,
+        cache=cache
     )

json_syntax/attrs.py

@@ -10,6 +10,7 @@
 )
 from . import pattern as pat
 from .product import build_attribute_map, build_named_tuple_map, build_typed_dict_map
+from .types import is_generic, get_origin, get_argument_map
 
 from functools import partial
 
@@ -47,7 +48,13 @@ def attrs_classes(
     """
     if verb not in _SUPPORTED_VERBS:
         return
-    inner_map = build_attribute_map(verb, typ, ctx)
+    if is_generic(typ):
+        typ_args = get_argument_map(typ)
+        typ = get_origin(typ)
+    else:
+        typ_args = None
+
+    inner_map = build_attribute_map(verb, typ, ctx, typ_args)
     if inner_map is None:
         return
 
@@ -115,11 +122,11 @@ def named_tuples(verb, typ, ctx):
 
 def typed_dicts(verb, typ, ctx):
     """
-    Handle the TypedDict product type. This allows you to construct a dict with specific (string) keys, which
-    is often how people really use dicts.
+    Handle the TypedDict product type. This allows you to construct a dict with specific
+    (string) keys, which is often how people really use dicts.
 
-    Both the class form and the functional form, ``TypedDict('Name', {'field': type, 'field': type})`` are
-    supported.
+    Both the class form and the functional form,
+    ``TypedDict('Name', {'field': type, 'field': type})`` are supported.
     """
     if verb not in _SUPPORTED_VERBS:
         return

json_syntax/cache.py

@@ -103,7 +103,8 @@ def complete(self, verb, typ, action):
 
 class ThreadLocalCache(SimpleCache):
     """
-    Avoids threads conflicting while looking up rules by keeping the cache in thread local storage.
+    Avoids threads conflicting while looking up rules by keeping the cache in thread local
+    storage.
 
     You can also prevent this by looking up rules during module loading.
     """

json_syntax/errors.py

@@ -0,0 +1,94 @@
+class _Context:
+    """
+    Stash contextual information in an exception. As we don't know exactly when an exception
+    is displayed to a user, this class tries to keep it always up to date.
+
+    This class subclasses string (to be compatible) and tracks an insertion point.
+    """
+
+    __slots__ = ("original", "context", "lead")
+
+    def __init__(self, original, lead, context):
+        self.original = original
+        self.lead = lead
+        self.context = [context]
+
+    def __str__(self):
+        return "{}{}{}".format(
+            self.original, self.lead, "".join(map(str, reversed(self.context)))
+        )
+
+    def __repr__(self):
+        return repr(self.__str__())
+
+    @classmethod
+    def add(cls, exc, context):
+        args = exc.args
+        if args and isinstance(args[0], cls):
+            args[0].context.append(context)
+            return
+        args = list(exc.args)
+        if args:
+            args[0] = cls(args[0], "; at ", context)
+        else:
+            args.append(cls("", "At ", context))
+        exc.args = tuple(args)
+
+
+class ErrorContext:
+    """
+    Inject contextual information into an exception message. This won't work for some
+    exceptions like OSError that ignore changes to `args`; likely not an issue for this
+    library. There is a neglible performance hit if there is no exception.
+
+    >>> with ErrorContext('.foo'):
+    ...   with ErrorContext('[0]'):
+    ...     with ErrorContext('.qux'):
+    ...       1 / 0
+    Traceback (most recent call last):
+    ZeroDivisionError: division by zero; at .foo[0].qux
+
+    The `__exit__` method will catch the exception and look for a `_context` attribute
+    assigned to it. If none exists, it appends `; at ` and the context string to the first
+    string argument.
+
+    As the exception walks up the stack, outer ErrorContexts will be called. They will see
+    the `_context` attribute and insert their context immediately after `; at ` and before
+    the existing context.
+
+    Thus, in the example above:
+
+        ('division by zero',)  -- the original message
+        ('division by zero; at .qux',)  -- the innermost context
+        ('division by zero; at [0].qux',)
+        ('division by zero; at .foo[0].qux',) -- the outermost context
+
+    For simplicity, the method doesn't attempt to inject whitespace. To represent names,
+    consider surrounding them with angle brackets, e.g. `<Class>`
+    """
+
+    def __init__(self, *context):
+        self.context = context
+
+    def __enter__(self):
+        pass
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        if exc_value is not None:
+            _Context.add(exc_value, "".join(self.context))
+
+
+def err_ctx(context, func):
+    """
+    Execute a callable, decorating exceptions raised with error context.
+
+    ``err_ctx(context, func)`` has the same effect as:
+
+        with ErrorContext(context):
+            return func()
+    """
+    try:
+        return func()
+    except Exception as exc:
+        _Context.add(exc, context)
+        raise