Fix for #54 - allow regexes when matching expected message text (#55)

* Some refactoring before making the change - use an object for expected output rather than a string.

* Simple regex case now working

* A spot of clean-up.

* Document regex option.

* Specify default value for regex flag.

* Add test for regexes inb out section.

* Add test to ensure that regexes ony run if the flag is switched on.

* Allow regexes on specific messages.

* Fix regex - single line flag froup needed to be optional.

* Add simple failing simple cases.

* Add case for mismatching regex.

Author: brunns

.github/workflows/test.yml

@@ -26,7 +26,9 @@ jobs:
         pip install -U pip setuptools wheel
         pip install -r dev-requirements.txt
     - name: Run tests
-      run: pytest
+      run: pytest --ignore-glob="*.shouldfail.yml"
+    - name: Run test with expected failures
+      run: pytest pytest_mypy_plugins/tests/*.shouldfail.yml 2>&1 | grep "5 failed"
 
   lint:
     runs-on: ubuntu-latest

.gitignore

@@ -4,3 +4,5 @@
 __pycache__
 dist/
 build/
+.pytest_cache/
+venv/

README.md

@@ -52,16 +52,17 @@ You can also specify `PYTHONPATH`, `MYPYPATH`, or any other environment variable
 In general each test case is just an element in an array written in a properly formatted `YAML` file.
 On top of that, each case must comply to following types:
 
-| Property        | Type                                                   | Description                                                                                                     |
-| --------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |
-| `case`          | `str`                                                  | Name of the test case, complies to `[a-zA-Z0-9]` pattern                                                        |
-| `main`          | `str`                                                  | Portion of the code as if written in `.py` file                                                                 |
-| `files`         | `Optional[List[File]]=[]`\*                            | List of extra files to simulate imports if needed                                                               |
-| `disable_cache` | `Optional[bool]=False`                                 | Set to `true` disables `mypy` caching                                                                           |
-| `mypy_config`   | `Optional[Dict[str, Union[str, int, bool, float]]]={}` | Inline `mypy` configuration, passed directly to `mypy` as `--config-file` option                                |
-| `env`           | `Optional[Dict[str, str]]={}`                          | Environmental variables to be provided inside of test run                                                       |
-| `parametrized`  | `Optional[List[Parameter]]=[]`\*                       | List of parameters, similar to [`@pytest.mark.parametrize`](https://docs.pytest.org/en/stable/parametrize.html) |
-| `skip`          | `str`                                                  | Expression evaluated with following globals set: `sys`, `os`, `pytest` and `platform`                           |
+| Property        | Type                                                   | Description                                                                                                         |
+| --------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
+| `case`          | `str`                                                  | Name of the test case, complies to `[a-zA-Z0-9]` pattern                                                            |
+| `main`          | `str`                                                  | Portion of the code as if written in `.py` file                                                                     |
+| `files`         | `Optional[List[File]]=[]`\*                            | List of extra files to simulate imports if needed                                                                   |
+| `disable_cache` | `Optional[bool]=False`                                 | Set to `true` disables `mypy` caching                                                                               |
+| `mypy_config`   | `Optional[Dict[str, Union[str, int, bool, float]]]={}` | Inline `mypy` configuration, passed directly to `mypy` as `--config-file` option                                    |
+| `env`           | `Optional[Dict[str, str]]={}`                          | Environmental variables to be provided inside of test run                                                           |
+| `parametrized`  | `Optional[List[Parameter]]=[]`\*                       | List of parameters, similar to [`@pytest.mark.parametrize`](https://docs.pytest.org/en/stable/parametrize.html)     |
+| `skip`          | `str`                                                  | Expression evaluated with following globals set: `sys`, `os`, `pytest` and `platform`                               |
+| `regex`         | `str`                                                  | Allow regular expressions in comments to be matched against actual output. Defaults to "no", i.e. matches full text.|
 
 (*) Appendix to **pseudo** types used above:
 
@@ -126,6 +127,27 @@ Implementation notes:
     main:1: note: Revealed type is 'builtins.str'
 ```
 
+#### 4. Regular expressions in expectations
+
+```yaml
+- case: expected_message_regex_with_out
+  regex: yes
+  main: |
+    a = 'abc'
+    reveal_type(a)
+  out: |
+    main:2: note: .*str.*
+```
+
+#### 5. Regular expressions specific lines of output.
+
+```yaml
+- case: expected_single_message_regex
+  main: |
+    a = 'hello'
+    reveal_type(a)  # NR: .*str.*
+```
+
 ## Options
 
 ```

pytest_mypy_plugins/collect.py

@@ -101,16 +101,21 @@ def collect(self) -> Iterator["YamlTestItem"]:
                 test_name = f"{test_name_prefix}{test_name_suffix}"
                 main_file = File(path="main.py", content=pystache.render(raw_test["main"], params))
                 test_files = [main_file] + parse_test_files(raw_test.get("files", []))
+                regex = raw_test.get("regex", False)
 
-                output_from_comments = []
+                expected_output = []
                 for test_file in test_files:
-                    output_lines = utils.extract_errors_from_comments(test_file.path, test_file.content.split("\n"))
-                    output_from_comments.extend(output_lines)
+                    output_lines = utils.extract_output_matchers_from_comments(
+                        test_file.path, test_file.content.split("\n"), regex=regex
+                    )
+                    expected_output.extend(output_lines)
 
                 starting_lineno = raw_test["__line__"]
                 extra_environment_variables = parse_environment_variables(raw_test.get("env", []))
                 disable_cache = raw_test.get("disable_cache", False)
-                expected_output_lines = pystache.render(raw_test.get("out", ""), params).split("\n")
+                expected_output.extend(
+                    utils.extract_output_matchers_from_out(raw_test.get("out", ""), params, regex=regex)
+                )
                 additional_mypy_config = raw_test.get("mypy_config", "")
 
                 skip = self._eval_skip(str(raw_test.get("skip", "False")))
@@ -122,7 +127,7 @@ def collect(self) -> Iterator["YamlTestItem"]:
                         starting_lineno=starting_lineno,
                         environment_variables=extra_environment_variables,
                         disable_cache=disable_cache,
-                        expected_output_lines=output_from_comments + expected_output_lines,
+                        expected_output=expected_output,
                         parsed_test_data=raw_test,
                         mypy_config=additional_mypy_config,
                     )

pytest_mypy_plugins/item.py

@@ -32,8 +32,9 @@
 from pytest_mypy_plugins import utils
 from pytest_mypy_plugins.collect import File, YamlTestFile
 from pytest_mypy_plugins.utils import (
+    OutputMatcher,
     TypecheckAssertionError,
-    assert_string_arrays_equal,
+    assert_expected_matched_actual,
     capture_std_streams,
     fname_to_module,
 )
@@ -124,7 +125,7 @@ def __init__(
         *,
         files: List[File],
         starting_lineno: int,
-        expected_output_lines: List[str],
+        expected_output: List[OutputMatcher],
         environment_variables: Dict[str, Any],
         disable_cache: bool,
         mypy_config: str,
@@ -134,7 +135,7 @@ def __init__(
         self.files = files
         self.environment_variables = environment_variables
         self.disable_cache = disable_cache
-        self.expected_output_lines = expected_output_lines
+        self.expected_output = expected_output
         self.starting_lineno = starting_lineno
         self.additional_mypy_config = mypy_config
         self.parsed_test_data = parsed_test_data
@@ -279,7 +280,7 @@ def runtest(self) -> None:
                 for line in mypy_output.splitlines():
                     output_line = replace_fpath_with_module_name(line, rootdir=execution_path)
                     output_lines.append(output_line)
-                assert_string_arrays_equal(expected=self.expected_output_lines, actual=output_lines)
+                assert_expected_matched_actual(expected=self.expected_output, actual=output_lines)
         finally:
             temp_dir.cleanup()
             # remove created modules and all their dependants from cache

pytest_mypy_plugins/tests/test-regex_assertions.shouldfail.yml

@@ -0,0 +1,16 @@
+- case: rexex_but_not_turned_on
+  main: |
+    a = 'hello'
+    reveal_type(a)  # N: .*str.*
+
+- case: rexex_but_turned_off
+  regex: no
+  main: |
+    a = 'hello'
+    reveal_type(a)  # N: .*str.*
+
+- case: regext_does_not_match
+  regex: no
+  main: |
+    a = 'hello'
+    reveal_type(a)  # NR: .*banana.*

pytest_mypy_plugins/tests/test-regex_assertions.yml

@@ -0,0 +1,22 @@
+- case: expected_message_regex
+  regex: yes
+  main: |
+    a = 1
+    b = 'hello'
+
+    reveal_type(a)  # N: Revealed type is "builtins.int"
+    reveal_type(b)  # N: .*str.*
+
+- case: expected_message_regex_with_out
+  regex: yes
+  main: |
+    a = 'abc'
+    reveal_type(a)
+  out: |
+    main:2: note: .*str.*
+
+- case: expected_single_message_regex
+  regex: no
+  main: |
+    a = 'hello'
+    reveal_type(a)  # NR: .*str.*

pytest_mypy_plugins/tests/test-simple-cases.shouldfail.yml

@@ -0,0 +1,12 @@
+- case: fail_if_message_does_not_match
+  main: |
+    a = 'hello'
+    reveal_type(a)  # N: Some other message
+
+- case: fail_if_message_from_outdoes_not_match
+  regex: yes
+  main: |
+    a = 'abc'
+    reveal_type(a)
+  out: |
+    main:2: note: Some other message