Merge pull request #66 from spencer-tb/tf-updates

Feature: Update `tf` to fill tests within sub-directories with depth > 1, change path defaults, add --no-output-structure option.

Author: spencer-tb

README.md

@@ -9,7 +9,8 @@ Ethereum tests. [Further documentation](https://execution-spec-tests.readthedocs
 
 The following are required to either generate or develop tests:
 
-1. Python `3.10.0`.
+1. Python >=`3.10.0`.
+   - For dists. with the `apt` package manager ensure you have python `-dev` & `-venv` packages installed.
 2. [`go-ethereum`](https://github.com/ethereum/go-ethereum) `geth`'s `evm` utility must be accessible in the `PATH`, typically at the latest version. To get it:
    1. Install [the Go programming language](https://go.dev/doc/install) on your computer.
    2. Clone [the Geth repository](https://github.com/ethereum/go-ethereum).
@@ -36,7 +37,7 @@ After the installation, run this sanity check to ensure tests are generated.
 If everything is OK, you will see the beginning of the JSON format filled test.
 
 ```console
-tf --output="fixtures" --test-case yul
+tf --test-case yul
 head fixtures/example/example/yul.json
 ```
 
@@ -46,24 +47,31 @@ head fixtures/example/example/yul.json
 To generate all the tests defined in the `./fillers` sub-directory, run the `tf` command:
 
 ```console
-tf --output="fixtures"
+tf --filler-path="fillers" --output="fixtures" 
 ```
 
+This is equivalent to running `tf` with no arguments. The paths`fillers/` and `fixtures/` are both defaults for the respective command.
+
 Note that the test `post` conditions are tested against the output of the `geth` `evm` utility during test generation.
 
-To generate all the tests in the `./fillers/vm` sub-directory (category), for example, run:
+To generate all the tests within the `./fillers/vm` directory (category), for example, run:
 ```console
-tf --output="fixtures" --test-categories vm
+tf --test-categories vm
+```
+
+This extends to sub-directories. To generate all specific tests within the `./fillers/vm/vm_arith/vm_add` sub-directory, run:
+```console
+tf --output="fixtures" --test-categories vm/vm_arith/vm_add
 ```
 
 To generate all the tests in the `./fillers/*/dup.py` modules, for example, run:
 ```console
-tf --output="fixtures" --test-module dup
+tf --test-module dup
 ```
 
 To generate specific tests, such as `./fillers/*/*.py::test_dup`, for example, run (remove the `test_` prefix from the test case's function name):
 ```console
-tf --output="fixtures" --test-case dup
+tf --test-case dup
 ```
 
 ### Testing the Execution Spec Tests Framework
@@ -156,6 +164,9 @@ A new test can be added by either:
   the new test function(s).
 - Creating an entirely new category by adding a subdirectory in
   `fillers` with the appropriate source files and test functions.
+    - Tests within multiple sub-directories must have a `__init__.py` file
+      within each directory above it (and it own), to ensure the test is found by the test filler `tf`.
+
 
 ### Test Spec Generator Functions
 

docs/getting_started/01_quick_start.md

@@ -5,6 +5,7 @@
 The following are required to either generate or develop tests:
 
 1. Python `3.10.0`.
+   - For dists. with the `apt` package manager ensure you have python `-dev` & `-venv` packages installed.
 2. [`go-ethereum`](https://github.com/ethereum/go-ethereum) `geth`'s `evm` utility must be accessible in the `PATH`, typically at the latest version. To get it:
      1. Install [the Go programming language](https://go.dev/doc/install) on your computer.
      2. Clone [the Geth repository](https://github.com/ethereum/go-ethereum).
@@ -31,7 +32,7 @@ After the installation, run this sanity check to ensure tests are generated.
 If everything is OK, you will see the beginning of the JSON format filled test.
 
 ```console
-tf --output="fixtures" --test-case yul
+tf --test-case yul
 head fixtures/example/example/yul.json
 ```
 
@@ -41,24 +42,32 @@ head fixtures/example/example/yul.json
 To generate all the tests defined in the `./fillers` sub-directory, run the `tf` command:
 
 ```console
-tf --output="fixtures"
+tf --filler-path="fillers" --output="fixtures" 
 ```
 
-Note that the test `post` conditions are tested against the output of the `geth` `evm` utility during test generation.
+This is equivalent to running `tf` with no arguments. The paths`fillers/` and `fixtures/` are both defaults for the respective command.
+
+!!! note
+    The test `post` conditions are tested against the output of the `geth` `evm` utility during test generation.
 
 To generate all the tests in the `./fillers/vm` sub-directory (category), for example, run:
 ```console
-tf --output="fixtures" --test-categories vm
+tf --test-categories vm
+```
+
+This extends to sub-directories. To generate all specific tests within the `./fillers/vm/vm_arith/vm_add` sub-directory, run:
+```console
+tf --output="fixtures" --test-categories vm/vm_arith/vm_add
 ```
 
 To generate all the tests in the `./fillers/*/dup.py` modules, for example, run:
 ```console
-tf --output="fixtures" --test-module dup
+tf --test-module dup
 ```
 
 To generate specific tests, such as `./fillers/*/*.py::test_dup`, for example, run (remove the `test_` prefix from the test case's function name):
 ```console
-tf --output="fixtures" --test-case dup
+tf --test-case dup
 ```
 
 ## Testing the Execution Spec Tests Framework

docs/getting_started/03_writing_tests.md

@@ -49,6 +49,8 @@ A new test can be added by either:
   the new test function(s).
 - Creating an entirely new category by adding a subdirectory in
   `fillers` with the appropriate source files and test functions.
+    - Tests within multiple sub-directories must have a `__init__.py` file
+      within each directory above it (and it own), to ensure the test is found by the test filler `tf`.
 
 ## Test Spec Generator Functions
 

src/ethereum_test_filling_tool/main.py

@@ -42,13 +42,17 @@ def parse_arguments() -> argparse.Namespace:
 
         parser.add_argument(
             "--filler-path",
-            help="path to filler directives",
+            help="path to filler directives, default: ./fillers",
+            default="fillers",
+            type=Path,
         )
 
         parser.add_argument(
             "--output",
-            help="directory to store filled test fixtures",
-            default="out",
+            help="directory to store filled test fixtures, \
+                  default: ./fixtures",
+            default="fixtures",
+            type=Path,
         )
 
         parser.add_argument(
@@ -75,6 +79,12 @@ def parse_arguments() -> argparse.Namespace:
             + "transition tool",
         )
 
+        parser.add_argument(
+            "--no-output-structure",
+            action="store_true",
+            help="removes the folder structure from test fixture output",
+        )
+
         return parser.parse_args()
 
     options: argparse.Namespace
@@ -88,10 +98,7 @@ def fill(self) -> None:
         """
         Fill test fixtures.
         """
-        pkg_path = "fillers"
-
-        if self.options.filler_path is not None:
-            pkg_path = self.options.filler_path
+        pkg_path = self.options.filler_path
 
         fillers = []
 
@@ -131,7 +138,9 @@ def fill(self) -> None:
             name = filler.__filler_metadata__["name"]
             output_dir = os.path.join(
                 self.options.output,
-                *(filler.__filler_metadata__["module_path"]),
+                *(filler.__filler_metadata__["module_path"])
+                if self.options.no_output_structure is None
+                else "",
             )
             os.makedirs(output_dir, exist_ok=True)
             path = os.path.join(output_dir, f"{name}.json")
@@ -157,17 +166,35 @@ def find_modules(root, include_pkg, include_modules):
         root,
         include=include_pkg if include_pkg is not None else ("*",),
     ):
-        for info in iter_modules([os.path.join(root, package)]):
-            if not info.ispkg:
-                module_full_name = package + "." + info.name
-                if module_full_name not in modules:
-                    if not include_modules or include_modules in info.name:
-                        yield (
-                            package,
-                            info.name,
-                            info.module_finder.find_module(module_full_name),
-                        )
-                    modules.add(module_full_name)
+        package = package.replace(
+            ".", "/"
+        )  # sub_package tests i.e 'vm.vm_tests'
+        for info, package_path in recursive_iter_modules(root, package):
+            module_full_name = package_path + "." + info.name
+            if module_full_name not in modules:
+                if not include_modules or include_modules in info.name:
+                    yield (
+                        package,
+                        info.name,
+                        info.module_finder.find_module(module_full_name),
+                    )
+                modules.add(module_full_name)
+
+
+def recursive_iter_modules(root, package):
+    """
+    Helper function for find_packages.
+    Iterates through all sub-packages (packages within a package).
+    Recursively navigates down the package tree until a new module is found.
+    """
+    for info in iter_modules([os.path.join(root, package)]):
+        if info.ispkg:
+            yield from recursive_iter_modules(
+                root, os.path.join(package, info.name)
+            )
+        else:
+            package_path = package.replace("/", ".")
+            yield info, package_path
 
 
 def main() -> None: