Fix TOML configuration errors (#3388)

Author: gaborbernat

.pre-commit-config.yaml

@@ -14,11 +14,6 @@ repos:
     hooks:
       - id: codespell
         additional_dependencies: ["tomli>=2.0.1"]
-  - repo: https://github.com/tox-dev/tox-ini-fmt
-    rev: "1.4.1"
-    hooks:
-      - id: tox-ini-fmt
-        args: ["-p", "fix"]
   - repo: https://github.com/tox-dev/pyproject-fmt
     rev: "2.2.4"
     hooks:

docs/changelog/3386.bugfix.rst

@@ -0,0 +1 @@
+Fix error when using ``requires`` within a TOML configuration file - by :user:`gaborbernat`.

docs/changelog/3387.bugfix.rst

@@ -0,0 +1 @@
+Fix error when using ``deps`` within a TOML configuration file - by :user:`gaborbernat`.

docs/changelog/3388.bugfix.rst

@@ -0,0 +1,7 @@
+Multiple fixes for the TOML configuration by :user:`gaborbernat`.:
+
+- Do not fail when there is an empty command within ``commands``.
+- Allow references for ``set_env`` by accepting list of dictionaries for it.
+- Do not try to be smart about reference unrolling, instead allow the user to control it via the ``extend`` flag,
+  available both for ``posargs`` and ``ref`` replacements.
+- The ``ref`` replacements ``raw`` key has been renamed to ``of``.

docs/config.rst

@@ -1296,6 +1296,39 @@ others to avoid repeating the same values:
 If the target table is one of the tox environments variable substitution will be applied on the replaced value,
 otherwise the text will be inserted as is (e.g., here with extra).
 
+Configuration reference
+~~~~~~~~~~~~~~~~~~~~~~~
+.. versionadded:: 4.21
+
+You can reference other configurations via the ``ref`` replacement. This can either be of type:
+
+
+- ``env``, in this case the configuration is loaded from another tox environment, where string substitution will happen
+  in that environments scope:
+
+    .. code-block:: toml
+
+        [env.src]
+        extras = ["A", "{env_name}"]
+        [env.dest]
+        extras = [{ replace = "ref", env = "src", key = "extras", extend = true }, "B"
+
+  In this case ``dest`` environments ``extras`` will be ``A``, ``src``, ``B``.
+
+- ``raw``, in this case the configuration is loaded as raw, and substitution executed in the current environments scope:
+
+    .. code-block:: toml
+
+        [env.src]
+        extras = ["A", "{env_name}"]
+        [env.dest]
+        extras = [{ replace = "ref", of = ["env", "extras"], extend = true }, "B"]
+
+  In this case ``dest`` environments ``extras`` will be ``A``, ``dest``, ``B``.
+
+The ``extend`` flag controls if after replacement the value should be replaced as is in the host structure (when flag is
+false -- by default) or be extended into. This flag only operates when the host is a list.
+
 Positional argument reference
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. versionadded:: 4.21
@@ -1305,9 +1338,12 @@ You can reference positional arguments via the ``posargs`` replacement:
 .. code-block:: toml
 
     [env.A]
-    commands = [["python", { replace = "posargs", default = ["a", "b"] } ]]
+    commands = [["python", { replace = "posargs", default = ["a", "b"], extend = true } ]]
 
 If the positional arguments are not set commands will become ``python a b``, otherwise will be ``python posarg-set``.
+The ``extend`` option instructs tox to unroll the positional arguments within the host structure. Without it the result
+would become ``["python", ["a", "b"]`` which would be invalid.
+
 Note that:
 
 .. code-block:: toml
@@ -1318,48 +1354,60 @@ Note that:
 Differs in sense that the positional arguments will be set as a single argument, while in the original example they are
 passed through as separate.
 
-Environment variable reference
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-.. versionadded:: 4.21
-
-You can reference environment variables via the ``env`` replacement:
+Empty commands groups will be ignored:
 
 .. code-block:: toml
 
     [env.A]
-    set_env.COVERAGE_FILE = { replace = "env", name = "COVERAGE_FILE", default = "ok" }
+    commands = [[], ["pytest]]
 
-If the environment variable is set the the ``COVERAGE_FILE`` will become that, otherwise will default to ``ok``.
+will only invoke pytest. This is especially useful together with posargs allowing you to opt out of running a set of
+commands:
 
-Other configuration reference
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+  .. code-block:: toml
+
+    [env.A]
+    commands = [
+        { replace = "posargs", default = ["python", "patch.py"]},
+        ["pytest"]
+    ]
+
+When running ``tox run -e A`` it will invoke ``python patch.py`` followed by pytest. When running ``tox run -e A --`` it
+will invoke only pytest.
+
+
+Environment variable reference
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. versionadded:: 4.21
 
 You can reference environment variables via the ``env`` replacement:
 
 .. code-block:: toml
 
-    [env_run_base]
-    extras = ["A", "{env_name}"]
-    [env.ab]
-    extras = [{ replace = "ref", raw = ["env_run_base", "extras"] }, "B"]
+    [env.A]
+    set_env.COVERAGE_FILE = { replace = "env", name = "COVERAGE_FILE", default = "ok" }
 
-In this case the ``extras`` for ``ab`` will be ``A``, ``B`` and ``ab``.
+If the environment variable is set the the ``COVERAGE_FILE`` will become that, otherwise will default to ``ok``.
 
-Reference replacement rules
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+References within set_env
+~~~~~~~~~~~~~~~~~~~~~~~~~
+.. versionadded:: 4.21.1
 
-When the replacement happens within a list and the returned value is also of type list the content will be extending the
-list rather than replacing it. For example:
+When you want to inherit ``set_env`` from another environment you can use the feature that if you pass a list of
+dictionaries to ``set_env`` they will be merged together, for example:
 
 .. code-block:: toml
 
-    [env_run_base]
-    extras = ["A"]
-    [env.ab]
-    extras = [{ replace = "ref", raw = ["env_run_base", "extras"] }, "B"]
+        [tool.tox.env_run_base]
+        set_env = { A = "1", B = "2"}
+
+        [tool.tox.env.magic]
+        set_env = [
+            { replace = "ref", of = ["tool", "tox", "env_run_base", "set_env"]},
+            { C = "3", D = "4"},
+        ]
 
-In this case the ``extras`` will be ``'A', 'B'`` rather than ``['A'], 'B'``.  Otherwise the replacement is in-place.
+Here the ``magic`` tox environment will have both ``A``, ``B``, ``C`` and ``D`` environments set.
 
 INI only
 --------

docs/user_guide.rst

@@ -33,18 +33,18 @@ these. The canonical file for this is either a ``tox.toml`` or ``tox.ini`` file.
                 "pytest>=8",
                 "pytest-sugar"
             ]
-            commands = [["pytest", { replace = "posargs", default = ["tests"] }]]
+            commands = [["pytest", { replace = "posargs", default = ["tests"], extend = true }]]
 
             [env.lint]
             description = "run linters"
             skip_install = true
             deps = ["black"]
-            commands = [["black", { replace = "posargs", default = ["."]} ]]
+            commands = [["black", { replace = "posargs", default = ["."], extend = true} ]]
 
             [env.type]
             description = "run type checks"
             deps = ["mypy"]
-            commands = [["mypy", { replace = "posargs", default = ["src", "tests"]} ]]
+            commands = [["mypy", { replace = "posargs", default = ["src", "tests"], extend = true} ]]
 
 
     .. tab:: INI
@@ -133,18 +133,18 @@ When ``<env_name>`` is the name of a specific environment, test environment conf
             "pytest>=8",
             "pytest-sugar"
         ]
-        commands = [["pytest", { replace = "posargs", default = ["tests"] }]]
+        commands = [["pytest", { replace = "posargs", default = ["tests"], extend = true }]]
 
         [env.lint]
         description = "run linters"
         skip_install = true
         deps = ["black"]
-        commands = [["black", { replace = "posargs", default = ["."]} ]]
+        commands = [["black", { replace = "posargs", default = ["."], extend = true} ]]
 
         [env.type]
         description = "run type checks"
         deps = ["mypy"]
-        commands = [["mypy", { replace = "posargs", default = ["src", "tests"]} ]]
+        commands = [["mypy", { replace = "posargs", default = ["src", "tests"], extend = true} ]]
 
 .. tab:: INI
 
@@ -217,7 +217,7 @@ Basic example
             "pytest>=7",
             "pytest-sugar",
         ]
-        commands = [[ "pytest", "tests", { replace = "posargs"} ]]
+        commands = [[ "pytest", "tests", { replace = "posargs", extend = true} ]]
 
 .. tab:: INI
 

src/tox/config/loader/convert.py

@@ -58,6 +58,8 @@ def _to_typing(self, raw: T, of_type: type[V], factory: Factory[V]) -> V:  # noq
         if origin in {list, List}:
             entry_type = of_type.__args__[0]  # type: ignore[attr-defined]
             result = [self.to(i, entry_type, factory) for i in self.to_list(raw, entry_type)]
+            if isclass(entry_type) and issubclass(entry_type, Command):
+                result = [i for i in result if i is not None]
         elif origin in {set, Set}:
             entry_type = of_type.__args__[0]  # type: ignore[attr-defined]
             result = {self.to(i, entry_type, factory) for i in self.to_set(raw, entry_type)}
@@ -160,7 +162,7 @@ def to_path(value: T) -> Path:
 
     @staticmethod
     @abstractmethod
-    def to_command(value: T) -> Command:
+    def to_command(value: T) -> Command | None:
         """
         Convert to a command to execute.
 

src/tox/config/loader/memory.py

@@ -51,7 +51,7 @@ def to_path(value: Any) -> Path:
         return Path(value)
 
     @staticmethod
-    def to_command(value: Any) -> Command:
+    def to_command(value: Any) -> Command | None:
         if isinstance(value, Command):
             return value
         if isinstance(value, str):

src/tox/config/loader/str_convert.py

@@ -72,7 +72,7 @@ def _win32_process_path_backslash(value: str, escape: str, special_chars: str) -
         return "".join(result)
 
     @staticmethod
-    def to_command(value: str) -> Command:
+    def to_command(value: str) -> Command | None:
         """
         At this point, ``value`` has already been substituted out, and all punctuation / escapes are final.
 

src/tox/config/loader/toml/__init__.py

@@ -96,8 +96,10 @@ def to_path(value: TomlTypes) -> Path:
         return Path(TomlLoader.to_str(value))
 
     @staticmethod
-    def to_command(value: TomlTypes) -> Command:
-        return Command(args=cast(List[str], value))  # validated during load in _ensure_type_correct
+    def to_command(value: TomlTypes) -> Command | None:
+        if value:
+            return Command(args=cast(List[str], value))  # validated during load in _ensure_type_correct
+        return None
 
     @staticmethod
     def to_env_list(value: TomlTypes) -> EnvList: