Avoid UnicodeDecodeError from command output (#2970)

* test_sync_write_decode_surrogate: utf-8 decode

When SyncWrite decodes bytes as utf-8, it should replace unknown
sequences with the unicode surrogate codepoint instead of crashing the
program.

Test case for #2969

* SyncWrite: decode with errors='surrogateescape'

Avoid bubbling UnicodeDecodeError up from stream handling internals.

Tox has no way of knowing that the bytestream emitted by a command will
be valid utf-8, even if utf-8 is ostensibly the "correct" encoding for
the stream. It's always possible for an arbitrary command to return
non-utf-8 bytes, and this situation should not break tox.

Fix #2969

Author: masenf

docs/changelog/2969.bugfix.rst

@@ -0,0 +1,3 @@
+Instead of raising ``UnicodeDecodeError`` when command output includes non-utf-8 bytes,
+``tox`` will now use ``surrogateescape`` error handling to convert the unrecognized bytes
+to escape sequences according to :pep:`383` - by :user:`masenf`.

src/tox/execute/stream.py

@@ -100,7 +100,7 @@ def colored(self) -> Iterator[None]:
     @property
     def text(self) -> str:
         with self._content_lock:
-            return self._content.decode("utf-8")
+            return self._content.decode("utf-8", errors="surrogateescape")
 
     @property
     def content(self) -> bytearray:

tests/execute/test_stream.py

@@ -8,3 +8,9 @@
 def test_sync_write_repr() -> None:
     sync_write = SyncWrite(name="a", target=None, color=Fore.RED)
     assert repr(sync_write) == f"SyncWrite(name='a', target=None, color={Fore.RED!r})"
+
+
+def test_sync_write_decode_surrogate() -> None:
+    sync_write = SyncWrite(name="a", target=None)
+    sync_write.handler(b"\xed\n")
+    assert sync_write.text == "\udced\n"