`TestCase.subTest` does not isolate tests

[jakelishman]

In Python 3.4, unittest gained the TestCase.subTest context manager, which is useful for writing several related tests in one in cases when fixtures may be too awkward. When deriving from testtools.TestCase, these sub-test contexts don't work; the code behaves exactly (as best as I could tell) as if they were not used at all.

For example, if I have a test file test_subtest.py:

import unittest
import testtools

class Unittest(unittest.TestCase):
    def test_failure(self):
        with self.subTest("failure1"):
            self.assertTrue(False)
        with self.subTest("failure2"):
            self.assertTrue(False)

class Testtools(testtools.TestCase):
    def test_failure(self):
        with self.subTest("failure1"):
            self.assertTrue(False)
        with self.subTest("failure2"):
            self.assertTrue(False)

then the output is:

$ python -munittest test_subtest.py
F
======================================================================
FAIL: test_failure (test_subtest.Testtools)
test_subtest.Testtools.test_failure
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/Users/jake/tmp/test_subtest.py", line 14, in test_failure
    self.assertTrue(False)
  File "/Users/jake/.miniconda3/envs/qiskit/lib/python3.9/unittest/case.py", line 682, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true


======================================================================
FAIL: test_failure (test_subtest.Unittest) [failure1]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jake/tmp/test_subtest.py", line 7, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

======================================================================
FAIL: test_failure (test_subtest.Unittest) [failure2]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jake/tmp/test_subtest.py", line 9, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=3)

The unittest version shows both failures (and puts the extra message in the report), while the testtools version only displays the first one. I also tried a modified version of this file, where each subtest also touched a separate file, which showed that the second subtest in the testtools case is not run at all.