Many tests leak temporary resources if a setup/setupclass fails. #1546
Description
I'm working on adding encryption/decryption routines to pgcontents, which means I'm breaking all my tests pretty regularly.
One of the things I've noticed in the process is that many of the contents API tests (which I subclass to ensure that I have a conforming implementation) leak temporary state (in particular, temporary directories) if a setUp test method fails.
This happens because unittest doesn't call a test suite's tearDown method if its setUp method fails, so if we get halfway through a setUp and fail, we don't clean up those resources in tearDown. For per-method resources, the right way to fix this is generally to use addCleanup to register a destructor for each allocated resource (functions registered with addCleanup are guaranteed to be called even if setUp fails).
Unfortunately, there's no good equivalent to addCleanup for resources created in an setupClass. Correct handling of class-level resources generally involves manually maintaining a stack of destructor callbacks and ensuring that they're fired properly even if setupClass fails. (We have a pretty intricate example of this over in Zipline.)
I'd like to at least update some of the setUp methods to use addCleanup. Whether or not the project has the test-refactoring stomach to do fix usages of setUpClass is a more interesting question.
At any rate, I figured I should open an issue before I go through and refactor a whole bunch of tests.