Skip to content

Commit 372ccef

Browse files
committed
rewrite edgecase as extensive input testing
1 parent 6a27607 commit 372ccef

File tree

1 file changed

+55
-12
lines changed

1 file changed

+55
-12
lines changed

docs/pages/principles/testing.md

Lines changed: 55 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -482,21 +482,64 @@ def test_pytest(mocker):
482482
dangerous_sideffects()
483483
```
484484

485-
## Testing Edgecases
485+
## Extensive Input Testing
486486

487-
While writing unit tests, you may be tempted to test edgecases. You may have a
488-
critical private function or algorithm, which is not part of the public API (so
489-
not a good candidate for External testing) and you are concerned about many
490-
edgecases that you want to defend against using tests.
487+
The range of inputs that test cases validate is an important decision.
491488

492-
It is perfectly valid to write extensive edgecase testing for private code, but
493-
these tests should be kept separate from the unit test suite. Extensive edgecase
494-
testing makes tests long, and difficult to read (tests are documentation). They
495-
can slow down execution, we want unit tests to run first, fast, and often.
489+
When the need for extensive testing starts to conflict with the readability of test cases
490+
and their usefulness as documentation for users and other developers, the tests should
491+
be re-organized into public-facing (concise, expressive, easily readable),
492+
and technical (complex, extensive) test files.
493+
494+
- To maintain readability:
495+
- tests may need to include fewer inputs
496+
- extensive edgecase testing may be moved into different sections or files
497+
- the code itself may need to be refactored
498+
- Avoid testing invalid input; the range of invalid input is infinite, and
499+
is the programming equivalent of trying to prove a negative.
500+
- Focus on inputs relevant to the code under test, and avoid testing the code in dependencies.
501+
Some integration testing of specific behaviors your code relies on is justifiable.
502+
- [Fuzz Tests](#fuzz-tests) are the best place to handle extensive and exhaustive input testing.
503+
504+
505+
### In Public Interface Tests
506+
507+
These are the most appropriate place to test certain invalid inputs and dependencies.
508+
Public Interface tests act like a contract with users; each behavior that is tested is like a promise that
509+
users can rely on, and expect that it will not change without warning (and probably a major version bump).
510+
So any input/output and side-effects included in these tests should be considered *officially supported behavior*
511+
and given careful consideration.
512+
513+
### In project level integration tests
514+
515+
These are a good place to handle more extensive input testing. Integration tests already tend to be more verbose,
516+
with a lot of setup and teardown, and much more behavior to cover than other kinds of tests.
517+
These are the kinds of tests that should focus on edgecases.
518+
519+
520+
### In Unit Tests
521+
522+
Unit Tests should focus on the "happy-path" of execution. In most cases one representative
523+
example of the *expected* input is sufficient. The test case should illustrate how the unit
524+
is expected to be used.
525+
526+
Invalid input should only be tested when the unit itself includes logic to handle that invalid input.
527+
528+
for example, this code:
529+
```python
530+
def foo(x: int):
531+
return x + 1
532+
```
533+
should not test its behavior when passed a string (the type annotation already covers that).
534+
535+
This code should be tested with a string, to cover the exception path.
536+
```python
537+
def bar(x):
538+
if type(x) is str:
539+
raise RuntimeError('invalid input')
540+
return x + 1
541+
```
496542

497-
- Place them in separate files from unit tests, to improve readability
498-
- [mark them](https://docs.pytest.org/en/stable/how-to/mark.html) so that they
499-
can be run as a separate test suite, after your unit test pass
500543

501544
## Additional Types of Test Suites
502545

0 commit comments

Comments
 (0)