TEST: add indirect test for different condition failures

[ubunatic]

Summary

Add a failing + skipped testing.T based test to simulate real world testing errors. Allow running the failing test with a go test companion test that analyzes the logs.

The test and test tool can be used to test this PR: stretchr#1809 with minimal adjustments:

• disable/comment the extra Panic tests
• adjust the counts: 2 panics -> 0 panics, tests: 8 -> 6

Changes

• add TestFailInsideEventually (skipped, because failing)
• use explicit logs inside TestFailInsideEventually
• catch logs from companion test TestFailInsideEventuallyViaCommandLine
• added draft for tool for command line testing in a structured way using new:
  assert.CommandLineTest(t, assert.CommandLineTestSpec{ ... }) to run go test

Motivation

• avoid MockT issues and simulate real world tests
• properly test unclean test exits

Related issues

• IDEA: Handle panics and make condition result explicit #1 (extended version of the condition result handling + extended test)
• Do not hang Eventually on failed condition stretchr/testify#1809 minimal viable change to stop hanging tests

Output

$ go test -v -race -run '^TestFailInsideEventuallyViaCommandLine$' ./...`

testing: warning: no tests to run
PASS
ok      github.com/stretchr/testify     (cached) [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/stretchr/testify/assert      (cached) [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/stretchr/testify/assert/internal/unsafetests (cached) [no tests to run]
?       github.com/stretchr/testify/assert/yaml [no test files]
?       github.com/stretchr/testify/http        [no test files]
testing: warning: no tests to run
PASS
ok      github.com/stretchr/testify/internal/difflib    (cached) [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/stretchr/testify/mock        (cached) [no tests to run]
=== RUN   TestFailInsideEventuallyViaCommandLine
=== RUN   TestFailInsideEventuallyViaCommandLine/TestFailInsideEventually
    requirements_test.go:798: Running test TestFailInsideEventually
    requirements_test.go:798: Running test TestFailInsideEventually/require.Fail_must_stop
    requirements_test.go:798: Test TestFailInsideEventually/require.Fail_must_stop logged expected success:
    requirements_test.go:916: ✅ FINISHED require.Fail must stop
    requirements_test.go:798: Running test TestFailInsideEventually/require.Fail_must_stop_even_if_told_not_to
    requirements_test.go:798: Test TestFailInsideEventually/require.Fail_must_stop_even_if_told_not_to logged expected success:
    requirements_test.go:916: ✅ FINISHED require.Fail must stop even if told not to
    requirements_test.go:798: Running test TestFailInsideEventually/assert.Fail_must_stop_if_told_to
    requirements_test.go:798: Test TestFailInsideEventually/assert.Fail_must_stop_if_told_to logged expected success:
    requirements_test.go:916: ✅ FINISHED assert.Fail must stop if told to
    requirements_test.go:798: Running test TestFailInsideEventually/assert.Fail_must_not_stop_if_told_not_to
    requirements_test.go:798: Test TestFailInsideEventually/assert.Fail_must_not_stop_if_told_not_to logged expected success:
    requirements_test.go:916: ✅ FINISHED assert.Fail must not stop if told not to
    requirements_test.go:798: Running test TestFailInsideEventually/runtime.Goexit_must_stop
    requirements_test.go:798: Test TestFailInsideEventually/runtime.Goexit_must_stop logged expected success:
    requirements_test.go:916: ✅ FINISHED runtime.Goexit must stop
    requirements_test.go:798: Running test TestFailInsideEventually/runtime.Goexit_must_stop_even_if_told_not_to
    requirements_test.go:798: Test TestFailInsideEventually/runtime.Goexit_must_stop_even_if_told_not_to logged expected success:
    requirements_test.go:916: ✅ FINISHED runtime.Goexit must stop even if told not to
    requirements_test.go:798: Running test TestFailInsideEventually/panic_must_stop
    requirements_test.go:798: Test TestFailInsideEventually/panic_must_stop logged expected success:
    requirements_test.go:916: ✅ FINISHED panic must stop
    requirements_test.go:798: Running test TestFailInsideEventually/panic_must_stop_even_if_told_not_to
    requirements_test.go:798: Test TestFailInsideEventually/panic_must_stop_even_if_told_not_to logged expected success:
    requirements_test.go:916: ✅ FINISHED panic must stop even if told not to
    requirements_test.go:798: Found expected line match "Condition never satisfied" the expected number of times: 1
    requirements_test.go:798: Found expected line match "Condition failed" the expected number of times: 4
    requirements_test.go:798: Found expected line match "Condition panicked" the expected number of times: 2
    requirements_test.go:798: Found expected line match "💥 mark as failed" the expected number of times: 4
    requirements_test.go:798: Found expected line match "💥 fail now" the expected number of times: 2
    requirements_test.go:798: Found expected line match "💥 goexit" the expected number of times: 2
--- PASS: TestFailInsideEventuallyViaCommandLine (1.48s)
    --- PASS: TestFailInsideEventuallyViaCommandLine/TestFailInsideEventually (1.48s)
PASS
ok      github.com/stretchr/testify/require     (cached)
testing: warning: no tests to run
PASS
ok      github.com/stretchr/testify/suite       (cached) [no tests to run]