Battle-test on kotlinx-coroutines

[diffitask]

When rewriting stress tests, it would be great to reproduce old (already fixed) bugs and check that we can detect manually embedded bugs.

[eupp]

Approximate plan for this issue.

• Prepare the infrastructure to run the tests (create separate branch of kotlinx-coroutines, add new tests, etc).

• Stage 1 testing: select N (~10 cases) stress tests from kotlinx-coroutines and rewrite them to run under model checker.

• Ensure tests are functional.

  • Tests are run without any Lincheck bugs.
  • If some simple bug is injected into the code, the test reveals it.

• Submit bug reports for any Lincheck bugs revealed by Stage 1 testing.

  • If a bug can be minimized and extracted from kotlinx-coroutines then create a minimal reproducer as a separate test in Lincheck repo. Create a draft PR with failing test and create an issue with the bug description.
  • If a bug cannot be extracted from kotlinx-coroutines just create an issue with the bug description.

• Fix critical Lincheck bugs revealed on previous step.

• Reproduce some known/fixed bugs from kotlinx-coroutines with the help of model checker.

  • We have a list of previously fixed know bugs (4 cases, see the list below) --- reproduce at least 1 of them.
  • We might need to create a separate branch of kotlinx-coroutines for each bug of this category (since we need to test it on specific version of kotlinx-coroutines).

• Stage 2 testing (optional): select M (M > N) stress tests from kotlinx-coroutines and rewrite them to run under model checker.

  • Same as stage 1, but more tests are added (and new tests can be more complex than on stage 1).

Constants N and M should be adjusted based on the analysis of stress tests in kotlinx-coroutines repository.

List of known bugs to try reproduce them with the model checker:

• Properly detect non-released reusable continuations in non-reusable o… Kotlin/kotlinx.coroutines#2772
• Fix DefaultExecutor not being able to exit Kotlin/kotlinx.coroutines#1876
• Fix of a bug in the semaphore Kotlin/kotlinx.coroutines#1477
• Reach target parallelism in .limitedParallelism of failing dispatchers Kotlin/kotlinx.coroutines#4330

[dmitrii-artuhov]

Custom kotlinx.coroutines branch with tests written with gpmc: https://github.com/dmitrii-artuhov/kotlinx.coroutines/tree/gpmc-tests.
Folder: https://github.com/dmitrii-artuhov/kotlinx.coroutines/tree/gpmc-tests/kotlinx-coroutines-core/jvm/test/gpmc

[dmitrii-artuhov]

Update on tests when all work-arounds are applied to surpass #542, #543, #409, #576.
With tick marked tests which now pass:

• channels:

  1. BroadcastChannelMultiReceiveModelCheckingTest
     • Hangs, issue is unknown.
  2. BufferedChannelModelCheckingTest
     • testModelCheck
     • testBurst
  3. ConflatedChannelCloseModelCheckingTest
     • works but only after adding status flags to every coroutine (in original stress test this functionality was implemented via delay(...). See the comment in the test.
  4. DoubleChannelCloseModelCheckingTest
  5. SimpleSendReceiveJvmTest

• internal:

  1. ConcurrentWeakMapCollectionModelCheckingTest
     • The test hangs. Tests a custom implementation of a ConcurrentWeakMap.
     • Could be related to Support WeakReference in model checking mode #279.
     • De-prioritize for now until we properly support WeakReference.
  2. LockFreeLinkedListLongModelCheckingTest
     • testModelCheckingCustom: simplified version of the second test
     • testModelChecking: invalid thread switch attempt (cause is unknown, probably some non-transformed classes), fixed when disabling local object manager.
  3. LockFreeTaskQueueModelCheckingTest
     • The test hangs. The test uses CyclicBarrier which might be the cause of the hang (some uninstrumented internal classes).
     • Actually, the same invalid thread switch attempt happens here, but this it is overridden with = Concurrent test has hung = assertion error. Fixed when disabling local object manager

• scheduling:

  1. WorkQueueModelCheckingTest
     • testStealing: invalid thread switch attempt (uses CountDownLatch internally, which might be the cause, since such behaviour already was observed on more simple CountDownLatch test). Fixed when disabling local object manager.
     • testSingleProducerSingleStealer: invalid thread switch attempt. Fixed when disabling local object manager.

---

Invalid thread switch attempt is usually caused by having threads appended to lists and accessed via them. There are two hypothesis why this happens:

1. Local objects detector incorrectly classifies something as local and do not insert switch points
2. Something is not instrumented in Arrays that leads to bug

[ndkoval]

@dmitrii-artuhov, let's also reproduce this failure with Lincheck: Kotlin/kotlinx.coroutines#3654

[dmitrii-artuhov]

Kotlin Coroutines issues reproducing:

• issue-1477: unable to rewrite, too old kotlin version
• issue-1876: unable to rewrite, too old kotlin version
• issue-2772: unable to rewrite, too old kotlin version
• issue-3654 in branch: does not reproduce
• issue-4330 in branch: reproduces, but it is a single-threaded error

Closing this issue