[LLDB] Flakey DAP tests on builders

[shafik]

It looks like the DAP tests are pretty flakey. Right now it looks like TestDAP_attachByPortNum.py is failing a bunch now e.g.:

https://lab.llvm.org/buildbot/#/builders/18/builds/15174

but if we keep going back various other DAP tests are also failing regularly.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Shafik Yaghmour (shafik)

It looks like the DAP tests are pretty flakey. Right now it looks like `TestDAP_attachByPortNum.py` is failing a bunch now e.g.:

https://lab.llvm.org/buildbot/#/builders/18/builds/15174

but if we keep going back various other DAP tests are also failing regularly.

[llvmbot]

@llvm/issue-subscribers-lldb-dap

Author: Shafik Yaghmour (shafik)

It looks like the DAP tests are pretty flakey. Right now it looks like `TestDAP_attachByPortNum.py` is failing a bunch now e.g.:

https://lab.llvm.org/buildbot/#/builders/18/builds/15174

but if we keep going back various other DAP tests are also failing regularly.

[JDevlieghere]

Looking at the bot history, it looks like there are two flakey tests:

• TestDAP_attachByPortNum.py
• TestDAP_evaluate.py

Additionally, @kusmour is working on making TestDap_attach.py more reliable. As someone who has looked into these issues, historically they've all been different issues rather than an inherent issue with "the DAP tests".

[felipepiovezan]

Also happening on the mac bots: https://green.lab.llvm.org/job/llvm.org/view/LLDB/job/lldb-cmake/11703/console

[felipepiovezan]

Should we disable this test for now? It is creating a lot of noise in the bots

[felipepiovezan]

Skipping that test for now.

[JDevlieghere]

I have a theory but I need to do some more investigating. I think the attach test is failing because we're seeing the original stop form the attach, which lldb-dap is trying to hide until dap.configuration_done_sent is set to true. However, that's racy because nothing guarantees that that stop event will have been handled before then. We need to find a better way to synchronize this. My plan is to test my theory by explicitly ignoring the first stop event rather than relying on dap.configuration_done_sent.

CC: @ashgti in case you're also looking into this.

[felipepiovezan]

TestDAP_attach.py is also consistently failing on the incremental bots (and others).
Going to skip that one too for now because of the noise.