LLDB tests hanging on premerge runners

[boomanaiden154]

We're seeing some LLDB tests hang on the premerge runners, with the default lit timeouts not working. Going into a hanging instance, we can see the following process from ps aux that has been running for 10+ hours:

gha      1597203  0.0  0.0 1562396 108092 ?      Sl   Oct30   0:08 /home/gha/llvm-project/build/bin/lldb-dap --connection accept:///tmp/lit-tmp-yp5x_cuj/dap-connection-1585649

Running lsof -a -p 1597203 we can see the following open file:

lldb-dap 1597203  gha    3w   REG              0,128    23969  1047917 /home/gha/llvm-project/build/lldb-test-build.noindex/tools/lldb-dap/server/TestDAP_server.test_server_unix_socket/dap.txt

Which seems to imply that this test is hanging:

llvm-project/lldb/test/API/tools/lldb-dap/server/TestDAP_server.py

Line 71 in e5f3d7f

def test_server_unix_socket(self):

Looking at a couple other instances, it's not exclusive to this test. I'm also seeing failures in test_server_interrupt and test_server_port. These are all DAP tests though.

[boomanaiden154]

I'm seeing the first failure in 242ebcf, but that looks unrelated. A couple commits before this starts occurring, it looks like a49bfbe landed, which definitely looks related.

[boomanaiden154]

Looking at the premerge dashboard, we're also starting to see timeouts after this change landed (we have the limit set to two hours).

[llvmbot]

@llvm/issue-subscribers-lldb-dap

Author: Aiden Grossman (boomanaiden154)

We're seeing some LLDB tests hang on the premerge runners, with the default lit timeouts not working. Going into a hanging instance, we can see the following process from `ps aux` that has been running for 10+ hours: ``` gha 1597203 0.0 0.0 1562396 108092 ? Sl Oct30 0:08 /home/gha/llvm-project/build/bin/lldb-dap --connection accept:///tmp/lit-tmp-yp5x_cuj/dap-connection-1585649 ```

Running lsof -a -p 1597203 we can see the following open file:

lldb-dap 1597203  gha    3w   REG              0,128    23969  1047917 /home/gha/llvm-project/build/lldb-test-build.noindex/tools/lldb-dap/server/TestDAP_server.test_server_unix_socket/dap.txt

Which seems to imply that this test is hanging:

llvm-project/lldb/test/API/tools/lldb-dap/server/TestDAP_server.py

Line 71 in e5f3d7f

def test_server_unix_socket(self):

Looking at a couple other instances, it's not exclusive to this test. I'm also seeing failures in test_server_interrupt and test_server_port. These are all DAP tests though.

[ashgti]

I think #165784 is related to this and #165823 I think should help this situation.

[JDevlieghere]

@boomanaiden154 Any way we can exclude the DAP tests form premerge until the reliability is improved, and not remove check-lldb altogether? The premerge testing has been incredibly valuable and I'd hate to lose it.

The commit you linked was one of the efforts to improve the reliability, but instead made it worse. But we are working on it. I also want to make sure we can keep experimenting with this (again, without affecting everyone through premerge). I think we have enough post-commit CI nodes to get some good signal to prove when the DAP tests are ready for prime-time again. It's a priority for me (because it's affecting the reliability of the Swift premerge as well) and John's been actively working on this as well.

Does that seem like a reasonable compromise?

[ashgti]

We can revert #165496 / f205be0 as well. It hasn't been fully stable on all platforms so, we may need to resolve other issues before re-applying that patch.

[boomanaiden154]

and not remove check-lldb altogether? The premerge testing has been incredibly valuable and I'd hate to lose it.

This was specifically for our postcommit runners. We have more issues there like custom scripts handling things very poorly related to timeouts, and it's also not super user facing outside of our systems. I've kept it on inside premerge. I'm not interested in removing this from the premerge configuration either.

Removing DAP tests for now would be okay. I'd want to reenable them at least on our postcommit runners as soon as the timeout issue as fixed even if they are still flaky so we can get large scale coverage to see if the improvements are working. The premerge advisor should be done soon, so just test flakiness won't be that big of a deal. The timeouts need to be fixed though.

Does that seem like a reasonable compromise?

Sounds reasonable enough. I think I would be a bit more aggressive on keep things enabled. I think we want to disable DAP tests, but reenable as soon as either:

1. The test timeouts actually trigger and the test fails after exceeding the timeout.
2. The timeouts are no longer occurring.

Reverting the patch in question also SGTM. We should make sure we investigate the timeouts not firing issue though, because that's the real problem with enabling this widely in CI.

[JDevlieghere]

Absolutely. Thanks for the clarification, I appreciate all the work you're doing in this space and I want to support it however we can.