Skip to content

Commit 004960e

Browse files
Deflake acceptance tests (#3672)
## Changes This PR makes some tactical changes to the acceptance test configuration to deflake them. They've been quite flaky for me, especially in agent settings. I asked an agent to stress test them and address problems: * The default timeout for tests is now 30s instead of 20s (common cause of failures) * run-local-node tests now automatically use a free port (very common cause of failures) * The `timeout` self-test is less timing sensitive (rare cause of failures) ## Tests * Stress tested by AI
1 parent 408779d commit 004960e

File tree

5 files changed

+15
-11
lines changed

5 files changed

+15
-11
lines changed

acceptance/cmd/workspace/apps/run-local-node/script

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,14 @@ cd app
44
# so we don't need to start it in background. It will install the dependencies as part of the command
55
trace $CLI apps run-local --prepare-environment --entry-point test.yml 2>&1 | grep -w "Hello, world"
66

7-
PORT=8080
8-
DEBUG_PORT=5252
9-
PROXY_PORT=8081
7+
# Function to find a free port in safe range
8+
get_free_port() {
9+
python3 -c "import socket,random; port=random.randint(49152,65535); s=socket.socket(); s.bind(('',port)); print(port); s.close()" 2>/dev/null || python3 -c "import socket; s=socket.socket(); s.bind(('',0)); print(s.getsockname()[1]); s.close()"
10+
}
11+
12+
PORT=$(get_free_port)
13+
DEBUG_PORT=$(get_free_port)
14+
PROXY_PORT=$(get_free_port)
1015

1116
cleanup() {
1217
# Kill any still running processes on these ports when the script exits
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11

2-
>>> sleep 2
2+
>>> sleep 4
33

44
Error: Test script killed due to a timeout

acceptance/selftest/timeout/script

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
trace sleep 2
1+
trace sleep 4
22
echo "This is never printed"
Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
1-
# Manually check that test takes about 0.1s:
2-
# PASS: TestAccept/selftest/timeout (0.10s)
3-
Timeout = '100ms'
4-
TimeoutWindows = '300ms'
5-
TimeoutCloud = '300ms'
1+
# Test that timeout mechanism works (script sleeps 4s but should be killed much sooner)
2+
Timeout = '500ms'
3+
TimeoutWindows = '500ms'
4+
TimeoutCloud = '500ms'

acceptance/test.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ Local = true
33
Cloud = false
44

55
# default timeouts
6-
Timeout = '20s'
6+
Timeout = '30s'
77
TimeoutWindows = '60s'
88

99
# Slowest test I saw:

0 commit comments

Comments
 (0)