Commit ee9c89e
committed
Fix linux-sandbox actions being killed prematurely when using virtual threads
This change fixes a bug where linux-sandbox actions that take longer
than 30 seconds can be killed prematurely and cause the build to fail.
There are two things that contribute to this bug:
Part one:
When using virtual threads with the JavaSubprocessFactory, the process
is forked from the currently running virtual thread's carrier thread.
The virtual thread then waits for the subprocess to complete, which
causes the carrier thread to detach. If there is no other work for the
carrier thread to perform, it becomes idle. If it is idle for long
enough it can be killed by the ForkJoinPool it is a member of. The
default virtual thread scheduler in JDK 25 uses a ForkJoinPool with an
idle TTL of 30 seconds.
Part two:
The linux-sandbox process sends SIGKILL to the child process when its
parent dies. This is setup like so: `prctl(PR_SET_PDEATHSIG, SIGKILL)`.
These two things combined cause problems for Bazel builds if you:
* Use the linux-sandbox strategy
* Use virtual threads via --experimental_async_execution
* Have an action which takes longer than 30 seconds to complete
If the Bazel server isn't very busy while that 30 second action is
running, the parent carrier thread can be killed due to inactivity,
which causes the linux-sandbox process to die, which causes the build to
fail.
This is only an issue if you're using virtual threads. When using
platform threads they directly wait on the subprocess to complete. The
thread is considered busy while waiting, which prevents it from being
killed.
This change fixes the issue by always forking from a long-lived platform
thread. That way there is no risk of the platform thread being killed
prematurely. The fix does so while still enabling the linux-sandbox
processes to be properly killed when the Bazel server is killed.1 parent 01cacec commit ee9c89e
File tree
1 file changed
+43
-1
lines changed- src/main/java/com/google/devtools/build/lib/shell
1 file changed
+43
-1
lines changedLines changed: 43 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
18 | 19 | | |
| 20 | + | |
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
| |||
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
27 | 33 | | |
28 | 34 | | |
29 | 35 | | |
| |||
118 | 124 | | |
119 | 125 | | |
120 | 126 | | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
121 | 138 | | |
122 | 139 | | |
123 | 140 | | |
| |||
143 | 160 | | |
144 | 161 | | |
145 | 162 | | |
146 | | - | |
| 163 | + | |
147 | 164 | | |
148 | 165 | | |
149 | 166 | | |
| |||
159 | 176 | | |
160 | 177 | | |
161 | 178 | | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
162 | 204 | | |
163 | 205 | | |
164 | 206 | | |
| |||
0 commit comments