Skip to content

Commit 925f620

Browse files
Ngone51HyukjinKwon
authored andcommitted
[SPARK-28302][CORE] Make sure to generate unique output file for SparkLauncher on Windows
## What changes were proposed in this pull request? When using SparkLauncher to submit applications **concurrently** with multiple threads under **Windows**, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. The issue can be reproduced by this [demo](https://issues.apache.org/jira/secure/attachment/12973920/Main.scala). After digging into the code, I find that, Windows cmd `%RANDOM%` would return the same number if we call it instantly(e.g. < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it tries to write the same file which has already been opened for writing by another app. We should make sure to generate unique output file for SparkLauncher on Windows to avoid this issue. ## How was this patch tested? Tested manually on Windows. Closes apache#25076 from Ngone51/SPARK-28302. Authored-by: wuyi <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent f15102b commit 925f620

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

bin/spark-class2.cmd

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,12 @@ if not "x%JAVA_HOME%"=="x" (
6363

6464
rem The launcher library prints the command to be executed in a single line suitable for being
6565
rem executed by the batch interpreter. So read all the output of the launcher into a variable.
66+
:gen
6667
set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt
68+
rem SPARK-28302: %RANDOM% would return the same number if we call it instantly after last call,
69+
rem so we should make it sure to generate unique file to avoid process collision of writing into
70+
rem the same file concurrently.
71+
if exist %LAUNCHER_OUTPUT% goto :gen
6772
"%RUNNER%" -Xmx128m -cp "%LAUNCH_CLASSPATH%" org.apache.spark.launcher.Main %* > %LAUNCHER_OUTPUT%
6873
for /f "tokens=*" %%i in (%LAUNCHER_OUTPUT%) do (
6974
set SPARK_CMD=%%i

0 commit comments

Comments
 (0)