Skip to content

Commit 339859c

Browse files
huangtengfei02ivoson
authored andcommitted
[SPARK-25261][MINOR][DOC] update the description for spark.executor|driver.memory in configuration.md
## What changes were proposed in this pull request? As described in [SPARK-25261](https://issues.apache.org/jira/projects/SPARK/issues/SPARK-25261),the unit of spark.executor.memory and spark.driver.memory is parsed as bytes in some cases if no unit specified, while in https://spark.apache.org/docs/latest/configuration.html#application-properties, they are descibed as MiB, which may lead to some misunderstandings. ## How was this patch tested? N/A Closes apache#22252 from ivoson/branch-correct-configuration. Lead-authored-by: huangtengfei02 <[email protected]> Co-authored-by: Huang Tengfei <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent 8d9495a commit 339859c

File tree

1 file changed

+5
-4
lines changed

1 file changed

+5
-4
lines changed

docs/configuration.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -152,8 +152,9 @@ of the most common options to set are:
152152
<td><code>spark.driver.memory</code></td>
153153
<td>1g</td>
154154
<td>
155-
Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in MiB
156-
unless otherwise specified (e.g. <code>1g</code>, <code>2g</code>).
155+
Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in the
156+
same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t")
157+
(e.g. <code>512m</code>, <code>2g</code>).
157158
<br />
158159
<em>Note:</em> In client mode, this config must not be set through the <code>SparkConf</code>
159160
directly in your application, because the driver JVM has already started at that point.
@@ -175,8 +176,8 @@ of the most common options to set are:
175176
<td><code>spark.executor.memory</code></td>
176177
<td>1g</td>
177178
<td>
178-
Amount of memory to use per executor process, in MiB unless otherwise specified.
179-
(e.g. <code>2g</code>, <code>8g</code>).
179+
Amount of memory to use per executor process, in the same format as JVM memory strings with
180+
a size unit suffix ("k", "m", "g" or "t") (e.g. <code>512m</code>, <code>2g</code>).
180181
</td>
181182
</tr>
182183
<tr>

0 commit comments

Comments
 (0)