Skip to content

Commit 0d77d57

Browse files
committed
[MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent on 'resource'
## What changes were proposed in this pull request? This PR adds a note that explicitly `spark.executor.pyspark.memory` is dependent on resource module's behaviours at Python memory usage. For instance, I at least see some difference at apache#21977 (comment) ## How was this patch tested? Manually built the doc. Closes apache#23664 from HyukjinKwon/note-resource-dependent. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
1 parent 308996b commit 0d77d57

File tree

1 file changed

+6
-3
lines changed

1 file changed

+6
-3
lines changed

docs/configuration.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -190,8 +190,10 @@ of the most common options to set are:
190190
and it is up to the application to avoid exceeding the overhead memory space
191191
shared with other non-JVM processes. When PySpark is run in YARN or Kubernetes, this memory
192192
is added to executor resource requests.
193-
194-
NOTE: Python memory usage may not be limited on platforms that do not support resource limiting, such as Windows.
193+
<br/>
194+
<em>Note:</em> This feature is dependent on Python's `resource` module; therefore, the behaviors and
195+
limitations are inherited. For instance, Windows does not support resource limiting and actual
196+
resource is not limited on MacOS.
195197
</td>
196198
</tr>
197199
<tr>
@@ -223,7 +225,8 @@ of the most common options to set are:
223225
stored on disk. This should be on a fast, local disk in your system. It can also be a
224226
comma-separated list of multiple directories on different disks.
225227

226-
NOTE: In Spark 1.0 and later this will be overridden by SPARK_LOCAL_DIRS (Standalone), MESOS_SANDBOX (Mesos) or
228+
<br/>
229+
<em>Note:</em> This will be overridden by SPARK_LOCAL_DIRS (Standalone), MESOS_SANDBOX (Mesos) or
227230
LOCAL_DIRS (YARN) environment variables set by the cluster manager.
228231
</td>
229232
</tr>

0 commit comments

Comments
 (0)