Skip to content

Commit 7187663

Browse files
shahidki31dongjoon-hyun
authored andcommitted
[SPARK-25583][DOC] Add history-server related configuration in the documentation.
## What changes were proposed in this pull request? Add history-server related configuration in the documentation. Some of the history server related configurations were missing in the documentation.Like, 'spark.history.store.maxDiskUsage', 'spark.ui.liveUpdate.period' etc. ## How was this patch tested? ![screenshot from 2018-10-01 20-58-26](https://user-images.githubusercontent.com/23054875/46298568-04833a80-c5bd-11e8-95b8-54c9d6582fd2.png) ![screenshot from 2018-10-01 20-59-31](https://user-images.githubusercontent.com/23054875/46298591-11a02980-c5bd-11e8-93d0-892afdfd4f9a.png) ![screenshot from 2018-10-01 20-59-45](https://user-images.githubusercontent.com/23054875/46298601-1533b080-c5bd-11e8-9689-e9b39882a7b5.png) Closes apache#22601 from shahidki31/historyConf. Authored-by: Shahid <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 5114db5 commit 7187663

File tree

2 files changed

+41
-0
lines changed

2 files changed

+41
-0
lines changed

docs/configuration.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -793,6 +793,13 @@ Apart from these, the following properties are also available, and may be useful
793793
Buffer size to use when writing to output streams, in KiB unless otherwise specified.
794794
</td>
795795
</tr>
796+
<tr>
797+
<td><code>spark.ui.dagGraph.retainedRootRDDs</code></td>
798+
<td>Int.MaxValue</td>
799+
<td>
800+
How many DAG graph nodes the Spark UI and status APIs remember before garbage collecting.
801+
</td>
802+
</tr>
796803
<tr>
797804
<td><code>spark.ui.enabled</code></td>
798805
<td>true</td>
@@ -807,6 +814,15 @@ Apart from these, the following properties are also available, and may be useful
807814
Allows jobs and stages to be killed from the web UI.
808815
</td>
809816
</tr>
817+
<tr>
818+
<td><code>spark.ui.liveUpdate.period</code></td>
819+
<td>100ms</td>
820+
<td>
821+
How often to update live entities. -1 means "never update" when replaying applications,
822+
meaning only the last write will happen. For live applications, this avoids a few
823+
operations that we can live without when rapidly processing incoming task events.
824+
</td>
825+
</tr>
810826
<tr>
811827
<td><code>spark.ui.port</code></td>
812828
<td>4040</td>

docs/monitoring.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,13 +185,38 @@ Security options for the Spark History Server are covered more detail in the
185185
Job history files older than this will be deleted when the filesystem history cleaner runs.
186186
</td>
187187
</tr>
188+
<tr>
189+
<td>spark.history.fs.endEventReparseChunkSize</td>
190+
<td>1m</td>
191+
<td>
192+
How many bytes to parse at the end of log files looking for the end event.
193+
This is used to speed up generation of application listings by skipping unnecessary
194+
parts of event log files. It can be disabled by setting this config to 0.
195+
</td>
196+
</tr>
197+
<tr>
198+
<td>spark.history.fs.inProgressOptimization.enabled</td>
199+
<td>true</td>
200+
<td>
201+
Enable optimized handling of in-progress logs. This option may leave finished
202+
applications that fail to rename their event logs listed as in-progress.
203+
</td>
204+
</tr>
188205
<tr>
189206
<td>spark.history.fs.numReplayThreads</td>
190207
<td>25% of available cores</td>
191208
<td>
192209
Number of threads that will be used by history server to process event logs.
193210
</td>
194211
</tr>
212+
<tr>
213+
<td>spark.history.store.maxDiskUsage</td>
214+
<td>10g</td>
215+
<td>
216+
Maximum disk usage for the local directory where the cache application history information
217+
are stored.
218+
</td>
219+
</tr>
195220
<tr>
196221
<td>spark.history.store.path</td>
197222
<td>(none)</td>

0 commit comments

Comments
 (0)