Skip to content

Commit 1521889

Browse files
shahidki31srowen
authored andcommitted
[SPARK-29902][DOC][MINOR] Add listener event queue capacity configuration to documentation
### What changes were proposed in this pull request? Add listener event queue capacity configuration to documentation ### Why are the changes needed? We some time see many event drops happening in eventLog listener queue. So, instead of increasing all the queues size, using this config we just need to increase eventLog queue capacity. ``` scala> sc.parallelize(1 to 100000, 100000).count() [Stage 0:=================================================>(98299 + 4) / 100000]19/11/14 20:56:35 ERROR AsyncEventQueue: Dropping event from queue eventLog. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler. 19/11/14 20:56:35 WARN AsyncEventQueue: Dropped 1 events from eventLog since the application started. ``` ### Does this PR introduce any user-facing change? No ### How was this patch tested? Existing tests Closes apache#26529 from shahidki31/master1. Authored-by: shahid <[email protected]> Signed-off-by: Sean Owen <[email protected]>
1 parent ee4784b commit 1521889

File tree

1 file changed

+45
-0
lines changed

1 file changed

+45
-0
lines changed

docs/configuration.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1857,6 +1857,51 @@ Apart from these, the following properties are also available, and may be useful
18571857
driver using more memory.
18581858
</td>
18591859
</tr>
1860+
<tr>
1861+
<td><code>spark.scheduler.listenerbus.eventqueue.shared.capacity</code></td>
1862+
<td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
1863+
<td>
1864+
Capacity for shared event queue in Spark listener bus, which hold events for external listener(s)
1865+
that register to the listener bus. Consider increasing value, if the listener events corresponding
1866+
to shared queue are dropped. Increasing this value may result in the driver using more memory.
1867+
</td>
1868+
</tr>
1869+
<tr>
1870+
<td><code>spark.scheduler.listenerbus.eventqueue.appStatus.capacity</code></td>
1871+
<td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
1872+
<td>
1873+
Capacity for appStatus event queue, which hold events for internal application status listeners.
1874+
Consider increasing value, if the listener events corresponding to appStatus queue are dropped.
1875+
Increasing this value may result in the driver using more memory.
1876+
</td>
1877+
</tr>
1878+
<tr>
1879+
<td><code>spark.scheduler.listenerbus.eventqueue.executorManagement.capacity</code></td>
1880+
<td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
1881+
<td>
1882+
Capacity for executorManagement event queue in Spark listener bus, which hold events for internal
1883+
executor management listeners. Consider increasing value if the listener events corresponding to
1884+
executorManagement queue are dropped. Increasing this value may result in the driver using more memory.
1885+
</td>
1886+
</tr>
1887+
<tr>
1888+
<td><code>spark.scheduler.listenerbus.eventqueue.eventLog.capacity</code></td>
1889+
<td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
1890+
<td>
1891+
Capacity for eventLog queue in Spark listener bus, which hold events for Event logging listeners
1892+
that write events to eventLogs. Consider increasing value if the listener events corresponding to eventLog queue
1893+
are dropped. Increasing this value may result in the driver using more memory.
1894+
</td>
1895+
</tr>
1896+
<tr>
1897+
<td><code>spark.scheduler.listenerbus.eventqueue.streams.capacity</code></td>
1898+
<td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
1899+
<td>
1900+
Capacity for streams queue in Spark listener bus, which hold events for internal streaming listener.
1901+
Consider increasing value if the listener events corresponding to streams queue are dropped. Increasing
1902+
this value may result in the driver using more memory.
1903+
</td>
1904+
</tr>
18601905
<tr>
18611906
<td><code>spark.scheduler.blacklist.unschedulableTaskSetTimeout</code></td>
18621907
<td>120s</td>

0 commit comments

Comments
 (0)