Skip to content

Commit ae37c79

Browse files
authored
docs: example configuration for the history server (#458)
1 parent 6d95d80 commit ae37c79

File tree

1 file changed

+46
-0
lines changed

1 file changed

+46
-0
lines changed

docs/modules/spark-k8s/pages/usage-guide/logging.adoc

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,49 @@ The Spark operator installs a https://vector.dev/docs/setup/deployment/roles/#ag
55
It is the user's responsibility to install and configure the vector aggregator, but the agents can discover the aggregator automatically using a discovery ConfigMap as described in the xref:concepts:logging.adoc[logging concepts].
66

77
NOTE: Only logs produced by the application's driver and executors are collected. Logs produced by `spark-submit` are discarded.
8+
9+
== History server
10+
11+
The following snippet shows how to configure log aggregation for the history server:
12+
13+
[source,yaml]
14+
----
15+
apiVersion: spark.stackable.tech/v1alpha1
16+
kind: SparkHistoryServer
17+
metadata:
18+
name: spark-history
19+
spec:
20+
vectorAggregatorConfigMapName: spark-vector-aggregator-discovery # <1>
21+
nodes:
22+
roleGroups:
23+
default:
24+
config:
25+
logging:
26+
enableVectorAgent: true # <2>
27+
containers:
28+
spark-history: #<3>
29+
console:
30+
level: INFO
31+
file:
32+
level: INFO
33+
loggers:
34+
ROOT:
35+
level: INFO
36+
...
37+
----
38+
<1> Name of a ConfigMap that referenced the vector aggregator. See example below.
39+
<2> Enable the vector agent in the history pod.
40+
<3> Configure log levels for file and console outputs.
41+
42+
Example vector aggregator configuration.
43+
44+
[source,yaml]
45+
----
46+
---
47+
apiVersion: v1
48+
kind: ConfigMap
49+
metadata:
50+
name: spark-vector-aggregator-discovery
51+
data:
52+
ADDRESS: spark-vector-aggregator:6123
53+
----

0 commit comments

Comments
 (0)