Skip to content

Commit cdd2790

Browse files
jiangzhoPeter Toth
authored andcommitted
[SPARK-53187] Support SparkCluster event related metrics set
### What changes were proposed in this pull request? This PR adds SparkCluster as a supported type in metrics and therefore supports publishing execution metrics for it. ### Why are the changes needed? Operator by default publish metrics by resource type (SparkApplication) - by adding this support, we support the same set of counter and histogram for received event and timed execution ### Does this PR introduce _any_ user-facing change? More Metrics becomes available for SparkClusters ### How was this patch tested? From dev sandbox we can see metrics like ``` metrics_operator.sdk_sparkapplication_added_resource_event_Count{type="counters"} 9010 metrics_operator.sdk_sparkapplication_reconciliation_failed_Count{type="counters"} 9 metrics_operator.sdk_sparkapplication_reconciliation_finished_Count{type="counters"} 182841 metrics_operator.sdk_sparkapplication_reconciliation_retries_Count{type="counters"} 9 metrics_operator.sdk_sparkcluster_added_resource_event_Count{type="counters"} 9009 metrics_operator.sdk_sparkcluster_reconciliation_failed_Count{type="counters"} 0 metrics_operator.sdk_sparkcluster_reconciliation_finished_Count{type="counters"} 182821 metrics_operator.sdk_sparkcluster_reconciliation_retries_Count{type="counters"} 0 ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#295 from jiangzho/cluster_metrics. Authored-by: Zhou JIANG <[email protected]> Signed-off-by: Peter Toth <[email protected]>
1 parent c95a8c0 commit cdd2790

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/source/OperatorJosdkMetrics.java

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444

4545
import org.apache.spark.k8s.operator.BaseResource;
4646
import org.apache.spark.k8s.operator.SparkApplication;
47+
import org.apache.spark.k8s.operator.SparkCluster;
4748
import org.apache.spark.metrics.source.Source;
4849
import org.apache.spark.util.Clock;
4950
import org.apache.spark.util.SystemClock;
@@ -305,6 +306,8 @@ private Counter getCounter(Class<?> klass, String... names) {
305306

306307
if (resourceGvk.getKind().equals(SparkApplication.class.getSimpleName())) {
307308
resourceClass = SparkApplication.class;
309+
} else if (resourceGvk.getKind().equals(SparkCluster.class.getSimpleName())) {
310+
resourceClass = SparkCluster.class;
308311
} else {
309312
return Optional.empty();
310313
}

0 commit comments

Comments
 (0)