Skip to content

Commit ecdb51b

Browse files
committed
added comment about checking executors as well as driver
1 parent f58a524 commit ecdb51b

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

rust/operator-binary/src/connect/controller.rs

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -315,6 +315,12 @@ pub async fn reconcile(
315315
let cluster_operation_cond_builder =
316316
ClusterOperationsConditionBuilder::new(&scs.spec.cluster_operation);
317317

318+
// TODO: This StatefulSet only contains the driver. We should probably also
319+
// consider the state of the executors to determine if the
320+
// SparkConnectServer is ready. This depends on the availability and
321+
// resilience properties of Spark and could e.g. be "driver and more than
322+
// 75% of the executors ready". Special care needs to be taken about
323+
// auto-scaling executors in this case (if/once supported).
318324
let status = SparkConnectServerStatus {
319325
conditions: compute_conditions(scs, &[&ss_cond_builder, &cluster_operation_cond_builder]),
320326
};

0 commit comments

Comments
 (0)