[Bug] autoscale up and down are slow

[caican00]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ray-operator

What happened + What you expected to happen

1. At around 09:59:20 on July 23, 2025, raycluster-219843 has updated the CRD (as shown in Figure 3), and the number of replicas needs to be scaled to 500

2. However, during the period from 09:59 to 10:08 on July 23, 2025, kuberay was constantly handling the expansion operation of another raycluster (RayCluster-538811), which lasted for 10 minutes.
   The time point for the first pod scaling up of raycluster-538811 is 2025-07-23 09:57:47.520

3. The time point of the last pod scaling up of raycluster-538811 is : 2025-07-23 10:08:32

4. The scaling up requirement for raycluster-219843 were not processed until 10:08 on July 23, 2025

5. The current ReconcileConcurrency value is 1, and the entire process has been pending for 10 minutes

Reproduction script

None

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[KunWuLuan]

Can you show the configuration of ray-operator? Concurrency of operator is 1 by default, so any slow reconcile can lead to long latency of other RayClusters/RayJobs. You can set ReconcileConcurrency in configuration to overwrite the value. @caican00

[Future-Outlier]

Hi, @caican00

I created a PR to update the KubeRay documentation about how to configure the reconcile concurrency.
Please take a look. Thank you!
Maybe in the future, we can add this field to the Helm chart.

ray-project/ray#55236

[win5923]

Maybe in the future, we can add this field to the Helm chart.

#3751

[rueian]

Hi @caican00, have you tried increasing the ReconcileConcurrency?