[server] Change the default value of 'coordinator.io-pool.size' from 1 to 10

[swuferhong]

Purpose

Linked issue: #616

When I creating a partitioned kv table with a large number of buckets like 512. I found that the CoordinatorServe heap memory kept increasing, eventually reaching around 15 GB. Further inspection of the memory usage revealed that most of it was occupied by KvFileHandler and KvSnapshotHandle.

The main reason for the accumulation of KvFileHandler and KvSnapshotHandle is that the speed of deleting remote snapshot files is very slow (the remote storage is OSS). Additionally, since the current concurrency level for deletion threads is only 1, the backlog becomes increasingly severe, eventually leading to frequent full GCs and even OOM. The preliminary solution is to increase the 'coordinator.io-pool.size' to 10, allowing 10 concurrent threads to handle the deletion tasks. From the test results, CPU usage increases by only about 3%, but the accumulation of KvFileHandler and KvSnapshotHandle no longer occurs.

In the long term, we need to further optimize the memory usage of KvFileHandler and KvSnapshotHandle. Issue #528 will continue to track this.

Brief change log

Tests

API and Format

Documentation

[swuferhong]

@wuchong @luoyuxia comments addressed, pr ready.

[wuchong]

Upon further review, I realized that all server.* configurations pertain to the TabletServer. Therefore, using server.background.threads for Coordinator is misleading. The original configuration, coordinator.io-pool.size, is more appropriate and self-explanatory.

Additionally, I found that Flink uses a similar configuration, cluster.io-pool.size, which defaults to 4 * CPU cores for both the Master and TaskExecutor. Given this, I suggest simply increasing the coordinator.io-pool.size to 10. This adjustment is straightforward and aligns with common practices.

Sorry for the back-and-forth on this matter. @swuferhong

[swuferhong]

Upon further review, I realized that all server.* configurations pertain to the TabletServer. Therefore, using server.background.threads for Coordinator is misleading. The original configuration, coordinator.io-pool.size, is more appropriate and self-explanatory.

Additionally, I found that Flink uses a similar configuration, cluster.io-pool.size, which defaults to 4 * CPU cores for both the Master and TaskExecutor. Given this, I suggest simply increasing the coordinator.io-pool.size to 10. This adjustment is straightforward and aligns with common practices.

Sorry for the back-and-forth on this matter. @swuferhong

Upon further review, I realized that all server.* configurations pertain to the TabletServer. Therefore, using server.background.threads for Coordinator is misleading. The original configuration, coordinator.io-pool.size, is more appropriate and self-explanatory.

Additionally, I found that Flink uses a similar configuration, cluster.io-pool.size, which defaults to 4 * CPU cores for both the Master and TaskExecutor. Given this, I suggest simply increasing the coordinator.io-pool.size to 10. This adjustment is straightforward and aligns with common practices.

Sorry for the back-and-forth on this matter. @swuferhong

Ok, I will change it back, but I feel that the ioExecutor can still be moved to be managed within the CoordinatorServer. I will keep this part of the code.

[swuferhong]

@wuchong comments addressed.

[wuchong]

COORDINATOR_IO_POOL_SIZE ?

[swuferhong]

TabletServer use BACKGROUND_THREADS as io param. Here I just noticed that there was no such check before, so I added it