Title: Excessive CPU and RAM usage during compaction operations

[cagricakiroglu]

Compaction operations are causing extreme resource consumption, significantly impacting system performance.
Environment

Milvus version: 2.6.7
RAM: 256 GB

Questions

Is this resource consumption expected during compaction?
What configuration parameters can reduce CPU/RAM usage during compaction?
Are there best practices for compaction settings with high-volume batch inserts?

I have RAM available, but CPU appears to be the bottleneck throttling the compaction process. I need to scale to approximately 100M vectors with 512 dimensions.

index_params.add_index(
field_name="vector",
index_name="vector_index",
index_type="HNSW",
metric_type="COSINE",
params={"M": 30, "efConstruction": 360},
)

My current dataCoord is

dataCoord:
channel:
watchTimeoutInterval: 300
legacyVersionWithoutRPCWatch: 2.4.1
balanceSilentDuration: 300
balanceInterval: 360
checkInterval: 1
notifyChannelOperationTimeout: 5
segment:
maxSize: 1024
diskSegmentMaxSize: 2048
sealProportion: 0.12
sealProportionJitter: 0.1
assignmentExpiration: 2000
allocLatestExpireAttempt: 200
maxLife: 86400
compactableProportion: 0.85
expansionRate: 1.25

if you need my milvus.yaml :
milvus.yaml

And also;

[yhmo]

In the "CPU Usage", the bluse curve is high, but seems all nodes are marked as "-collection". Which node is blue?
The same with "Memory", I can't see which node is in high memory usage.

A normal chart is like:

[cagricakiroglu]

reduce index concurrency, reduce indexnode slot (if index build is too slow, it is not able to cover ingestion heavy case)

If I reduce the indexTaskSlot, isn't there more room for other indexes? Shouldn't I increase the indexTaskSlot so there's less room for other index tasks?

[cagricakiroglu]

Based on the milvus_datanode_compaction_latency_bucket metrics, I'm seeing frequent data compaction latency. Which parameters should I tune to reduce this latency?"

[yhmo]

For standalone, by default, a datanode, can process: 1 index task or 4 mix-compaction tasks or 2 L0 compaction tasks. It is configured in the milvus.yaml:

dataCoord:
  slot:
    clusteringCompactionUsage: 65535 # slot usage of clustering compaction task, setting it to 65536 means it takes up a whole worker.
    mixCompactionUsage: 4 # slot usage of mix compaction task.
    l0DeleteCompactionUsage: 8 # slot usage of l0 compaction task.
    indexTaskSlotUsage: 64 # slot usage of index task per 512mb
    statsTaskSlotUsage: 8 # slot usage of stats task per 512mb

dataNode:
  slot:
    slotCap: 16 # The maximum number of tasks(e.g. compaction, importing) allowed to run concurrently on a datanode

index task ---------------------------- build index for a segment
mix-compaction task -------------- merge several small segments into a larger segment
L0 compaction task ---------------- find delete entities in segments and mark them as "soft-deleted"
"slot" ----------------------------------- virtual unit for estimating the workload of task and the ability of datanodes

dataNode.slot.slotCap=16 ------------- each datanode's ability is 16 units
dataCoord.slot.mixCompactionUsage=4 ---------------- each mix-compaction task workload is 4 units
indexTaskSlotUsage = 64 ------------------------------------ each index task workload is 64 units

At the same time, each datanode can only process one index task, or 4 mix-compaction tasks.
If you enlarge the mixCompactionUsage to 16, then each datanode can only process one compaction task at the same time.

The latency of compaction task mainly depends on the IO of the storage, maybe reduce compaction concurrency can slightly reduce compaction latency.

[cagricakiroglu]

thanks for your response, how can i reduce the compaction concurrency ?

[yhmo]

Enlarge the dataCoord.slot.mixCompactionUsage from 4 to 16, then the compaction concurrency decreases from 4 to 1.