ethdb/pebble: lower the compaction debt (ethereum#31988)

This pull request reduces the threshold for triggering compaction at
level0, leading to less compaction debt. This change is helpful in the
case of heavy write-load, mitigating the case of heavy write stalls
caused by compaction.

closes ethereum#31830

Author: rjl493456442

ethdb/pebble/pebble.go

@@ -280,6 +280,17 @@ func New(file string, cache int, handles int, namespace string, readonly bool) (
 		// By setting the WALBytesPerSync, the cached WAL writes will be periodically
 		// flushed at the background if the accumulated size exceeds this threshold.
 		WALBytesPerSync: 5 * ethdb.IdealBatchSize,
+
+		// L0CompactionThreshold specifies the number of L0 read-amplification
+		// necessary to trigger an L0 compaction. It essentially refers to the
+		// number of sub-levels at the L0. For each sub-level, it contains several
+		// L0 files which are non-overlapping with each other, typically produced
+		// by a single memory-table flush.
+		//
+		// The default value in Pebble is 4, which is a bit too large to have
+		// the compaction debt as around 10GB. By reducing it to 2, the compaction
+		// debt will be less than 1GB, but with more frequent compactions scheduled.
+		L0CompactionThreshold: 2,
 	}
 	// Disable seek compaction explicitly. Check https://github.com/ethereum/go-ethereum/pull/20130
 	// for more details.