server: raise DefaultCacheSize to 256MiB

Increase the default size of the storage engine's block cache from 128MiB to
256MiB. The 128MiB default is grossly inadequate, especially in the context of
Pebble's 64MiB memtable size which subtracts from the block cache.

With the old 128 MiB cache size, the block cache size is effectively zero. We
maintain one 64MiB memtable as the mutable memtable and one 64MiB memtable for
recycling. Both draw down from the block cache's memory, leaving nothing for
caching blocks.

We should consider updating the default to be a percentage of available memory
(see #128190), but updating to a percentage is a more involved change and runs
afoul of expectations of some tests. As a temporary stopgap, we double the
fixed default size of the block cache.

Informs: #128190
Epic: none
Release note (ops change): Raises the cache size for the storage engine's block
cache to 256MiB. Note production systems should always configure this setting.

Author: jbowens

pkg/server/config.go

@@ -56,12 +56,12 @@ import (
 // Context defaults.
 const (
 	// DefaultCacheSize is the default size of the Pebble cache. We default the
-	// cache size to 128MiB and SQL memory pool size to 256 MiB. Larger values
-	// might provide significantly better performance, but we're not sure what
-	// type of system we're running on (development or production or some shared
+	// cache size and SQL memory pool size to 256 MiB. Larger values might
+	// provide significantly better performance, but we're not sure what type of
+	// system we're running on (development or production or some shared
 	// environment). Production users should almost certainly override these
 	// settings and we'll warn in the logs about doing so.
-	DefaultCacheSize         = 128 << 20 // 128 MiB
+	DefaultCacheSize         = 256 << 20 // 256 MiB
 	defaultSQLMemoryPoolSize = 256 << 20 // 256 MiB
 	defaultScanInterval      = 10 * time.Minute
 	defaultScanMinIdleTime   = 10 * time.Millisecond