pin index and bloom filter blocks to reader

[RaduBerinde]

I'm finding that 1%+ of CPU in TPCC (on c4-standard-32-lssd) is "loading" bloom filters and index blocks from cache (read lock, value ref counter, map lookup, updating category stats). We should "pin" these blocks inside the sstable reader for (at least) L0-L4.

Jira issue: PEBBLE-1303

[jbowens]

Once upon a time the block cache exposed 'weak handles' for exactly this purpose, but I think @petermattis removed them because they seemed to be a pessimization at least in some workloads at the time.

[jbowens]

#686

[RaduBerinde]

Interesting. In normal configurations and circumstances, these things take up a small fraction of the block cache for the upper levels, so we can get away with something much simpler and more efficient where we keep them around as long as the files are open. We just need a fail-safe limiting mechanism for unusual circumstances.

[petermattis]

#686 is the PR which removed weak handles. Simply pinning blocks in the cache would be simpler than that approach.