[server][improve] Add WAL cache to optimize replication.

[dao-jun]

Add WAL LogEntry cache to improve replication.
Bypass page-cache and eliminate deserialization overhead when tailing-read the WAL.

Under ideal conditions, the oxia_server_wal_read_latency_milliseconds_sum can be 0.

Test the WAL via wal-perf
before:

after:

The Read/Write throughput increase about 32%

[merlimat]

That's a cool addition. just a couple of comments

[merlimat]

It would be better if we could set weighter and use max number of bytes instead of entries here.

[dao-jun]

Resolved. But I don't understand what does set weighter mean, could you please explain in detail?

[dao-jun]

Perf test after address review comments, the performance is still OK

[mattisonchao]

LGTM +1

It would be better if you could consider this.

Oxia is a sharding systems. we need to consider more of cache memory control. currently, every shard has their own WAL. and I am not sure if we will go or when we will go for sharding WAL(IMO, we should go to avoid mMap cost by many opened segment). but anyway, we should pay attention on the cost.

• We need a global cache to avoid shards_num * 2MB. (100 shards = 200MiB， 1_000 shards ~ 2GiB). this is still very useful when we migrate to sharding WAL.

[dao-jun]

LGTM +1

It would be better if you could consider this.

Oxia is a sharding systems. we need to consider more of cache memory control. currently, every shard has their own WAL. and I am not sure if we will go or when we will go for sharding WAL(IMO, we should go to avoid mMap cost by many opened segment). but anyway, we should pay attention on the cost.

• We need a global cache to avoid shards_num * 2MB. (100 shards = 200MiB， 1_000 shards ~ 2GiB). this is still very useful when we migrate to sharding WAL.

I've considered this. Even in future single WAL instances, we still strive to distribute memory evenly among each shard. Otherwise, there will still be cache penetration, which is not much different from the current implementation

[mattisonchao]

I've considered this. Even in future single WAL instances, we still strive to distribute memory evenly among each shard. Otherwise, there will still be cache penetration, which is not much different from the current implementation

well... after deep thinking. I think the write cache could not help us very much in this case. because reader always happened after the data sync. If the write traffic is very large, the 2MiB buffer will never work as expected.

Indeed, your benchmark proof some improvement, but that logic is different as oxia. let me change some logics to make it match the implementation of Oxia. Plus, you could also use your implementation in the cluster benchmarking to see if any improvement on oxia_server_wal_read_latency_milliseconds_sum .