Topic read session: problems with memory consumption in case of high compression ratio #554
Description
There is a special field that is used for memory control in read session: https://github.com/ydb-platform/ydb/blob/main/ydb/public/api/protos/ydb_topic.proto#L429
The server code sets it as a serialized proto size: https://github.com/ydb-platform/ydb/blob/f38187482f0084fe7bc5ab264dbd7817a983fef9/ydb/services/persqueue_v1/actors/read_session_actor.cpp#L2083
As the messages in topic are stored in compressed state, this size presents a compressed data, that can differ a lot in case of high compression ratio. At the same time, YDB SDK sets bytes limit as settings.getMaxMemoryUsageBytes(), the limit considered by users as a memory limit for the session.
If we decompress all the received batch, it can lead to significant increase of the memory used by the session.
The proposed solution is not to decompress all messages, rather do it partially, so that the used amount of memory would be more controlable.