Skip to content

Organize WAL logs of all indexes into one file #210

@xiangwanpeng

Description

@xiangwanpeng

Describe This Problem

Now Tatris maintains a WAL file for each shard for the sake of clear classification, but this brings the I/O resource consumption of the same order of magnitude as the number of shards, the waste seems particularly unreasonable especially when most of shards only have sporadic write traffic.

Proposal

These scattered WAL logs can be organized into the same file, as long as we can distinguish the index and shard it belongs to for each line of log.

image

Additional Context

Please consider this issue together with issue #209 : when multiple shards' WAL logs are converged to a single file, maintaining a POINT for truncating expired logs seems to become more complicated, since you need to know exactly how many logs are consumed by all indexes in a certain period of time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceAll issues related to Tatris performance, including reporting, optimization, and regressions.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions