Skip to content

Need to validate auto-partition time when creation and write new partition #2939

@loserwang1024

Description

@loserwang1024

Search before asking

  • I searched in the issues and found nothing similar.

Fluss version

0.9.0 (latest release)

Please describe the bug 🐞

Currently, when a user manually creates a partition via the Admin API or when the client dynamically creates a partition during a write, there is no validation that the partition time value is:

  1. Correctly formatted — matching the pattern expected by the configured TABLE_AUTO_PARTITION_TIME_UNIT (e.g., yyyyMMdd for DAY, yyyy for YEAR).
  2. Not out-of-date — i.e., not older than the earliest partition retained by the auto-partition retention policy (TABLE_AUTO_PARTITION_NUM_RETENTION).

This means a user can currently:

  • Create a partition like 2024-03-25 on a table with YEAR unit (wrong format), which will never be managed by auto-partition.
  • Create a partition that is already beyond the retention window, which will be immediately eligible for deletion.

Solution

Add a shared utility method PartitionUtils.validateAutoPartitionTime in fluss-common that:

  1. Returns early if auto-partition is not enabled for the table.
  2. Validates the partition value matches the format string for the configured time unit (e.g., yyyyMMdd for DAY).
  3. Validates the partition value is not older than currentTime - numToRetain units, throwing InvalidPartitionException with a descriptive message in both failure cases.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions