AIP-76 Implement PartitionBySequence and PartitionByProduct

[uranusjr]

Body

AIP-76

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[anishgirianish]

Hi @uranusjr I'd like to pick this up if it's available. I've been contributing to Airflow and have been following the AIP-76 work closely.

I've gone through the existing merged PRs (#57360, #58289, #59115, #60934) and the AIP spec to understand the current partition infrastructure. Here's my read of the implementation path , happy to be corrected:

PartitionBySequence

• New PartitionMapper subclass accepting an explicit ordered list of string keys (e.g.,["us", "eu", "asia"])
• Defined in airflow-core/src/airflow/partition_mapper/ with a mirrored definition in task sdk/src/airflow/sdk/definitions/partition_mapper/
• to_downstream() would validate the key exists in the sequence and pass it through
• serialize()/deserialize() to persist the sequence list

PartitionByProduct

• Combines up to one time-based dimension with multiple segment-based dimensions as the AIP describes
• Composes existing partition types and generates Cartesian product keys
• Key format would need a convention for compound keys (e.g., tuple serialization)

Shared work

• Register both in BUILTIN_PARTITION_MAPPERS in encoders.py
• Export from task-sdk/src/airflow/sdk/init.py
• Tests and documentation

A couple of questions before I start:

1. For PartitionBySequence should to_downstream() enforce that incoming keys are members
   of the defined sequence, or should it be more permissive?
2. For PartitionByProduct compound keys is there a preferred serialization format for multi-dimensional keys, or is that still an open design question?
3. I see @Lee-W's work on per-asset partition mapping (Extend PartitionedAssetTimetable to allow per asset partition #60966) I believe this is independent since I'm adding new mapper types rather than changing how mappers are assigned. Just want to confirm there's no coordination needed.

I'll open a draft PR early so you can steer the direction before I go too deep. Looking forward to contributing to AIP-76.

[anishgirianish]

Quick update, I noticed PR #61522 renames partition_mapper to partition_mappers and PR #61706 removed to_upstream from the base class. I'll base my work on top of these changes against the latest main.

[anishgirianish]

Hi, I've opened two PRs implementing the mappers described in this AIP:

• AIP-76: Implement SequenceMapper for partition key validation  #61931 — SequenceMapper for partition key validation
• AIP-76: Add ProductMapper for multi-dimensional partition keys #61937 — ProductMapper for multi-dimensional partition keys

Would appreciate a review when you get a chance. Thanks

[Lee-W]

It's the Lunar New Year this week. Neither of us will be available this week. Will try to take a look next week Thanks!