Skip to content

Is it feasible to implement sequence parallel?Β #3

@egangu

Description

@egangu

Since packing is usually combined with sequence parallel (SP). Is is feasible to add SP for your dpo packing repo?
Additionally, I am not sure if FlexAttention would conflict with SP-attention like ring-attention.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions