Skip to content

Re-enable dedup pushdown #4790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Re-enable dedup pushdown #4790

wants to merge 2 commits into from

Conversation

@LantaoJin
Copy link
Member

@LantaoJin LantaoJin commented Nov 12, 2025 •
edited
Loading

Description

The dedup pushdown was disabled previously, because of the plan of join with max option can be optimized to collapse search. But the collapse search didn't support search_after which cause the join return an incorrect result.

This PR re-enable the dedup pushdown by distinguishing the plan of dedup command and join with max option command.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@LantaoJin
Enable dedup pushdown
8a1bf11 
Signed-off-by: Lantao Jin <[email protected]>
@LantaoJin LantaoJin added the enhancement New feature or request label Nov 12, 2025
@LantaoJin
fix doctest
d2a197f 
Signed-off-by: Lantao Jin <[email protected]>
@penghuo
Copy link
Collaborator

penghuo commented Nov 12, 2025

dedup optimization depend on opensearch-project/OpenSearch#19261. this improvement can not been backport to 2.19-dev?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ps48 ps48 Awaiting requested review from ps48 ps48 will be requested when the pull request is marked ready for review ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm will be requested when the pull request is marked ready for review kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho will be requested when the pull request is marked ready for review derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 will be requested when the pull request is marked ready for review joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen will be requested when the pull request is marked ready for review dai-chen is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB will be requested when the pull request is marked ready for review YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric will be requested when the pull request is marked ready for review mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar vamsimanohar will be requested when the pull request is marked ready for review vamsimanohar is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis will be requested when the pull request is marked ready for review Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo will be requested when the pull request is marked ready for review penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az will be requested when the pull request is marked ready for review seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz will be requested when the pull request is marked ready for review MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand will be requested when the pull request is marked ready for review Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha will be requested when the pull request is marked ready for review anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey will be requested when the pull request is marked ready for review forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto will be requested when the pull request is marked ready for review acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG will be requested when the pull request is marked ready for review GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 will be requested when the pull request is marked ready for review ykmr1224 is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger will be requested when the pull request is marked ready for review noCharger is a code owner

@qianheng-aws qianheng-aws Awaiting requested review from qianheng-aws qianheng-aws will be requested when the pull request is marked ready for review qianheng-aws is a code owner

@yuancu yuancu Awaiting requested review from yuancu yuancu will be requested when the pull request is marked ready for review yuancu is a code owner

@RyanL1997 RyanL1997 Awaiting requested review from RyanL1997 RyanL1997 will be requested when the pull request is marked ready for review RyanL1997 is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

backport 2.19-dev enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@LantaoJin @penghuo