Skip to content

Mvexpand feature #4675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open

Mvexpand feature #4675

wants to merge 32 commits into from

Conversation

@srikanthpadakanti
Copy link

@srikanthpadakanti srikanthpadakanti commented Oct 27, 2025

Description

This pull request adds native support for the mvexpand command in PPL to OpenSearch SQL, enabling users to expand multivalue fields (arrays) into separate rows directly within queries. This functionality is analogous to Splunk's mvexpand command and streamlines analytics, dashboarding, and data preparation involving arrays or multivalue fields.

Key features introduced:

Native mvexpand command for PPL queries to expand array fields into separate rows/events.
Optional limit parameter to restrict the number of expanded values per event/document.
Robust handling of empty/null arrays, large arrays (with memory/resource limits), and non-array fields.
Streaming/distributable execution for performance and scalability.
Comprehensive documentation and edge case coverage.
This feature makes OpenSearch SQL more powerful and user-friendly for log analytics, data exploration, and migration from platforms like Splunk.

Related Issues

Resolves #4439
#4439

Check List

  • [ X] New functionality includes testing.
  • [ X] New functionality has been documented.
  • [ X] New functionality has javadoc added.
  • [ X] New functionality has a user manual doc added.
  • [ X] New PPL command checklist all confirmed.
  • [ X] API changes companion pull request created.
  • [ X] Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Srikanth Padakanti and others added 9 commits October 27, 2025 10:44
Initial checkpoint - following calcite way and commented legacy way
445799f 
Signed-off-by: Srikanth Padakanti <[email protected]>
Removed the build.gradle dependency opensearch-common
427af0f 
Signed-off-by: Srikanth Padakanti <[email protected]>
Ready to submit this PR
83d7786 
Signed-off-by: Srikanth Padakanti <[email protected]>
Ready to submit this PR
7f6e127 
Signed-off-by: Srikanth Padakanti <[email protected]>
Ready to submit this PR
dcbf56b 
Signed-off-by: Srikanth Padakanti <[email protected]>
Add mvexpand.rst
a3c1384 
Signed-off-by: Srikanth Padakanti <[email protected]>
Add Tests
148ccc5 
Signed-off-by: Srikanth Padakanti <[email protected]>
Add the mvexpand.rst to the index.rst
f5a9e82 
Signed-off-by: Srikanth Padakanti <[email protected]>
@srikanthpadakanti
Merge branch 'opensearch-project:main' into main
2cc60ad
Srikanth Padakanti added 2 commits November 3, 2025 13:00
Fix the failing test
6c6e0ec 
Signed-off-by: Srikanth Padakanti <[email protected]>
Fix the failing test
069d52e 
Signed-off-by: Srikanth Padakanti <[email protected]>
@srikanthpadakanti
Copy link
Author

srikanthpadakanti commented Nov 3, 2025 •
edited
Loading

Hello @dai-chen There are some tests failed in Security Plugin IT - unrelated to my mvexpand changes.
@ykmr1224

dai-chen
dai-chen reviewed Nov 4, 2025
View reviewed changes
@ykmr1224
Copy link
Collaborator

ykmr1224 commented Nov 6, 2025

#4675 (comment)
From this comment, it looks like an improvement for existing expand command.
Why don't we improve expand at the same time, and consider mvexpand as and alias of expand?

ykmr1224
ykmr1224 reviewed Nov 6, 2025
View reviewed changes
@srikanthpadakanti
Copy link
Author

srikanthpadakanti commented Nov 7, 2025

#4675 (comment) From this comment, it looks like an improvement for existing expand command. Why don't we improve expand at the same time, and consider mvexpand as and alias of expand?

Refactored as requested.
Improved the "expand" command and aliased the - "mvexpand" to it.

Address the PR comments
bf018b7 
Signed-off-by: Srikanth Padakanti <[email protected]>
dai-chen
dai-chen previously approved these changes Nov 11, 2025
View reviewed changes
Copy link
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!

Merge upstream changes to my main
a6ccb5a 
Signed-off-by: Srikanth Padakanti <[email protected]>
Srikanth Padakanti added 2 commits November 13, 2025 17:39
Address the PR comments
7743ad0 
Signed-off-by: Srikanth Padakanti <[email protected]>
Address the PR comments
1e03131 
Signed-off-by: Srikanth Padakanti <[email protected]>
Address the PR comments
dbea3a2 
Signed-off-by: Srikanth Padakanti <[email protected]>
@dai-chen
Copy link
Collaborator

dai-chen commented Nov 14, 2025

Please check CI failure.

@srikanthpadakanti
Copy link
Author

srikanthpadakanti commented Nov 15, 2025

Please check CI failure.

@dai-chen The CI failures are not related to mvexpand. I merged the upstream changes to. my main. CI failure is pointing to -
testMvindexSingleElementPositive, testMvindexSingleElementNegative, testMvindexRangePositive, testMvindexRangeNegative in CalcitePPLArrayFunctionTest.
The assertion failures are string-equality checks of generated SQL (PPL → SparkSQL text). The RelNode / logical plan printed in the local run shows ITEM(...) and ARRAY_SLICE(...) expressions — i.e. the array functions are present and being produced, but the exact string formatting (parentheses, unary +/-, expression grouping) differs from the test expectation used in CI.

My local run shows those tests pass. But, CI failed. That implies either:
A difference in environment (JDK/OS/Calcite pretty-printer) between my local and CI, or
Different code in CI (different branch/commit or cached build) than my local, or
Tests were updated in one place and translator/pretty printing in another.method in CalcitePPLArrayFunctionTest. @ahkcs Can you please confirm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dai-chen dai-chen dai-chen left review comments

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar vamsimanohar is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@LantaoJin LantaoJin Awaiting requested review from LantaoJin LantaoJin is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

@qianheng-aws qianheng-aws Awaiting requested review from qianheng-aws qianheng-aws is a code owner

@yuancu yuancu Awaiting requested review from yuancu yuancu is a code owner

@RyanL1997 RyanL1997 Awaiting requested review from RyanL1997 RyanL1997 is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

enhancement New feature or request PPL Piped processing language

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add native support for mvexpand command in PPL

3 participants

@srikanthpadakanti @ykmr1224 @dai-chen