Skip to content

Avoid wasting a thread when building a hash table in parallel (#15919) #1433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
prestodb-ci wants to merge 32 commits into
base: oss-baseline
Choose a base branch
from
Open

Avoid wasting a thread when building a hash table in parallel (#15919) #1433

prestodb-ci wants to merge 32 commits into from

Conversation

@prestodb-ci
Copy link
Collaborator

@prestodb-ci prestodb-ci commented Nov 25, 2025 •
edited
Loading

Test PR for branch staging-7740ffc40-pr with head 7740ffc

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Nov 26, 2025

Test passed for commit 1777c7523, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/394/display/redirect for details

@prestodb-ci prestodb-ci deleted the staging-1777c7523-pr branch November 26, 2025 01:29
@zhouyuan @FelixYBW
fix: Enable CI on IBM repo
e962632 
Signed-off-by: Yuan <[email protected]>

Set ccache maximum size to 1G

Remove sed command from Gluten workflow

Removed a sed command that replaces 'oap-project' with 'IBM' in the get-velox.sh script.

Modify get-velox.sh to change 'ibm' to 'ibm-xxx'

Update the get-velox.sh script to replace 'ibm' with 'ibm-xxx'.

Update sed command to be case-insensitive

Update gluten.yml

fix iceberg unit test

Signed-off-by: Yuan <[email protected]>

Update gluten.yml

Enable enhanced features in gluten build script

Update cache keys for Gluten workflow
@FelixYBW FelixYBW restored the staging-1777c7523-pr branch November 27, 2025 09:14
@FelixYBW FelixYBW reopened this Nov 27, 2025
@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Nov 27, 2025

❌ Test commit 2a50cd131 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/2/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Nov 28, 2025

❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/4/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 1, 2025

❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/5/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 1, 2025

❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/6/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 2, 2025

❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/7/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 3, 2025

❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/8/display/redirect for details

@FelixYBW FelixYBW changed the title fix: Avoid TSAN data race during cache entry initialization (#15623) refactor: Extract common BaseSerializedPage API (#15626) Dec 3, 2025
@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 3, 2025

❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/9/display/redirect for details

@prestodb-ci prestodb-ci changed the title refactor: Extract common BaseSerializedPage API (#15626) fix(build): Ambiguity caused by long literal (#15670) Dec 3, 2025
@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 3, 2025

❌ Test commit a017fca78 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/427/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 3, 2025

❌ Test commit a236cf5b3 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/428/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 3, 2025

❌ Test commit 0f21ff954 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/429/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 3, 2025

❌ Test commit e01ca2187 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/430/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 3, 2025

❌ Test commit 4f39eef54 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/431/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 4, 2025

❌ Test commit eba05296b failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/437/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 4, 2025

❌ Test commit bf9e9fd6d failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/440/display/redirect for details

@prestodb-ci
Copy link
Collaborator Author

prestodb-ci commented Dec 5, 2025

❌ Test commit 4ff28c6bf failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/449/display/redirect for details

rui-mo and others added 28 commits January 8, 2026 19:44
@rui-mo
[OAP][5962] Support struct schema evolution matching by name
35161f7 
Alchemy-item: (ID = 917) [OAP] Support struct schema evolution matching by name commit 1/1 - c816a39
@rui-mo
[OAP][15173][15343]Allow reading integers into smaller-range types
64d92ea 
Alchemy-item: (ID = 883) [OAP] [13620] Allow reading integers into smaller-range types  commit 1/1 - 4cae2f5
@zhouyuan
[OAP][11771]fix: Fix smj result mismatch issue in semi, anit and full…
5877d1b 
… outer join

Signed-off-by: Yuan <[email protected]>

Alchemy-item: (ID = 882) [OAP] [11771] Fix smj result mismatch issue commit 1/1 - ada7dd2
@rui-mo
[OAP][14722] Fix memory leak caused by asynchronous prefetch
da27f22 
Alchemy-item: (ID = 954) [OAP] [14722] Fix memory leak caused by asynchronous prefetch commit 1/1 - aedf6b0
@PingLiuPing
Revert "feat: Support iceberg partition transform (facebookincubator#…
5ef34b6 
…15509)"

This reverts commit d791d84.

Alchemy-item: (ID = 983) Iceberg staging hub commit 1/18 - ec46102
@PingLiuPing
Revert "refactor: Add Iceberg connector (facebookincubator#15581)"
e050191 
This reverts commit fd0682b.

Alchemy-item: (ID = 983) Iceberg staging hub commit 2/18 - f7d02da
@PingLiuPing
Revert "feat: Add Iceberg partition name generator (facebookincubator…
c5bc57a 
…#15461)"

This reverts commit 7576f4e.

Alchemy-item: (ID = 983) Iceberg staging hub commit 3/18 - d7d8655
@PingLiuPing
Revert "feat: Add support for evaluating Iceberg partition transforms (…
0239c22 
…facebookincubator#15477)"

This reverts commit 1895711.

Alchemy-item: (ID = 983) Iceberg staging hub commit 4/18 - aee2a67
@PingLiuPing
Revert "feat: Add Iceberg partition name generation utility (facebook…
0b2ec4e 
…incubator#15443)"

This reverts commit 51d4a94.

Alchemy-item: (ID = 983) Iceberg staging hub commit 5/18 - 9ed4008
@PingLiuPing
Revert "feat: Add iceberg partition specification (facebookincubator#…
1d68dc2 
…15423)"

This reverts commit 600524b.

Alchemy-item: (ID = 983) Iceberg staging hub commit 6/18 - 710ba8d
@yingsu00
refactor: Move toValues from InPredicate.cpp to Filter.h
de5f391 
The function toValues removes duplicated values from the vector and
return them in a std::vector. It was used to build an InPredicate. It
will be needed for building NOT IN filters for Iceberg equality delete
read as well, therefore moving it from velox/functions/prestosql/InPred
icate.cpp to velox/type/Filter.h. This commit also renames it to
deDuplicateValues to make it easier to understand.

Alchemy-item: (ID = 983) Iceberg staging hub commit 7/18 - 8ff8f5b
@yingsu00
feat(connector): Support reading Iceberg split with equality deletes
4da89a6 
This commit introduces EqualityDeleteFileReader, which is used to read
Iceberg splits with equality delete files. The equality delete files
are read to construct domain filters or filter functions, which then
would be evaluated in the base file readers.

When there is only one equality delete field, and when that field is
an Iceberg identifier field, i.e. non-floating point primitive types,
the values would be converted to a list as a NOT IN domain filter,
with the NULL treated separately. This domain filter would then be
pushed to the ColumnReaders to filter our unwanted rows before they
are read into Velox vectors. When the equality delete column is a
nested column, e.g. a sub-column in a struct, or the key in a map,
such column may not be in the base file ScanSpec. We need to add/remove
these subfields to/from the SchemaWithId and ScanSpec recursively if
they were not in the ScanSpec already. A test is also added for such
case.

If there are more than one equality delete field, or the field is not
an Iceberg identifier field, the values would be converted to a typed
expression in the conjunct of disconjunts form. This expression would
be evaluated as the remaining filter function after the rows are read
into the Velox vectors. Note that this only works for Presto now as
the "neq" function is not registered by Spark. See https://github.com/
facebookincubator/issues/12667

Note that this commit only supports integral types. VARCHAR and
VARBINARY need to be supported in future commits (see
facebookincubator#12664).

Co-authored-by: Naveen Kumar Mahadevuni <[email protected]>

# Conflicts:
#	velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp

# Conflicts:
#	velox/dwio/common/ScanSpec.h

# Conflicts:
#	velox/type/Filter.h

Alchemy-item: (ID = 983) Iceberg staging hub commit 8/18 - 117b7f9
@PingLiuPing
Support insert data into iceberg table.
641624c 
# Conflicts:
#	velox/connectors/hive/HiveConnectorUtil.cpp

Alchemy-item: (ID = 983) Iceberg staging hub commit 9/18 - fd66e1e
@PingLiuPing
Add iceberg partition transforms.
cc29046 
Co-authored-by: Chengcheng Jin <[email protected]>

Alchemy-item: (ID = 983) Iceberg staging hub commit 10/18 - d501ccc
@PingLiuPing
Add NaN statistics to parquet writer.
d35468d 
Alchemy-item: (ID = 983) Iceberg staging hub commit 11/18 - 157015b
@PingLiuPing
Collect Iceberg data file statistics in dwio.
a3659ea 
# Conflicts:
#	velox/dwio/parquet/writer/Writer.cpp
#	velox/dwio/parquet/writer/Writer.h
#	velox/dwio/parquet/writer/arrow/ArrowSchema.cpp
#	velox/dwio/parquet/writer/arrow/ArrowSchema.h
#	velox/dwio/parquet/writer/arrow/Metadata.cpp

Alchemy-item: (ID = 983) Iceberg staging hub commit 12/18 - 6d3be8c
@PingLiuPing
Integrate Iceberg data file statistics and adding unit test.
2ddfe93 
Alchemy-item: (ID = 983) Iceberg staging hub commit 13/18 - 306dd1c
@PingLiuPing
Support write field_id to parquet metadata SchemaElement.
e5d413c 
Alchemy-item: (ID = 983) Iceberg staging hub commit 14/18 - 5a76f95
@PingLiuPing
Implement iceberg sort order
aefd6e3 
Alchemy-item: (ID = 983) Iceberg staging hub commit 15/18 - ea55d14
@PingLiuPing
Add clustered Iceberg writer mode.
d0722bb 
Alchemy-item: (ID = 983) Iceberg staging hub commit 16/18 - 973a1e4
@PingLiuPing
Fix parquet writer ut
03d2ce4 
Alchemy-item: (ID = 983) Iceberg staging hub commit 17/18 - bfbd027
@PingLiuPing
Add IcebergConnector
e73bf2f 
Alchemy-item: (ID = 983) Iceberg staging hub commit 18/18 - 836264c
@zhouyuan
adding daily tests
539f5c3 
Signed-off-by: Yuan <[email protected]>

Alchemy-item: (ID = 906) fix: Adding daily tests commit 1/2 - e2eb2c6
@FelixYBW
remote gluten daily build
f20b5f7 
we can cache ccache on every build even on failure, since ibm/velox is
always incremental build

Alchemy-item: (ID = 906) fix: Adding daily tests commit 2/2 - 0899ddc
@zhouyuan
fix: remove website folder to bypass the security issues
0a76915 
Signed-off-by: Yuan <[email protected]>

Alchemy-item: (ID = 956) fix: Remove website folder to bypass the security issues commit 1/1 - 42debeb
@zhouyuan
Disable flaky test: partialAggregateWithTableScan
2582cff 
Signed-off-by: Yuan <[email protected]>

Alchemy-item: (ID = 902) fix: Disable flaky test temporary commit 1/1 - 0bc5c50
@Zouxxyy
1
0e9fc69 
Alchemy-item: (ID = 988) Add fileNameGenerator to the constructor of IcebergInsertTableHandle commit 1/1 - a5f7e46
@wanglinsong
Merge branch 'ci-fix-pr' into staging-7740ffc40-pr
c203fd5 
Signed-off-by: Linsong Wang <[email protected]>
@prestodb-ci prestodb-ci changed the title Pass string buffers from encodings in selective reader (#15912) Avoid wasting a thread when building a hash table in parallel (#15919) Jan 8, 2026
@prestodb-ci prestodb-ci force-pushed the staging-1777c7523-pr branch from 1745050 to c203fd5 Compare January 8, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@majetideepak majetideepak Awaiting requested review from majetideepak majetideepak is a code owner

@jinchengchenghh jinchengchenghh Awaiting requested review from jinchengchenghh

@rui-mo rui-mo Awaiting requested review from rui-mo

Assignees

No one assigned

Labels

do_not_review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.