Skip to content

GH-3163: Reduce memory and time overhead of ParquetRewriterTests #3164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wgtmac merged 3 commits into from
Mar 4, 2025
Merged

GH-3163: Reduce memory and time overhead of ParquetRewriterTests #3164

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bf01d89
Reduce number of records in ParquetRewriterTest to 10000
rahulketch Feb 26, 2025
d20ed14
Parameterize the tests on numRecord (default=10000) and rowsPerPage (…
rahulketch Feb 28, 2025
abe04fd
fix formatting by doing spotless:apply
rahulketch Mar 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion parquet-hadoop/src/test/java/org/apache/parquet/hadoop/rewrite/ParquetRewriterTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@
@RunWith(Parameterized.class)
public class ParquetRewriterTest {

private final int numRecord = 100000;
private final int numRecord = 10000;
Copy link
Member

@wgtmac wgtmac Feb 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may result in a single page for each column chunk. Could you try following things:

Copy link
Contributor Author

@rahulketch rahulketch Feb 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wgtmac : thanks for your comment. Do you happen to know how I can run a single test in the repository after making my changes? I wanted to run only the ParquetRewriterTest, but have not figured out a good way to achieve that via mvn.

Copy link
Member

@wgtmac wgtmac Feb 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

cd ~/Projects/parquet-java   # replace with your project root directory

cd parquet-hadoop

mvn test -Dtest=org.apache.parquet.hadoop.rewrite.ParquetRewriterTest

Copy link
Contributor Author

@rahulketch rahulketch Feb 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I understand, the relevant config parameter is parquet.page.row.count.limit and I have changed that to be num_records / 5. Hope it looks good!

private final Configuration conf = new Configuration();
private final ParquetConfiguration parquetConf = new PlainParquetConfiguration();
private final ParquetProperties.WriterVersion writerVersion;
Expand Down