Skip to content

[query] reduce uses of fs.open with compression - #15196

Draft
ehigham wants to merge 1 commit into
ehigham/scala-fs-extract-primitivesfrom
ehigham/limit-open-compression-uses
Draft

[query] reduce uses of fs.open with compression#15196
ehigham wants to merge 1 commit into
ehigham/scala-fs-extract-primitivesfrom
ehigham/limit-open-compression-uses

Conversation

@ehigham

@ehigham ehigham commented Nov 18, 2025

Copy link
Copy Markdown
Member

Change Description

This PR refactors stream handling in the codebase by introducing buffering and byte tracking utilities as extension methods. It adds RichCodeInputStream and RichCodeOutputStream classes with methods like buffer and trackBytes to simplify stream creation and management.

The changes remove direct instantiation of BufferedInputStream, BufferedOutputStream, and ByteTrackingOutputStream throughout the codebase, replacing them with the new extension methods. This standardizes stream handling and makes the code more consistent.

The PR also updates various file operations in EmitClassBuilder, MatrixWriter, TableWriter, and other classes to use these new utilities, ensuring consistent stream handling across the codebase.

Security Assessment

This change cannot impact the Hail Batch instance as deployed by Broad Institute in GCP

ehigham commented Nov 18, 2025

Copy link
Copy Markdown
Member Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@ehigham
ehigham force-pushed the ehigham/limit-open-compression-uses branch from 10dec30 to cb7c670 Compare November 20, 2025 04:01
@ehigham
ehigham force-pushed the ehigham/scala-fs-extract-primitives branch from 6079d3d to 0bcda90 Compare November 20, 2025 04:01
@ehigham
ehigham force-pushed the ehigham/scala-fs-extract-primitives branch from 0bcda90 to a460934 Compare January 5, 2026 18:04
@ehigham
ehigham force-pushed the ehigham/limit-open-compression-uses branch from cb7c670 to 2dbd0c9 Compare January 5, 2026 18:04
@ehigham
ehigham force-pushed the ehigham/scala-fs-extract-primitives branch from a460934 to 8909888 Compare February 19, 2026 15:14
@ehigham
ehigham force-pushed the ehigham/limit-open-compression-uses branch from 2dbd0c9 to ece11fd Compare February 19, 2026 15:14
@ehigham
ehigham force-pushed the ehigham/limit-open-compression-uses branch from ece11fd to 6eed16c Compare February 19, 2026 15:39
@ehigham
ehigham force-pushed the ehigham/scala-fs-extract-primitives branch from 8909888 to 1af1120 Compare February 19, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant