Skip to content

fix: correct LZ0 to LZO in compression options #995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jan 14, 2025
Merged

fix: correct LZ0 to LZO in compression options #995

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5cb1c0e
fix: correct LZ0 to LZO in compression options
kosiew Jan 13, 2025
0a00bd3
fix: disable LZO compression option and update tests to reflect its u…
kosiew Jan 13, 2025
e53844b
fix: ruff format expected string in test_execution_plan
kosiew Jan 13, 2025
bceef41
fix: update test for execution plan and add validation for invalid LZ…
kosiew Jan 13, 2025
3c7b68a
fix: remove LZO compression option and related test cases
kosiew Jan 13, 2025
af67813
ruff autoformat
kosiew Jan 14, 2025
1437ea0
fix: remove TODO comment regarding LZO compression implementation
kosiew Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions python/datafusion/dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,10 @@ class Compression(Enum):
GZIP = "gzip"
BROTLI = "brotli"
LZ4 = "lz4"
LZ0 = "lz0"
# TODO
# lzo is not implemented yet
# https://github.com/apache/arrow-rs/issues/6970
# LZO = "lzo"
ZSTD = "zstd"
LZ4_RAW = "lz4_raw"

Expand Down Expand Up @@ -696,10 +699,10 @@ def write_parquet(
- "snappy": Snappy compression.
- "gzip": Gzip compression.
- "brotli": Brotli compression.
- "lz0": LZ0 compression.
- "lz4": LZ4 compression.
- "lz4_raw": LZ4_RAW compression.
- "zstd": Zstandard compression.
Note: LZO is not yet implemented in arrow-rs and is therefore excluded.
compression_level: Compression level to use. For ZSTD, the
recommended range is 1 to 22, with the default being 4. Higher levels
provide better compression but slower speed.
Expand Down
3 changes: 3 additions & 0 deletions python/tests/test_dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1115,6 +1115,9 @@ def test_write_compressed_parquet_invalid_compression(df, tmp_path, compression)
df.write_parquet(str(path), compression=compression)


# TODO
# not testing lzo because it it not implemented yet
# https://github.com/apache/arrow-rs/issues/6970
@pytest.mark.parametrize("compression", ["zstd", "brotli", "gzip"])
def test_write_compressed_parquet_default_compression_level(df, tmp_path, compression):
# Test write_parquet with zstd, brotli, gzip default compression level,
Expand Down
2 changes: 1 addition & 1 deletion src/dataframe.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -491,7 +491,7 @@ impl PyDataFrame {
ZstdLevel::try_new(verify_compression_level(compression_level)? as i32)
.map_err(|e| PyValueError::new_err(format!("{e}")))?,
),
"lz0" => Compression::LZO,
"lzo" => Compression::LZO,
"lz4" => Compression::LZ4,
"lz4_raw" => Compression::LZ4_RAW,
"uncompressed" => Compression::UNCOMPRESSED,
Expand Down
Loading