Skip to content

Commit 87cd5aa

Browse files
authored
Add note on Parquet compression (#14154)
1 parent fe89360 commit 87cd5aa

File tree

1 file changed

+2
-0
lines changed
  • contents/docs/cdp/batch-exports

1 file changed

+2
-0
lines changed

contents/docs/cdp/batch-exports/s3.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,8 @@ We intend to add support for other common formats, and format-specific configura
7979

8080
Each file format supports a variety of compression methods. The compression method you choose can have a significant effect on the exported file size and the overall time taken to export the data. From our own internal testing, we would recommend using Parquet with zstd compression for the best combination of speed and file size.
8181

82+
> **Note on Parquet compression:** The compression type is included in the file extension, even for Parquet files. For example, files compressed with zstd will have the extension `parquet.zst`. Since compression is embedded in the format itself, the file should be read directly as a Parquet file and not uncompressed first.
83+
8284
### Manifest file
8385

8486
If you specify a max file size in your configuration, several files may be exported. In order to know when the export is complete, we send a `manifest.json` file (with the same prefix as the other files) once all the data files have been exported. This manifest file contains the key names of all the files exported.

0 commit comments

Comments
 (0)