-
|
Is it currently possible to configure sling to write to parquet in different encodings. It seems like the default encoding used is DELTA_LENGTH_BYTE_ARRAY, which is not supported when reading into Redshift Spectrum. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
Hi, no it's not currently possible. |
Beta Was this translation helpful? Give feedback.
-
|
Its interesting that this encoding issue is happening since I imagine that the same data would be produced in the intermediate AWS Bucket if I were to do a direct transfer from BigQuery to Redshift. I might have to test that approach. I'm still learning the tools so I appreciate you taking the time to answer |
Beta Was this translation helpful? Give feedback.
-
|
Yea, CSV is still the most compatible format. Parquet can have compatibility issues depending on the engine reading/writing it. BigQuery, for example, cannot export a JSON column into a Parquet file. You'd just have to manually cast it to a string. But it exports a JSON column to CSV no problem. So that's why Sling uses CSV as the temporary file format for bulk loading into Redshift/BigQuery (via COPY). |
Beta Was this translation helpful? Give feedback.
-
|
That makes a lot of sense. Thanks for the extra context |
Beta Was this translation helpful? Give feedback.
Yea, CSV is still the most compatible format. Parquet can have compatibility issues depending on the engine reading/writing it. BigQuery, for example, cannot export a JSON column into a Parquet file. You'd just have to manually cast it to a string. But it exports a JSON column to CSV no problem. So that's why Sling uses CSV as the temporary file format for bulk loading into Redshift/BigQuery (via COPY).