autoconverted parquet file has too big cells

[severo]

See https://huggingface.co/datasets/imvladikon/hebrew_speech_coursera/discussions/1#6523d448b623a04e6c2f118a

From the logs I see this error

TooBigRows: Rows from parquet row groups are too big to be read: 313.33 MiB (max=286.10 MiB)

It looks like an issue on our side: the row groups in the parquet files at https://huggingface.co/datasets/imvladikon/hebrew_speech_coursera/tree/refs%2Fconvert%2Fparquet/default/train are too big to be read by the api. We'll investigate this, thanks for reporting

[severo]

Launched the recreation of imvladikon/hebrew_speech_coursera.

[severo]

-> JobManagerCrashedError 😮

[lhoestq]

UnexpectedApiError for https://huggingface.co/datasets/danielz01/landmarks

libcommon.parquet_utils.TooBigRows: Rows from parquet row groups are too big to be read: 958.13 MiB (max=286.10 MiB)

[severo]

Note that the issue is that the cells are too big (in bytes) and it's not related to the row groups (I was mistaken in the title)

[lhoestq]

Same UnexpectedApiError for https://huggingface.co/datasets/osunlp/Mind2Web, row group is 564MB for 100 rows

[severo]

row group is 564MB for 100 rows

The issue is that we don't allow big "cells". What should we do? Improve the error message? Allow big cells? Truncate?

[lhoestq]

For the UI the best is to truncate, and a bonus would be to let the user click to expand a row

[severo]

so: I think we should add a query parameter, like: "full: boolean", or "truncate: boolean", to /rows, /search, /filter.