-
Notifications
You must be signed in to change notification settings - Fork 538
Labels
⛃ re_datastoreaffects the datastore itselfaffects the datastore itself🏹 arrowApache ArrowApache Arrow💣 crashcrash, deadlock/freeze, do-no-startcrash, deadlock/freeze, do-no-start🔩 data modelSorbetSorbet🪳 bugSomething isn't workingSomething isn't working
Description
Describe the bug
Large video files cause failures.
To Reproduce
Steps to reproduce the behavior:
- Acquire or generate a large (2.1GB or more) video. E.g.
ffmpeg -f lavfi -i testsrc -t 36000 -pix_fmt yuv420p -c:v libx264 -g 1 large.mp4
- Attempt to open this with rerun:
cargo run -r -- large.mp4
- See error:
thread 'rayon-1' panicked at 'offset overflow'
arrow-buffer-55.2.0/src/buffer/offset.rs:132
Expected behavior
Large video should display in rerun viewer.
Backtrace
arrow-buffer-55.2.0/src/buffer/offset.rs:132
stack backtrace:
8: core::panicking::panic_fmt
9: core::option::expect_failed
10: <re_types::datatypes::_datatypes::blob::Blob as re_types_core::loggable::Loggable>::to_arrow_opt
Rerun version
current main branch ( now at 204025d ) and previous, including releases.
Additional context
This seems to be related to video data being stored as a re_chunk::Blob
which currently is serialized using arrow's arrow::array::ListArray
which takes i32 offset which are too small for blobs greater than 2.1GB.
Metadata
Metadata
Assignees
Labels
⛃ re_datastoreaffects the datastore itselfaffects the datastore itself🏹 arrowApache ArrowApache Arrow💣 crashcrash, deadlock/freeze, do-no-startcrash, deadlock/freeze, do-no-start🔩 data modelSorbetSorbet🪳 bugSomething isn't workingSomething isn't working