Add H.265 support for native & VideoStream
#10778
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hi! Thanks for opening this pull request.
Because this is your first time contributing to this repository, make sure you've read our Contributor Guide and Code of Conduct.
ntjohnson1
added
include in changelog
feat-video
ntjohnson1
added
enhancement
There was a problem hiding this comment.
thank you so much for looking into this! <3
I did a quick diff between the new ffmpeg.rs and the old one and it's pretty clear that we should just parameterize the old as almost all the code is c&p. I haven't dived deep but it may be as easy as passing in a codec parameter and then branching accordingly
Integration tests in crates/viewer/re_view_spatial/tests/video.rs need to be extended for the new codec.
Speaking of testing, how much manual testing have you done so far in terms of Browser and OS support - you mention Ubuntu, so only Ubuntu native then? That's of course something I can assist with as well.
Once we're done with all that the reference docs (docs/content/reference/video.md) also need some updates
| @@ -8501,6 +8510,7 @@ dependencies = [ | |||
| "bit-vec", | |||
| "cfg_aliases", | |||
| "criterion", | |||
| "cros-codecs", | |||
There was a problem hiding this comment.
neat library, and nice that it doesn't have any dependencies we don't already have. It seems it has the potential of unifying some of our other codec parsing as well - having h264-reader separately doesn't seem great when this one also offers all sort of parsing as well. I was previously considering to instead switch over to scuffle cloud's crates https://github.com/ScuffleCloud/scuffle/tree/main/crates for the same purpose as they looked very promising to me and are well maintained (albeit nascent)
Do you have any particular rationale for picking cros-codecs?
|
@Wumpf I moved all the code into a single ffmpeg_cli that receives the encoder as an argument. I still need to move the h264 sps parsing into cros_codecs, I started this but it looks like it's also being used for mp4, so I'll need to take a look to see what's needed there. I did only test on Ubuntu native, so I would be thankful for any further testing. |
|
Awesome! Looks really compact now! I'll try to have a look later today at the code and do some testing. |
There was a problem hiding this comment.
code lgtm (minus nits) but when I tested it with this simple video things didn't work out, giving me:
[2025-08-07T14:32:42Z WARN re_video::decode::ffmpeg_cli::ffmpeg] /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4, codec: H.265 HEV1 (hev1.1.6.L120.90) decoder: [hevc] [warning] Invalid NAL unit 5, skipping.
[2025-08-07T14:32:42Z DEBUG re_renderer::video::chunk_decoder] Error during decoding of /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4, codec: H.265 HEV1 (hev1.1.6.L120.90): FFmpeg error: [hevc] [error] missing picture in access unit with size 652356
[2025-08-07T14:32:42Z WARN re_video::decode::ffmpeg_cli::ffmpeg] /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4, codec: H.265 HEV1 (hev1.1.6.L120.90) decoder: [extract_extradata] [warning] Invalid NAL unit 5, skipping.
[2025-08-07T14:32:42Z WARN re_video::decode::ffmpeg_cli::ffmpeg] /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4, codec: H.265 HEV1 (hev1.1.6.L120.90) decoder: [hevc] [warning] Could not find codec parameters for stream 0 (Video: hevc, none): unspecified size
[2025-08-07T14:32:42Z DEBUG re_renderer::video::chunk_decoder] Error during decoding of /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4, codec: H.265 HEV1 (hev1.1.6.L120.90): FFmpeg error: [error] Cannot determine format of input 0:0 after EOF
[2025-08-07T14:32:42Z DEBUG re_renderer::video::chunk_decoder] Error during decoding of /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4, codec: H.265 HEV1 (hev1.1.6.L120.90): FFmpeg error: [vf#0:0] [error] Task finished with error code: -1094995529 (Invalid data found when processing input)
[2025-08-07T14:32:42Z DEBUG re_renderer::video::chunk_decoder] Error during decoding of /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4, codec: H.265 HEV1 (hev1.1.6.L120.90): FFmpeg error: [vf#0:0] [error] Terminating thread with return code -1094995529 (Invalid data found when processing input)
[2025-08-07T14:32:42Z DEBUG re_renderer::video::chunk_decoder] Error during decoding of /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4, codec: H.265 HEV1 (hev1.1.6.L120.90): FFmpeg error: [vost#0:0/rawvideo] [error] Could not open encoder before EOF
[2025-08-07T14:32:42Z DEBUG re_renderer::video::chunk_decoder] Error during decoding of /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4, codec: H.265 HEV1 (hev1.1.6.L120.90): FFmpeg error: [vost#0:0/rawvideo] [error] Task finished with error code: -22 (Invalid argument)
[2025-08-07T14:32:42Z DEBUG re_renderer::video::chunk_decoder] Error during decoding of /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4, codec: H.265 HEV1 (hev1.1.6.L120.90): FFmpeg error: [vost#0:0/rawvideo] [error] Terminating thread with return code -22 (Invalid argument)
seems that the annexb conversion isn't quite right
Big_Buck_Bunny_1080_1s_h265.mp4
This is the same video snippet that's used in the integration tests crates/viewer/re_view_spatial/tests/video.rs I've mentioned
Might be encoding specific, can you share what you used for testing?
For testing I modified the example from the docs to do on-the-fly encoding: """Video encode images using av and stream them to Rerun."""
import av
import numpy as np
import numpy.typing as npt
import rerun as rr
import time
fps = 30
duration_seconds = 240
width = 1280
height = 720
ball_radius = 30
def create_example_video_frame(frame_i: int) -> npt.NDArray[np.uint8]:
img = np.zeros((height, width, 3), dtype=np.uint8)
for h in range(height):
img[h, :] = [0, int(100 * h / height), int(200 * h / height)]
x_pos = width // 2
y_pos = height // 2 + 80 * np.sin(2 * np.pi * frame_i / fps)
y, x = np.ogrid[:height, :width]
r_sq = (x - x_pos) ** 2 + (y - y_pos) ** 2
img[r_sq < ball_radius**2] = [255, 200, 0]
return img
rr.init("rerun_example_video_stream_synthetic_h265", spawn=True)
# Setup encoding pipeline for H.265 (HEVC).
av.logging.set_level(av.logging.VERBOSE)
container = av.open("/dev/null", "w", format="hevc")
stream = container.add_stream("libx265", rate=fps)
stream.width = width
stream.height = height
stream.max_b_frames = 0
# Log codec as static metadata.
rr.log("video_stream", rr.VideoStream(codec=rr.VideoCodec.H265), static=True)
# record when we started
start_time = time.time()
# Generate frames and stream them.
for frame_i in range(fps * duration_seconds):
img = create_example_video_frame(frame_i)
frame = av.VideoFrame.from_ndarray(img, format="rgb24")
for packet in stream.encode(frame):
if packet.pts is None:
continue
rr.set_time("time", duration=float(packet.pts * packet.time_base))
rr.log("video_stream", rr.VideoStream.from_fields(sample=bytes(packet)))
target = start_time + frame_i / fps
now = time.time()
to_sleep = target - now
if to_sleep > 0:
time.sleep(to_sleep)
# Flush any remaining packets.
for packet in stream.encode():
if packet.pts is None:
continue
rr.set_time("time", duration=float(packet.pts * packet.time_base))
rr.log("video_stream", rr.VideoStream.from_fields(sample=bytes(packet))) |
emmadrigal
force-pushed
the
h265-support
branch
from
5ce0249 to
cd8b3ac
Compare
There was a problem hiding this comment.
nice! totally missed that the on-the-fly annex-b conversion was missing, makes sense and confirmed it working now against my sample mp4 and some others.
However, I still get this warning
[2025-08-12T13:42:56Z WARN re_video::decode::ffmpeg_cli::ffmpeg] /\//Users/andreas/dev/rerun-io/rerun/tests/assets/video/Big_Buck_Bunny_1080_1s_h265_nobframes.mp4, codec: H.265 HEV1 (hvc1.01.L120) decoder: [hevc] [warning] Invalid NAL unit 5, skipping.
for both https://github.com/rerun-io/rerun/blob/main/tests/assets/video/Big_Buck_Bunny_1080_1s_h265_nobframes.mp4 and https://github.com/rerun-io/rerun/blob/main/tests/assets/video/Big_Buck_Bunny_1080_1s_h265.mp4
Might be that the data is indeed not sound, either way I won't block from landing because of this (let's create a new issue for that if we land without a fix)
However, tests don't build yet and there's some style nits I'd like to see fixed. Once that's sorted out I'll do a x-platform round of testing and we can land. Thank you!! :)
emmadrigal
force-pushed
the
h265-support
branch
from
56447c0 to
38b484b
Compare
crates/viewer/re_view_spatial/tests/snapshots/video_stream_H264_beyond_end.png
Show resolved
Hide resolved
crates/viewer/re_view_spatial/tests/snapshots/video_asset_H265_beyond_end.png
Show resolved
Hide resolved
crates/viewer/re_view_spatial/tests/snapshots/transform_tree_origins__sun_planet.png
Show resolved
Hide resolved
|
I'm having trouble pushing to this PR, creating a new one here |
REOPENED AT #10994
Related
VideoStreamarchetype #10185What
This adds support for h265 using the same approach as for h264 throught ffmpeg, just changing the expected format.
Mostly this duplicates the code for h264 so I welcome any feedback into how to refactor this.
Tested this on Ubuntu.