High h265 decoding latency ( decoder buffers frames)

[Consti10]

Hello,

I am using your example code to investigate the behaviour of the rpi h265 decoder regarding frame buffering and decoding latency.

So far I have made the following observations:

1. after feeding a frame with avcodec_send_packet() , and sequentially polling avcodec_receive_frame() afterwards for one Second,
   The decoder still returns AVERROR(EAGAIN)
   The only interpretation for me here is that the decoder buffers at least one frame
   Aka frame N only becomes available after you feed frame N+1 to the decoder (or more).
   The used test file decodes with zero frame buffering on other platforms, for example ffmpeg sw decoder.

hacky testing code: https://github.com/Consti10/hello_drmprime/blob/0882f18e2e4bb3d5b5e6087cc95ac0b74d9d1c9b/hello_drmprime.cpp#L211

2. By looking at my debug output, it looks as if the decoder buffers exactly 2 frames. E.g. frame N becomes available after frame N+2
   has been fed to the decoder.

[Consti10]

pi@raspberrypi:~/hello_drmprime $ ./hello_drmprime ../ffmpeg_decoder/in/rv1126.h265 
No connector ID specified.  Choosing default from list:
Connector 32 (crtc 80): type 11, 1920x1080 (chosen)
Connector 40 (crtc 0): type 11, 0x0
Decode packet:0 size:90381 B
[hevc @ 0x5aa680] Hwaccel V4L2 HEVC stateless V1; devices: /dev/media3,/dev/video19
Timeout of 1 second reached
Fake fps:0.983284
Decode packet:90381 size:1078 B
Timeout of 1 second reached
Fake fps:0.98912
Decode packet:91459 size:1506 B
Decode delay:5.937 ms
(True) decode delay:2028.41 ms
Timeout of 1 second reached
Fake fps:0.990753
Decode packet:92965 size:16858 B
Decode delay:4.541 ms
(True) decode delay:2015.88 ms
Timeout of 1 second reached
Fake fps:0.991818
Decode packet:109823 size:2752 B
Decode delay:3.599 ms
(True) decode delay:2014.27 ms
Timeout of 1 second reached
Fake fps:0.992852

In this test I made the following two changes:

1. disable display output
2. After feeding a new h265 frame with avcodec_send_packet(), poll for up to 1 second to check if this frame becomes available (or rather if previosu frames become available with buffering)
3. some more debug output.

When interpreting this data, it looks as if the decoder buffers exactly 2 frames.

[Consti10]

(Fixed title, I sometimes write h264 out of habbit though meaning h265)

[Consti10]

While I am not 100% sure yet, I might have found a way to reduce the frame buffering to zero:

//decoder_ctx->thread_count = 3;
decoder_ctx->thread_count = 1;

Needs some verification though.