Crash if ffmpeg does not report speed or maxrss

[gingerbreadassassin]

Classifications

• Running python ./jellybench_py/hwi.py manually (within the Poetry shell) works and produces correct results.
• This happens on a Virtual Hardware (VM, WSL, ...)
• The issue occurs during the benchmark section of the script (after Starting Benchmark...).
• A server issue or faulty server response is causing the problem.
• jellybench was run in --debug mode (recommended for issue reports).
• I am using the latest version (branch develop) of jellybench. I understand older versions are not supported.

Description

When running jellybench in normal mode (non-debug), the script crashes during part of the progress bar rendering step. If ffmpeg does not output a "speed" or "maxrss", python raises a KeyError. This does not happen in debug mode as the progress bar isn't rendered.

Error Code or Output

Traceback (most recent call last):
  File "/usr/local/bin/jellybench", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 948, in main
    return cli()
           ^^^^^
  File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 910, in cli
    valid, runs, result = benchmark(
                          ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 357, in benchmark
    "single_worker_speed": max_pass_run_data["speed"],
                           ~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'speed'

OR

Traceback (most recent call last):
  File "/usr/local/bin/jellybench", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 948, in main
    return cli()
           ^^^^^
  File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 910, in cli
    valid, runs, result = benchmark(
                          ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/jellybench_py/core.py", line 358, in benchmark
    "single_worker_rss_kb": max_pass_run_data["rss_kb"],
                            ~~~~~~~~~~~~~~~~~^^^^^^^^^^
KeyError: 'rss_kb'

Steps to Reproduce

1. Start jellybench normally on a system where ffmpeg fails in a specific manner (see additional context)
2. ffmpeg fails and does not print speed or maxrss
3. Observe traceback

Operating System

Linux

System Architecture

x86_64 (64-bit)

Python Version

3.12.3

CPU[s]

AMD Ryzen 7 5700G with Radeon Graphics

GPU[s]

Cezanne [Radeon Vega Series / Radeon Vega Mobile Series]

Additional Context

I am running this in docker, according to my discussion here

Working ffmpeg benchmark

root@0d182ef0c537:/usr/src/jellybench_py# /usr/src/jellybench_py/jellybench_data/ffmpeg/ffmpeg_files/ffmpeg -init_hw_device vaapi=va:/dev/dri/by-path/pci-0000:0d:00.0-render -hwaccel vaapi -hwaccel_output_format vaapi -c:v hevc -i /usr/src/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20HEVC%2010bit%20150M.mp4 -noautoscale -an -sn -vf "scale_vaapi=-1:528:format=nv12" -c:v hevc_vaapi -b:v 79616000 -maxrate 79616000 -f null - -benchmark
ffmpeg version 7.0.2-Jellyfin Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 13 (Ubuntu 13.3.0-6ubuntu2~24.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto=auto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libharfbuzz --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100
amdgpu: os_same_file_description couldn't determine if two DRM fds reference the same file description.
If they do, bad things may happen!
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/usr/src/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20HEVC%2010bit%20150M.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf61.1.100
  Duration: 00:00:29.95, start: 0.000000, bitrate: 147664 kb/s
  Stream #0:0[0x1](und): Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(tv, bt709/bt709/unknown, progressive), 3840x2160 [SAR 1:1 DAR 16:9], 147661 kb/s, 60 fps, 60 tbr, 15360 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.3.100 hevc_videotoolbox
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> hevc (hevc_vaapi))
Press [q] to stop, [?] for help
[hevc_vaapi @ 0x5581a1a4e9c0] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf61.1.100
  Stream #0:0(und): Video: hevc (Main), vaapi(tv, bt709, progressive), 939x528 [SAR 2816:2817 DAR 16:9], q=2-31, 79616 kb/s, 60 fps, 60 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.3.100 hevc_vaapi
[out#0/null @ 0x5581a1a4a740] video:288661KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame= 1797 fps=127 q=-0.0 Lsize=N/A time=00:00:29.95 bitrate=N/A speed=2.11x
bench: utime=1.247s stime=0.995s rtime=14.199s
bench: maxrss=530632KiB

Broken ffmpeg benchmark

root@0d182ef0c537:/usr/src/jellybench_py# /usr/src/jellybench_py/jellybench_data/ffmpeg/ffmpeg_files/ffmpeg -c:v hevc -i /usr/src/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20HEVC%2010bit%20150M.mp4 -noautoscale -an -sn -vf "scale=trunc(min(max(iw\,ih*a)\,528)/2)*2:trunc(ow/a/2)*2,format=yuv420p" -c:v libx265 -preset veryfast -b:v 79616000 -maxrate 79616000 -f null - -benchmark
ffmpeg version 7.0.2-Jellyfin Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 13 (Ubuntu 13.3.0-6ubuntu2~24.04)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto=auto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libxml2 --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libharfbuzz --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/usr/src/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20HEVC%2010bit%20150M.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf61.1.100
  Duration: 00:00:29.95, start: 0.000000, bitrate: 147664 kb/s
  Stream #0:0[0x1](und): Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(tv, bt709/bt709/unknown, progressive), 3840x2160 [SAR 1:1 DAR 16:9], 147661 kb/s, 60 fps, 60 tbr, 15360 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.3.100 hevc_videotoolbox
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> hevc (libx265))
Press [q] to stop, [?] for help
x265 [info]: HEVC encoder version 3.5+1-f0c1022b6
x265 [info]: build info [Linux][GCC 13.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main profile, Level-5 (High tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 4 / wpp(5 rows)
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [warning]: VBV maxrate specified, but no bufsize, ignored
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
set_mempolicy: Operation not permitted
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 1 / 2
x265 [info]: Keyframe min / max / scenecut / bias  : 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt        : 15 / 4 / 0
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 2 / on / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : ABR-79616 kbps / 0.60
x265 [info]: tools: rd=2 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: fast-intra strong-intra-smoothing deblock sao
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf61.1.100
  Stream #0:0(und): Video: hevc, yuv420p(tv, bt709/bt709/unknown, progressive), 528x296 [SAR 296:297 DAR 16:9], q=2-31, 79616 kb/s, 60 fps, 60 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.3.100 libx265
      Side data:
        cpb: bitrate max/min/avg: 79616000/0/79616000 buffer size: 0 vbv_delay: N/A
[out#0/null @ 0x55d1559e4200] video:69831KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame= 1797 fps= 59 q=0.3 Lsize=N/A time=00:00:29.91 bitrate=N/A speed=0.988x
bench: utime=180.647s stime=2.023s rtime=30.286s
bench: maxrss=1003308KiB
x265 [info]: frame I:      8, Avg QP:0.00  kb/s: 37592.58
x265 [info]: frame P:    353, Avg QP:0.00  kb/s: 27251.53
x265 [info]: frame B:   1436, Avg QP:1.18  kb/s: 16984.90
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: consecutive B-frames: 0.6% 0.0% 0.0% 0.0% 99.4%

encoded 1797 frames in 30.13s (59.64 fps), 19093.40 kb/s, Avg QP:0.94

Note the missing speed and maxrss output. I have not looked into why they are missing, I just wanted to get the python script to better handle failures. As this is happening during the progress bar steps, I don't think it's all that critical to the output of the script.

Possible Solution

Minor code changes; I will open a PR with the fixes I used to get the tool to finish working.

[BotBlake]

If possible, try to find out WHY this info is missing!
I highly suspect its related to issues that ffmpeg is having!

Therefore handling this would be somewhat pointless, as it indicates a faulty runtime environment - In my oppinion at leat.

[theAeon]

If possible, try to find out WHY this info is missing! I highly suspect its related to issues that ffmpeg is having!

Therefore handling this would be somewhat pointless, as it indicates a faulty runtime environment - In my oppinion at leat.

just ran into this one w/ the ffmpeg that the program itself downloaded! so something's up