Skip to content

Crash is run with 1 worker reports speed <1Β #102

@zwogorg

Description

@zwogorg

Classifications

  • Running python ./jellybench_py/hwi.py manually (within the Poetry shell) works and produces correct results.
  • This happens on a Virtual Hardware (VM, WSL, ...)
  • The issue occurs during the benchmark section of the script (after Starting Benchmark...).
  • A server issue or faulty server response is causing the problem.
  • jellybench was run in --debug mode (recommended for issue reports).
  • I am using the latest version (branch develop) of jellybench. I understand older versions are not supported.

Description

Hi,

I'm testing Jellybench on various systems. If the system can't handle a run with a speed of >= 1 Jellybench crashes with the following error:

Error Code or Output

> > > > Failed: ['performance']
| > > Current Test: 2160p - 720p
| > > > Current Device: cpu
| > > > ffmpeg command: ./jellybench_data/ffmpeg/ffmpeg_files/ffmpeg -c:v h264 -i /home/redacted/jellybench_py/jellybench_data/videos/Test%20Jellyfin%204K%20AVC%20150M.mp4 -noautoscale -an -sn -vf "scale=trunc(min(max(iw\,ih*a)\,528)/2)*2:trunc(ow/a/2)*2,format=yuv420p" -c:v libx265 -preset veryfast -b:v 3616000 -maxrate 3616000 -f null - -benchmark
| > > > > Starting run with 1 Workers... completed with speed 0.99
| > > > > Failed: ['limited']
Traceback (most recent call last):
  File "/home/redacted/jellybench_py/bin/jellybench", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/redacted/jellybench_py/jellybench_py/core.py", line 1045, in main
    return cli()
           ^^^^^
  File "/home/redacted/jellybench_py/jellybench_py/core.py", line 998, in cli
    valid, runs, result = benchmark(
                          ^^^^^^^^^^
  File "/home/redacted/jellybench_py/jellybench_py/core.py", line 360, in benchmark
    "single_worker_speed": max_pass_run_data["speed"],
                           ~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'speed'

Steps to Reproduce

Run Jellybench on a system that can't handle cpu intensive transcoding tasks

Operating System

Linux

System Architecture

x86_64 (64-bit)

Python Version

3.12.3

CPU[s]

Intel(R) Core(TM) i5-7500T CPU @ 2.70GHz

GPU[s]

Intel HD Graphics 630

Additional Context

No response

Possible Solution

Hotfix is to set "speed" and "rss_kb"to 0 if not set. Don't know how the API will handle this, though. Change line 360 and 361 in core.py from

"single_worker_speed": max_pass_run_data["speed"],
"single_worker_rss_kb": max_pass_run_data["rss_kb"],

to

    "single_worker_speed": max_pass_run_data.get("speed", 0),
    "single_worker_rss_kb": max_pass_run_data.get("rss_kb", 0),

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions