Resize frame to avoid "Unequal size tensor" error

[tjandrasg]

More details on #586
Update of #591

[nagadomi]

Maybe it would be better to resize using
frame_dim = [video_input_stream.codec_context.width, video_input_stream.codec_context.height]
before applying fps_filter (and user-specified vf filters), since the output resolution is determined in advance based on this.

nunif/nunif/utils/video.py

Lines 430 to 454 in b61ff5f

	def test_output_size(test_callback, video_stream, vf):
	video_filter = FixedFPSFilter(video_stream, fps=60, vf=vf, deny_filters=SIZE_SAFE_FILTERS)
	empty_image = Image.new("RGB", (video_stream.codec_context.width,
	video_stream.codec_context.height), (128, 128, 128))
	test_frame = av.video.frame.VideoFrame.from_image(empty_image).reformat(
	format=video_stream.pix_fmt,
	src_color_range=ColorRange.JPEG, dst_color_range=video_stream.codec_context.color_range)
	pts_step = int((1. / video_stream.time_base) / 30) or 1
	test_frame.pts = pts_step
	try_count = 0
	while True:
	while True:
	frame = video_filter.update(test_frame)
	test_frame.pts = (test_frame.pts + pts_step)
	if frame is not None:
	break
	try_count += 1
	if try_count * video_stream.codec_context.width * video_stream.codec_context.height * 3 > 2000 * 1024 * 1024:
	raise RuntimeError("Unable to estimate output size of video filter")
	output_frame = get_new_frames(test_callback(frame))
	if output_frame:
	output_frame = output_frame[0]
	break
	return output_frame.width, output_frame.height

However, it's unclear whether the input stream metadata represents the correct target(main?) resolution.

[nagadomi]

There's a discussion about this issue here, so I'll try creating various test data later.
https://superuser.com/a/1867062

[nagadomi]

The simplest solution would be to always add
f"scale={video_input_stream.codec_context.width}:{video_input_stream.codec_context.height}"
in FPSFilter. It's assumed that no resizing will occur if the resolution is the same. Well, I'll check this later too.

[tjandrasg]

Maybe it would be better to resize using
frame_dim = [video_input_stream.codec_context.width, video_input_stream.codec_context.height]
before applying fps_filter ...

Yes, after some test with variable frame size video, seems that the value of codex_context is the majority of frame size in the video (not necessarily the largest), this lead to minimal frame to be resized 👍 I also think that this is way better than using the first valid frame as reference size, so I applied the change ✅

Some question still remain though:

• Is it still need two pass to determine the largest frame size of the video?
• How to automatically preserve the aspect ratio (and padding black bar)? as discussed in the https://superuser.com/a/1867062

The simplest solution would be to always add
f"scale={video_input_stream.codec_context.width}:{video_input_stream.codec_context.height}"
in FPSFilter ...

Yes it could be simpler, but I'm unfamiliar with internal ffmpeg filter so I didn't test this yet. And also this will not warn the user if video has variable size frame.