Track selection is barely usable if at all

[lkiesow]

Since #1230 has been merged, the track selection is a mess. It's very hard to understand what is going on and in some cases it's now unusable by default. That's not great. (That's partially my fault for stopping to review/complain about the patch due to lack of time). Still, right now the track selection has been deliberately broken for more than two tracks. This worked before:

But even if you have only two tracks, it's completely unclear to end-users what is supposed to happen:

Selecting just one video flavor

You have selected 1 audio track. It will be duplicated onto all videos.

What? I've selected just one stream? The text suggests that this selection will result in multiple videos. It shouldn't. It should just remove the presentation track.

Selecting one video but two audio streams

The message about the stream being copied over has been removed. But does that mean that the audio streams will be mixed and this will result in one video with one audio stream? Or will this result in an audio-only stream for presentation? That could lead to confusion when playing back the video later since our player should ignore the audio stream.

Additional Thoughts

• Before you could just remove an entire track (video + audio). This made it pretty clear what's happening. Personally, I would be happy to go back to that.
• This all becomes more complicated and more confusing if you deal with more than two video flavors. Let's say you have three tracks, you selected three video streams and two audio streams. What will be the resulting audio stream of the third video? No idea.
• Just always mixing all audio streams would theoretically fix the issue. Unfortunately, this also can easily lead to problems if two streams have the same audio and are ever so slightly out of sync (echo effect). Still, this may be the solution?
• If we don't want to mix audio and want users to define which stream goes where, we need a different interface.

[mtneug]

Could we go into the direction of assuming a single main and separate audio track (+ maybe audio tracks in other languages if people use that)? Late binding of video and audio in the player is a standard feature of HLS and DASH. In fact, it is required by modern formats such as CMAF.

[lkiesow]

So that would be the idea of having one video track and mixing all audio streams (assuming you don't have streams for separate languages) into one audio? That would at least make it clear to users what is happening and what to expect as an end result.

There are a few technical issues you need to solve for this though:

• How would you deal with downloads if you have separate A/V files?
• Mixing the (more or less) identical audio streams can lead to side effects like an echo effect if it's not 100% in sync. I remember this being a problem in the past and why “just mixing” was dropped
  • If we know it's the same audio, we could just drop one stream earlier in the workflow
  • Can we auto-detect that it's the same audio? That would make things a lot simpler.
  • We need to remember that we have a lot of material with duplicated audio streams in the archive

Since this issue is more about the UI/UX, it's we also need to think about how this would affect the UI – if at all.

[mtneug]

mixing all audio streams

No, never do that. I assume that recordings have the same audio track (maybe slightly shifted by some milliseconds). So in the long term, I could us see ingesting audio/source from the CA again, but we can also extract one audio track from presenter/source or presentation/source.

Question: isn't the editor already assuming one audio track? Do we support separate waveforms for presenter/source and presentation/source? I might be wrong.

How would you deal with downloads if you have separate A/V files?

If we go the HLS/DASH route, we would need to have a separate track for downloads anyways. Even when using fragmented MP4s, I don't want to guarantee that all native players support these files. I have seen Firefox only playing the first fragment when directly opening a local fragmented MP4, but when testing this again, this seems to be fixed today. Nonetheless, I already create non-fragmented tracks specifically for download. If you don't want to pay the storage price, you could also require a prior "export" workflow to run that muxes audio and video for downloads.

Since this issue is more about the UI/UX, it's we also need to think about how this would affect the UI – if at all.

Fair. Probably more of a data model discussion.

[audiehe]

Hi, I have created a redesign of the track editor for multiple videos. There are two current versions, in the process of getting user opinions. In the pdf attached, you can see each feature described in the prototypes.

Version 1 (simplistic)

Version 2 (visually oriented)

Prototype Features for Opencast Track Editor.pdf

[snoesberger]

Some thoughts from my side about this:

It is still important to us to have the option to deselect a single audio or video track. This is mainly due to our main use case: we have lecture rooms that can record a camera in addition to the slides. If the camera isn't used, a black screen with the room's audio is recorded. We record the audio with the camera to ensure perfect lip sync. Afterwards, we have to deactivate the presenter/camera video so that the camera audio is published with the presentation video.
It should also be possible to publish an audio-only (e.g. with a picture as video placeholder).

Regarding multiple audio tracks:
Muxing several audio tracks doesn't make sense to me. In most cases, only one audio track should be published. The only time it would make sense to publish audio tracks separately is if they are in different languages. So far, our university hasn't had any demand for videos with multiple language audio tracks. Are there any institutions that are already processing such videos in Opencast? Are single videos with multiple audio tracks currently supported at all? Could this be an edge case that we can ignore in the editor for now to keep things simple?

[audiehe]

Hi, thanks for your feedback! I understand your point about allowing users to deselect specific audio/visual tracks, and I agree that audio mixing doesn’t add much value in this context. Taking that into account, along with additional input from current video editors at Osnabrück University, I’ve created another iteration of the design.

Prototype: Opencast Track Editor Version 4

This version includes the following updates:

• Video preview: Users can preview the video in both default and custom settings. This provides a quick visual check of the final output and ensures the correct audio is being used.

• Refined visuals: The interface now features a more minimalistic aesthetic that aligns with the existing site design.

• Improved video selection: The video selection button has been replaced with a drop-down menu, making it visually easier to preview, add, or remove videos.

• Simplified audio selection: Audio tracks are now selectable via a drop-down menu, allowing only one to be active at a time.