Fallback `ffmpeg` codecs support

[hidmic]

As of today, and to the best of my knowledge, a single codec may be chosen for the ffmpeg image transport plugin. This limitation goes all the way up to the ffmpeg_encoder_decoder package.

It would be ideal if there was support to specify a list of possible codecs, prioritized, with later codecs used as fallback for earlier ones. This would help accommodate systems that may run on multiple platforms, some of which may not have e.g. hardware support for specific codecs.

[hidmic]

@berndpfrommer would you accept such a feature if we were to contribute it?

FYI @calderpg @frneer.

[berndpfrommer]

In general I'm pretty open to PRs.
In this particular case though I do wonder how many people will use the proposed feature? Usually you will know what codecs are supported on the robots, and if the robots are heterogeneous, there normally is a specific set of parameters configured for each class of robots. Can you talk more about the specific use case you have in mind?
When/if you touch the ffmpeg_encoder_decoder, please make sure to keep the number of lines per function low. One of them is now already >120 lines long, very border line for my taste. So please feel free to clean up and package things such that the code remains readable.
FYI, these packages also depend on ffmpeg_encoder_decoder:

• foxglove_compressed_video_transport
• ffmpeg_image_transport_tools

[calderpg]

In this particular case though I do wonder how many people will use the proposed feature? Usually you will know what codecs are supported on the robots, and if the robots are heterogeneous, there normally is a specific set of parameters configured for each class of robots. Can you talk more about the specific use case you have in mind?

We (TRI) are the motivating case here: we are using AV1 via hardware encoding on robots (i.e. av1_nvenc or av1_qsv), and want to be able to specify a prioritized set of decoders on the user workstations/laptops so that the best available decoder for their hardware is selected (e.g. av1_qsv, libdav1d in that order). (It would also be nice to have similar encoder fallback, but decoder fallback is more important).

[hidmic]

@berndpfrommer thoughts? I can open a draft PR to discuss over code if it helps.

[berndpfrommer]

@hidmic Sure! Please submit a draft and I'll look at it!

[berndpfrommer]

@hidmic @calderpg have you started working on this project? I'm working on a project now where probing for the best decoder would be useful. In the process of fixing this we could also address issue #43 i.e. rather than puting "hevc_nvenc" in the message "encoding" field, we put just "hevc".

[hidmic]

@berndpfrommer I already parsed the codebase and have a sense of how I'll go about implementing it. Should have something up later this week (hopefully). If you need it sooner, feel free to pick it up -- do let me know though!

[berndpfrommer]

I'm working on it right now. Hope to get the first pieces working by tomorrow.

[berndpfrommer]

Started to work on the decoder side of things. Now moving to the ffmpeg image transport itself. Here is a link to the branch of the decoder

[hidmic]

Looking good. We can ping pong on the implementation once it makes it into a PR.

[berndpfrommer]

Here are the first steps towards using the fallback decoder in the ffmpeg_image_transport. I just completed the code but it failed on the first test. Will get to debugging probably Friday/Saturday.

[hidmic]

@berndpfrommer any updates? Do you need someone else to pick it up?

[berndpfrommer]

The fallback decoding kinda works. Please try this and this branch together. Still in very experimental stage.
Right now I'm working on getting rid the special casing for vaapi encoding. I'm struggling with finding the right (best?) pixel format for a given encoder. Somehow the command line ffmpeg figures it out automatically. I have to retrace how that is done.

I spent way too much time on trying to get parameters set for rqt_image_view and rqt_gui, only to learn that you cannot remap any topics or set parameters on those two programs. It works for the image_transport republish node, so I added a sample launch file for that.
What happens for the rqt_* programs right now is that the decoder, lacking any config hints from the parameter space, tries all available decoders, first hardware accelerated, then software. Since I have no laptop available right now with nvidia card, the nvidia decoder fails, and it falls back to software decoder. If you could test what happens if indeed *_cuvid decoding is selected, that would be great.

[berndpfrommer]

Updated the ffmpeg_encoder_decoder again. Please use this PR now.

[berndpfrommer]

Addressed with PR #45 . Closing this issue. Please open new issues for problems discovered during testing.