receiver.hardwareAcceleration attribute instead of disableHardware[En|De]coding() static method

[jan-ivar]

It doesn't look like RTCRtpReceiver.disableHardwareDecoding() [update: and RTCRtpSender.disableHardwareEncoding()!] have shipped anywhere.

Mozilla feedback in mozilla/standards-positions#728 was originally negative (though a formal position never formed).

But after recent review, we'd like to propose a slightly different API shape that Mozilla would get behind:

partial interface RTCRtpReceiver {
  HardwareAcceleration hardwareAcceleration = "no-preference";  
}

// update: also
partial interface RTCRtpSender {
  HardwareAcceleration hardwareAcceleration = "no-preference";  
}

It's a receiver[ / sender] attribute instead of a static method.

Importantly for us, HardwareAcceleration is defined in WebCodecs, which has a page-long note about when the UA might override webpage preference.

We think this would resolve our concerns raised in the issue, specifically around the UAs ability to override website preference in situations like after a hardware codec fix.

[aboba]

Related: #98

TPAC 2022 slides

[henbos]

This is much better, a global static method had some obvious limitations, but being able to control a preference on a receiver does make sense.

[fippo]

Let me give you a brief recap since it has been a while:

• the API was intentionally designed to be cumbersome. As also noted in the mozilla position letting JS decide is not a great idea. But as the list-of-doom of hardware-acceleration related issues showed (and it has grown since) it is a practical problem.
• The API was for both senders and receivers (or encoders and decoders respectively). Encoders are typically the source of the issues like undershooting
• there was enough consensus to merge the PR, yet the "Mozilla position" (a conglomerate!) went the other way?
  2+ years later the suggestion is to go back to the drawing board?

@jan-ivar, three questions:

• how do you plan to address the (theoretical) fingerprinting concerns, e.g. by comparing decode times with and without hardware on different receivers (or senders)
• Does Firefox have hardware encoder and decoder support? This is somewhat hard to determine since Firefox still lacks both decoderImplementation as well as powerEfficientDecoder stats but the result of disabling the OpenH264 plugin suggests "no".
• What concrete issue do you have with decoders? Most of the time the issues are related to rate control and undershooting but that is on the encoder side

[jan-ivar]

• the API was intentionally designed to be cumbersome. As also noted in the mozilla position letting JS decide is not a great idea. But as the list-of-doom of hardware-acceleration related issues showed (and it has grown since) it is a practical problem.

Understood. The updated Mozilla position is we'd rather align with WebCodecs at this point.

Trading a prescriptive method for a declarative API should suffice to let UAs that so wish apply the same fingerprinting mitigation heuristics here and in WebCodecs.

This might include not honoring combinations of different values in the same top-level context, giving the same protections as the one-way top-level pageload gate suggested for the existing API.

• The API was for both senders and receivers (or encoders and decoders respectively). Encoders are typically the source of the issues like undershooting

Thanks, I've updated the proposal to include RTCRtpSender as well!

• Does Firefox have hardware encoder and decoder support?

Playback decode on all platforms. WebRTC on android where allow-listed. Behind pref on other platforms right now due to issues. Not sure about hardware encode.

• What concrete issue do you have with decoders? Most of the time the issues are related to rate control and undershooting but that is on the encoder side

I meant to include encode as well. Now updated.

[jan-ivar]

(@fippo sorry I accidentally edited your comment when I meant to reply! github pilot error on my part. Now restored I hope.)

[dontcallmedom-bot]

This issue was discussed in WebRTC November 19 2024 meeting – 19 November 2024 (Issue 229: receiver.hardwareAcceleration attribute instead of disableHardware[En|De]coding() static methods Issue 229: receiver.hardwareAcceleration attribute instead of disableHardware[En|De]coding() static methods)

[henbos]

There is also the risk that without this API assumptions such as "L1T3 will give us SW" and "L1T1 will give us HW" are more likely to be cemented, but so far nobody has wired up "fallback due to corruption" yet and even if L1T3 gave corruption there is always the option of switching to a different codec

[aboba]

This issue was discussed at the November 2024 virtual interim.

Resolution: no consensus on the proposal.