WebCodecs Guides / Better WebCodecs documentation

[sb2702]

Hi everyone, my name is Sam,

I have a few years of experience writing WebCodecs+WebGPU applications like this, which are used in production by 200k+ users per month.

I'm interested in getting involved in MDN as a contributor to better document WebCodecs. LLMs routinely hallucinate broken WebCodecs code and a big reason is that the API is very poorly documented. A very basic example is that you need to specify a 'codec' parameter to use the WebCodecs APIs, but there is no list of valid values for that parameter, on MDN or W3C or anywhere else.

I built https://webcodecsfundamentals.org/, an actual real-world guide to building production WebCodecs applications, both from my experience and with contributions from other devs who've built popular WebCodecs libraries. I also have a dataset for valid values for 'codec' here: https://webcodecsfundamentals.org/datasets/codec-support-table/.

Beyond that standalone website though, is it possible to contribute the most relevant info on MDN iself? Like, the WebAudio API has a bunch of good practical guides, but the WebCodecs API has none.

My proposal would be to build a similar guide or series of articles under the WebCodecsAPI section, that go beyond hello world, and actually talk about production concerns, so developers getting started with WebCodecs can actually build working WebCodecs apps.

Starting this discussion to see if it's okay and/or discuss what the scope of such a series could be?

Tentative proposal:

• Intro to WebCodecs (what are codecs, use cases for the WebCodecs API)
• Basics (VideoEncoder/Decoder, VideoFrame, EncodedChunk - not the API, but conceptually what they are, how they work)
• Muxing and Streaming
• Working with Audio
• Working with Images
• Production concerns (GPU vs CPU, file handling, memory)
• Tutorial 1: How to Transcode
• Tutorial 2: How to build a player
• Tutorial 3: How to stream media with WebCodecs
• Tutorial n: whatever else there's demand for?
• Helpful Resources (https://webcodecsfundamentals.org/, https://mediabunny.dev/)

[hamishwillee]

Hi Sam,

Wow. The lack of responses here is not due to a lack of interest - the hearts on your post are from well established MDN reviewers and team members.

What you are suggesting seems to make sense, but if you aren't a domain expert (like me), it is hard to be sure. I certainly think it is worth starting building out the guide, in the same way as you have suggested above. I would use standard kinds of names/structures as the rest of MDN. Anything you did would be useful and go through a review process - I would be happy to look at your contributions.

That said, if it were me:

• I would update https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API with high level concepts.
• Add the first two topics you are suggesting and get reviews.
• Propose best way to indicate codecs that should be used
  • https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/Audio_codecs, https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/Video_codecs, https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/WebRTC_codecs all talk about codecs, but at way too high a level to know what you should use.
  • As a developer I would want to know what codecs my site must, should, could support.
  • For MDN I would tend to document the must-support codecs and maybe some should-support codecs - but after that I'd point to your list for the exhaustive reference (since we can't maintain it).

I am not so sure about the tutorials. Some of our guides are tutorial like, but they tend to be in the Learn section. I'm sure by the time you get to those we'll be able to better understand how that might integrate.

[sb2702]

Awesome, and no worries about the replies, I understand the team is very busy.

Your suggestion makes complete sense, I will start with the first two topics and focus on core concepts rather than "how-to" tutorials. I can also do a quick primer on codec selection since that is one of the first questions developers ask when setting up VideoEncoder.

I'll submit a PR with those first articles as a starting point for review.