Feature: a `useVideo()` addon

[sc0ttj]

Create a useVideo addon that returns a video object that can be filtered, cropped, edited, etc.

It should extend Ctx canvas module, adding the extra methods needed for video editing:

• split audio and video into separate channels/streams
  • can control both at same time (for playback etc)
  • can control each on its own (for filters, etc)
  • control the audio channel/stream using useAudio
• filters:
  • cropping, zooming, panning, stretching, etc
  • colours/hues/sepia/etc
  • chromakey ("green screen", remove a colour)
• overlays:
  • timestamps, titles, branded lower thirds, etc
• can change all settings, even during playback
• save the edited video to video file or video element

Keep it small & simple:

• whole useVideo addon to be less then 15kb minified & gzipped
• work in browser and NodeJS (with node-canvas)
• no ffmpeg, only use the regular HTML5 Canvas API
• very similar API and usage with useAudio addon
• no other dependencies

Example:

const myVideo = useVideo({
  src: 'path/to/file.mp4',   // could also be a stream, File blob, <video> Element, webcam device, etc
  name: 'myVideo',
  container: canvasElem, // if not given, create an off-screen canvas to use instead
  startOffset: 0,
  endOffset: 0,
  filters: { 
    // ...filters and their settings here
  },
});

myVideo.play();

myVideo.settings({
  // ...change any settings defined above
});

myVideo.toElement(videoElem => { ... })

myVideo.saveAs('filename.mp4'); // raises the browsers "save file" dialog

Notes:

• Using WebM and HEVC video formats, you can have native transparent HTML5 video in all browsers as of late 2020.

Further reading:

• https://developer.mozilla.org/en-US/docs/Web/API/Media_Capture_and_Streams_API - main docs to look at
• https://developer.mozilla.org/en-US/docs/Web/API/MediaStream - media streams
• https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices - access webcams, microphones, screen shares
• https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection - for remote sources (rtc, media streams, etc)
• https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video - the video element docs
• https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API and video processing with WebCodecs

Other projects:

• https://github.com/bbc/videocontext (canvas video node objects, like AudioNodes)
• https://github.com/foo123/FILTER.js (video filters for browser and node) demo
• https://github.com/wix/kampos (real time effects using webGL) blog
• http://videocortex.io/2017/opencv-web-app/ (OpenCV compiled to WebAssembly via Emscripten) demo
• https://github.com/MattKetmo/darkroomjs (JS based image editor)
• https://github.com/saygiselim/video-pixel-processor (small examples)
• https://github.com/alyssaxuu/motionity (full editor app, works in browser, didn't load videos into canvas for me)
• https://github.com/brianchirls/Seriously.js/ (real time effects compositor, nodeJS based)
• https://github.com/catseye/Canvas-Feedback (real time effects)
• https://github.com/lindelof/awesome-web-effect (lots of repos, many video/image filter related)
• https://github.com/cloud-annotations/object-tracking-js (uses TensorFlow, works in browser)
• https://github.com/rendro/vintageJS (a "vintage look" hipster bullshit filter, but with lots of options)
• https://github.com/kilobtye/Potrace (JS port of Potrace - generates vectors from raster/bitmap data)

[sc0ttj]

Also may be useful:

• https://www.npmjs.com/package/offscreen-canvas (OffScreenCanvas polyfill, < 400 bytes)

[sc0ttj]

• https://github.com/lo-th/neo - nice timeline thing, with video example
• https://github.com/mrdoob/frame.js/ - another nice one, nice UI