Surfacing binary timed metadata from a video file in a web browser

[rjksmith]

Complexity and bandwidth are significant challenges to accessing metadata embedded within a video file. Detailed knowledge of the video encoding format is required to successfully access the metadata packets within its structure. For example, OGC Testbed-16: Full Motion Video to Moving Features Engineering Report outlines the five steps required to access MISB metadata packets from a MPEG-2 Transport Stream file.

A useful first step for analysis is to export metadata verbatim from a video file to a VMT file. Decoupling metadata packets from a video file facilitates access - as described in the Testbed-16 video - and requires no knowledge of the metadata content itself.

Binary metadata can be exported as text to a VMT file in several ways, including:

1. Hexadecimal: Each data byte is encoded as two text characters that represent the hexadecimal value of the 8-bit number, such as '3F' to represent a value of 63. This format is verbose, though easy to edit.
2. Base64: Each 8-bit data byte is encoded into a 6-bit range using only printable ASCII characters. This format is cheap to compute and more compact than hexadecimal, but less easy to edit.

Integration with HTML allows WebVMT to synchronise the exported metadata with a video element in a web browser. Those metadata can then be surfaced with either DataCue or a custom cue based on TextTrackCue for rapid analysis of metadata content using reuseable tools and integrated with web services.

Timed metadata can be exposed in its original binary form in HTML so the data consumer may be agnostic of the text encoding. Once initial analysis has interpreted the binary metadata content, this step can be modified to expose the decoded metadata content in a more accessible form.

graph TD;
  A(Video file) --> |Export| B;
  B(Metadata packets) --> |Hex| C;
  B --> |Base64| C;
  C(VMT file) --> |DataCue| D;
  C --> |Custom cue| D;
  D(HTML) --> |Timed metadata| E;
  E(Data consumer);

Loading

[rjksmith]

Motion Imagery Standards Board (MISB) Example

MISB is a binary metadata standard designed to be embedded within a video stream using key-length-values (KLVs) and was discussed in OGC Testbed-16: Full Motion Video to Moving Features Engineering Report.

Integration of a prototype TextTrackCue polyfill based on the proposal in WICG DataCue with the WebVMT Javascript engine using the approach outlined above has enabled MISB metadata packets to surfaced as binary data in a web browser.

Figure 1: MISB metadata synchronised with video in a browser and surfaced in binary form using DataCue

This example is based on the MISB video file that was used in the Testbed-20 aerial observation video.

Binary metadata have been exported to a VMT file using Base64 encoding, then synchronised and decoded back to binary data by the WebVMT engine. DataCue has been used to successfully deliver the binary metadata - displayed as green text in the figure above - to web application code in a web browser.

This DataCue polyfill works in modern web browsers including Firefox, Chrome and Safari.

[rjksmith]

Dash Cam Example

Dash cameras typically store telemetry - such as location and accelerometer data - in a proprietary format embedded within the video files that they create. Most manufacturers provide a custom video player that can display this information, but few offer a feature to export these metadata or access them directly.

Vehicle telemetry may be stored in a metadata stream or subtitle track within the video file, though these details are often not published and some experimentation may be required to identify them. Open source tools such as exiftool and ffmpeg can help with this process.

Figure 2: Front dash cam video synchronised with binary Nextbase 322GW telemetry

This example is based on video recorded by a Nextbase dash cam during a collaboration at Ordnance Survey's headquarters in April 2024.

Binary metadata were exported to a VMT file with Base64 encoding as described above. These data are decoded back to their original binary form by the WebVMT engine and delivered to a web application using DataCue in a web browser. Most modern browsers are supported by this DataCue polyfill - including Firefox, Chrome and Safari.

Figure 3: Rear dash cam video synchronised with binary Nextbase 322GW telemetry