Use case for lighting

[KensakuKOMATSU]

@rjksmith , I'm now developing live screening system which synchronize lighting effect for each remote venues [1]. I'm realizing light synchronization makes great user experience. And, I may think that such an experience will be provided more casual way. Web page can realize such a great and immersive experience using WebVMT and DataCue.

Use case : When someone is watching live concert video on web page, lighting effect which is synchronized wth video is served automatically. The method of lighting effect will be varied. Simple scenario will be the background of the web page will be changed with sync. Highly immersive experience will be realized when external lighting devices, for instance dmx or smart light, will be changed their color or moving effect.

My proposal of VMT is like this.

00:00:07.000 --> 00:00:09.500
{
    "sync":
    { "type": "org.webvmt.example.lighting", "id": "light1", "data": 
        { "pan": 32, "tilt": 160, "dimmer": 255, "color": "rgb( 0, 0, 255 )" }
    }
}

My current scenario is user agent will parse the vmt files likewise shown above and added as DataCues. Then makes lighting effect synchronized with video.

The demo of "Simple lighting w/ DataCues" : [2] (polyfill)
The repository of above demo : [3]

[1] https://www.tvbeurope.com/media-delivery/ntt-com-partners-with-yamaha-to-trial-interactive-live-viewing-technology
[2] https://dmx-control-example.netlify.app/
[3] https://github.com/KensakuKOMATSU/dmx-control-example

[rjksmith]

@KensakuKOMATSU That's a fantastic demo of a novel use case. Many thanks for raising this.

1. Since your application involves live-streamed data, you may want to consider using unbounded cues which are supported by WebVMT for streaming. Your current VMT files use bounded cues that include end times for each cue. Unbounded cues allow the end time to remain unknown at the moment when the cue starts - such as the time of the next change after the current lighting change.
2. You might also consider designing your own custom cue to simplify access to the lighting data. Custom cues can be derived from DataCue or directly from TextTrackCue with methods tailored to the requirements of the particular use case. For example, a lighting cue might include custom methods such as:
   • getColorAsRGB() that returns the light's colour formatted as rgb(red, green, blue)
   • getColorAsHex() that returns the light's colour formatted as #rrggbb
   • getColorAsHSL() that returns the light's colour formatted as hsl(hue, saturation, lightness)

Further details of streaming and custom cues can be found in the vmt-sync-examples repo.

Please let me know if you have any questions. I'm happy to discuss the details of your use case and look forward to seeing how this progresses.

[KensakuKOMATSU]

@rjksmith Thanks!

I'm interested in Streaming use case of lighting scenario. Thanks for your comment. It helps a lot for considering details about it.

Custom cues are interesting way! Actually, some lighting device supports hue value, e.g. Phillips Hue, tailored methods are convenient for many cases.

Additionally, I'm interested in binary data for Data Cue. For instance, interface of DMX is 512 bytes of binary data. Though it is specific, in some case using binary data will be convenient.

I'll keep working with this use case. I'll post progress of this work in this issue.

[rjksmith]

@KensakuKOMATSU I've updated the vmt-sync-examples code variations to make the layout easier to read.

Support for binary data with DataCue is discussed in issue #15 and includes a simple demo as proof of concept. Exporting DMX binary data to VMT using Base64 may reduce the bandwidth in comparison to JSON encoding, and using custom cues could simplify access to those binary data. It would be interesting to compare the processing and bandwidth differences between JSON and binary encodings, and how these affect overall latency.

I also notice that your demo uses cuechange track events, which I agree is the correct choice for the lighting use case requirements so far. I'll consider adding more examples to vmt-sync-examples repo that explore the differences between cuechange track events and enter/exit cue events to highlight their strengths and weaknesses.

Thanks for your input. I'll keep you posted with progress.

[rjksmith]

@KensakuKOMATSU I've updated the vmt-sync-examples repo with a set of demos that use cuechange track events.

Hope this helps with your lighting use case. Please let me know if you have any questions.

[rjksmith]

@KensakuKOMATSU I’ve added a dash cam example to issue #15 that demonstrates synchronisation with two video streams - front and rear cameras. This may be useful for live screening events where footage is streamed from more than one camera simultaneously, and could allow a remote user to select their preferred viewpoint for greater interactivity.

For example, cheering or applause from a remote user could be routed to a speaker near their current camera in the live venue. Live performers could then hear from which direction that feedback was coming and may look towards that camera which would seamlessly integrate with feedback from the audience present at the venue.