RFC: Shared Control of hardware between Automation Systems and Users

[mint-dewit]

About Me

This RFC is posted on behalf of the BBC.

Use Case

We are addressing a scenario where a user may wish to control a video and/or audio mixer directly while the same system is otherwise being controlled by Sofie.

In this scenario a user does a take in Sofie which takes input A on air. The user proceeds to directly control the vision mixer and cut to input B. After this the user does another take in Sofie which should take input A on air again.

Note that in the current architecture Sofie is not aware of any external changes and will not send a command for the second take as it assumes input A is already on-air.

Proposal

Note: most of these changes are to be done in the Timeline State Resolver library, I am posting here as the use case is from Sofie's perspective and I believe posting here may give more visibility

Short gist: introduce tracking of both Actual State (i.e. what is currently on the device) and the Expected State (ie. what the state should be according to the timeline) for portions of the device (i.e. track each channel of an audio mixer separately). When the Actual State is updated to be different from the Expected State we may assume the device is being controlled externally and we “block” our control of this part of the device until some signal is received to retake control.

In order to achieve this:

• Introduce a StateTracker class in the TSR to track both the expected/internal and actual/external state of a device
• The Actual State is to be derived from information the device sends us, the Expected State is to be derived from the Timeline
• Track small parts of the entire state individually, i.e. a mix effect or audio channel. Call this an AddressState
• Each small part of state, the AddressState, must have an Address, an index by which we can look up the stored state
• Each Address can be blocked individually, when an Address is blocked the TSR integration is not allowed to control the part of the device described by the AddressState
• Each Address has a ControlValue, when this value changes the TSR integration should resume control of the part of the device described by the AddressState. This ControlValue can be a string expressed on a timeline object. Changing this string is the only way to unblock an Address currently. The external device can only block control and never unblock as we assume the external device does not understand what a TSR is.
• When the Actual State for an Address is updated, it shall be compared to the Expected State for that Address and when they are not the same the Address shall be blocked

An integration wanting to implement Shared Control shall implement the following:

• A function that can compare 2 AdressStates and returns a boolean value that shall be true if and only if the 2 states are not the same
• A mechanism that takes updates from the external device and converts into AddressStates
• A function that can generate AddressStates from an internal Device State (Note - a Device State is currently the output of the diffState method on the integration)
  • Alternatively, the Address States may be generated in the same call that currently converts a Timeline State into a Device State
• A way to integrate an AddressState into it’s state diff mechanism
  • This will be in the form of a function that can take a Device State and an array of Address States and override the Device State with the contents of the Address State.
  • This process must also be able to understand when the Device State will not be overridden - i.e. the Address shall be unblocked by this state change.

The TSR shall provide

• Tracking of the current Expected State for every Address
• Blocking Addresses
  • Based on the output of the diff function provided by the integration

The general flow shall be as follows

1. A new timeline state is generated from the timeline
2. The integration transforms the timeline state into a device state, as well as Address States. Note - there may be devices that wish to only use AddressStates so perhaps an option here to omit the device state completely?
3. The StateHandler overrides any parts in the device state that are blocked using the Address States

• For each address, a function is called to decide whether the address should still be blocked after the state change (As of yet I’m a bit uncertain whether the Control Value should be something inside of the Address State or separately tracked in the StateTracker, I’ll probably try out both for the final implementation and see what works better)

4. The integration performs a diff on the current expected state and the next expected state, resulting in commands

• The blocked addresses are already overridden in the states here, the integration does not have to understand whether something is blocked nor why it is.

5. When the next state becomes active, the integration sends each command to the device

• The command sending logic has no knowledge of the current state, any blocking is done by adjusting the states fed to the diff
• Internally, the StateHandler will update the Expected State on the State Tracker by using the Address States provided in step 2.

Note: we could provide some standard state and diff functions that could be replaced by integrations if they want to? i.e. atem needs a special device state object for the libraries but sisyfos might just be able to only use AddressStates

The reverse flow (ie. the device feeding updates to TSR) shall be as follows:

1. The integration receives an update from the device on its connection
2. The integration creates an Address State corresponding to this update
3. The created address state is stored in the state tracker, if the address is not blocked it is fed to a diff function together with the expected state. If the new actual state is different from the expected state the address is blocked

Sequence Diagram

Timing and edge cases

Race conditions are inevitable when dealing with concurrent systems. When the integration updates the actual state we must decide whether that update came from TSR or an external system. The current prototype implementation aims to mitigate some of these issues by introducing a delay. It always waits 200ms before deciding if an Address is to be blocked. The exact amount of how long to wait is not set in stone for this RFC and perhaps should be configurable per integration. A lower value could reduce delay in blocking the Address, that being said this is only relevant when a user is controlling the device just before the automation sends a command. In the current use case that is not as likely.

If the user manually sets an input while Sofie expects another but then switches back before Sofie has unblocked the Address the Address currently remains blocked. Once we are in a blocked state for the Address only a change in Control Value can unblock the Address.

An important thing to consider here is that the integration needs to be able to receive consistent updates from the external device for this to work. If the integration sends a command based on the the state changing and the device sends a response back that is inconsistent with the integration's expectation the Address would be blocked until a new control value is received. Depending on how the Control Value is used by the library user (or blueprint developer in Sofie's case) this may cause problematic and unexpected behaviour. One solution could be to expose an option to disable Shared Control, this would not be ideal but at least give users a mitigation that does not involve a bug fix in code. Alternatively, we could consider ignoring any updates that follow directly after we send a command although that could also introduce additional race conditions if the user were to override an erroneous automation decision. (TBD what "Directly" means)

Future extensions

• This RFC describes the Control Value being expressed on the Timeline, currently this in the form of a timeline object but it could also be using a timeline object's id, start time or some other property. Additionally, an extension could be made where a Control Value could be changed by using a TSR Action. This would give the user more explicit control over when control should be given back to Sofie without having to manipulate the timeline. The integration developer would have to take special care to understand how the control values coming from the timeline and the values coming from the action would interact in that case.
• It may be useful for the user to be able to explicitly be informed in the Sofie UI that a certain part of a device is not fully under control of the automation. Currently Sofie is not aware of Addresses so either it needs to become aware of this or alternatively we could inform Sofie that a mapping is no longer fully under control. Sofie could then attach this information to pieces that are affected by this.
• If the blueprints would be aware of how a device's state has been changed by the user it could react to it by manipulating pieces in Sofie and it would be possible to implement features such as AFV. This could perhaps be done by exposing the interfaces of the Address States in the timeline-state-resolver-types package. I think this would be a larger undertaking and probably quite far out of scope for now.

Prototype

This RFC is based on the work done in a prototype done for BBC. Note that this prototype doesn't have the same abstractions as described in this RFC but the basic mechanisms of state tracking, diffing and blocking are the same.

Process

The Sofie Team will evaluate this RFC and open up a discussion about it, usually within a week.

• RFC created
• Sofie Team has evaluated the RFC
• A workshop has been planned
• RFC has been discussed in a workshop
• A conclusion has been reached, see comments in thread

[nytamin]

Thank you for opening this RFC!

Having shared control of devices is something that has been in the back of our minds since the very beginning of Sofie. This RFC just might be the push we need to make a serious effort in making it happen!

We have had a discussion internally in the NRK team, and while we’re positive to move in a shared-control direction, we believe that it would be a paradigm shift in terms of playout logic philosophy and user experience. As a next step, we’re going to do an internal investigation to talk to our end-users and map up our use cases. We’ll be considering both our users’ perspective and technical abilities of the controlled hardware and software. We plan to have concluded the investigation by February 24th, and by that time we’ll share the results with you.

We propose to have a workshop Monday, March 3rd at 14:00 CET, with the agenda to share our respective use cases and make a plan to go forward.

If anyone else wishes to join the workshop, please email me at johan.nyman@nrk.no.

[nytamin]

Notes from the workshop, 2025-03-03

Nomenclature:

• Agenda points & headlines in regular font.
• Notes from the meeting in italics…

Participants: Representatives from the respective Sofie projects of NRK, BBC, TV2 Norway and EVS.

Presenting our worldviews

The participants present their current Sofie status, workflows and wishes from users that come under the banner of “Shared Control”.

BBC: Our initial focuses are smaller studios , regional variations, sports. Then if all goes well, we might aim for the larger productions, “starting small, dreaming big”.

NRK: We’re concentrating our efforts on our big galleries and treat our smaller galleries as “trimmed down” versions, to keep our efforts streamlined. Our users see Sofie as their main GUI during production, the place where they interact with “everything in the studio”. In the cases where we have multiple users, they are interested in “collaborative control”, an ability to have a shared workspace, ie to have multiple user interfaces (like both Sofie panels and the switcher panels) where the Sofie State (in the Sofie GUI) reflects the actual state, no matter where the input comes from.
For example, if a user presses camera 2 on the vision mixer panel, Sofie would pick up the change and display camera 2 in its GUI.

TV2No: No concrete use cases at the moment. One thought is that this could be useful for lighting or routing, where Sofie controls parts of the routing and where feedback from external source telling Sofie that cameras have been routed in specific ways.

EVS: We’re looking into having all controlled devices being abstracted and all control commands from Sofie are going through one single TSR integration. A part of the plan is to allow end users to use their control panels, while also using Sofie.

Question to the group: What is your vision for shared control? Is Sofie one of the tools to affect change to devices among others, or is Sofie a single workspace where end users can collaborate and share state?

BBC: We’d need both of those eventually.
For example: Sofie controls a vision mixer and a user switches the cameras on the hardware panel. How does Sofie control state it it’s out of touch with reality? So this is more about when and how Sofie should be “regaining control”.
Example scenario: User has faded up lights manually. When entering next segment, Sofie needs to be aware that lights have been faded up and now needs to reset them. This has been done using “TriggerValues” previously.

Our opinion on the “TriggerValues” approach are that that while they do work, it would be better if TSR had a better understanding of reality.

From a technical perspective, the ideal solution is of course that every input goes via Sofie and then Sofie controls everything. However our users do want to be able to control things manually as well..

One issue in the current Sofie is that the Timeline is the state executor and TSR doesn’t know when a “Take” has been done. It would be better if TSR is aware of that so that it can avoid sending updates for manually controlled things until next Take.

EVS: Sofie should know that something has changed and “do the right thing” when it is the right time to do so. In our current implementation, it is “the last command wins”.

BBC: “The last wins” is good enough for us as well, we don’t do many time-sensitive things anyway.
The ”Blocking concept” is the idea of being able to see the reality, and also know that it has been manually changed.

TV2No: One interesting situation though, is if we’re on cam1 and user press cam2 manually. Sofie should now be aware that cam2 is on and adjust the audio accordingly.

NRK: Yes, that is what we’d like to aim for eventually. One use case for us is that we currently have a studio with a physical Atem control panel in combination with Sofie control and many of the shows there are produced manually via the control panel. Since Sofie is unaware of these changes in state, the users typically prepare a butterfly using Sofie and put it on the program output with Sofie, and then continue cutting to lives and cameras on the Atem control panel. The next time they plan to use a butterfly, Sofie still thinks that it is on air, and either does nothing or prepare the next butterfly in the background without showing it on preview. Users then think that somethings wrong and stop using the butterfly effect – or worse, stops using Sofie for the remainder of the show. A way for Sofie to pick up changes in state and display them in the UI to reflect the actual state of things would gain the users' confidence during production.

BBC: It’s a tricky situation to solve gracefully, because when Sofie receives a message that a device has change its state, one would actually have to go to Blueprints to actually understand what that entails. This would be a huge change and we should not make such a large change in scope right now.

(nods all around in agreement)

NRK: If we focus on the “reasserting control” feature for now, that solves a lot of low-hanging fruits. Though we should keep the “shared control” goal in mind, as there are several ways we can achieve it in the end.

Question to the group: What are your plans regarding level of involvement on this?

NRK: No immediate need for this, we’ll be focusing our efforts in other areas for the next year or so.

EVS: We’ll have something working short term for us using a private, external software. We’re interested in the question of “Sofie doing the right thing at next timeline update / Take”.

BBC: We’re actively looking into this, we’re kind of fighting the logic already.

TV2No: No concrete plans, but this would be nice to have.

Other notes

NRK: It is important for us that this feature is opt-in and that any added functionality doesn’t affect existing functionality.

BBC: The prototype PR is made so that it is easy to opt-in.

Conclusion

There was some confusion in the meeting around the meaning of “Shared Control” in the original RFC. The meeting found it helpful to clarify this by renaming the proposed feature to “Reasserting Control” as this better describes the RFC’s technical proposal for how Sofie could be made to cope with users' direct interaction with devices that are under Sofie’s control.

As this is a fundamental change to the current model where Sofie assumes it has singular control over all devices, the meeting raised concerns over stability and risk of critical bugs affecting all broadcasters, even those that choose not to use the proposed feature.

The meeting recommended that the BBC should move forward on the RFC.

[hummelstrand]

Addendum
Notes from an internal post-meeting discussion in SuperFly.tv

An aspect that contributed to the confusion of the meeting was that the RFC was proposing a detailed, technical implementation of a feature, while the workshop invitation and the assembled participants indicated that this would be a general discussion of how a “shared control” feature could work, and how such a feature might affect the Sofie system.

Since some participants arrived expecting to discuss general ideas, while others expected to get a green light for a proposed technical implementation, these different worldviews caused a lot of misunderstandings.

To learn from this, it might be helpful to clarify the RFC process so that all parties are allowed to discuss and raise concerns before a detailed technical solution is presented.

[rjmunro]

This sounds related to SuperFlyTV/SuperConductor#122