FrameProcessor protocol

[pblazej]

Python reference livekit/python-sdks#533

This is a compromise between existing Audio/Video interfaces, mostly to get uniform "add" behavior from the Room/Session.

The ideal solution in pure Swift would be kinda:

public protocol FrameProcessor: Sendable {
    associatedtype Input
    associatedtype Output

    func process(frame: Input) -> Output
}

This encapsulates the idea of Audio/Video-agnostic interface.

However, the entry points in audio stack must remain ObjC compatible anyway (no generics/associated types), and you cannot get "one protocol conforming to the other" for free (without wrapping/adapters).

Alternative solution

We can just remove explicit FrameProcessor protocol and move RoomDelegate conformance to both Audio and Video processor protocols or typealiases. Everything is optional there anyway, so this is a non-braking change.

Then we're left with these 2 "dangling" methods that don't have any explicit representation anywhere, but also we cannot call them directly from the RoomDelegate, so the benefits are negligible:

    def _update_stream_info(
        self,
        *,
        room_name: str,
        participant_identity: str,
        publication_sid: str,
    ): ...

    def _update_credentials(self, *, token: str, url: str): ...

[github-actions]

⚠️ This PR does not contain any files in the .changes directory.

[pblazej]

The minimal implementation is:

    private static let noiseFilter = AICAudioFilter()
    private let session = Session(
        tokenSource: ...,
        options: SessionOptions(audioProcessor: Self.noiseFilter)
    )

[pblazej]

This IS a breaking change for Krisp, we can go for @objc optional but then this requirement does not make much sense to me.

[lukasIO]

the alternative would be to not define it here and keep isEnabled on the processor implementations, right?

Maybe that's preferable if the alternative is a breaking change

[pblazej]

You can also use @objc optional annotation, but then you get Bool? you need to default to ?? true for legacy things like Krisp.

[pblazej]

The motivation here is obviously to avoid casting in the audio/video thread.

[lukasIO]

but implementations could still implement an isEnabled check in the process function and just return early?

[pblazej]

or should WE do it internally (not passing frames) (see below)

[pblazej]

That's a little blind python copy, not sure how can we really utilize that outside deinit (destruction).

[lukasIO]

currently the python code will allow to close the aic processor and implicitly recreate it on a next .process call.
The reason why it works this way right now is that on the agents side we need the ability to switch tracks for a processor.

[pblazej]

We cannot magically call it, so this is basically just a hint that "you need to get the credentials from the RoomDelegate somehow".

[lukasIO]

should we remove it then?

[pblazej]

See Alternative solution above 🙏

[pblazej]

76c1ebe

[pblazej]

Alternative solution 76c1ebe

[pblazej]

Closing this for now due to pausing new filter work ⏯️ Let's revisit if needed.