Camera and Reducer

[kakhaberikiknadze]

I've been wondering what would be the best practice of using camera with reducer?

Let's say we need to start the camera session and get a buffer from video output delegate method captureOutput(_ output:, didOutput sampleBuffer:, from connection:). We need to update state's buffer each time we receive a buffer. State's buffer is then used by some kind of SwiftUI-wrapped Metal view. E.g,

PassthroughMetalView(
    device: device, 
    commandQueue: queue,
    buffer: viewStore.binding(
        get: \.buffer,
        set: Action.receiveBuffer
    )
)

I know this example is not enough but what I'm trying to achieve is to have a specific metal view with its internal renderer and inject only the buffer. I don't want to inject renderer in PassthroughMetalView as it would make the client able to inject un-contextual renderer (Draw a triangle for example).

This is what I quickly tried and of course doesn't work. It will receive the first buffer but even though buffer.send gets triggered, task closure is already finished.

final class Camera: NSObject, ObservableObject {
   // Some properties ..
  
    // Buffer publisher
    var buffer: PassthroughSubject<CVPixelBuffer?, Never> = .init()
   // Async buffer
    var pixelBuffer: CVPixelBuffer? {
        get async {
            var iterator = buffer.values.makeAsyncIterator()
            let buffer = await iterator.next()
            return buffer ?? nil
        }
    }
    
    // Some setup code here including setting self as AVCaptureVideoDataOutputSampleBufferDelegate ...
    
    func start() {
        session.startRunning()
    }
}

extension CameraModel: AVCaptureVideoDataOutputSampleBufferDelegate {
    public func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        let timestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
        let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
        buffer.send(imageBuffer)
    }

struct CameraReducer: CameraReducing {
    @Dependency(\.camera) var camera
    
    struct State: CameraViewState {
        var buffer: CVPixelBuffer?
    }
    
    enum Action: CameraAction {
        case startSession
        case sessionStarted
        case receiveBuffer(CVPixelBuffer?)
    }
    
    var body: some ReducerProtocol<State, Action> {
        Reduce { state, action in
            switch action {
            case .startSession:
                return .task {
                    camera.start()
                    return .sessionStarted
                }
            case .sessionStarted:
                return .task {
                    let newBuffer = await camera.pixelBuffer
                    return .receiveBuffer(newBuffer)
                }
            case let .receiveBuffer(buffer):
                state.buffer = buffer
                return .none
            default:
                return .none
            }
        }
    }
}

[kakhaberikiknadze]

I tried this now and it works but I don't think it's that performant as in theory some buffers might be missed

struct CameraReducer: CameraReducing {
    @Dependency(\.camera) var camera
    
    struct State: CameraViewState {
        var buffer: CVPixelBuffer?
    }
    
    enum Action: CameraAction {
        case startSession
        case receiveBuffer(CVPixelBuffer?)
        case waitForTheNextBuffer
        case startCapturing
        case endCapturing
    }
    
    var body: some ReducerProtocol<State, Action> {
        Reduce { state, action in
            switch action {
            case .startSession:
                return .task {
                    camera.start()
                    return .waitForTheNextBuffer
                }
            case let .receiveBuffer(buffer):
                state.buffer = buffer
                return .task {
                    return .waitForTheNextBuffer
                }
            case .waitForTheNextBuffer:
                return .task {
                    let newBuffer = await camera.pixelBuffer
                    return .receiveBuffer(newBuffer)
                }
            default:
                return .none
            }
        }
    }
}

[kakhaberikiknadze]

It's definitely laggy 😬

RPReplay_Final1676832462.mov

[tgrapperon]

Hey @kakhaberikiknadze! TCA reducers are not particularly well suited for this kind of high-frequency and CPU intensive task. ViewStore aren't adapted either to diff very large data types efficiently. Both factors are maybe explaining the performance bottleneck you're observing.

You can check the "Performance" section of the documentation for more informations, but the idea would be to model your high frequency domain as a dependency. You should ask yourself what kind of data is really important for the reducer. For example, are you performing some business logic on each frame using TCA? If this is not the case, you can maybe find a way to make this information travel up to the View using something else than a ViewStore state directly. You can for example imagine passing a stable reference type from your Dependency up to the view where you bind it to a ViewModel like you suggested in your first approach. You can do this by passing the reference through the ViewStore (passing a stable publisher of buffers is lightweight), or even through SwiftUI's environment if you conform your DependencyKey value to EnvironmentKey (you need to make sure to keep them in sync if you override them).

In other words, the reducer should handle the start and stop, but it has very little to do with the CVPixelBuffer for example. Imagine that you're detecting features on your video feed using AI, you would typically model your dependency so it would produce an async sequence of arrays of features with their coordinates (that the reducer can be interested in), but which would handle the detection and buffer requests internally.

This kind of performance-heavy and high-frequency domains requires some finesse in their implementation, and you can't unfortunately throw everything into TCA.

[mbrandonw]

Yep, I agree with everything @tgrapperon, and just to give an analogy of why we recommend this style: if you were making a network request you would not model that as an effect that emits for each single byte that is received and then accumulates the bytes in the reducer, unless you truly needed to execute logic for each individual byte. Instead, most likely, you would simply tell the dependency to start and it would notify you when it's done with all the bytes.

And even if you did need to execute logic for bytes received, maybe it's not for literally every byte received. Maybe you can just do something for every 1,000 bytes or every percentage of completion.

[kakhaberikiknadze]

Thank you guys! It all makes sense. I think I'll try to pass some kind of an observable object buffer provider reference as a separate dependency. If I'm not missing something, I should still be able to keep the renderer internal to the PassthroughMetalView. (It, as well as the scene, lives in a different module than the one the reducer implementation lives in).