Memory being retained during concurrency

[p-larson]

Problem

View "ComposableCameraApp.swift" Gist

In my dependency, CMSampleBuffer's is being retained in memory somehow throughout the concurrency, leading to samples being dropped. This has taken my attention for the day, I'm seeking new eyes to shed light into what I'm doing wrong.

Side Quest

I'm sending the CameraAction.receive(CGImage) potentially a lot to the reducer, I couldn't find anything saying I shouldn't in the ComposableArchitecture's documentation but I still feel like I shouldn't.

Is there a more efficient approach to updating view dependent state without running it through the reducer for high volume state changes?

I greatly appreciate any corrections to my approach. Thank you for your time.

Ouput

received action:
  CameraAction.open
  (No state changes)

received action:
  CameraAction.authorizationResponse(AVAuthorizationStatus.AVAuthorizationStatus)
  (No state changes)

received action:
  CameraAction.start
  (No state changes)

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: nil,
+   feed: __NSCFType(),
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

received action:
  CameraAction.receive(__NSCFType())
  CameraState(
    numberOfFrames: 0,
-   feed: __NSCFType()
+   feed: __NSCFType()
    isRecording: false
  )

Optional(OutOfBuffers)
Wimyx_08262022/ContentView.swift:78: Fatal error
2022-08-28 18:53:12.202177-0500 Wimyx 08262022[48818:5157375] Wimyx_08262022/ContentView.swift:78: Fatal error

[tgrapperon]

Hey @p-larson!
I've checked your Gist, and after fixing a few issues around authorization status (it seems you're not calling AVCaptureDevice.requestAccess(for: .video) when the status is unauthorized), I've managed to get it in a working state by updating a little your delegate. I never really used the video part of AVFoundation, so I may be wrong, but I'm not getting why you keep a reference to the feed image in your delegate. If you write your buffer in a local variable instead, it seems to work:

// private var feed: CGImage! // Removed
func captureOutput(
  _ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer,
  from connection: AVCaptureConnection
) {
  if let imageBuffer = sampleBuffer.imageBuffer {
    var image: CGImage?
    VTCreateCGImageFromCVPixelBuffer(imageBuffer, options: nil, imageOut: &image)
    if let image = image {
      self.continuation.yield(image)
    }
  }
}

As I'm getting it, the objective is to exit this function the fastest possible, so the sample buffer can be reused.
On my iPhone Xs, it does not drop frames most of the time (only a few ones when the app starts, before reaching its cruising speed). This is likely not related to TCA, as the issue seems to also happen if you don't yield the image to the continuation, hence to TCA. These old C-style methods with pointers as output are really inconvenient to use (and good luck finding what a returned OSStatus stands for in some 2008 mailing list archive). I'm wondering if there are no newer API's in this area, but again, I'm not a specialist. As I'm getting it, it's OK to do so. The CGImage is managed under ARC, so it shouldn't pose any issue down the line.

Deciding what to do with these images depends on your use case and you may find a few remarks about high-frequency actions here. Fortunately, you've already transformed this feed into an AsyncStream<CGImage>, so it should be relatively convenient to handle its output in the way it's more convenient to you. As an aside, you can maybe tweak its buffering policy to .bufferingNewest(1) if you only need the latest image in your use case.

Another remark: you did set your queue to .concurrent, but you were reading/writing to the feed property in a potentially racy way. The local variable avoids the issue, but you can also maybe try with a serial queue. Furthermore, if for some reason the call to VTCreateCGImageFromCVPixelBuffer takes a different time from one buffer to another, you may end up yielding images to your stream in a different order, which is likely not what you want.

[p-larson]

@thomasmarsh you're the man!

Thank you for the help, that seemed to be the fix. I appreciate you sharing your wisdom, your responses were super insightful. I'm setting the queue to serial for now, hoping to avoid any unpredictable behavior. I'm adopting all of your tips, especially keeping away from the 2008 mailing list archives!