11/27/2024 Wednesday Meeting Notes

[himanshunaidu]

The following notes are the progress and next steps for each sub-task:

1. Computer Vision Model Update
   Assigned to Himanshu Naidu.
   In order to accommodate the segmentation post-processing, some architectural changes had to be done. These changes revealed massive issues (that tbf should have been spotted by me earlier) in the CPU requirements by the deeplabv3 model we are using.
   Thus, it is not feasible to use the current deeplabv3 model for real-time segmentation, hence I have temporarily moved back to using ESPNetv2 Pascal model. I will try to get the ESPNetv2 cityscapes on CoreML, although weirdly enough the conversion from PyTorch to CoreML has failed to work properly only for ESPNetv2.

Nonetheless, I don't see this as a back-to-square-one situation because we finally figured out the conversion logic for many of the existing models.

2. Segmentation Mask Post-Processing
   Assigned to Ihor and Dmytro (with Himanshu working on the architectural changes).

The architectural changes to make the contiguous segmented frames available, are mostly completed.
Now we need to decide which format the frames should be saved in. Currently, it is CVPixelBuffer, but we may want to move to a more uniform format such as CIImage, as this is the current formats for the different images we are using:

class SharedImageData: ObservableObject {
    @Published var cameraImage: CGImage?
    @Published var depthData: CVPixelBuffer?
    
    @Published var classImages: [CIImage] = []
...
}

class SegmentationModel: ObservableObject {
    @Published var segmentationResults: UIImage?
...
}

.... not systematic at all.
The tradeoff is that a lot of existing functions in our app are hard-tied to these formats. So we will soon decide on proceeding with CVPixelBuffer and adding more to the refactoring backlog, or doing the refactoring for uniformity now itself and delay the post-processing.

3. API Transmission
   Assigned to Mariana.

For logout and changeset functionality, Mariana has raised PRs for login, change sets and logout.
The PRs have been reviewed by Himanshu. There are some action items pending on it.

For the refresh token logic, the life of the auth token is now 86400 seconds instead of 1800, so this task is now on the backseat. Nonetheless, Mariana is going to raise a PR that implements the refresh functionality as a background task.

Also, regarding the KartaView Api, we will try to take reference from the Gaussian team on how to implement it (although this is a stretch goal for now).

4. Other Issues
   Himanshu has identified a bug with the AnnotationView, where the index sometimes exceeds the length of the classImages array, which crashed the application. This is despite the fact that there are checks to prevent this.
   Still need to work on this.