Accuracy difference with tflite quant8 model

[akash4562800]

Please do not disclose security vulnerabilities as issues. See our security policy for responsible disclosures.

Before opening an issue

Issue Checklist

• I have checked the compatibility table in the documentation.
• I have verified that the SDK version I am using is compatible with the firmware version installed on my device.
• I have included the SDK version and firmware version in my issue description for reference.

Describe the bug

I am using "AXIS M4317-PLR Panoramic Camera" for detection of an object detection model. I trained a mbilenet ssd model, exported to tflite quant8 and able to successfully infer on the Axis Camera"
For inference on axis camera, I'm using: https://github.com/AxisCommunications/acap-computer-vision-sdk-examples/tree/main/object-detector-python.

Issue: Issue is I'm seeing the difference in detection accuracy when I'm running the same model on camera and my local.
I understand I'm using different methods and I want to understand how to resolve this difference in accuracy.
Local Inference steps:

self.inference_client = tf.lite.Interpreter(model_path=model_path)
self.inference_client.allocate_tensors()
self.input_details = self.inference_client.get_input_details()
self.output_details = self.inference_client.get_output_details()
def pre_process(self, image):
        image = cv2.resize(image, (320,320),interpolation=cv2.INTER_AREA)
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        image = np.expand_dims(image, axis=0)
        image = image.astype(np.uint8)
        
        return image
resized_image = self.pre_process(image)
self.inference_client.set_tensor(self.input_details[0]['index'], resized_image)
self.inference_client.invoke()
        
result = {}
for i, detail in enumerate(self.output_details):
result[detail['name']] = self.inference_client.get_tensor(detail['index'])
bounding_boxes, classes, confidences = tuple([np.squeeze(result[key]) for key in [
'StatefulPartitionedCall:3', 'StatefulPartitionedCall:2', 'StatefulPartitionedCall:1']])

for Inference on axis camera, I'm using
https://github.com/AxisCommunications/acap-computer-vision-sdk-examples/blob/main/object-detector-python/app/detector.py and https://github.com/AxisCommunications/acap-computer-vision-sdk/blob/main/sdk/tfserving/tf_proto_utils.py (not entirely sure) InferenceClient.infer method.

def detect(self, image):
        image = np.expand_dims(image, axis=0)
        image = image.astype(np.uint8)
        success, result = self.inference_client.infer({'data': image}, self.model_path)
        if not success:
            return False, 0, 0, 0
        bounding_boxes, classes, confidences = tuple([np.squeeze(result[key]) for key in [
            'StatefulPartitionedCall:3', 'StatefulPartitionedCall:2', 'StatefulPartitionedCall:1']])
def run_camera_source(self):
        stream_width, stream_height, stream_framerate = (800, 800, 10)
        capture_client = VideoCaptureClient(socket=self.grpc_socket,
                                            stream_width=stream_width,
                                            stream_height=stream_height,
                                            stream_framerate=stream_framerate)
         while True:
            frame = capture_client.get_frame()
            if frame is None:
                print("Error: Failed to capture frame")
                time.sleep(1)
                continue
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            frame_resized = cv2.resize(frame,(320,320),cv2.INTER_AREA )
            succeed, bounding_boxes, obj_classes, confidence = self.detect(frame_resized)

Axis camera is using tfserve and to serve the model and from this I'm guessing the issue might come. I have tried to mimic the pre-processing steps from detector.py but issue still persists.
My local inference is giving higher accuracy. Please help us.

To reproduce

Please provide as much context as possible and describe the reproduction steps that someone else can follow to recreate the issue on their own.

A team member will try to reproduce the issue with your provided steps. If there are no reproduction steps or no obvious way to reproduce the issue, the team will ask you for those steps. Bugs without steps will not be addressed until they can be reproduced.

Steps to reproduce the behavior:

1. Set up '...'
2. Do this '...'
3. See error

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

• Axis device model: [e.g. Q1615 Mk III]
• Axis device firmware version: [e.g. 10.5]
• Stack trace or logs: [e.g. Axis device system logs]
• OS and version: [e.g. macOS v12.2.1, Ubuntu 20.04 LTS, Windows build 21996.1]
• Version: [version of the application, SDK, runtime environment, package manager, interpreter, compiler, depending on what seems relevant]

Additional context

Add any other context about the problem here.

[Corallo]

Hi @akash4562800

From your code snippet I see you are converting BGR to RGB, unless I misunderstood your intent that doesn't seem correct.
The VideoCaptureClient will return an RGB image.
Maybe that is the reason of the performance difference?
If you want to dig more, take a look at this discussion: AxisCommunications/axis-model-zoo#50
It has some guideline on how to debug another model, but you might find it useful.

In particular, you can try it on a fixed image both on tensorflow and on the camera.
You can save the image as a binary file and use the larod-client to test the model in isolation.
Here is a guide that shows how to do it: https://developer.axis.com/computer-vision/computer-vision-on-device/test-your-model

[akash4562800]

Hi @Corallo ,
Thanks for your prompt response.
I checked the VideoCaptureClient code and found it's indeed returning RGB image.
The frame upon looking seemed BGR that's why I used cv2.cvtColor() method.
Although logically it's doesn't make sense but this was the issue and now I'm getting same results from both local and axis camera.
Thanks a lot.

I have another question, Which tracker will be the best for tracking person that can run on this camera.
I am using centriod based kf tracker but I see tracks are failing many times.
You can mark this complete and move to discussion section.