Integrate OCSORT to supervision

[tgbaoo]

Search before asking

• I have searched the Supervision issues and found no similar feature requests.

Description

My project require another tracking algorithm is OCSORT (https://github.com/noahcao/OC_SORT) beside bytetracker into supervision lib?

I tend to migrate thể algorithm and write some custom code for update_with_detection method to tracking the Detection objects of supervision lib.

Like this boxmot repo already integrate ocsort into their repo:
https://github.com/mikel-brostrom/yolo_tracking/blob/master/boxmot%2Ftrackers%2Focsort%2Focsort.py

Is there anything that I have to pay attention when plug some custom tracker into supervision lib beside the update_with_detection method?

I want to custom tracker to use in this file of code replace bytetracker to ocsorttracker:
https://github.com/roboflow/supervision/blob/develop/examples%2Ftime_in_zone%2Fultralytics_stream_example.py

Use case

No response

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[SkalskiP]

Hi @tgbaoo 👋🏻 Sorry for the late response, but I traveled a lot at the end of last week, and my access to GitHub was limited.

I want to make sure I understand correctly. Do you want to add OCSORT to the Supervision codebase or just use it in combination with Supervision?

[tgbaoo]

Yes, I want to integrate ocsort to use with supervision library by sync the code and setup update_with_detections method

[SkalskiP]

In that case, let me start by converting this issue into a discussion and placing it in the Q&A section.

[SkalskiP]

Your tracker should implement the update_with_detection method, which takes sv.Detections and returns sv.Detections. After passing through update_with_detection, the sv.Detections.tracker_id field should be filled with a numpy array of shape (N,), where N is the number of detections. The values of this numpy array should be IDs. Important: every detection should have a tracker_id. Returning None instead of int is unacceptable. That's all.

[tgbaoo]

hi @SkalskiP, I have Plug OCSORT into supervision source code, and when I run the ultralytic_stream_example from time_in_zone example, i have this log
`WARNING Error in results dispatching - The truth value of an array with inference_pipeline.py:714
more than one element is ambiguous. Use a.any() or a.all()

                WARNING  Error in results dispatching - 'NoneType' object is not           inference_pipeline.py:714
                         subscriptable
                
                WARNING  Error in results dispatching - 'NoneType' object is not           inference_pipeline.py:714
                         subscriptable
                
                WARNING  Error in results dispatching - 'NoneType' object is not           inference_pipeline.py:714
                         subscriptable
                
                WARNING  Error in results dispatching - The truth value of an array with   inference_pipeline.py:714
                         more than one element is ambiguous. Use a.any() or a.all()
                
                WARNING  Error in results dispatching - The truth value of an array with   inference_pipeline.py:714
                         more than one element is ambiguous. Use a.any() or a.all()
                
                WARNING  Error in results dispatching - The truth value of an array with   inference_pipeline.py:714
                         more than one element is ambiguous. Use a.any() or a.all()
                
                WARNING  Error in results dispatching - The truth value of an array with   inference_pipeline.py:714
                         more than one element is ambiguous. Use a.any() or a.all()
                
                WARNING  Error in results dispatching - The truth value of an array with   inference_pipeline.py:714
                         more than one element is ambiguous. Use a.any() or a.all()
                
                WARNING  Error in results dispatching - The truth value of an array with   inference_pipeline.py:714
                         more than one element is ambiguous. Use a.any() or a.all()
                
                WARNING  Error in results dispatching - The truth value of an array with   inference_pipeline.py:714
                         more than one element is ambiguous. Use a.any() or a.all()

               [05/21/24 06:37:10] WARNING  Error in results dispatching - index 5 is out of bounds for axis  inference_pipeline.py:714
                         1 with size 3

                WARNING  Error in results dispatching - index 5 is out of bounds for axis  inference_pipeline.py:714
                         1 with size 3

                WARNING  Error in results dispatching - index 5 is out of bounds for axis  inference_pipeline.py:714
                         1 with size 3

                WARNING  Error in results dispatching - index 4 is out of bounds for axis  inference_pipeline.py:714
                         1 with size 3                    

`
This is my code for update_with_detection method:

def update_with_detections(
    self, detections: Detections, img: np.ndarray
) -> Detections:
    """
    Updates the tracker with new detections and returns updated detections with tracker IDs.
    """
    if detections is None or img is None:
        print("Detections or image frame is None.")
        detections = Detections.empty()
        detections.tracker_id = np.array([], dtype=int)
        return detections

    self.frame_count += 1
    h, w = img.shape[:2]

    # Split detections into primary and secondary based on confidence
    confs = detections.confidence
    primary_mask = confs > self.det_thresh
    secondary_mask = (confs > 0.1) & (confs < self.det_thresh)

    primary_detections = Detections(
        xyxy=detections.xyxy[primary_mask],
        mask=detections.mask[primary_mask] if detections.mask is not None else None,
        confidence=detections.confidence[primary_mask],
        class_id=detections.class_id[primary_mask],
        tracker_id=detections.tracker_id[primary_mask] if detections.tracker_id is not None else None,
        data={key: np.array(value)[primary_mask] for key, value in detections.data.items() if value is not None}
    )
    secondary_detections = Detections(
        xyxy=detections.xyxy[secondary_mask],
        mask=detections.mask[secondary_mask] if detections.mask is not None else None,
        confidence=detections.confidence[secondary_mask],
        class_id=detections.class_id[secondary_mask],
        tracker_id=detections.tracker_id[secondary_mask] if detections.tracker_id is not None else None,
        data={key: np.array(value)[secondary_mask] for key, value in detections.data.items() if value is not None}
    ) if self.use_byte else None

    # Predict states for existing trackers and prepare for association
    trks = np.zeros((len(self.trackers), 5))
    to_del = []
    for t, trk in enumerate(self.trackers):
        pos = trk.predict()[0]
        trks[t, :] = [pos[0], pos[1], pos[2], pos[3], 0]
        if np.any(np.isnan(pos)):
            to_del.append(t)
    trks = np.ma.compress_rows(np.ma.masked_invalid(trks))
    for t in reversed(to_del):
        self.trackers.pop(t)

    velocities = np.array(
        [trk.velocity if trk.velocity is not None else np.array((0, 0)) for trk in self.trackers]
    )
    last_boxes = np.array([trk.last_observation for trk in self.trackers])
    k_observations = np.array(
        [k_previous_obs(trk.observations, trk.age, self.delta_t) for trk in self.trackers]
    )

    # First round of association
    matched, unmatched_dets, unmatched_trks = associate(
        primary_detections.xyxy,
        trks,
        self.asso_func,
        self.asso_threshold,
        velocities,
        k_observations,
        self.inertia,
        w,
        h,
    )

    for m in matched:
        self.trackers[m[1]].update(
            primary_detections.xyxy[m[0]],
            primary_detections.class_id[m[0]],
            primary_detections.tracker_id[m[0]],
        )

    # Handle secondary detections if use_byte is enabled
    if self.use_byte and secondary_detections is not None:
        secondary_trks = np.array([trks[i] for i in unmatched_trks])
        iou_left = self.asso_func(secondary_detections.xyxy, secondary_trks)
        secondary_matched = linear_assignment(-iou_left)
        for det_idx, trk_idx in secondary_matched:
            if iou_left[det_idx, trk_idx] < self.asso_threshold:
                continue
            self.trackers[trk_idx].update(
                secondary_detections.xyxy[det_idx],
                secondary_detections.class_id[det_idx],
                secondary_detections.tracker_id[det_idx],
            )
            unmatched_trks = np.setdiff1d(unmatched_trks, trk_idx)
            secondary_detections.xyxy = np.delete(secondary_detections.xyxy, det_idx, axis=0)
            secondary_detections.class_id = np.delete(secondary_detections.class_id, det_idx, axis=0)
            secondary_detections.tracker_id = np.delete(secondary_detections.tracker_id, det_idx, axis=0)

    if unmatched_dets.shape[0] > 0 and unmatched_trks.shape[0] > 0:
        left_dets = primary_detections.xyxy[unmatched_dets]
        left_trks = last_boxes[unmatched_trks]
        iou_left = run_asso_func(self.asso_func, left_dets, left_trks, w, h)
        rematched_indices = linear_assignment(-iou_left)
        for m in rematched_indices:
            if iou_left[m[0], m[1]] < self.asso_threshold:
                continue
            self.trackers[unmatched_trks[m[1]]].update(
                primary_detections.xyxy[unmatched_dets[m[0]]],
                primary_detections.class_id[unmatched_dets[m[0]]],
                primary_detections.tracker_id[unmatched_dets[m[0]]],
            )
            unmatched_trks = np.setdiff1d(unmatched_trks, unmatched_trks[m[1]])
            unmatched_dets = np.setdiff1d(unmatched_dets, unmatched_dets[m[0]])

    for trk_idx in unmatched_trks:
        self.trackers[trk_idx].update(None, None, None)

    # Create and initialise new trackers for unmatched detections
    for i in unmatched_dets:
        trk = KalmanBoxTracker(
            primary_detections.xyxy[i],
            primary_detections.class_id[i],
            i,  # Using index as a stand-in for detection ID
            delta_t=self.delta_t,
        )
        self.trackers.append(trk)

    updated_tracker_ids = np.full(len(detections), -1, dtype=int)
    for det_idx, trk_idx in matched:
        updated_tracker_ids[det_idx] = self.trackers[trk_idx].id

    for trk in self.trackers:
        if trk.time_since_update < 1 and (
            trk.hit_streak >= self.min_hits or self.frame_count <= self.min_hits
        ):
            updated_tracker_ids[trk.det_ind] = trk.id + 1

    detections.tracker_id = updated_tracker_ids
    return detections

`

I just copy the source code from the ocsort of boxmot repo and pase into this, file name convention inspired by the bytetracker:

From your expertise, What problem am I facing and what should I do now? Sorry for lack of knowledge on math and CV ...
Thanks you

[LinasKo]

Hi @tgbaoo 👋

Let's see if I can help out.

First of all, integrations into supervision are held to a very high standard. Simply copying it would typically not be accepted. I'm not sure whether you intend to contribute, but if so - we'd typically want the style adapted to ours, tested, etc.

Regarding the issue you're getting, we'd need to track to exact line the error happens in.

• What version of inference are you using? (error in Line 714 makes little sense in latest release)
• At least for now, could you try running with a method simpler than InferencePipeline? This example will stop when it hits an error: https://supervision.roboflow.com/latest/how_to/track_objects/#tracking

[SkalskiP]

Awesome tips @LinasKo; I think that @tgbaoo said above Yes, I want to integrate ocsort to use with supervision library, so I assume he is not planning to contribute, but just make it work. Other than this I fully agree. Simplify your code as much as possible. https://supervision.roboflow.com/latest/how_to/track_objects/#tracking is a great start.

[tgbaoo]

Yeah, because I want to integrate for my project first, if It work good I will plan to deep integrate and contribute to supervision if you guys allow me to do and I my code fit with the lib and meet the standard of the code quality

[SkalskiP]

Awesome! Can't wait to see the result!