Tracking objects when there are no objects detected

[cnavarrete]

Hello while using the SORT object from sort_tracker.py I realized that, after I stop detecting objects, if I send empty boxes, confidence and classes to the update function I still get 1 or more objects as being tracked. I thought the max_lost parameter would prevent this? the paper indicates that the tracks should be terminated if they are not detected for Tlost frames, so shouln't an empty detection count as all tracks not being detected in that frame?

Adding something like this should take care of the tracks if no object is detected in a frame:

for t in unmatched_tracks:
    track_id = track_ids[t]
    bbox = bbox_tracks[t, :]
    confidence = self.tracks[track_id].detection_confidence
    cid = self.tracks[track_id].class_id
    self._update_track(track_id, self.frame_count, bbox, detection_confidence=confidence, class_id=cid, lost=1)
    
    if self.tracks[track_id].lost > self.max_lost:
        self._remove_track(track_id)

# In case there are no detections, update tracks with the current prediction, if lost > max_lost remove track.
if len(bboxes) == 0:
    for i in range(len(bbox_tracks)):
        track_id = track_ids[i]
        bbox = bbox_tracks[i, :]
        confidence = self.tracks[track_id].detection_confidence
        cid = self.tracks[track_id].class_id
        self._update_track(track_id, self.frame_count, bbox, detection_confidence=confidence, class_id=cid, lost=1)
        if self.tracks[track_id].lost > self.max_lost:
            self._remove_track(track_id)

Also talking about the parameter "lost" for the track class isn't that line odd? if you set max_lost to let's say 5, then the condition lost > max_lost cannot be reached since the lost parameter is always reset to 1. In my case I commented that line out to make it work.

Here's a little demo that I wrote, I recommend runing it (without the changes I made) with a video where after the tracked object leaves the screen there are no more objects to detect, to see what I mean.

...

detector = Yolov4Detector()
tracker = CentroidTracker()

for i in range(len(images)):
    detection_bboxes = []
    detection_confidences = []
    detection_class_ids = []

    im = Image.open(images[i])
    transform = detector.get_transform()
    new_im = transform(im)
    new_im.unsqueeze_(0)
    dets = detector.detect(new_im)[0]

    for det in dets:
        xmin = int(det['bbox']['x1'] * im.size[0])
        ymin = int(det['bbox']['y1'] * im.size[1])
        xmax = int(det['bbox']['x2'] * im.size[0])
        ymax = int(det['bbox']['y2'] * im.size[1])

        width = abs(xmin - xmax)
        height = abs(ymin - ymax)

        detection_bboxes.append([xmin, xmax, width, height])
        detection_confidences.append(det['confidence'])
        detection_class_ids.append(0) #There's only one class

    detection_bboxes = np.array(detection_bboxes)
    detection_confidences = np.array(detection_confidences)
    detection_class_ids = np.array(detection_class_ids)

    output_tracks = tracker.update(detection_bboxes, detection_confidences, detection_class_ids)


    for track in output_tracks:
        frame, id, bb_left, bb_top, bb_width, bb_height, confidence, x, y, z = track
        assert len(track) == 10
        print(track)

Looks like someone else had the same issue #21 (comment)

[adipandas]

@cnavarrete Thank you for figuring out this bug. I haven't gotten a chance to look at this project over last few months. Can you also check if there is similar issue for other trackers or this was happening only in SORT?

Please feel free to send a pull request.

Thank you.

[cnavarrete]

Your welcome, will try to send a pull request on the weekend.

[adipandas]

I have included the suggested changes in the SORT implementation. I tested it with the single video and may be more formal testing required. In case you find any bugs, let me know.
I will close this issue for now.

You can refer this commit for further details.

Let me know if there are any other issues.

Thank you.