Use my own detection model

[Tetsujinfr]

Hi
Thanks for sharing this project.
If I want to use my own detection model with your tracker, what/where is the main entry point for me to adapt your code in order to replace the yolox detection model?
Do I need retrain everything or can I inject the detections into your pretrained model for inference?
Thanks

[ifzhang]

You can add your detection model here:
https://github.com/ifzhang/ByteTrack/blob/main/yolox/evaluators/mot_evaluator.py
And pass your detection results to BYTETracker to get the tracking results.

[Tetsujinfr]

So I have been trying to use my own detections , and it works. However my results are less stable on occlusion scenarios than using your default detector in this repo.
I am testing on the video below, and with your detector + bytetrack I got 3 ids tracked (perfect, using match_thresh=0.85 and the x model).
When using my detections (yolov5, only 1 class and not many low conf score/false positives) though, I end up with 6 (the last 3 occlusions generate new tracks).

I have played with track_buffer, match_thresh, with my detection conf threshold (I went as low as 0.001% to have more than 2 or 3 detections per frame) but without success.

Which parameter do you recommend I should test further? E.g. I did not observe any impact when increasing track_buffer, but not sure why.

thanks

video1.mp4

.

[Tetsujinfr]

Ok so there is a scale post processing factor in the tracker.update() method, which downscale detections by a factor of 1.48 in my test case, and this scale factor is related to the exp.test_size of the yolox detector of the repo I guess. But if one uses detections from another detector, then this scale factor needs to be turned off I think.
Also, there is a hard coded floor confidence level at 0.1 in the update() methode to discard detections with very low conf levels, but I did expect this threshold not to be hard coded.
Nonetheless, even after tweaking those details, I can not get as accurate results as the Yolox detector so I guess it is down to my detector which is not as good as the default one of the repo.

One question for @ifzhang : is there a parameter to discard tracks with very short durations, e.g. tracks with lower than 2 frames of existence or should I implement one if I need this?

thanks

[Tetsujinfr]

Hi @ifzhang
Is there a way to get the correspondence between the detection input id and the tracked id returned from ByteTracker?
I provide the tracker with a lot of bboxes given the low conf scores used as inputs, however the tracker returns much fewer bboxes (as expected), but how do I know which detection bbox have been retained and which have been discarded by the tracker, for a given frame?

Basically how do I reconcile between the "output_results" and the "online_targets returned by the tracker? Is there a property in the online_targets object which contain the original detection indices for example?

thanks a lot

[tungdop2]

Hi @Tetsujinfr
How can you adapt your code in order to replace the yolox detection model? Help me to put yolov5 instead of yolox please!

[Tetsujinfr]

You need to feed a n x 5 array as the first argument to the ByteTracker.update(). The first 4 columns of the matrix represent the bbox coordinates of your yolov5 bbox detections (so x1, y1, x2, y2) and the fifth column is the confidence score of the bbox.
Pay attention though, there are a couple of hardcoded thresholds in the class code, namely 0.1 as a low score confidence filter and 0.5 as a tracking threshold (if I remember well). You may need to tweak those based on the distribution of the confidence scores of the detector you use.
It does work but for some reason I have much less stable results than the ones from yolox, the tracker does not acquire some detections that I feed to it, and I have no idea why, those detection having high confifencecscores. I need to debug this further. Good luck.

[tungdop2]

Thanks, I load yolo5s from torch hub and put its detection to ByteTrack update function, seems it works.

[tungdop2]

Hi @Tetsujinfr , any process?
I use yolov5 instead of yolox, the detections work well, but the track is pretty bad, I still don't know why.

[Tetsujinfr]

I am still debuging to understand why some tracks are much worse when using my y5 detections rather than the yolox ones. I suspect it may come from the kalman filter initial parameters: my detections have an aspect ratio close to 1.0 (that is normal and ok on my side) while the yolox ones from the repo are > 1.6 (human bodies), and I see that the detection ratio plays a specific role in the kalman filter formulation. Still searching on this ...
@ifzhang if I am tracking round or square like detection shapes, shall I modify some of the kalman filters params you use? Shall I change even the filter formulation? Thanks

[NaifahNurya]

I am still debuging to understand why some tracks are much worse when using my y5 detections rather than the yolox ones. I suspect it may come from the kalman filter initial parameters: my detections have an aspect ratio close to 1.0 (that is normal and ok on my side) while the yolox ones from the repo are > 1.6 (human bodies), and I see that the detection ratio plays a specific role in the kalman filter formulation. Still searching on this ... @ifzhang if I am tracking round or square like detection shapes, shall I modify some of the kalman filters params you use? Shall I change even the filter formulation? Thanks

@Tetsujinfr To solve your problem you can modify line number 270 and 271 of https://github.com/ifzhang/ByteTrack/blob/1926ce65c1c3b1320e229cc91b60e5867fb0244b/tools/demo_track.py
as follows:

from:

                vertical = tlwh[2] / tlwh[3] > args.aspect_ratio_thresh
                if tlwh[2] * tlwh[3] > args.min_box_area and not vertical:

To

                    #vertical = tlwh[2] / tlwh[3] > args.aspect_ratio_thresh
                    if tlwh[2] * tlwh[3] > args.min_box_area: #and not vertical:

[Tetsujinfr]

Yes I saw the vertical bool flag and I did de-activate it already but that did not change much if anything at the time. I think there is something more profound which is optimised for the yolox detections shapes and dynamics.
Thanks anyway for your comment

[tungdop2]

@Tetsujinfr I run on MOT17, so the aspect ratio is not the problem. Do you have a solution now?

[Tetsujinfr]

From what I have seen while debuging through the code, I think that the problem is actually trivial : my detections bbox areas are quite small, and in the video I am testing the objects move fast so that there is no IOU between two frames for the same bbox. Consequently the tracker loose tracking of the object I think, even with th help of the kalman filter predictions.
When using the Yolox tracker the bbox area are larger (I am not tracking persons as opposed to the default Yolox tracker of the repo), so there is more likelihood that bboxes overlap over two consecutive frames. I have tried to increase the default KalmanF velocity value but that did not help.
So now I am working on resizing up my objects detection areas and on adding camera motion compensation to the Kalman filter to improve the next frame predictions of the objects. Standard next steps I think.

[MoaazAbdulrahman]

Hi @Tetsujinfr,
I am facing the same issue as I am running yolox to detect moving vehicles. The detection is pretty well but the tracker misses many tracks. Do you have any update about that issue.

Thanks

[Tetsujinfr]

It really depends if your vehicles are moving quite fast w.r.t their size on the image. If they are moving fast and can move by more than the bbox size within one frame, then if you artificially expand the bbox sizes input to the tracker, it should definitely help (but you may have more id duplicates as well). i did it and it worked quite well for me. You could remove the condition in the code which prevents predictions to be accepted if they are too far away from the current position, but I am not sure if this is a safe move, I did not analyse the code enough.

I think the proper way to manage all this is to tweak the Kalman filter parameters, or the Kalman filter model itself. I am looking into it but I have to do some homework first to do that properly in the code.

Also, if your camera is moving, compensating for those moves is definitely a must. You can use optical flow with Lucas-Kanade to give you an approximation of xy shifts, frame by frame (you will add at least one frame of latency though, and some extra compute load). You would have to mask out the dynamic parts of the image for a robust result, so masking out your moving vehicle in your use case.