Automatic Detection and Tracking of Objects of Interest in Video Data with Global Motion

Introduction. At present, automatic capture and tracking of moving objects in video data obtained by a video camera mounted on a mobile carrier represents a relevant research task. Its successful solution is challenged by such factors, as a non-uniform background, object overlapping between one another and the background, significant and rapid changes in the size of the object of interest, abrupt changes in the movement trajectory of the mobile carrier.

Aim. To develop an automatic method for detecting moving objects followed by their tracking in video data obtained under difficult observation conditions. An additional requirement imposed on the tracking stage consists in the restriction of computing resources.

Materials and methods. The method is based on a convolutional neural network with a YOLO architecture. Due to the restriction of computing resources, object tracking is implemented without neural network solutions. In order to ensure stable tracking, several detectors are used simultaneously with the subsequent analysis of the data obtained. The tracking stage involves a detector based on histograms of oriented gradients (HOG), supplemented by a detector based on correlation filtering and motion trajectory prediction based on the Kalman filter.

Results. At the automatic detection stage, the TPR, averaged over all video files participating in the experiments, was equal to 0.81, with the FPR corresponding to 0.10. At the tracking stage, the failure rate (tracking failures) was 6·10 ^–5.

Conclusion. The proposed method can be successfully used to detect and track objects at a distance of 1500 m with an object projection size on the frame of 5 × 5 pixels under the conditions of global motion, a non-uniform background, and significant changes in the properties of the object of interest.

1. Yu Zhang, Xiangzhi Bai, Tao Wang. Boundary Finding Based Multi-Focus Image Fusion through MultiScale Morphological Focus-Measure. Information Fusion. 2017, vol. 35, pp. 81–101. doi: 10.1016/j.inffus.2016.09.006

2. Hossain M. D., Chen D. Segmentation for ObjectBased Image Analysis (OBIA): A review of Algorithms and Challenges from Remote Sensing Perspective. ISPRS J. of Photogrammetry and Remote Sensing. 2019, vol. 150, pp. 115–134. doi: 10.1016/j.isprsjprs.2019.02.009

3. Mobahi H., Rao S., Yang A., Sastry S., Ma Y. Segmentation of Natural Images by Texture and Boundary Compression. Intern. J. of Computer Vision. 2011, vol. 95, pp. 86–98.

4. Rao S. R., Mobahi H., Yang A. Y., Sastry S. S., Ma Y. Natural Image Segmentation with Adaptive Texture and Boundary Encoding. Computer Vision – ACCV 2009. Lecture Notes in Computer Science. Ed. by H. Zha, R.-i. Taniguchi, S. Maybank. Berlin, Springer, 2010, vol. 5994, pp. 135–146. doi: 10.1007/978-3-642-12307-8_13

5. Gao M., Chen H., Zheng Sh., Fang B., Zhang L. Texture Image Segmentation Using Fused Features and Active Contour. 23 Intern. Conf. on Pattern Recognition, Cancun, Mexico, 04–08 Dec. 2016. IEEE, 2016, pp. 520–526. doi: 10.1109/ICPR.2016.7899935

6. Bouaynaya N., Charif-Chefchaouni M., Schonfeld D. Spatially Variant Morphological Restoration and Skeleton Representation. IEEE Transactions on Image Processing. 2006, vol. 15, iss. 11, pp. 3579–3591. doi: 10.1109/TIP.2006.877475

7. Maggiani L., Salvadori C., Petracca M., Pagano P., Saletti R. Reconfigurable Architecture for Computing Histograms in Real-Time Tailored to FPGA-Based Smart Camera. IEEE 23 Intern. Symp. on Industrial Electronics (ISIE), Istanbul, Turkey, 01–04 June 2014. IEEE, 2014, pp. 1042–1046. doi: 10.1109/ISIE.2014.6864756

8. Dalal N., Triggs B. Histograms of Oriented Gradients for Human Detection. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition (CVPR'05), San Diego, USA, 20–25 June 2005. IEEE, 2005, vol. 1, pp. 886–893. doi: 10.1109/CVPR.2005.177

9. Kumar S., Yadav J. S. Video Object Extraction and its Tracking using Background Subtraction in Complex Environments. Perspectives in Science. 2016, vol. 8, pp. 317–322. doi: 10.1016/j.pisc.2016.04.064

10. Shaikh S. H, Saeed K., Chaki N. Chapter-2, Moving Object Detection Approaches, Challenges and Object Tracking. In: Moving Object Detection using Background Subtraction. SpringerBriefs in Computer Science. Cham, Springer, 2014, pp. 5–11. doi: 10.1007/978-3-319-07386-6_2

11. Barnich O., Droogenbroeck M. V. ViBe: A Universal Background Subtraction Algorithm for Video Sequences. IEEE Transactions on Image Processing. 2011, vol. 20, iss. 6, pp. 1709–1724. doi: 10.1109/TIP.2010.2101613

12. Obuhova N. A. Segmentation of Objects of Interest Based on Motion Feature in Video Computer Systems. Infocommunication Technologies. 2007, no. 1, pp. 77–85. (In Russ.)

13. Aslani S., Mahdavi-Nasab H. Optical Flow Based Moving Object Detection and Tracking for Traffic Surveillance. Intern. J. of Electrical, Computer, Energetic, Electronic and Communication Engineering. 2013, vol. 7, no. 9, pp. 1252–1256.

14. Kale K., Pawar S., Dhulekar P. Moving Object Tracking Using Optical Flow And Motion Vector Estimation. 4th Intern. Conf. on Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future Directions), Noida, India, 02–04 Sept. 2015. IEEE, 2015, pp. 1–6. doi: 10.1109/ICRITO.2015.7359323

15. Obukhova N. A. Apriority Priori Estimation of Optical Flow Vectors (Motion Vectors) Reliability. J. of the Russian Universities. Radioelectronics. 2006, no. 3, pp. 30–36. (In Russ.)

16. Obukhova N. A. Optical Flow Vectors in Tasks of Moving Objects Segmentation and Tracking. J. of the Russian Universities. Radioelectronics. 2006, no. 2, pp. 42–51. (In Russ.)

17. Zhao Z.-Q., Zheng P., Xu S.-T., Wu X.Object Detection with Deep Learning: a Review. IEEE Transaction on Neural Networks and Learning Systems. 2019, vol. 30, iss. 11, pp. 3212–3232. doi: 10.1109/TNNLS.2018.2876865

18. Terven J., Córdova-Esparza D.-M., RomeroGonzález J.-A. A. Comprehensive Review of YOLO Architectures in Computer Vision: From YOLOv1 to YOLOv8 and YOLO-NAS. Machine Learning and Knowledge Extraction. 2023, no. 5, pp. 1680–1716. doi: 10.3390/make5040083

19. Brabec J., Komarek T., Franc V., Machlica L. On Model Evaluation under Non-Constant Class Imbalance. Intern. Conf. on Computational Science. Lecture Notes in Computer Science. Cham, Springer, 2020, pp. 74–87. doi: 10.1007/978-3-030-50423-6_6

20. Verma R. A Review of Object Detection and Tracking Methods. Intern. J. of Advance Engineering and Research Development. 2017, vol. 4, iss. 10, pp. 569–578.

21. Lukezic A., Voj'ir T., Cehovin Zajc L., Matas J., Kristan M. Discriminative Correlation Filter Tracker with Channel and Spatial Reliability. Intern. J. of Computer Vision. 2018, vol. 126, iss. 8, pp. 671–688. doi: 10.1007/s11263-017-1061-3

22. Van de Weijer J., Schmid C., Verbeek J., Larlus D. Learning Color Names for Real-World Applications. IEEE Trans. Image Proc. 2009, vol. 18, iss. 7, pp. 1512–1523. doi: 10.1109/TIP.2009.2019809

23. Henriques J. F., Caseirio R., Martins P., Batista J. High-Speed Tracking with Kernelized Correlation Filters. IEEE Trans on PAMI. 2015, vol. 37, iss. 3, pp. 583–596. doi: 10.1109/TPAMI.2014.2345390

24. Kirillov A., Mintun E., Ravi N., Mao H., Rolland Ch., Gustafson L. Segment Anything. IEEE/CVF Intern. Conf. on Computer Vision (ICCV), Paris, France, 01–06 Oct. 2023. IEEE, 2023, pp. 3992–4003. doi: 10.1109/ICCV51070.2023.00371

25. Shchelkunov A. E., Kovalev V. V., Morev K. I., Sidko I. V. The Metrics for Tracking Algorithms Evaluation. Izvestiya SFEDU. Engineering Sciences. 2020, no. 1, pp. 233–245. doi: 10.18522/2311-3103-2020-1233-245 (In Russ.)