The Object Tracker for Infrared and Visual Bands based on Channel-Independent Spatially-Regularized Discriminative Correlation Filter

Authors

DOI:

https://doi.org/10.20535/RADAP.2020.83.5-16

Keywords:

visual object tracking, multispectral images, discriminative correlation filters (DCF), alternating direction method of multipliers (ADMM)

Abstract

The method of visual object tracking intended for the application on multispectral video sequences is considered.

Introduction. The possible techniques of multispectral information fusion for visual object tracking are considered and the use of feature based fusion approach is justified. The tracker is suggested to be implemented using the discriminative correlation filters (DCF), since this approach is known to provide the compromise in terms of tracking quality and speed.

Theoretic results. The method for channel-independent discriminative correlation filter with spatial regularization calculation based on the use of alternating direction method of multipliers (ADMM) is proposed. The calculation of DCF filter and the object localization is suggested to be performed in special feature space, which employs the multichannel FHOG features and the features that are based on the backprojection of object weighted histogram. In particular, we propose to calculate the mentioned features for each channel of the respective frame of the multispectral video sequence with subsequent concatenation of obtained features into a single tensor, which forms the joint feature space.

Conclusions. Using the VOT Challenge RGBT2019 subchallenge, it was shown that the implementation of the suggested method is competitive in terms of tracking robustness with more sophisticated approaches, including the ones that are based on the convolutional neural networks. During the experiments, it was additionally established that the increasing of context-background information gives slight tracking quality improvement compared to the basic method implementation, even when only FHOG features are used.

Author Biographies

A. Y. Varfolomieiev, National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute"

Cand. of Sci (Tech), Department of design of electronic digital equipment

I. V. Korotkyi , National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute"

Cand. of Sci. (Techn), Associate Professor

References

Перелік посилань

Multi-Sensor Image Fusion and Its Applications / Ed. by R. S. Blum, Z. Liu. — CRC Press, 2006. — 528 p. ISBN: 9780849334177.

Image Fusion and Its Applications / Ed. by Y. Zheng. — InTechOpen, 2011. — 242 p. ISBN: 9789533071824.

Гривачевський А. П. Пiдвищення iнформативностi мультиспектральних зображень шляхом мультимодального комплексування / А. П. Гривачевський, І. Н. Прудиус // Вісник НТУУ <<КПІ>>. Серія Радіотехніка, Радіоапаратобудування. — 2018. — № 73. — C. 40--49. DOI: 10.20535/RADAP.2018.73.40-49.

Maggio E., Cavallaro A. Video tracking. Theory and Practice. — John Wiley & Sons Ltd, 2011. — 294 p. ISBN: 9780470749647.

Kristan M. The Visual Object Tracking VOT2015 challenge results / M.Kristan, J.Matas, A.Leonardis, M.Felsberg та ін. // IEEE International Conference on Computer Vision Workshop (ICCVW), . — 2015. — P. 564-586.

Kristan M. The Visual Object Tracking VOT2016 challenge results / M.Kristan, A.Leonardis, J.Matas та ін. // European Conference on Computer Vision — ECCV 2016 Workshops. ECCV 2016. Lecture Notes in Computer Science. — Vol. 9914. — 2016. — pp. 777-823.

Kristan M. The visual object tracking VOT2017 challenge results / M.Kristan, A.Leonardis, J.Matas, M.Felsberg та ін. // ICCV2017 Workshops, Workshop on visual object tracking challenge. — 2017.

Kristan M. The sixth Visual Object Tracking VOT2018 challenge results / M.Kristan, A.Leonardis, J.Matas, M.Felsberg та ін. // Computer Vision — ECCV 2018 Workshops. ECCV 2018. Lecture Notes in Computer Science. — Vol. 11129. — 2018. — P. 3-53.

Nam H. Learning multi-domain convolutional neural networks for visual tracking / H.Nam, B.Han // IEEE Conference on Computer Vision and Pattern Recognition (CVPR). — 2016. — P. 4293-4302.

Danelljan M. ECO: Efficient convolution operators for tracking / M.Danelljan, G.Bhat, F.S.Khan, M.Felsberg // IEEE Conference on Computer Vision and Pattern Recognition (CVPR). — 2017. — P. 6931-6939.

Bertinetto L. Fully-convolutional siamese networks for object tracking / L.Bertinetto, J.Valmadre, J.F.Henriques, A.Vedaldi, P.H.S.Torr // European Conference on Computer Vision — ECCV 2016 Workshops. ECCV 2016. Lecture Notes in Computer Science. — Vol. 9914. — 2016. — P. 850-865.

Henriques J.F. High-speed tracking with kernelized correlation filters / J.F.Henriques, R.Caseiro, P.Martins, J.Batista // IEEE Trans. Pattern Anal. Mach. Intell. (PAMI). — Vol. 37, Iss. 3. — 2015. — P. 583-596.

Kristan M. The Seventh Visual Object Tracking VOT2019 Challenge Results / M.Kristan, J.Matas, A.Leonardis, M.Felsberg та ін. // 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). — 2019.

Bolme D.S. Visual Object Tracking using Adaptive Correlation Filters / D.S.Bolme, R.J.Beveridge, B.A.Draper, Y.M.Lui // IEEE Conference on Computer Vision and Pattern Recognition (CVPR). — 2010. — P. 2544-2550.

Galoogahi H.K. Correlation Filters with Limited Boundaries / H.K.Galoogahi, T.Sim, S.Lucey // IEEE Conference on Computer Vision and Pattern Recognition (CVPR). — 2015. — P. 4630-4638.

Danelljan M. Learning spatially regularized correlation filters for visual tracking / M.Danelljan, G.Häger, F.Khan, M.Felsberg // IEEE International Conference on Computer Vision (ICCV). — 2015. — P. 4310-4318.

Galoogahi H.K. Learning background-aware correlation filters for visual tracking / H.K.Galoogahi, A.Fagg, S.Lucey // IEEE International Conference on Computer Vision (ICCV). — 2017. — P. 1144-1152.

Lukezic A. Discriminative correlation filter tracker with channel and spatial reliability / A.Lukezic, T.Vojir, L.C.Zajc, J.Matas, M.Kristan // IEEE Conference on Computer Vision and Pattern Recognition (CVPR). — 2017. — P. 6309-6318.

Danelljan M. Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking / M.Danelljan, A.Robinson, F.S.Khan, M.Felsberg // 14th European Conference on Computer Vision — ECCV 2016. ECCV 2016. Lecture Notes in Computer Science. — Vol. 9909. — 2016. — P. 472-488.

Feng L. Learning spatial-temporal regularized correlation filters for visual tracking / L.Feng, C.Tian, W.Zuo, L.Zhang, M.-H.Yang // IEEE Conference on Computer Vision and Pattern Recognition (CVPR). — 2018. — P. 4904-4913.

Varfolomieiev A. Channel-independent spatially regularized discriminative correlation filter for visual object tracking / A. Varfolomieiev // Journal of Real-Time Image Processing (RTIP). — 2020. — P. 1-11.

Boyd S. Distributed optimization and statistical learning via the Alternating Direction Method of Multipliers / S.Boyd, N.Parikh, E.Chu, B.Peleato, J.Eckstein // Foundations and Trends in Machine Learning. — Vol. 3, Iss. 1. — 2010. — P. 1-122. DOI: 10.1561/2200000016

Messerschmitt D. Stationary points of a real-valued function of a complex variable. Tech. Report, EECS, U.C. Berkeley. — 2006.

Felzenszwalb P.F. Object Detection with Discriminatively Trained Part Based Models / P.F.Felzenszwalb, R.B.Girshick, D.McAllester, D.Ramanan // IEEE Trans. Pattern Anal. Mach. Intell. (PAMI). — Vol. 32, Iss. 9. — 2010. — P. 1627-1645.

Bertinetto L. Staple: Complementary learners for real-time tracking / L.Bertinetto, J.Valmadre, S.Golodetz, O.Miksik, P.H.S.Torr // IEEE Conference on Computer Vision and Pattern Recognition (CVPR). — 2016. — P. 1401-1409.

Varfolomieiev A. Modification of the KCF tracking method for implementation on embedded hardware platforms / A.Varfolomieiev, O.Lysenko // International Conference Radio Electronics & Info Communications (UkrMiCo), Kyiv. — 2016.

References

Blum R.S., Liu Z. (2006) Multi-Sensor Image Fusion and Its Applications. CRC Press , 528 p. ISBN: 9780849334177.

Zheng Y. (2011) Image Fusion and Its Applications. InTechOpen, 242 p. ISBN: 9789533071824.

Hryvachevskyi A.P., Prudyus I.N. (2018) Enhancing the Informativeness of Multi-spectral Images by means of Multimodal Image Fusion. Visnik NTUU KPI. Ser. Radioteh. radioaparatobuduv., Iss. 73, pp. 40-49. DOI: 10.20535/RADAP.2018.73.40-49.

Maggio E., Cavallaro A. (2011) Video tracking. Theory and Practice. John Wiley & Sons Ltd, 294 p. ISBN: 9780470749647.

Kristan M., Matas J., Leonardis A., Felsberg M. et al (2015) The Visual Object Tracking VOT2015 challenge results. IEEE International Conference on Computer Vision Workshop (ICCVW)>, pp. 564-586. DOI: 10.1109/ICCVW.2015.79.

Kristan M., Leonardis A., Matas J. et al (2016) The Visual Object Tracking VOT2016 challenge results. European Conference on Computer Vision - ECCV 2016 Workshops. ECCV 2016. Lecture Notes in Computer Science, vol. 9914, pp. 777-823. DOI: 10.1007/978-3-319-48881-3_54.

Kristan M., Leonardis A., Matas J., Felsberg M. et al (2017) The visual object tracking VOT2017 challenge results. ICCV2017 Workshops, Workshop on visual object tracking challenge. DOI: 10.1109/ICCVW.2017.230.

Kristan M., Leonardis A., Matas J., Felsberg M. et al (2018) The sixth Visual Object Tracking VOT2018 challenge results. Computer Vision - ECCV 2018 Workshops. ECCV 2018. Lecture Notes in Computer Science, vol. 11129, pp. 3-53. DOI: 10.1007/978-3-030-11009-3_1.

Nam H., Han B. (2016) Learning multi-domain convolutional neural networks for visual tracking. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4293-4302. DOI: 10.1109/CVPR.2016.465.

Danelljan M., Bhat G., Khan F.S., Felsberg M. (2017) ECO: Efficient convolution operators for tracking. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6931-6939. DOI: 10.1109/CVPR.2017.733.

Bertinetto L., Valmadre J., Henriques J.F., Vedaldi A., Torr P.H.S. (2016) Fully-convolutional siamese networks for object tracking. European Conference on Computer Vision - ECCV 2016 Workshops. ECCV 2016. Lecture Notes in Computer Science, Vol. 9914, pp. 850-865. DOI: 10.1007/978-3-319-48881-3_56.

Henriques J.F., Caseiro R., Martins P., Batista J. (2015) High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), Vol. 37, Iss. 3, pp. 583-596. DOI: 10.1109/TPAMI.2014.2345390.

Kristan M., Matas J., Leonardis A., Felsberg M. et al (2019) The Seventh Visual Object Tracking VOT2019 Challenge Results. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). DOI: 10.1109/ICCVW.2019.00276.

Bolme D.S., Beveridge R.J., Draper B.A., Lui Y.M. (2010) Visual Object Tracking using Adaptive Correlation Filters. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2544-2550. DOI: 10.1109/CVPR.2010.5539960.

Galoogahi H.K., Sim T., Lucey S. (2015) Correlation Filters with Limited Boundaries. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4630-4638. DOI: 10.1109/CVPR.2015.7299094.

Danelljan M., Häger G., Khan F., Felsberg M. (2015) Learning spatially regularized correlation filters for visual tracking. IEEE International Conference on Computer Vision (ICCV), pp. 4310-4318. DOI: 10.1109/ICCV.2015.490.

Galoogahi H.K., Fagg A., Lucey S. (2017) Learning background-aware correlation filters for visual tracking. IEEE International Conference on Computer Vision (ICCV), pp. 1144-1152. DOI: 10.1109/ICCV.2017.129.

Lukezic A., Vojir T., Zajc L. C., Matas J., Kristan M. (2017) Discriminative correlation filter tracker with channel and spatial reliability. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6309-6318. DOI: 10.1007/s11263-017-1061-3.

Danelljan M., Robinson A., Khan F.S. and Felsberg M. (2016) Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking. 14th European Conference on Computer Vision - ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, Vol. 9909, pp. 472-488. DOI: 10.1007/978-3-319-46454-1_29.

Li F., Tian C., Zuo W., Zhang Lei, Yang M.-H. (2018) Learning spatial-temporal regularized correlation filters for visual tracking. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4904-4913. DOI: 10.1109/CVPR.2018.00515.

Varfolomieiev A. (2020) Channel-independent spatially regularized discriminative correlation filter for visual object tracking. Journal of Real-Time Image Processing (RTIP), DOI: 10.1007/s11554-020-00967-y.

Boyd S., Parikh N., Chu E, Peleato B., Eckstein J. (2010) Distributed optimization and statistical learning via the Alternating Direction Method of Multipliers. Foundations and Trends in Machine Learning, Vol. 3, Iss. 1, pp. 1-122. DOI: 10.1561/2200000016.

Messerschmitt D. (2006) Stationary points of a real-valued function of a complex variable. Tech. Report, EECS, U.C. Berkeley.

Felzenszwalb P.F., Girshick R.B., McAllester D., Ramanan D. (2010) Object Detection with Discriminatively Trained Part Based Models. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), Vol. 32, Iss. 9, pp. 1627-1645. DOI: 10.1109/TPAMI.2009.167.

Bertinetto L., Valmadre J., Golodetz S., Miksik O., Torr P.H.S. (2016) Staple: Complementary learners for real-time tracking. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1401-1409. DOI: 10.1109/CVPR.2016.156.

Varfolomieiev A., Lysenko O. (2016) Modification of the KCF tracking method for implementation on embedded hardware platforms. International Conference Radio Electronics & Info Communications (UkrMiCo). DOI: 10.1109/UkrMiCo.2016.7739644.

Published

2020-12-30

How to Cite

Варфоломєєв, А. Ю. and Короткий , Є. В. (2020) “The Object Tracker for Infrared and Visual Bands based on Channel-Independent Spatially-Regularized Discriminative Correlation Filter”, Visnyk NTUU KPI Seriia - Radiotekhnika Radioaparatobuduvannia, (83), pp. 5-16. doi: 10.20535/RADAP.2020.83.5-16.

Issue

Section

Radio Circuits and Signals

Most read articles by the same author(s)