Pedestrian Attributes and Activity Recognition Using Deep Learning: A Comprehensive Survey

Mohammed  Fadhil Asghar; Mudhafar Hussein Ali; Jumana Waleed

doi:10.58564/IJSER.1.2.2022.51

Authors

Mohammed Fadhil Asghar Department of Computer Engineering, College of Engineering, Al-Iraqia University, Iraq
Mudhafar Hussein Ali Department of Computer Engineering, College of Engineering, Al-Iraqia University, Iraq
Jumana Waleed Department of Computer Science ,College of Science, University of Diyala, Iraq

DOI:

https://doi.org/10.58564/IJSER.1.2.2022.51

Keywords:

Pedestrian Attribute (PA) recognition, Pedestrian Activity recognition, Deep learning

Abstract

In recent years, pedestrian attributes and activity recognition has attracted expanding research emphasis due to their considerable research importance and value of application in the intelligent civil and military domains. Owing to the inadequate image or frame quality of inexpensive cameras and the absence of obvious and stable feature information, direction of pedestrian movement, and so on, the complication of pedestrian attributes and activity recognition is expanded. However, with the comprehensive implementation of deep learning techniques, pedestrian attributes and activity recognition has made a substantial advance. This paper is the first of its kind because it combines the recognition of pedestrian attributes and activities separately, and reviews the works on pedestrian attributes and activities using deep learning in relation to datasets. The fundamental concepts, corresponding challenges, and popular solutions are also explained. Furthermore, in this community, metrics of evaluation and concise performance comparisons are given. In the end, the hotspots of the present research and the directions of the future research are summarized.

References

I. N. Junejo, “Pedestrian attribute recognition using two-branch trainable Gabor wavelets network,” PLoS One, vol. 16, no. 6, 2021. DOI: https://doi.org/10.1371/journal.pone.0251667

Chollet, F. (2021). Deep learning with Python. Simon and Schuster.‏

T. M. Hasan, S. D. Mohammed, J. Waleed, "Development of Breast Cancer Diagnosis System Based on Fuzzy Logic and Probabilistic Neural Network", Eastern-European Journal of Enterprise Technologies, Information and Controlling System, Vol. 4, No. 9 (106), pp. 6-13, 2020. DOI: https://doi.org/10.15587/1729-4061.2020.202820

Deng, Y.; Luo, P.; Loy, C.C.; Tang, X. Pedestrian attribute recognition at far distance. In Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA, 3–7 November 2014; pp. 789–792. DOI: https://doi.org/10.1145/2647868.2654966

Xiao Wang, Shaofei Zheng, Rui Yang, Aihua Zheng, Zhe Chen, Jin Tang, Bin Luo,Pedestrian attribute recognition: A survey, Pattern Recognition, Vol. 121,2022. DOI: https://doi.org/10.1016/j.patcog.2021.108220

Chen, X., Zhuang, S., Zheng, X., & Wang, Z. (2021, December). Pedestrian Attribute Recognition Based On Deep Learning: A Survey. In 2021 International Conference on Information Technology and Biomedical Engineering (ICITBE) (pp. 140-144). IEEE.‏ DOI: https://doi.org/10.1109/ICITBE54178.2021.00039

Fang W., Chen J., Lu T., Hu R. (2018) Pedestrian Attributes Recognition in Surveillance Scenarios with Hierarchical Multi-task CNN Models. Advances in Multimedia Information Processing – PCM 2018. PCM 2018. Lecture Notes in Computer Science, vol 11165. Springer, Cham. DOI: https://doi.org/10.1007/978-3-030-00767-6_70

Han, K., Wang, Y., Shu, H., Liu, C., Xu, C., & Xu, C. (2019). Attribute aware pooling for pedestrian attribute recognition. arXiv preprint arXiv:1907.11837.‏ DOI: https://doi.org/10.24963/ijcai.2019/341

Qiaozhe Li, Xin Zhao, Ran He, Kaiqi Huang. Visual-semantic graph reasoning for pedestrian attribute recognition. In: Proceedings of the AAAI conference on artificial intelligence. 2019. p. 8634-8641.‏ DOI: https://doi.org/10.1609/aaai.v33i01.33018634

Qiaozhe Li, Xin Zhao, Ran He, Kaiqi Huang. "Pedestrian Attribute Recognition by Joint Visual-semantic Reasoning and Knowledge Distillation." IJCAI. 2019.‏

H. An, H. Fan, K. Deng and H. -M. Hu, "Part-guided Network for Pedestrian Attribute Recognition," 2019 IEEE Visual Communications and Image Processing (VCIP), 2019, pp. 1-4, DOI: https://doi.org/10.1109/VCIP47243.2019.8965957

X. He, Q. Shi, F. Su, Z. Zhao and B. Zhuang, "Pedestrian Attribute Recognition Based on Mtcnn with Online Batch Weighted Loss," 2019 IEEE International Conference on Image Processing (ICIP), 2019, pp. 2461-2465. DOI: https://doi.org/10.1109/ICIP.2019.8803227

Li, Y.; Xu, H.; Bian, M.; Xiao, J. Attention Based CNN-ConvLSTM for Pedestrian Attribute Recognition. Sensors 2020, 20, 811 DOI: https://doi.org/10.3390/s20030811

Zhong Ji, Zhenfei Hu, Erlu He, Jungong Han,Yanwei Pang,Pedestrian attribute recognition based on multiple time steps attention, Pattern Recognition Letters, Volume 138,2020,Pages 170-176, DOI: https://doi.org/10.1016/j.patrec.2020.07.018

Tan, Z., Yang, Y., Wan, J., Guo, G., & Li, S. Z. (2020, April). Relation-aware pedestrian attribute recognition with graph convolutional networks. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 07, pp. 12055-12062).‏ DOI: https://doi.org/10.1609/aaai.v34i07.6883

Yang, Y., Tan, Z., Tiwari, P. et al. Cascaded Split-and-Aggregate Learning with Feature Recombination for Pedestrian Attribute Recognition. International Journal of Computer Vision, 129, 2731–2744, 2021. DOI: https://doi.org/10.1007/s11263-021-01499-z

Jia, J., Chen, X., & Huang, K. (2021). Spatial and Semantic Consistency Regularizations for Pedestrian Attribute Recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 962-971).‏ DOI: https://doi.org/10.1109/ICCV48922.2021.00100

Liu, X.; Zhao, H.; Tian, M.; Sheng, L.; Shao, J.; Yi, S.; Yan, J.; Wang, X. Hydraplus-net: Attentive deep features for pedestrian analysis. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 350–359. DOI: https://doi.org/10.1109/ICCV.2017.46

Liu, P., Liu, X., Yan, J., & Shao, J. (2018). Localization guided learning for pedestrian attribute recognition. arXiv preprint arXiv:1808.09102.‏

X. Zheng, Z. Yu, L. Chen, F. Zhu and S. Wang, "Multi-label Contrastive Focal Loss for Pedestrian Attribute Recognition," 2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 7349-7356. DOI: https://doi.org/10.1109/ICPR48806.2021.9411959

Chen, WC., Yu, XY. & Ou, LL. Pedestrian Attribute Recognition in Video Surveillance Scenarios Based on View-attribute Attention Localization. Mach. Intell. Res. (2022). DOI: https://doi.org/10.1007/s11633-022-1321-8

A. Rasouli, I. Kotseruba, T. Kunic and J. Tsotsos, "PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction," 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 6261-6270 DOI: https://doi.org/10.1109/ICCV.2019.00636

A. Rasouli, M. Rohani and J. Luo, "Bifold and Semantic Reasoning for Pedestrian Behavior Prediction," 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 15580-15590..‏ DOI: https://doi.org/10.1109/ICCV48922.2021.01531

Javier Lorenzo, Ignacio P. Alonso, Rubén Izquierdo, Augusto L. Ballardini, Álvaro H. Saz, David F. Llorca, and Miguel Á. Sotelo, "CAPformer: Pedestrian Crossing Action Prediction Using Transformer", Sensors, Vol. 21, No. 17: 5694, 2021.‏ DOI: https://doi.org/10.3390/s21175694

R. Quan, L. Zhu, Y. Wu and Y. Yang, "Holistic LSTM for Pedestrian Trajectory Prediction," in IEEE Transactions on Image Processing, vol. 30, pp. 3229-3239, 2021. DOI: https://doi.org/10.1109/TIP.2021.3058599

Li, D., Zhang, Z., Chen, X., Ling, H., & Huang, K. (2016). A richly annotated dataset for pedestrian attribute recognition. arXiv preprint arXiv:1603.07054.‏.

Ehsan Yaghoubi, Diana Borza, João Neves, Aruna Kumar, Hugo Proença, “An attention-based deep learning model for multiple pedestrian attributes recognition”, Image and Vision Computing, Vol. 102,2020. DOI: https://doi.org/10.1016/j.imavis.2020.103981

Imran N. Junejo, Naveed Ahmed, Mohammad Lataifeh, “Pedestrian attribute recognition using trainable Gabor wavelets”, Heliyon, Vol. 7, No. 6, 2021. DOI: https://doi.org/10.1016/j.heliyon.2021.e07422

Imran N. Junejo, and Naveed Ahmed, “Depthwise Separable Convolutional Neural Networks for Pedestrian Attribute Recognition”, SN Computer Science, Vol. 2, no. 100, 2021. DOI: https://doi.org/10.1007/s42979-021-00493-z

Bouchabou, D.; Nguyen, S.M.; Lohr, C.; LeDuc, B.; Kanellos, I. A Survey of Human Activity Recognition in Smart Homes Based on IoT Sensors Algorithms: Taxonomies, Challenges, and Opportunities with Deep Learning. Sensors 2021, 21, 6037. DOI: https://doi.org/10.3390/s21186037

P. V. K. Borges, N. Conci and A. Cavallaro, "Video-Based Human Behavior Understanding: A Survey," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 23, no. 11, pp. 1993-2008, Nov. 2013. DOI: https://doi.org/10.1109/TCSVT.2013.2270402

Schneider N., Gavrila D.M. (2013), Pedestrian Path Prediction with Recursive Bayesian Filters: A Comparative Study. Pattern Recognition. GCPR 2013. Lecture Notes in Computer Science, vol 8142. Springer, Berlin, Heidelberg DOI: https://doi.org/10.1007/978-3-642-40602-7_18

P. Xue, J. Liu, S. Chen, Z. Zhou, Y. Huo and N. Zheng, "Crossing-Road Pedestrian Trajectory Prediction via Encoder-Decoder LSTM," 2019 IEEE Intelligent Transportation Systems Conference (ITSC), 2019, pp. 2027-2033.‏ DOI: https://doi.org/10.1109/ITSC.2019.8917510

Fang, Z., Vázquez, D., & López, A. M. (2017). On-board detection of pedestrian intentions. Sensors, 17(10), 2193.‏ DOI: https://doi.org/10.3390/s17102193

H. Kataoka, Y. Aoki, Y. Satoh, S. Oikawa and Y. Matsui, "Fine-Grained Walking Activity Recognition via Driving Recorder Dataset," 2015 IEEE 18th International Conference on Intelligent Transportation Systems, 2015, pp. 620-625. DOI: https://doi.org/10.1109/ITSC.2015.107

H. Kataoka, Y. He, S. Shirakabe, Y. Satoh, “Motion Representation with Acceleration Images”. Computer Vision, ECCV 2016 Workshops. ECCV 2016. Lecture Notes in Computer Science, Vol. 9915. Springer, Cham, 2016. DOI: https://doi.org/10.1007/978-3-319-49409-8_3

H. Kataoka, Y. Miyashita, M. Hayashi, K. Iwata, Y. Satoh, "Recognition of Transitional Action for Short-Term Action Prediction using Discriminative Temporal CNN Feature", Proceedings of the British Machine Vision Conference (BMVC), BMVA Press, pp. 1-12, 2016.‏ DOI: https://doi.org/10.5244/C.30.12

Wongun Choi and S. Savarese, “A unified framework for multitarget tracking and collective activity recognition’, In Computer Vision–ECCV 2012, pages 215–230. Springer, 2012. DOI: https://doi.org/10.1007/978-3-642-33765-9_16

Zhiwei Deng, Mengyao Zhai, Lei Chen, Yuhao Liu, Srikanth Muralidharan, Mehrsan Javan Roshtkhari, Greg Mori, "Deep structured models for group activity recognition." arXiv preprint arXiv:1506.04191 (2015).‏ DOI: https://doi.org/10.5244/C.29.179

Z. Deng, A. Vahdat, H. Hu and G. Mori, "Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4772-4781.‏ DOI: https://doi.org/10.1109/CVPR.2016.516

M. S. Ibrahim, S. Muralidharan, Z. Deng, A. Vahdat and G. Mori, "A Hierarchical Deep Temporal Model for Group Activity Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1971-1980. DOI: https://doi.org/10.1109/CVPR.2016.217

L. Kong, J. Qin, D. Huang, Y. Wang and L. Van Gool, "Hierarchical Attention and Context Modeling for Group Activity Recognition," 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 1328-1332.‏ DOI: https://doi.org/10.1109/ICASSP.2018.8461770

C. Zalluhoglu, N. Ikizler-Cinbis, "Region based multi-stream convolutional neural networks for collective activity recognition", Journal of Visual Communication and Image Representation, Vol. 60, pp. 170-179, 2019.‏ DOI: https://doi.org/10.1016/j.jvcir.2019.02.016

Waleed, J., Albawi, S., Flayyih, H. Q., & Alkhayyat, A. (2021, September). An Effective and Accurate CNN Model for Detecting Tomato Leaves Diseases. In 2021 4th International Iraqi Conference on Engineering Technology and Their Applications (IICETA) (pp. 33-37). IEEE.‏ DOI: https://doi.org/10.1109/IICETA51758.2021.9717816

Waleed, J., Abbas, T., & Hasan, T. M. (2022, March). Facemask Wearing Detection Based on Deep CNN to Control COVID-19 Transmission. In 2022 Muthanna International Conference on Engineering Science and Technology (MICEST) (pp. 158-161). IEEE.‏ DOI: https://doi.org/10.1109/MICEST54286.2022.9790197