Naga Prameela; Marri Swamy Das; Raiza D. Borreo

Multi-head Network based Students Behaviour Prediction with Feedback Generation for Enhancing Classroom Engagement and Teaching Effectiveness

PDF (1294KB), PP.81-100

Views: 0 Downloads: 0

Author(s)

Naga Prameela ^1,* Marri Swamy Das ² Raiza D. Borreo ³

1. Department of Computer Science and Engineering, University College of Engineering, Osmania University, Hyderabad, 500 007, India

2. Department of Computer Science and Engineering, Chaitanya Bharathi Institute of Technology, Hyderabad, 500 075, India

3. Division of Computer Information Science, Higher Colleges of Technology, Abu Dhabi, UAE

* Corresponding author.

DOI: https://doi.org/10.5815/ijitcs.2024.05.06

Received: 20 May 2024 / Revised: 1 Aug. 2024 / Accepted: 7 Sep. 2024 / Published: 8 Oct. 2024

Index Terms

Deep Learning, Cognitive Emotions, Pre-processing, Facial Expression Recognition (FER), Post-processing, Circle-inspired Optimization Algorithm (CIOA), Group Engagement Level (GEL)

Abstract

Emotions are pivotal in the learning process, highlighting the importance of identifying students' emotional states within educational settings. While neural network models, particularly those rooted in deep learning, have demonstrated remarkable accuracy in detecting primary emotions like happiness, sadness, fear, disgust, and anger from facial expressions in videos, these emotions occur infrequently in learning environments. Conversely, cognitive emotions such as engagement, confusion, frustration, and boredom are significantly more prevalent, transpiring five times more frequently than basic emotions. However, unlike basic emotions which are relatively distinct, cognitive emotions present a subtler distinction, necessitating the utilization of more sophisticated models for accurate recognition. The proposed work presents an efficient Facial Expression Recognition (FER) model for monitoring the student engagement in a learning environment by considering their facial expressions like boredom, frustration, confusion and engagement. The proposed methodology includes certain pre-processing steps followed by facial expression recognition founded on Efficient-Net B3 CNN in which the learning parameters are optimized using Circle-Inspired Optimization Algorithm (CIOA). Finally, the post processing stage estimates the frame-wise group engagement level (GEL) of students based on certain expression labels. Based on the acquired results, it is noted that the suggested Efficient-Net B3 CNN-CIOA based FER model provides promising results in terms of accuracy by 99.5%, precision by 99.2%, recall by 99.5% and f1-score by 99.6%, when compared with some state-of-art facial expression recognition approaches. Also, the suggested approach computational complexity is very much less than the compared existing approaches.

Cite This Paper

Naga Prameela, Marri Swamy Das, Raiza D. Borreo, "Multi-head Network based Students Behaviour Prediction with Feedback Generation for Enhancing Classroom Engagement and Teaching Effectiveness", International Journal of Information Technology and Computer Science(IJITCS), Vol.16, No.5, pp.81-100, 2024. DOI:10.5815/ijitcs.2024.05.06

Reference

[1]Q. Li, Y.Q. Liu, Y.Q. Peng, C. Liu, J. Shi, F. Yan, and Q. Zhang, “Real-time facial emotion recognition using lightweight convolution neural network,” In Journal of Physics: Conference Series (Vol. 1827, No. 1, p. 012130). IOP Publishing. 2021, March.
[2]S. Fakhar, J. Baber, S.U. Bazai, S. Marjan, M. Jasinski, E. Jasinska, M.U. Chaudhry, Z. Leonowicz, and S. Hussain, “Smart classroom monitoring using novel real-time facial expression recognition system,” Applied Sciences, 12(23), p.12134, 2022.
[3]I. Lasri, A.R. Solh, and M. El Belkacemi, “Facial emotion recognition of students using convolutional neural network,” In 2019 third international conference on intelligent computing in data sciences (ICDS) (pp. 1-6). IEEE. 2019, October.
[4]V.U. Pinjarkar, U.S. Pinjarkar, H.N. Bhor, Y.V. Mahajan, V.R. Patil, S.D. Rajput, P. Kothari, D. Ghori, and H.P. Bhabad, “Student Engagement Monitoring in Online Learning Environment,” International Journal of Intelligent Systems and Applications in Engineering, 12(1), pp.292-298,2024.
[5]Y. Ma, Y. Wei, Y. Shi, X. Li, Y. Tian, and Z. Zhao, “Online learning engagement recognition using bidirectional Long-Term recurrent convolutional networks,” Sustainability, 15(1), p.198, 2022.
[6]S. Khan, and S. Safa, “Revisiting Annotations in Online Student Engagement,” In Proceedings of the 2024 10th International Conference on Computing and Data Engineering (pp. 111-117). 2024, January.
[7]P.R. Komaravalli, and B. Janet, “Detecting Academic Affective States of Learners in Online Learning Environments Using Deep Transfer Learning,” Scalable Computing: Practice and Experience, 24(4), pp.957-970, 2023.
[8]S. Gupta, P. Kumar, and R. Tekchandani, “EDFA: Ensemble deep CNN for assessing student's cognitive state in adaptive online learning environments,” International Journal of Cognitive Computing in Engineering, 4, pp.373-387,2023.
[9]S. Fakhar, J. Baber, S.U. Bazai, S. Marjan, M. Jasinski, E. Jasinska, M.U. Chaudhry, Z. Leonowicz, and S. Hussain, “Smart classroom monitoring using novel real-time facial expression recognition system,” Applied Sciences, 12(23), p.12134,2022.
[10]S. Kavitha, G. Raghuraman, A. Kavithasri, S. Aishvarya, and B. Janani, “Learning behaviour analysis of online course learners using EEG and facial expression data,” Measurement: Sensors, 25, p.100669,2023.
[11]X. Xu, D.M. Dugdale, X. Wei, and W. Mi, “Leveraging artificial intelligence to predict young learner online learning engagement,” American Journal of Distance Education, 37(3), pp.185-198, 2023.
[12]A. Sukumaran, and A. Manoharan, “Multimodal Engagement Recognition from Image traits using Deep Learning Techniques,” IEEE Access. 2024.
[13]Q. Li, Y.Q. Liu, Y.Q. Peng, C. Liu, J. Shi, F. Yan, and Q. Zhang, “Real-time facial emotion recognition using lightweight convolution neural network,” In Journal of Physics: Conference Series (Vol. 1827, No. 1, p. 012130). IOP Publishing. 2021, March.
[14]T. Selim, I. Elkabani, and M.A. Abdou, “Students engagement level detection in online e-learning using hybrid efficientnetb7 together with tcn, lstm, and bi-lstm,” IEEE Access, 10, pp.99573-99583,2022.
[15]S. Malekshahi, J.M. Kheyridoost, and O. Fatemi, “A General Model for Detecting Learner Engagement: Implementation and Evaluation,” arXiv preprint arXiv:2405.04251,2024.
[16]P. Buono, B. De Carolis, F. D’Errico, N. Macchiarulo, and G. Palestra, “Assessing student engagement from facial behavior in on-line learning,” Multimedia Tools and Applications, 82(9), pp.12859-12877, 2023.
[17]Chaudhary, Snehal, “Training Convolutional Neural Network with Logistic Regression Model for Facial Recognition to Monitor Attentiveness in Classrooms," International Journal of Intelligent Systems and Applications in Engineering 12.8s (2024): 592-598.
[18]Z. abed Almoussawi, K.S. Kumar, T. Veena, M.M. Adnan, and M.B. Jabar, “Face Detection and Classification using Convolutional Neural Network with Channel Attention Module,” In 2023 3rd International Conference on Mobile Networks and Wireless Communications (ICMNWC) (pp. 1-6). IEEE. 2023, December.
[19]I. Alkabbany, A.M. Ali, C. Foreman, T. Tretter, N. Hindy, and A. Farag, “An experimental platform for real-time students engagement measurements from video in stem classrooms,” Sensors, 23(3), p.1614, 2023.
[20]K. Watanabe, T. Sathyanarayana, A. Dengel, and S. Ishimaru, “Engauge: Engagement gauge of meeting participants estimated by facial expression and deep neural network,” IEEE Access, 11, pp.52886-52898, 2023.
[21]Z. Trabelsi, F. Alnajjar, M.M.A. Parambil, M. Gochoo, and L. Ali, “Real-time attention monitoring system for classroom: A deep learning approach for student’s behavior recognition,” Big Data and Cognitive Computing, 7(1), p.48, 2023.
[22]A.V. Savchenko, L.V. Savchenko, and N.S. Belova, “Group-Level Affect Recognition in Video Using Deviation of Frame Features,” In International Conference on Analysis of Images, Social Networks and Texts (pp. 199-207). Cham: Springer International Publishing. 2021, December.
[23]H. Alhichri, A.S. Alswayed, Y. Bazi, N. Ammour, and Alajlan, “Classification of remote sensing images using EfficientNet-B3 CNN model with attention,” IEEE access, 9, pp.14078-14094, 2021.
[24]X. Zhang, H. Zeng, S. Guo, and L. Zhang, “Efficient long-range attention network for image super-resolution. In European conference on computer vision (pp. 649-667),” Cham: Springer Nature Switzerland. 2022, October.
[25]O.A.P. de Souza, and L.F.F. Miguel, “CIOA: Circle-Inspired Optimization Algorithm, an algorithm for engineering optimization,” SoftwareX, 19, p.101192, 2022.
[26]F.M. Shiri, E. Ahmadi, M. Rezaee, and T. Perumal, “Detection of Student Engagement in E-Learning Environments Using EfficientnetV2-L Together with RNN-Based Models,”.
[27]Mukhopadhyay, Moutan, Aniruddha Dey, and Sayan Kahali. "A deep-learning-based facial expression recognition method using textural features." Neural Computing and Applications 35.9 (2023): 6499-6514.
[28]Tian, Xinran, et al. "Predicting student engagement using sequential ensemble model." IEEE Transactions on Learning Technologies (2023).
[29]Santoni, Mayanda Mega, T. Basaruddin, and KasiyahJunus. "Convolutional Neural Network Model based Students’ Engagement Detection in Imbalanced DAiSEE Dataset." International Journal of Advanced Computer Science and Applications 14.3 (2023).
[30]Mehta, Naval Kishore, et al. "Three-dimensional DenseNet self-attention neural network for automatic detection of student’s engagement." Applied Intelligence 52.12 (2022): 13803-13823.
[31]Mejia-Escobar, Christian, Miguel Cazorla, and Ester Martinez-Martin. "Improving Facial Expression Recognition through Data Preparation & Merging." IEEE Access (2023).
[32]Gupta, Swadha, Parteek Kumar, and Raj Kumar Tekchandani. "Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models." Multimedia Tools and Applications 82.8 (2023): 11365-11394.
[33]Meena, Gaurav, et al. "Identifying emotions from facial expressions using a deep convolutional neural network-based approach." Multimedia Tools and Applications 83.6 (2024): 15711-15732.
[34]Reghunathan, Resmi K., et al. "Facial Expression Recognition Using Pre-trained Architectures." Engineering Proceedings 62.1 (2024): 22.
[35]Gupta, Swadha, Parteek Kumar, and Rajkumar Tekchandani. "A multimodal facial cues based engagement detection system in e-learning context using deep learning approach." Multimedia Tools and Applications 82.18 (2023): 28589-28615.
[36]Aly, Mohammed. "Revolutionizing online education: Advanced facial expression recognition for real-time student progress tracking via deep learning model." Multimedia Tools and Applications (2024): 1-40.
[37]Fida, Alisha, et al. "Real time emotions recognition through facial expressions." Multimedia Tools and Applications (2023): 1-28.
[38]Yao, Lisha. "Facial Expression Recognition Based on Multiscale Features and Attention Mechanism." Automatic Control and Computer Sciences 58.4 (2024): 429-440.

International Journal of Information Technology and Computer Science (IJITCS)