Emotions are pivotal in the learning process, highlighting the importance of identifying students' emotional states within educational settings. While neural network models, particularly those rooted in deep learning, have demonstrated remarkable accuracy in detecting primary emotions like happiness, sadness, fear, disgust, and anger from facial expressions in videos, these emotions occur infrequently in learning environments. Conversely, cognitive emotions such as engagement, confusion, frustration, and boredom are significantly more prevalent, transpiring five times more frequently than basic emotions. However, unlike basic emotions which are relatively distinct, cognitive emotions present a subtler distinction, necessitating the utilization of more sophisticated models for accurate recognition. The proposed work presents an efficient Facial Expression Recognition (FER) model for monitoring the student engagement in a learning environment by considering their facial expressions like boredom, frustration, confusion and engagement. The proposed methodology includes certain pre-processing steps followed by facial expression recognition founded on Efficient-Net B3 CNN in which the learning parameters are optimized using Circle-Inspired Optimization Algorithm (CIOA). Finally, the post processing stage estimates the frame-wise group engagement level (GEL) of students based on certain expression labels. Based on the acquired results, it is noted that the suggested Efficient-Net B3 CNN-CIOA based FER model provides promising results in terms of accuracy by 99.5%, precision by 99.2%, recall by 99.5% and f1-score by 99.6%, when compared with some state-of-art facial expression recognition approaches. Also, the suggested approach computational complexity is very much less than the compared existing approaches.