Human Facial Expression Detection

Artificial intelligence field has been evolved so much and has brought a whole new revolution in the technology as it further majorly splits into deep learning and machine learning. AI is the ability of the machines to demonstrate the tasks and human behaviour just like a natural human being, but through the machines or the computers. Artificial technology and its parts have major significance in various sectors and tends to promote several developments, one of the most promising and unique human facial expression detection technique with the utilization of artificial intelligence.

The field is frequently active in recognizing and promoting the computational vision, further the research over sentiment assessment from the use of visual data and just looking at the expression of the human. Research over the interaction technology is now become popular and one of its advantages are facial expression recognition which is known as a very crucial sort of visual information which is used for understanding the emotional situation for the human (Ahmed et al., 2019). Detection of the emotions have become quite an urgent necessity as it applications are evolving in the field of artificial intelligence, for example computer-human collaboration, communication between robot and human, and data driven technology, these all offers benefits to the automation and advances the world and technology.

Since, the technology is in an utter demand and the objective of this research is to make a model with the help of artificial intelligence which can identify the facial expression and detect the emotion a human is feeling. The model is helpful in businesses, as well as even the police and surveillance industry are also using this technology in identifying the motions of the humans walking and detecting if they are feeling anxious or sad helping them out, however it is only a few countries only. However, in the businesses, it can be a helpful in identifying the moos of the employees and cheer them up, as it will not only lead to a good engagement at the workplace but also boost up the productivity of the employees.

The study shows the use of machine learning models and libraries and the use of deep learning methods including artificial neural network, convolutional neural network (CNN)- it’s major advantage is that it is used in analysing the imagery visuals, for example identifying and classifying the images and objects. In short, this feature is being extracted with the help of CNN and its presence in the development of this technological idea and the technology has been useful as it will aided in the data augmentation, as the data is being collected from various sources.

The further study contains a section which will tell about the working of this model, it will picture about how the model will work, in short, firstly the model will take a picture from the given dataset and go with the face detection by the cascade classifier. If the face detected is found then it will be further be sent for the processing, the data is augmented by the function named ImageData Generator of the API –Keras. This augmented dataset is fed to the Convolutional neural network so that class can be predicted.

Further it was observed that the, facial expression classification contains three convolutional layers having 32, 64, and 128 filters in it as well as the kernel size if of 3x3. However, to reach to a conclusion, the data needs to be processed, hence the study contains a section of data processing which will show how the image taken will be analysed and processed for further operations. The basic steps it will include are- the data is collected from various sources, which includes FER2013, CK+, KDEF, etc. further these are the steps which is being used for the processing of the data- face detection and cropping the image, followed by grayscale conversion, normalizing the image and lastly augmenting the image.

The study will follow the research methodology which includes hybrid, which means qualitative and quantitative. Qualitative research methodology refers to the method of pursuing the research which includes theoretical section, in this dissertation of human facial recognition using machine and deep learning & artificial intelligence, several research iver the previous studies will be conducted and mentioned in the form of literature review. This will aid in knowing the already existing studies and will help in conducting the further research and development of new libraries, as well as notify about what went bad and did not work. The study has typically taken reference from the existing sources such as research papers, data, website sources, etc.

However the research over the dissertation also follows the primary research method which includes conducting and finding the data by oneself. It typically includes observation, which means observing the data and predicting the possibilities and results, surveys including what is the early necessity of conducting this project. Most important the research over which the majority of the study relies on is quantitative research methodology which includes the collection of the data in the form of images and numbers which will be sent for the analysis and processing, and on the basis of that the emotion through the facial detection will be generate or results will come out.

Along with the literature review, working, research methodology, the study focuses on electronic framework or the physical framework which will process and show the results on the screen, it includes a huge server connection, unit rooms, processors, webcams, monitoring system, computer system, memory unit, and connectors. This all will help in collecting and storing the data efficiently and in an effective manner, along with the processing will be done as the coding will only be done in the computer system, after processing the servers will analyse it and coordinate with main unit to know the results of the detected facial emotion of the human. However, the main hardware resource which is required to carry out this entire process is a web camera, as the project’s name suggests human facial emotion detection, it requires to take a photo of the human face in a live manner, as there may be times when the data set s run out and the system needs to take a live pic and detect the emotion through their expression.

The dissertation further consists a section of discussion, which comes after the data analysis sections and it includes interpret and analyse, in simple language this section will picture about what the results of the data analysis mean and how they will be forma a relation among the research. Finding and recommendation section of the study aims at depicting what the dissertation over the topic “Human facial expression detection using Artificial intelligence” have offered to learn, such as what sort of data can be used, techniques and tools for the data analysis, libraries for the facial detection, along with the various programming languages. The recommendation section will focus on the parameters which can be included and enhanced with the help of research and it will aid in a better project development, as well as it will help in the further studies.

Along with, this section, the most important section that the study has is results section, it includes the outcomes of the whole research conducted for the data analysis of the techniques such as ML, CNN, ANN, DNN, etc. and checking if the model is predicting the right emotions through the facial detection or not. The aim of the project is developing a model which can detect the human emotion, either they are happy, sad, angry, etc. however, this ca be done with the help of machine learning, but this study is focused on the use of deep leaning as well as it offers various advancement and advantages over any other technology such as convolutional neural network and helps in the detection pf the emotion through just an expression, which is an unique idea.

The variability of diverse feelings is intricately tied to the context of human interaction. People's faces reveal a range of expression profiles when they are feeling basic emotions, each having its very own set of traits and dispersion scale. Face expression identification is a critical element of human-computer communication because it enables computers to comprehend facial expressions focusing on individual reasoning. Facial recognition, extraction of features, and categorization are the three main modules in the facial expression identification process (Song, 2021). The study will help in through developing an understanding over the artificial intelligence and its application in various fields, as this technology is advancing the world, for example the development of emotion identification through facial expression identification. Sone of the part of the study will also focus on geometric feature based convolutional neural network-based sentiment recognition, it will analyse each angle of the face and curves, on the basis of that identify the emotion as per the standard and well researched set of parameters.

Literature review

Sentiment recognition using CNN and edge computing

Facial expression recognition through the use of the technological method is a complicated tasks and hence this factor needed to be overlooked with the use of new implantation of the artificial intelligence in the industry. Hence, this study of face recognition comprises the major concept of deep learning, and majorly focuses on the involvement of convolutional neural network further the detection of image edge will be proposed. According to Zhang et al. 2019, in the convolution kernel, the edge from each tier of the photo is removed, and the face expression image is normalised. To retain the texture picture's edge shape information, the retrieved edge information is put on each feature image. The maximum pooling technique is then used to reduce the dimensions of the obtained implicit characteristics. Finally, a Softmax classifier is used to classify and recognise the sentiment of the test picture.

Figure 1: CNN For facial expression

(Source: Zhang et al., 2019)

A simulated experiment is constructed by properly blending the Fer-2013 face expression data with LFW data set to demonstrate the resilience of this approach for facial expression identification under a complex context. As per the research conducted by the author, they have observed that the rate of average recognition is 88.56%. The study includes the data processing of the facial expression following the steps-

Face detection and location- It uses Haar classifier, which is a small feature utilized for texture descriptor, the main features include diagonal, linear, and edge. It is helpful in reflecting the grey level change in the image which will affect the human face in the picture as the human face tends to have several contrasts and changes
Scale normalization- Initially the input photo is a fixed picture of size, earlier than the photo is inserted into the networks, hence it is necessary to normalize the original image in order to produce specified size of the photo.

Figure 2: Before and after normalization

(Source: Zhang et al., 2019)

Gray level equalization- in the acquisition process, it is noticed that the actual image may get affected by the other factors, such as illumination, these all things make the photo appear as unevenly and disturbed which might lead to problem in identifying the emotion detected. Hence, it is very crucial to average the level of gray in the image in order to advance the image contrast.

Figure 3: Before and after gray level equalization

(Source: Zhang et al., 2019)

The research utilizes the layers for the process, including full connection layer, convolution layer, pooling layer, CNN parameter, and softmax layer. Further, the author performed the performance analysis and comparative analysis experiments for ensuring the working of the model as well as identifying the efficiency of each model in the job (Yang et al., 2018).

Methods based on artificial intelligence which provides the human-computer intelligent interaction

According to the author human-computer interaction is a field of research is in that it helps people to make proper use of design and technical implementation in communication. The increased development of technology has also improved the concept of technical changes in the field of communication. The computer programs are used to provide the software based on the user’s requirement. The technology based on the HCI has increased in recent years that have modified the concept of communication. According to the HCI has become the most important topic in today’s time which helps to the basic understanding ding about the facial recognition methods (Sumak et al., 2021).

Figure 4: System flowchart

(Source: Ahmed et al., 2019)

This concept has increased the need for an intelligent user interface (IUI) for that it has included the section of artificial intelligence methods based on the algorithms. The study has given a basic understanding of the ML approaches. That included the different categories of the machine learning methodology like supervised learning, reinforcement learning, and unsupervised learning. Based on this concept the study has given information about different algorithms which includes ANN, deep learning, conventional neural network, RNN, genetic algorithms, Bayesian network so on (Hussain & Balushi, 2020).

It has concluded the section of related studies in which conventional information is provided based on the different author reports. Which has included the method based on the SLR and SMS studies the improvised the data collection methods which can be used to provide the knowledge in the particular field. A systematic mapping study has provided the gathering material list for the conventional development of the methods (Sumak et al., 2021).

Figure 5: SMS process adoption

(Source: sumac et al., 2022)

In almost all the concepts of human face interaction, the need for sensors is a must need that can helps to divide the capabilities. The development of AI has increased the inclusion of modified sensors. They are can be categorized based on face recognition, attitude recognition, and figure print recognition. And the device that is used in the process of it is the wrist control sensors, biometric sensors, electroencephalography sensors for processing the data. And different kind of multi-biological sensors is used for example poly GI, EEG, ECG, EMG, and BIOPAC MP 150. The result section has provided the methods of identifying and mapping the research systematically (Sumak et al., 2021).

A brief review based on the human emotion recognition by implementing visual information

According to the author, this concept is becoming more popular for making an effective interaction between humans and machines. They are can be said as the most important factor in making effective communication which can help to understand the intention of others. In this process, a person can explain their emotions feeling, sadness, anger, and vocal tones which gives emotional understanding to the other person. In this report, the survey has been considered to find the verbal components that can help to convey one-third of the human communication process (Ko., 2018).

Figure 6: Face detection using AI

(Source: Daws, 2019)

In this report FER term has been used that defines this process in different words which include facial emotion recognition. In the development, the face is executed to determine the emotion of the humans that gives the accessibility to understand the feeling of the peoples. For the implication of this technology many techniques are has been used such as virtual reality, advanced driver assistant systems, and augmented reality. On the other hand, they are based on the sensor implication which helps in collecting the data from the user. The sensors that are included in this process are the cameras, electrocardiogram, electromyograph, and electroencephalograph. These tools are can be used as an input method for collecting the data related to the human face functionality (Ko, 2018).

Figure 7: Procedure used in the FER in this study

(Source: Ko, 2018)

The report has divided into two different aspects which define the functionality of FER which are used to divide the features according to the category.

Conventional FER approach: According to the author, many pieces of research is studies done for the convenient implementation of the human recognition system. From the author’s point of view, the best method is the Artificial intelligence method which gives the automated system on the particular scenario. The approach has divided the concept into different featured categories. That includes the geometrical feature, appearance-based feature, and hybrid feature (Ko., 2018).
Contrast to the traditional approach: The traditional approach is the inclusion of CNN for the development of the software based on the FER. This approach gives the accessibility use the Deep learning methodology for making effective recognizing applications. This can help to understand the various dataset to test the capability of emotion detection and network training. According to the author, this concept contains three heterogeneous layers which are defined as the max-pooling layers, convolution layer, and fully connected layer. That helps to take the take images as an input and convert them into a set of filters. After that it represses gives the output based on the spatial arrangements of facial images (Zhang et al., 2019).

The study has also concluded the topic in which t has given the information about the FER databases. That data sets are being used to provide extensive and comparative experiments. The report has also discussed the popular databases sets based on the 2D and 3D video sequences (Ko., 2018).

Convolutional neural network for facial emotion recognition

According to the author facial expression plays an important role in understanding the feeling of people. It can be said the nonverbal way of expression is based on the emotions of humans. This study has provided a solution based on the convolutional neural network for identifying the thinking of the people. It has defined the study based on the five important facial expressions that include anger, sad, feared, astonished, and smiling. The FERC algorithms have given the manuscript based on the expressional examination. Furthermore, this is used to categorize the expression into the imaginary form which divides them into the five categorized parts of the emotion classes (Mehendal, 2020).

The fundamental of CNN is based on the two different aspects in which the first one is used to remove the background style from the image. And the second one is used to provide the features based on the facial vector extraction. In the EV model, it is used to find and define the category based on the five different regular expressions. It can provide the data of supervised categories that can be used to store the 10000 images in a single database. The two-level CNN works in a sequential manner in which the last layer is used for making adjustments on the exponents and weights values (Guo et al., 2018).

Figure 8: Block diagram for FERC

(Source: Mehendal, 2020)

Based on the theory that is given by the author this has divided the facial expression into two different categories or approaches. The first is used to distinguish the expression on the identified with the explicit identifiers. On the other hand, the second approach is used to develop a characterization that depends on the outer facial highlights. Expression markers are used for a facial action coding system that belongs to the different action units (Mehendal, 2020).

The study has conducted a literature review on facial expression in which the author has analyzed and studied the different concepts of the CNN algorithms. The frame extraction concept is amplified by taking the input from the video graphical format. For providing the proper stability to the imaginary video it has given the concept of a Canny edge detector. Which was used to provide the structure for calculating the white pixel sum. The concept is based on theological competency for the following algorithms. The author has also provided the list of software and hardware that should be used while performing the functionality of facial expressions (Mehendal, 2020).

Survey on facial expression recognition

The study has provided a survey-based study on facial expression recognition in which the author has surveyed people based on the survey questionnaires. According to the Huang et al., 2019, facial is becoming the most useful means of communication in the non-verbal field. in this, the survey is categorized into different fields which helps to define the concept of working of their methodology. In this, the study has concluded the section of the FER terminologies which gives brief knowledge about them.

The whole concept of the study is based on the valence Arousal space and action models which are works as two main models of the FER. The author’s survey has only focused on the visible expression related to facial expression recognition. The methodology has been applied based on the two models in which it has concluded the conventional and deep learning FER approaches (Guo et al., 2018). This author has concluded the facial landmarks tools which are used to define or highlight the key components of human faces. By capturing the facial data, the analysing process is implemented in that which gives the information of faces (Huang et al., 2019).

Figure 9: List of the emotions and VAS

(Source: Huang et al., 2019)

The points that are considered for the Conventional approach in the survey.

Processing image
Extraction of features
Feature extraction based on the Gabor
Implication of local binary patterns
Using optical flow method
Point tracking
Classification of expression based on the different algorithms

The points are focused on the deep learning-based FER approach-

Convolutional neural network
Deep Belief network
Implication of long- and short-term memory
Generative Adversarial network

The author has also surveyed the datasets which provide basic information about the FER-related datasets. In the early time for this process, the Two-D static image processing is used which doesn’t provide that much flexibility to the user. The survey-based information also concluded the 3D-based data sets which give the advanced version in the facial emotion detection. Based on n the research on different prospective related to the topic it has identified the opportunities and challenges that provide the scope information of different algorithms. For evaluating the methods and metrics it has given the performance metrics on the bases that the right algorithm can be chosen for providing better functionality. The study has given better implementation results for the CNN and different algorithms. Which has increased the understanding of face recognition methods (Huang et al., 2019).

Figure 10: Trainin session

(Source: Siqueira et al., 2020)

The hierarchical deep neural network structure for providing effective facial expression

The following study has given a brief on the technological concepts for communication. It has concluded the methodology of establishing the hearing sensors and AI speakers which give the effective voice structure in the communication (Awan et al., 2021). In this, the author has proposed a system based on the hierarchical deep neural network that can be used to classify the results of the process. In this, the author has highlighted almost all the concepts of facial detection which also hearing sensibility, the sensor which is used to perform actions, vision sensors, trackers, audio outputs, and so on. According to the author, the FER process is defined in four steps (Kim et al., 2019).

In the first step, it detects the face through different algorithms which will find out the height, length, colour of the face. After that, it finds out the main features by comparing the images to different concepts that can help to understand the functionality based on the muscle movement and face rotation. The next step is to clarify the imaginary things that can help to easily identify the accuracy level of the video. It concludes the recognizing process based on the facial expression for those an active appearance model is implemented (Kim et al., 2019).

The author has compared his work with different studies that can be used to provide the information model of different algorithms. That has concluded the classical feature based on the FER approaches and Conventional neural network FER approaches (Awan et al., 2021).

Figure 11: Procedure of the given FER algorithm

(Source: Kim et al., 2019)

According to the author, the aim of the proposed algorithm is to improve the recognition accuracy because many experiments are already executed based on the defined algorithms. in the first network defines the CNN which focuses on the AU model by considering the LBP features. Whereas the second network defines or extracts the changes on landmarks based on the geometrical theory. Which has focused on all the six concepts of emotions. The strep by the process has been executed in the identification which also recognizes the basic need that is required during the process. This has also defined the geometric network for reducing the errors there on the system this also gives the ability to perform the action frequently in a fast manner (Kim et al., 2019).

Figure 12: Using landmarks in geometric based CNN

(Source: Kim et al., 2019)

Influencing the activation of functions in the CNN model of facial expression recognition

According to the author in today’s world, it has seen that is a going an advancement in big data and many other technologies have increased the usability of different machinery day to day life. In this whole process deep learning has shown effective information processing capabilities mostly in the field of identification, target finding, and classification. The study has defined the practical aspect of all the applications in the study which gives information about the face recognition functionality. It has considered the activation function which provides its significance of it (Wang et al., 2020).

Figure 13: Single layer perception

(Source: Wang et al., 2020)

The implication of common activation functions helps in producing the output based on the sigmoid functions. This provides a consistent structure with the help of synapses based on neurological things but at present time this function is not that usable. In this, the ReLu function is implicated which gives the graphical representation on the curves that improvise the activation functions. This gives the understanding based on the different data sets. According to the author, the exterminate results based on FER are not that satisfactory in the defined paper (Damnich & Anbarjafari, 2021).

This paper has used the cross-validation technique to improvise the identification and reliability related to the project. By dividing the subsets into two different concepts of facial expression samples it gives a better understanding of the experimented results (Wang et al., 2020).

Figure 14: The experimental results based on the CNN model

(Source: Wang et al., 2020)

The paper has influenced the different activation function that gives understanding about the CNN models. It has also included the proposed system of the ReLu function that provides the different functionality of the working process of face recognition. This has solved the problem of over-fitting on different models. Using the above functionality of the different function give advantages over the adjustment of the log functions. In this report, the author has used the transfer learning method with the design CNN model (Hortensius et al., 2018). A TLM is used to classify the task based on the imaginary concept of the faces. It has defined the capabilities by comparing the two different learning models.

Methodology

Method of Research

In any research the important part is to attain the required assessment of the procedure to be followed in the proceedings of the research. It also helps in the continuation of the process with a complete structure and clarity of what is to be done next. It continues to reduce any kind of risk responsibility of risks that may occur in the process. The validation of whether the project will become a success or not is also obtained by the choice made of the structure made for research. This will help in the maintenance of the work such that the openness is attained in the proportionate validation.

The method used for the chosen topic of research is the applied research which will be useful in the testing of the concepts of Human expression recognition. This is the reason for the completion of the research in this chosen framework. This gives a base to the study and helps in validating the hypothesis along with testing it (Barnstaple et al. 2021). The applied method is attained through the application of the concepts to be used in the making of the system. It will continue to form the base for the research, providing evident growth.

Data Collection

The data that has been collected in the process is achieved from the secondary method of data collection. This method is chosen because of its ability to extract data without involvement and considering the time and resource, it seemed to be the most suitable approach to be taken into action. This is the reason for the adaption of this data collection method to be adapted and implemented in the system of research. The source is taken Kaggle and the operation seems suitable for the completion of the task. This is the reason to resort to this option in the process.

The data that has been collected in the process is essential in the validated proportion of the work, such that it may provide an open study to the collected data. The manipulation takes place by the concept of classification and then dividing it into the train and test data. This makes it more evident and valid for the managing of the work and the continuation to be attained, by the help of validated structure. The reason to choose classification is that it will help in the continuation of ML and integration of AI in the given platform created in the process.

Analysis of Data using Image Classification

The data that is acquired and intended to be acquired in the research procedure, which will form the base for the study done in the research and the project. This makes an open study to the work that is to be integrated in the system and the validation of the task and the operations done in the project procedures. It will comprehend the overall proceeding of the secondary research that is performed in the procedural estimation of the work. The validation of the task is also gained by the help of this assessment of these datasets that create the information to be extracted.

The completion of the project and its related research is best obtained when the images and differentiated and classified such that the images are in the procedure such that it will continue to change the consideration validation of the work. The wonder in the field of making an evaluate study of the design and the comprehensive data to be delivered by the caution of field to be expressed in the validation of classification (Pandey et al. 2019). The completion could understand in the nature as the classification would provide the type to the data by the use of class. The comprehension of the human expression and its pattern is expanded and elaborated in the process, such that it will convolute the comprehensive structure of the work.

The validation to expand the study of the ML system integrated by the use of various virtual machines, such that it may open the study to an expansion that may open the expressive study. It will also validate the return of values in to the classes, making it more comprehended to perform the expansion of the values. This is the reason for the comprehension of the human activity differentiation of the expressions and their geometry convolution by the use of cv2 library. This makes the consideration more accurate and open to study that will present the idea of comprehensive estimation.

Use of SVM

The Support Vector Machine or an SVM is machinery like structure that is created for SVM to make a more open study to the comprehensive understanding. It will continue to make an comprehension of the validated structure such that the procedure of Machine Learning could be changed into the vector. This is an active process making the overall expansion of the classification and the operations related to expansion in the data and its categories. It will continue to form the expansion and the study of the task that will deliver some better operations in the field, such that it will continue to falter the image recognition and processing program in the field of evaluation. This will make the operational independence for the user and then open the accessibility of the project and its programs.

This is the management of the technique, which is operated by the use of various comprehensive studies, performed in the process. This will make an studious expansion of the train data and make the use of sklearn more accurate in the procedure establishment. Thus, it will continue such consolation in the field of an expanded operation of the expanded study to be provided to the concerned estimation of the work (Sheykhmousa et al. 2020). This is the reason for the comprehension of the task, such that it may provide an open study to the expression on the human face and the comprehensive reaction to it. Thus, it will form a more loaded comprehension to the work of understanding.

Figure 15 Linear Separation using SVM

(Source: Sheykhmousa et al. 2020)

Implementation of CNN Model

The use of a CNN model is beneficial in the estimate exceptional study of the work, to complete the . It has extracted the concept of CNN because the data collection process is executed from different data sources. In the report, a section provides a piece of knowledge about the working model of the algorithms. this has shown the structure in a diagrammatical way. Or the process of this works on collecting the pictures from different data sets and performing the detection process by applying the cascade classifier. After this process, if it gets the requirements of further analysis the processing concept is performed again.

Figure 16 Block Layer in CNN

(Source: Sultana et al. 2018)

In the study, it has been observed that it works on the three convolutional layers that are defined in the category of 32, 64, and 124 filters. The study has implemented the section on data processing which has shown the analyzed and processed operation for performing the task. For the execution of this task, this has concluded the various data sources by the inclusion of the FER201, CK+, and KDEF. These are used in the procedure of cropping and detecting the images (Sultana et al. 2018). This study has defined the algorithms of machine learning and artificial intelligence in which it concludes the CNN, neural networks, deep learning, regression tolls, and so on.

For the perfect execution of the report, it has implemented the research methodology. That is based on the hydride methodology which includes research based on the qualitative and quantitative methodology. In the report, the qualitative research methodology is to give the theoretical data of Artificial methodology algorithms. The following dissertation has provided the knowledge for the human expression analysis for analyzing human nature. It has contributed in the form of deep and machine learning and AI. It has contributed in the form of a literature review that provides knowledge about its working structure of it. The conclusion of the literature review is based on the data, research paper, websites sources, and so on.

Findings

Data is an essential part of the research that is to be performed in the process, and also essential in any kind of research, that will continue to form the validation in the process. The convolution of the continuation in the measurement of the values, such that these values are to be validated in the process, and the certainty to be attained in the research. This is the reason for the comprehension of the research that will deliver the completion of the work, and the assessment to be created in the management of the information and its flow.

Figure 17 Test and Train Matrix

(Source: Ayman 2021)

This has shown the structure in a diagrammatical way. Or the process of this works on collecting the pictures from different data sets and performing the detection process by applying the cascade classifier. After this process, if it gets the requirements of further analysis the processing concept is performed. This is the reason for the comprehensive study of the emotions and expressions provided by the test and train data. This step of classification would help in the making of the proportionate study of the task. This is the reason for the adoption of Cv2 library and the mapping by the use of matplotlib. The proportionate analysis of the work is to be considered and then comprehended in the convolution of the provided data. The understanding of the expression is done by the locus of facial geometry, the comprehension would seem to be right.

Implementation

The SVM will prove to be useful in the making of the validated structure of the values, which may provide an open elaboration to the study of the emotional data. The equalisation of the work to be done in the procedure would help in the integration of the project with AI and provide a structure to it. This has also provided good platform for the practical implementation of the project. This is the major effect that holds the completion of the task and the assessment of the validation that may be obtained in validated structure. The implementation also proves to be an evident implementation of the work, such that it may open the complete validation to create human expression recognition to identify the emotion to it.

Figure 18 Validated Result

(Source: Ayman 2021)

The making of the ML system by the use of the recent available data provides a benefit in the making of the validated estimation of the work and the continuation in the attainment of a result. The implementation of this operation helps in testing the program which is a success, the preloaded data shows. The data is interpreted and then completion of the work, is then comprehended by the help of proportionate study of the task expanded. The data assessment that is done by the sample data is represented here, of how the system operates the given understanding of the expressions.

The image above shows the complete definition to the study of the expression that is showcased in the image as understood by the system. The system predicts the class to be angry, and the validation can be gained, as the actual class is also similar to the completion of the work. The comprehension of the task is to be attained, by the help of some certain condition may act otherwise, the expressions that may not be expressed well. The prediction in the below image is fear, but the actual class of expression is sad.

Figure 19 Failed Operation

(Source: Ayman 2021)

Discussion

This concept of human face emotion detection has provided an important methodology for understanding the behavior of humans. These are considered the most important and powerful tools for giving immediate results to communicate based on the intentions and emotions of human beings. The concept of this has received more attention based on the IoT technology that has the concept in many industries. This has helped in making smart homes, smart cities, and hospitals. The concept of this can be implemented with the automation tools like Siri, Alexia, and Cortana. It has provided better processing of communication for humans (Xiao et al. 2022).

T
he technical concept of these states that the first phase will collect the same category and analyze them with the help of CNN algorithms. This will help in identifying the similarities between the images. The concept is beneficial in understanding emotion by analyzing it based on happiness, fear, sadness, anger, disgust. In this, the conventional neural network will give effective information of data to the user. Based on that they can analyze the different executory tasks. The implication of the NumPy array will give the integration process over the categorization process. For the detection process, the 14 EEG signals are implemented. That converts them into the classification process based on the LSTM (Saffaryazdi et al. 2022)

Figure 20: Face detection and emotion detection

(Source: Mehta et al. 2018)

The benefit of using this technique is understanding the requirements of the user by analyzing the face and transforming those data into relevant information. This can also help in improving the security structure of user data. Which can be assessable through the proper detection of the face based on the emotions of the user. Thus, it can be an improvised concept of making non-verbal communication. Most of this can be seen in the business development growth. That can affect the growth of the business. But at some points, this has some limitations like this can increase the cost of implementation of technology because this kind of task can be executed with highly professional people (Tripoli et al. 2022).

It can be considered as a complex structure because it decreases the manipulation process or extended the concept of making the final structure of images. For using this it can be seen under the law full activities can be said to the violations of rights. This process affects the personal freedom of the user or the clients. In many countries, it comes in the category of law full activities. The technology can be used with different methods like with the implication of sensor based on the motion detection. Which gives the better understanding of the human face that can be analysed by the services provider. But some times this human face emotion detection technique can hurt the feelings of the person. This has concluded the support vector machine which will provide the user interface based on the video graphic form that also gives the report in the form of statical data.

Figure 21: Model output and image sample

(Source: Thripathi 2021)

Recommendation

Artificial intelligence has provided the important concept of work in the field of human face emotion detection. It has been used in different methodologies which helps in improving the concept of the face detection. Human face detection has given the better ways to understand the feelings and emotions. It has given the vast concept of enhancement in the communication network. The future implication for this is to give the simplified process off face detection that can helps in describing the complexity of the user. The best recommendation about this is that for the better implication of this can helps them to provide the effective structure to the application. for the better getting information it is must necessary to provide the tools affection in the application. An improvised version of hardware and software can be implemented that gives the better techniques for the application execution process (Mehat et al. 2018).

To provide a better structure to the face detection process it must necessary to give a secured environment to the user. There are fast methodologies or algorithms that can be used to enhance the performance of the software. This can include the different kind of techniques which are known as the machine learning and deep learning concepts. The has given the best way of understanding the artificial intelligences algorithms which has helped in making an effective study. which has helped thee student and the programmers to work effectively in the better environment. The study will provide the methodologies based in the machine learning algorithms which gives the knowledge about the implication of the data sets in the defined module (Matsuda and yoshimura 2022).

The above study has given the structure of the report has given the understanding about the NumPy libraries. For the better compilation and process of the code the other libraries can also implements along with this. The whole model of human face emotion detection gives the accessibility of understanding the feelings of the people and generates a report based imaginary videos. For better wide increment of this technique can be extended with the help of cloud computing. This can also be used in the hospital industry which gives the opportunity of identifying the issues related to their patients. The most advantages that are considered is in the security related data protection of the user (Kyranides et al. 2022).

Conclusion

Artificial intelligence is the ability to perform a task based on a defined computer system. The concept of this has been introduced to provide flexibility in doing the work automatically. This had simplified the work procedure of human work with the implication this it has introduced an advanced level of technology. The implication of the AI procedure, it has involved the four types of AI systems are mainly known as the reactive machine, theory of mind, limited memory, and the self-aware concept. The increment in technology has changed the concept of human facial detection by providing automation based on different algorithms. For the better implication of this methodology, it has provided ways to make the effective structure from the algorithms. Artificial intelligence has given the vast concept for the growth of understanding the nature of based on their requirements.

The study has provided information about human face detection in which it has given the methodologies for human facial expression detection using artificial intelligence. Finding the nature of the human has given the statical model of the face recognition method. The model has been acted as an important tool for defining the facial expression of humans. It has provided the ability to improve human coding skills for the implication of data centres in technology. AI has given a major effect on the different sectors of the industries which have improvised the working structure of the organization. The concept of human face emotion detection has made an urgency for providing the improvised version to different. The examples which are already based on this concept are human-computer collaboration, data-driven technology, and establishing an effective communication pathway between robots and humans. The study has achieved the aim effectively of the perfect implication of AI in identifying human facial expressions.

This report has completed the action based on the objectives of the report. That defines the model of facial emotion detection which helps in identifying the results using the machine learning algorithms. these are used in the business which acts on policies and surveillance industries. This model is used to identify felling based on emotions like sadness, anger, and happiness has helped in identifying and solving the personal issues of humans.

The research has shown the machine learning algorithms based on different models with the bits of help of deep learning methodology. It has also concluded the section on the artificial neural network and convolutional neural network (CNN). Using this functionality has given the advantages of analyzing the visual imagery of human faces. The concept is implemented in identifying and classifying the objects and images based on the given data. For the data augmentation process, it has extracted the concept of CNN because the data collection process is executed from different data sources. In the report, a section provides a piece of knowledge about the working model of the algorithms. this has shown the structure in a diagrammatical way. Or the process of this works on collecting the pictures from different data sets and performing the detection process by applying the cascade classifier. After this process, if it gets the requirements of further analysis the processing concept is reperformed.

In the study, it has been observed that it works on the three convolutional layers that are defined in the category of 32, 64, and 124 filters. The study has implemented the section on data processing which has shown the analyzed and processed operation for performing the task. For the execution of this task, this has concluded the various data sources by the inclusion of the FER201, CK+, and KDEF. These are used in the procedure of cropping and detecting the images. This study has defined the algorithms of machine learning and artificial intelligence in which it concludes the CNN, neural networks, deep learning, regression tolls, and so on.

For the perfect execution of the report, it has implemented the research methodology. That is based on the hydride methodology which includes research based on the qualitative and quantitative methodology. In the report, the qualitative research methodology is to give the theoretical data of artificial methodology algorithms. The following dissertation has provided the knowledge for the human expression analysis for analyzing human nature. It has contributed in the form of deep and machine learning and AI. It has contributed in the form of a literature review that provides knowledge about its working structure of it. The conclusion of the literature review is based on the data, research paper, websites sources, and so on.

It has also given a focus on the electronic and physical framework which shows the results on the screen in an effective form. That process has included server connections, processors, webcams, monitors, computer systems, and so on. These all technologies are used in storing and collecting the data effectively. For the implication of coding structure that has been imposed on the structure of the computer. This will take the live images and ten perform the coding procedures in them which helps in writing the coding in an effective structure. The section of the discussion has included the section on interpreting and analyzing the data. This has given the results that are obtained from the research by implementing the data analysis process.

The study has also introduced the section on finding and recommendations section has helped define the topic of human facial expression detection using artificial intelligence. It has described the tools and techniques that are used in making this procedure. The recommendation section has provided the future aspects that can be used for improvising the functionality of the working structure of the human interface. After the inclusion of the result, the section has given the solution results that are obtained during the research. This section has considered the information about the outcomes which are obtained during the research process. In this process, it has concluded the technologies based on CNN, DNN, ANN, and ML. which has helped in detecting the right emotion of the users.

The study has been perfectly implemented the study in an effective manner that provides the solution for the AI-based application. That will help the hospital, traveling, and communication industry. which gives the opportunity of making a better within human beings. The whole model of this is based on the CNN algorithms which have given the perfect example of advanced technology.

References

Ahmed, T. U., Hossain, S., Hossain, M. S., ul Islam, R., & Andersson, K. (2019, May). Facial expression recognition using convolutional neural network with data augmentation. In 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR) (pp. 336-341). IEEE. DOI: https://www.diva-portal.org/smash/get/diva2:1299002/FULLTEXT01.pdf

Awan, M. J., Raza, A., Yasin, A., Shehzad, H. M. F., & Butt, I. (2021). The customized convolutional neural network of face emotion expression classification. Annals of the Romanian Society for Cell Biology, 25(6), 5296-5304. https://annalsofrscb.ro/index.php/journal/article/view/6476/4901

Ayman B., 2021, Facial Recognition Data, Kaggle, viewed 01 April 2022, https://www.kaggle.com/code/basel99/facial-recognition/notebook

Barnstaple, R., Protzak, J., DeSouza, J.F. and Gramann, K., 2021. Mobile brain/body Imaging in dance: A dynamic transdisciplinary field for applied research. European Journal of Neuroscience, 54(12), pp.8355-8363. Doi: https://doi.org/10.1111/EJN.14866

Daws, R. (2019, January 17). Speech and facial recognition combine to boost AI emotion detection. AI News. https://artificialintelligence-news.com/2019/01/17/speech-facial-recognition-ai-emotion-detection/

Domnich, A., & Anbarjafari, G. (2021). Responsible AI: Gender bias assessment in emotion recognition. arXiv preprint arXiv:2103.11436. https://arxiv.org/pdf/2103.11436.pdf

Guo, J., Lei, Z., Wan, J., Avots, E., Hajarolasvadi, N., Knyazev, B., ... & Anbarjafari, G. (2018). Dominant and complementary emotion recognition from still images of faces. IEEE Access, 6, 26391-26403.DOI 10.1109/ACCESS.2018.2831927

Hortensius, R., Hekele, F., & Cross, E. S. (2018). The perception of emotion in artificial agents. IEEE Transactions on Cognitive and Developmental Systems, 10(4), 852-864.

Huang, Y., Chen, F., Lv, S., & Wang, X. (2019). Facial expression recognition: A survey. Symmetry, 11(10), 1189. DOI https://doi.org/10.3390/sym11101189

Hussain, S. A., & Al Balushi, A. S. A. (2020). A real time face emotion classification and recognition using deep learning model. In Journal of Physics: Conference Series (Vol. 1432, No. 1, p. 012087). IOP Publishing. https://iopscience.iop.org/article/10.1088/1742-6596/1432/1/012087/meta

Kim, J. H., Kim, B. G., Roy, P. P., & Jeong, D. M. (2019). Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE access, 7, 41273-41285.DOI 10.1109/ACCESS.2019.2907327

Ko, B. C. (2018). A brief review of facial emotion recognition based on visual information. sensors, 18(2), 401.DOI https://doi.org/10.3390/s18020401

Kyranides, M.N., Petridou, M., Gokani, H.A., Hill, S. and Fanti, K.A., 2022. Reading and reacting to faces, the effect of facial mimicry in improving facial emotion recognition in individuals with antisocial behavior and psychopathic traits. Current Psychology, pp.1-14. https://link.springer.com/article/10.1007/s12144-022-02749-0

Lee, Y.S. and Park, W.H., 2022. Diagnosis of Depressive Disorder Model on Facial Expression Based on Fast R-CNN. Diagnostics, 12(2), p.317. https://doi.org/10.3390/diagnostics12020317

Matsuda, S. and Yoshimura, H., 2022. Personal identification with artificial intelligence under COVID-19 crisis: a scoping review. Systematic reviews, 11(1), pp.1-6. https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-021-01879-z

Mehendale, N. (2020). Facial emotion recognition using convolutional neural networks (FERC). SN Applied Sciences, 2(3), 1-8. https://link.springer.com/article/10.1007/s42452-020-2234-1

Mehta, D., Siddiqui, M.F.H. and Javaid, A.Y., 2018. Facial emotion recognition: A survey and real-world user experiences in mixed reality. Sensors, 18(2), p.416. DOI:10.3390/s18020416

Pandey, S.K., Shekhawat, H.S. and Prasanna, S.R.M., 2019, April. Deep learning techniques for speech emotion recognition: A review. In 2019 29th International Conference Radioelektronika (RADIOELEKTRONIKA) (pp. 1-6). IEEE. Link: https://www.researchgate.net/publication/334457978

Saffaryazdi, N., Goonesekera, Y., Saffaryazdi, N., Hailemariam, N.D., Temesgen, E.G., Nanayakkara, S., Broadbent, E. and Billinghurst, M., 2022, March. Emotion recognition in conversations using brain and physiological signals. In 27th International Conference on Intelligent User Interfaces (pp. 229-242). https://doi.org/10.1145/3490099.3511148

Sheykhmousa, M., Mahdianpari, M., Ghanbari, H., Mohammadimanesh, F., Ghamisi, P. and Homayouni, S., 2020. Support vector machine versus random forest for remote sensing image classification: A meta-analysis and systematic review. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13, pp.6308-6325. Link: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9206124

Siqueira, H., Magg, S., & Wermter, S. (2020, April). Efficient facial feature learning with wide ensemble-based convolutional neural networks. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 04, pp. 5800-5809).

Song, Z. (2021). Facial Expression Emotion Recognition Model Integrating Philosophy and Machine Learning Theory. Frontiers in Psychology, 12. DOI: https://doi.org/10.3389/fpsyg.2021.759485

Sultana, F., Sufian, A. and Dutta, P., 2018, November. Advancements in image classification using convolutional neural network. In 2018 Fourth International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN) (pp. 122-129). IEEE. Link: https://arxiv.org/pdf/1905.03288

Šumak, B., Brdnik, S., & Pušnik, M. (2021). Sensors and Artificial Intelligence Methods and Algorithms for Human–Computer Intelligent Interaction: A Systematic Mapping Study. Sensors, 22(1), 20. DOI https://doi.org/10.3390/s22010020

Tripathi, M., 2023. Facial Emotion Recognition Using Convolutional Neural Network. ICTACT Journal on Image and Video Processing, 12(1), pp.2531-2536. DOI: 10.21917/ijivp.2021.0359

Tripoli, G., Quattrone, D., Ferraro, L., Gayer-Anderson, C., La Cascia, C., Richards, A. and O'Donovan, M., 2022. Facial emotion recognition in psychosis and associations with polygenic risk for schizophrenia. Findings from the multi-centre EU-GEI case-control study. Frontiers in Psychiatry. https://orca.cardiff.ac.uk/147963/1/2022%20AR%20Facial%20emotion%20combined.pdf

Wang, Y., Li, Y., Song, Y., & Rong, X. (2020). The influence of the activation function in a convolution neural network model of facial expression recognition. Applied Sciences, 10(5), 1897. DOI https://doi.org/10.3390/app10051897

Xiao, H., Li, W., Zeng, G., Wu, Y., Xue, J., Zhang, J., Li, C. and Guo, G., 2022. On-Road Driver Emotion Recognition Using Facial Expression. Applied Sciences, 12(2), p.807. https://doi.org/10.3390/app12020807

Yang, D., Alsadoon, A., Prasad, P. C., Singh, A. K., & Elchouemi, A. (2018). An emotion recognition model based on facial recognition in virtual learning environment. Procedia Computer Science, 125, 2-10.

Zhang, H., Jolfaei, A., & Alazab, M. (2019). A face emotion recognition method using convolutional neural network and image edge computing. IEEE Access, 7, 159081-159089. DOI: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8884205

Zhang, S., Pan, X., Cui, Y., Zhao, X., & Liu, L. (2019). Learning affective video features for facial expression recognition via hybrid deep learning. IEEE Access, 7, 32297-32304. DOI 10.1109/ACCESS.2019.2901521

Human Facial Expression Detection using AI

Table of Contents

Introduction

Literature review

Sentiment recognition using CNN and edge computing

Methods based on artificial intelligence which provides the human-computer intelligent interaction

A brief review based on the human emotion recognition by implementing visual information

Convolutional neural network for facial emotion recognition

Survey on facial expression recognition

The hierarchical deep neural network structure for providing effective facial expression

Influencing the activation of functions in the CNN model of facial expression recognition

Methodology

Method of Research

Data Collection

Analysis of Data using Image Classification

Use of SVM

Implementation of CNN Model

Findings

Implementation

Discussion

Recommendation

Conclusion

References