Real-time mask detection and face recognition using eigenfaces and local binary pattern histogram for attendance system

Received Nov 18, 2020 Revised Jan 27, 2021 Accepted Feb 20, 2021 Face recognition is gaining popularity as one of the biometrics methods for an attendance system in an organization. Due to the pandemic, the common face recognition system needs to be modified to meet the current needs, whereby facemask detection is necessary. The main objective of this paper is to investigate and develop a real-time face recognition system for the attendance system based on the current scenarios. The proposed framework consists of face detection, mask detection, face recognition, and attendance report generation modules. The face and facemask detection is performed using the haar cascade classifier. Two techniques for face recognition were investigated, the eigenfaces and local binary pattern histogram. The initial experimental results and implementation at Kuching Community College show the effectiveness of the system. For future work, an approach that is able to perform masked face recognition will be investigated.


INTRODUCTION
Face recognition is one of the biometrics promises that become more popular for the application of identification, access control and register in an organization. Manual and conventional methods have been replaced by biometrics due to their security, saving cost, fast and efficiency. Recently many attendance systems proposed using the face recognition advantage. It has reduced the burden of taking attendance manually, prevent fraud and embedded in the updated technology [1]. Among biometrics techniques, face recognition provides a contactless method that complies with current scenarios requirement compared to fingerprint analysis, palm prints and iris recognition. To prevent the spread of the COVID-19 pandemic, contactless and even wearing facemask has been made compulsory. The medium for diseases spread, such as sharing a paper and pen, also should be avoided. Besides, wearing a facemask to cover the nose and mouth has been identified as one of the prevention steps. Thus conventional attendance system should be replaced with a digitalization attendance system to provide a safe environment to the attendees. However, face recognition is hindered by the occlusion of facemask wearing by attendees. The detection of the facemask is required before face recognition is conducted by the system. The attendance record also needs to tailor to an organization's requirement, such as record entry and exit of the attendees, so that the time engaged in each event can be calculated. framework of the system. The system starts with an image captured from the video camera at the entrance. Two cameras were used, one at the entrance door for entry records and another at the exit door for exit records. Face image of attendees captured and fed to the system for face detection. Once the face is detected, the system will perform mask detection. If the attendee is not wearing a mask, the system will perform face recognition. Else, the system will print 'Using Mask' text on the screen to remind the attendee to remove his/ her mask. The face recognition module is conducted by comparing the face captured to the images saved in the face database. Finally, attendance information of recognized attendees and time captured during detection was recorded and saved to the database. The same process is repeated at the exit door, whereby two different times will be recorded for the final calculation of engaged time for each attendee.  Figure 1. The framework of real-time mask detection and face recognition attendance system

Face and mask detection
For the face detection task, we adopted the work of [12]. First, the image was represented as a Haarlike feature representation. Second, AdaBoost was used to boost the classification performance. For mask detection, we adopted the custom Haar cascade [13]. The cascade function was trained using positive and negative images. Positive images refer to the face images whereby a facemask occurs. Face without facemask images construct the negative images. For each of the images where the face is successfully detected, preprocessing that involves contrast adjustment, intensity and size normalization was performed. The preprocessed images were then sent to the face recognition module.

Face recognition
Eigenfaces and local binary pattern histogram (LBPH) were selected to perform face recognition. These algorithms were chosen as they require less processing time for implementation. Furthermore, both have been shown to be feasible for implementation in a real-time environment and produced good recognition results [14][15][16].

Face recognition using eigenfaces
Eigenfaces is a subspace or holistic based approach which used PCA to find a set of features that characterized variation among images. It has been used for face detection and recognition due to its superiority in extracting relevant facial information and efficient face image representation [17]. For each image in the training set, the eigenfaces were calculated and only images with the highest eigenvalues were selected. The weights for each of the selected image was then calculated and projected to the eigenspace. To recognize an unknown face, the eigenvectors need to be calculated, similar to the above steps. The new face image projected into eigenspace Γ is given by: where is the weight vector representation of the new face, ˆ is the set of significant eigenvectors, is the sum of vectors and is the average of the vectors. To determine which class face belongs to is by minimizing the Euclidean distance, ϵ : where Ω is representing the weight vector of the test image and Ω_k is the weight vector representing the kth face class in the training set. ϵ_k is corresponding to class k if the minimum ϵ_k is smaller than a predefined threshold, θϵ. If ϵ_k is greater than θϵ, the new face is unknown. The threshold, θϵ, can be manually chosen by defining maximum allowable distances from any face class or maximum distance from face space [17].

Face recognition using LBPH
LBP has been widely used in many applications due to its computational simplicity in real-time and robustness to monotonic greyscale changes such as illumination [18]. LBP is a local based appearance approach, which used the texture information as features extracted from the frontal view of an image. It labels the pixels of an image by thresholding the neighbor of each surrounding pixel. The LBP for a defined matrix size of 3x3 is given by: where and are the intensity value of the center pixel and neighborhood pixel. The notation , indicates the use of sample points in the neighborhood and defines the thresholding function. The pixel of this matrix is a threshold with the value of the center pixel; use the intensity value of the center pixel i( ) as a reference for thresholding to produce the binary code. If a neighbor pixel's value is lower than the center pixel value, it is given a zero; otherwise, it is given one. These processes are created for each part of the region in the image. Later, a histogram is extracted for each region, and all histograms of an image were then concatenated to produce only one histogram for each image. The new face image and training image are then compared using the chi-square comparison applied to the image's histogram [19].

RESULTS AND DISCUSSION
This section presented the experiments and implementation of the proposed system. The experiment was conducted to measure the performance of the described face recognition techniques, and the implementation was conducted to test the workability of the proposed system. The system has been experimented and implemented at the Kuching Community College (KCC), Malaysia. The system runs on a lightweight CPU of Intel i5 with 1.5GHz and 4GB RAM. Using Python and OpenCV, the system used an internal database for storage. The camera of a full HD with a 3.6mm lens was used with the support of LED ring light surrounding it. This is to ensure a better face image is captured [3] under control environment [20,21]. Section 4.1 presents the results of the mask detection and face recognition performance, and section 4.2 describes the implementation of the proposed face recognition system.

The mask detection and face recognition performance
In this section, the performance of mask detection and face recognition are reported. A group of 15 students from a class was chosen for face detection and mask detection. The face detection using the Haar cascade classifier with AdaBoost able to capture the faces. Using the Haar cascade classifier for a facemask for mask detection, the classifier effectively detects all the faces with the facemask. Figure 2 shows the mask detection of the subjects and face detection. A warning sign 'Using Mask' is shown on the screen if a facemask is detected.
For face recognition, two sets of experiments were conducted using eigenfaces and LBPH. The first is to identify the confidence value, C, for each algorithm that best represents the data used. The value C refers to the minimum distance between a new test image and the corresponding image in the recorded database or  [21]. The identified C was then used to set a threshold (θ) representing the set of data for recognition. The second experiment is to identify the best performing algorithm for face recognition using the identified C.

Figure 2. Mask detection and face detection
The first experiment was performed on a subset of data used in this paper. The data consist of a group of 30 students in a class. Under the control environment, the face images of each student were used to train and test both algorithms. The confidence C value for each algorithm was recorded. The highest C value, which represents the dataset used for the experiment, was chosen for each algorithm. Table 1 shows the identified C value for both of the algorithms used. These values represent the maximum C for the dataset used and are important parameters for the second experiment.
The identified C values from the first experiment were used as threshold θ in the second experiment. Both algorithms used their own θ set as the border for the recognition. The same group of students was chosen for the second experiment. Table 2 shows the accuracy of face recognition using eigenfaces and LBPH. From Table 2, LBPH outperformed eigenfaces for face recognition. Recognition accuracy for eigenfaces was recorded at 73.3% compared to 100% in LBPH. Concerning face recognition, LBPH outperformed eigenfaces using the proposed framework with the hardware and the control environment setting. The results show that LBPH is robust in greyscale and illumination in real time [15,18,22]. It is also observed that the eigenfaces have difficulties in recognizing faces in a situation where the position of the face is similar across images of different subjects. Figure 3 shows examples of error in recognizing the faces of an attendee using eigenfaces compared to LBPH. Eigenfaces with PCA in a holistic approach known to be good at data representation but not necessarily for class discrimination in face recognition [23,24]. The texture-based features for face recognition seem to be more effective in this context, as shown by the LBPH. Under the controlled environment, the correct person who was wrongly classified in Figure 3(a) was able to be recognized by LBPH in Figure 3

The implementation of the proposed system
The custom Haar cascade classifier and the best-performing face recognition algorithm, the LBPH from the experiment presented in section 4.1, is used for implementation. The implementation of the system consists of three main modules. The modules are data face recording, mask detection and face recognition, and attendance output generation.

Data face recording
The data face recording module involved collecting images of subjects and saving it to the database. The system first detects the face of members and capture the face according to the name or ID provided. Only a unique name or ID is saved to the database under the correspondence folder. The face image was resized, convert to greyscale and normalized before stored in the database. Figure 4 shows a sample of images captured, preprocessed and stored in the database.

Mask detection and face recognition
Mask detection and face recognition module involving two different steps, as shown in Figure 1. First, face detection of the subject is performed, followed by mask detection. If the subject is wearing a facemask, the system displays a warning 'using mask'. If no facemask is detected, the recognition process is performed using the LBPH. For each of the recognized faces, the name of the subject is printed on the screen and store the record of the subjects in the database. The system records information that includes the name of the subject, date and time of recognition, i.e., attendance-entry and attendance-exit. Figure 5 shows examples of mask detection and face recognition of subjects. The subject in Figure 5(a) was unknown due to occlusion caused by the facemask. It is advisable that subject should focus on the camera for still image [20,25], and any drastic change can cause a poorly captured image [26]. If a facemask is detected, the 'Using Mask' warning was shown, which indicates the subject has to remove her mask to allow face recognition takes place. The face later had been recognized once the subject removes the facemask, as shown in Figure 5 However, from the experiments, it is found that the non-frontal view of the face would also fail the face recognition. Figure 6(a) shows the unknown recognition due to a non-frontal image while the correct recognition is performed using a frontal image 6(b). The frontal view is one of the mandatory poses for better recognition performance even in a controlled environment [20,21,24].

Attendance output generation
The final module of the implementation is the attendance report generated. The report consists of information that includes the entry time, the exit time and the total time spent in a class or event. Figure 7 shows the screenshot of the attendance report generated.

CONCLUSION
This paper presents a real-time mask detection and face recognition for attendance system. A framework of mask detection embedded in face recognition was proposed for the system to comply with the current pandemic situation. The framework consists of image input, face detection, mask detection, face recognition, and attendance recording. Haar cascade was used for mask and face detection. Two techniques for face recognition were used, the eigenfaces and LBPH. In the experiments, the LBPH outperformed the Eigenfaces for face recognition. The system was implemented in KCC, Malaysia. Initial experiment and implementation show the effectiveness of the approach to use face recognition to register attendance. For future work, an investigation on an approach that is able to perform partial face occlusion recognition using a partial face image will be conducted. Such an approach would permit face recognition without the subjects' need to remove their facemask or masked face recognition, which is compulsory in some countries due to the COVID-19 pandemic.