Smart and real-time door lock system for an elderly user based on face recognition

Received Nov 4, 2020 Revised Jan 19, 2021 Accepted Apr 6, 2021 Many sophisticated smart door lock systems have been made. Still, most of them required the user to use an additional device such as a smartphone, tag, smartcard, or accessing some user interfaces, which is complicated to use for inexperienced elderly. This condition creates a gap between the elderly and technology which makes it difficult for the elderly to accept and use the technology. In this paper, we proposed a smart and real-time door lock system for an elderly user based on local binary pattern histogram as a face recognition algorithm with modular system architecture design. The novelty in our proposed system design, it does not require any additional device, it does not use any user interface, and the least user participation by automating the processes. All the user needs to do just walk toward the door and stand in front of it and the door will automatically unlock and locked back after the user enters the house and close the door. The system resulted in an accuracy of 98%, with an average processing time is 1.449 seconds for the entire process. Additional advantages, the system is designed with a modular approach that makes it flexible and scalable for further development.


INTRODUCTION
The number of elderly is growing globally. Solutions are needed to cope with the complex care demands of aging and to satisfy the needs of elderly persons to live longer in their own homes [1]. Technology has been an answer for assisting their life, especially for the ones who have been living alone without family to help and take care of them. However, it is essential for the elderly could accept and use the given supporting technology. The acceptance model of technology for the elderly are required two necessary conditions perceived ease of use and perceived usefulness [2].
The door is one of the most critical elements of a house because it is used as an entrance for the owner. According to the work from [3], Burglary cases involving the elderly and disabled are increasing because there is no proper security system for them in the house. The simplicity of the door lock system based on technology can be useful for the elderly and help them by protecting their home from thieves. Aging leads to an increased risk of developing chronic diseases, such as Alzheimer's disease [1]. The process of unlocking and locking the door of the house could be a problem if they lost the key by accidentally dropping or forgot the position where they left the key. The development and the innovations on the conventional lock are aimed at decreasing the weaknesses [4]. Digital door lock system equipment that uses

RELATED WORKS
Many sophisticated smart door lock systems have been made. Table 1 shows the comparison of the previous smart door lock research based on the authentication method, additional device requirement, and manual input requirement. The Table contains several types of smart door locks that have been made by previous researchers. Most of those systems require a smartphone as an additional device to operate the system and to achieve an authenticated condition by the system, the user still has to manually input commands to the systems via the interface provided by the smartphone. The problem could arise if a user is an elderly person who is not familiar with smartphone technology and its user interface, of course, this will make it difficult for them to use. Therefore, this research will produce a system that is not only safe with its real-time face recognition authentication methods but also easy and convenient to use because it does not require additional devices, does not require manual input commands from users, and does not require any user interface skills. The entire system of the smart door locks is embedded in the door and operated fully automatically. This is a novelty design of a smart door lock system which will be discussed further in this study. Password matching Keypad Required [7] NFC authentication and password matching Smartphone Required [8] NFC authentication and face recognition Smartphone Required [9] NFC authentication and password matching Smartphone Required [10] Mobile phone network authentication Smartphone Required [11] Mobile phone Bluetooth authentication Smartphone Required [12] Mobile phone Bluetooth authentication Smartphone Required [13] Captured image face recognition authentication Smartphone Required This research Real-time face recognition Not required Not Required

PROPOSED SYSTEM
This paper aims to propose easy to use and useful face recognition door lock with an automation approach to help the elderly securing their door and without the need for various additional devices to operate it. This paper describes the design of a system for supporting the elderly, which may have limited physical or cognitive capabilities. The prototype should be working automatically with the least participation from users and the user does not need to access any user interface. The prototype will be made of the low-cost hardware and modularity approach to make the system scalable with additional functionalities. This paper also compares two face recognition algorithm that most suitable for use on the prototype by evaluating the performances.

Proposed system architecture
The automatic face recognition door lock will be divided into three modules, movement detection module, processor module, and lock control module. This system uses a modular approach, so there will be three modules that are mutually integrated and form one system. Figure 1 is the picture of the proposed diagram for automatic face recognition door lock. The prototype will be made from low-cost hardware and a modularity approach. It consists of three modules that are movement detection, processor, and lock control

Proposed system flowchart
The automatic face recognition door lock starts to work by detecting movement using PIR Sensor. After getting input from the PIR sensor, Arduino will turn on the lamp and simultaneously send a message to Raspberry Pi to execute the face recognition program. If the detected person is recognized as the owner, Raspberry Pi will trigger the solenoid to unlock the door, and then the owner can open the door and enter the house. As long as the door is open, the magnetic switch state will become low. Raspberry Pi will monitor this state. If the door closed, the magnetic switch state would become high, and Raspberry Pi will stop the trigger to the solenoid to lock the door. This sequence will run automatically without the intervention from the owner as of the user. Figure 2 is a complete flowchart of the system.

Proposed system use case diagram
The use case diagram will be explaining how the relationship between actor and system. It will also give detailed information about what functions the system has. Figure 3 is the use case diagram of the automatic face recognition door lock.

Face recognition algorithm-local binary pattern histogram
The local binary pattern histogram (LBPH) is not only simple and low computation complexity but also has a rotational invariance, grayscale invariance, and other significant advantages for face analysis and recognition [22]. This makes LBPH can be used on low computational platforms like the Raspberry Pi. The first computational process of the LBPH algorithm is to convert the grayscale facial image (dataset) to a decimal value. Figure 4 shows the conversion process. The facial image will be divided into nine grid (3x3) pixels. Each pixel contains a pixel intensity value with a range of values 0-255. Before converting to a decimal value, all grid would be converted to a binary value. Centre pixel value would become the threshold. If the value is equal or greater than the threshold, the binary value is 1. If the value is smaller than the threshold, the binary value is 0. After this process, the system has a threshold result of LBP. The threshold result would be converted into pattern result to a form histogram by multiplying the threshold result to binary to decimal conversion mask. Figure 5 shows the process of generating the final histogram by dividing the image into an 8x8 grid. Each histogram from each grid contains a decimal value from 0 to 255. The final histogram is the concatenated result from each generated histogram.  After getting the histogram of each input image, the algorithm is already trained. Each histogram created will be used to represent each image from the training dataset. The face recognition process performing by input a new image by generating it using a face detection algorithm from a video in real-time and create a histogram that represents the image and calculates the distance between two histograms with a trained dataset histogram. The LBPH algorithm used Euclidean distance based on the formula: The formula output is the ID from the image with the closest histogram. The algorithm also returns the calculated distance, which can be used as a confidence measurement. As Euclidean measures the gap between the images, the smaller the distance, the higher the similarity [23].

Face recognition algorithm-dlib face_recognition
Dlib library face_recognition consists of some kind of recognition modes such as histogram of oriented gradient (HOG), support vector machine (SVM), and convolutional neural network (CNN). Even though dlib with HOG only detects a frontal image, and CNN can detect multiple angles, HOG used less resource or computational cost than CNN [24]. So, this paper used dlib face_recognition based on HOG because of its low computational cost and fit the use case of this research.
The first computational process of dlib face_recognition is creating face encoding from the facial image dataset. Histogram of oriented gradient (HOG) and used for finding a face in dataset image. This process is done by calculating the gradient value by using a one-dimensional centered mask with subsequent filtering of the color or saturation of an image [25]. After finding the face, the algorithm will use a facial landmark estimation algorithm to detect 68 landmarks that are present in the images, and the output is a centered facial image, as shown in Figure 6. After the finished facial landmark, the face will be trained to become a 128-dimensional vector space where images of the same person are near to each other, and pictures from different people are far apart. Face recognition is performed by mapping the input face to the 128-dimensional vector and checking the Euclidean distance. If the Euclidean distance is very small or below the threshold value, the system will authenticate the person. Still, if the euclidean distance is significant, the system would not authenticate the person. This entire process is shown in Figure 7.

Proposed system software architecture
This paper will be tested two face recognition algorithms on the same hardware architecture in realtime conditions. The first algorithm to be tested is the LBPH from the open curriculum vitae (CV) library, and the second algorithm to be tested is dlib face_recognition based on the histogram of oriented gradient (HOG) from dlib library. Figure 8 is software architecture to use LBPH, and Figure 9 is software architecture to use dlib face_recognition. A real-time face detection algorithm to be used for each architecture is the Haar cascade classifier, which is based on a viola-jones robust real-time face detection algorithm [26]. This reason is that viola-jones face detection is more suitable for real-time face detection, requires less CPU resource, and fast to be processed by Raspberry Pi [27], [28].

RESEARCH METHOD
This section is explained the systematic experimental processes utilized to uncover new information for analysis purposes and create a better understanding of the research.

Prototype design
The schematic diagram in Figure 10 shows the overall system design while each module is integrated. This prototype design will be tested with two face recognition algorithms. This aims to see whether the system can work with several types of algorithms and which face recognition algorithms are more suitable for use in this prototype design.

Test scenario
The testing process will involve four actors. Two actors act as true eve whose face image has been used as the dataset and will be authenticated by the system. The other two actors act as a non-owner (falseevent) whose face image has not been used as the dataset and will not be verified by the system. Dataset images are taken separately using the haar cascade algorithm program and converted to grayscale as many as 30 images. In this testing, there are two parameters to be tested that are accuracy and processing time. Accuracy is obtained from the evaluation of the system performance using a confusion matrix, and the processing time is the time that takes for the system to process incoming (human) input until the system unlocks the door. The calculated processing time interval can be seen in Figure 11.

Evaluation method
Evaluation is the process of determining the performance of the designed system. The evaluation will be done using the confusion matrix. Figure 12 is the confusion matrix. There are four terms as a representative result of the classification process. True positive (TP), while the input is the owner, and the system recognizes the input as owner. True negative (TN), while the input is not the owner and the system, does not recognize the input as owner. False positive (FP), while the input is not the owner, but the system recognizes the input as the owner. False negative (FN) while the input is the owner, but the system does not recognize the input as the owner.

RESULT AND DISCUSSION
This section has explained the implementation, test, and evaluation results of the automatic face recognition door lock. Figure 12 shows the prototype of the system hardware its implementation in the testing environment. The testing environment is the real place, which is the entrance door of a real house. Figure 12. System prototype and its implementation in the testing environment

Functional test
After installing the system in the testing environment, the system should be passed the functional test to determine whether the system is working as planned or not. This test is crucial to do before testing the system performance. This test will be done on two face recognition algorithms LBPH and dlib face_recognition. The system will be activated and entering idle status, waiting for the owner to walk towards the door, as shown in Figure 13. After the owner arrives at the door, the PIR sensor will detect movement, and the lights automatically turn on then simultaneously run the face recognition will be carried out. After the owner is authenticated, as shown in Figure 14. The owner just needs to turn the door handle to open the door, then enter the house. After the owner enters the house and the door is closed, the system will automatically lock the door. This process was considered as a true-positive condition and can be concluded the system is working as planned.

Performance evaluation
This section will be discussing how the results of testing accuracy and processing time. Table 2 is the confusion matrix for LBPH based system and Table 3 is dlib face_recognition based system. Figure 15 is the performance comparison of LBPH and dlib face_recognition. Figure 16 shows the average processing time comparison of LBPH and dlib face_recognition.   The results of the implementation and experimental tests on the prototype, it was found in Figure 16 that the LBPH has better accuracy than the dlib face_recognition. The face recognition process in testing is carried out in real-time, where the input from the system is a real-time video captured by the raspberry pi camera. It is converted into a frame, and from that frame, the face detection process is carried out. The face detection method is carried out using the haar cascade classifier. Haar cascade classifier results are the face that becomes the input system to continue the recognition process using the local binary pattern histogram or dlib face_recognition method.
LBPH algorithm is resistant to differences in expression, lighting, and poses. This makes the LBPH is very suitable for use in real-time applications where image input is dynamic [29]. The dlib face_recognition algorithm used is based on the histogram of oriented gradient (HOG). HOG is known as an algorithm that has resistance to the 2D rotation but has no resistance to lighting. [30] therefore this is a drawback of the HOG-based method in this system architecture where the lighting is given at a certain angle, and the user's position as the face input does not stand in the same position all the time. So, of course, there will be differences in the input frame captured by the system. This is what makes the accuracy of LBPH slightly better than dlib face_recognition. In Table 1, dlib face_recognition is superior in detecting positive input and but not superior in detecting negative input. Meanwhile, LBPH is not superior in detecting positive input, but it is very good at detecting negative input. In the context of the security system, a false positive is an error that should not exist on the system because it will allow non-authorized persons to have access due to system errors.
The next thing that was tested in the experiment was processing time. LBPH has a better processing time than dlib face_recognition. The last process in face recognition used by both methods is Euclidean distance. This process compares the trained features from the image dataset and features from real-time input. Dlib face_recognition stores the trained feature with the pickle data type so that the data is encoded, and during the face recognition process, the system must decode the pickle file before the distance between the trained feature and the input feature can be calculated. In contrast to LBPH, trained data is stored in the YAML format so that no encoding and decoding process is occurred and makes the recognition process faster and low computational cost. In this case, there is a trade-off in terms of security trained data and processing speed. Based on the use case in this study, the Raspberry Pi, which stores trained data, is designed to work standalone and headless so that trained data in formal YAML is still categorized as safe from theft and sabotage. Figure 16. Average processing time comparison

CONCLUSION
The face recognition algorithm tested has good accuracy of 98% for local binary pattern histogram and 96% for dlib face_recognition. The average processing time for the algorithms is 1.472 seconds for local binary pattern histogram and 2.221 seconds for dlib face_recognition. Based on these data, the local binary pattern histogram algorithm has better performance for the novel system architecture that has been designed. The research in this paper has succeeded in eliminating locking and unlocking the door activities and remove any additional object called the key. The process of locking and unlocking the door is done automatically, and the key to the door is the owner's face. This makes these activities require the least user participation. The prototype is designed in a modular system that would become flexible and scalable for further development. The result of this study can be used to provide technical support to the elderly who are not experienced with technology. Future work for this system is by adding automatic door opening and closing devices such as electrically controllable motors and hydraulics that makes the system can be used as a solution to prevent virus transmission during the COVID-19 Pandemic because users would not need to touch the door before entering the house.