Translation of Mexican sign language into Spanish using machine learning and internet of things devices

Abstract

This paper describes an application based on internet of things (IoT) and gesture recognition through the MediaPipe library to communicate people with hearing and speech disabilities who use Mexican sign language (MSL) with people who do not have a disability. The system is made up of two modules, on the one hand, there is an Apple watch with the ability to obtain text and voice inputs in Spanish to stand for the translation into sign language through video and images. The system can set up a connection with an application in charge of performing sign recognition by capturing images from a camera, the IoT user can connect so that communication between both modules is carried out bidirectionally through Firebase database. During the development of the experimental tests, the visual recognition module was able to recognize signs belonging to the Spanish alphabet, digits, and greetings, with a 96% of precision, while the IoT device correctly displayed voice translations and text to symbology compared to the MSL.