An assistive communication system for mute individuals using MATLAB-based image processing and sensor integration

Abstract

Interactions between non-verbal people and verbal communicators have always presented serious challenges. Although functional, current sign language systems often lack the flexibility and affordability needed for wider use. This document provides a novel speaking system created for those who are mute, supporting communication using two methodologies: i) recognition of hand motions with flex sensors and ii) detection of finger positions through image processing techniques combined with MATLAB. The setup in question uses a single camera and a Raspberry Pi to capture hand photos. Finger coordinates are then extracted from these photos and compared to an existing database for matching. An appropriate verbal output is then produced from the identified gesture. Flex sensor-based inputs are simultaneously connected to an Arduino, which uses a Bluetooth module to transmit text data to an external application for speech conversion. This dual-mode technology improves everyday communication for people who are silent by providing a customized and efficient framework for converting sign-based gestures into vocal communications. This approach makes a significant contribution to the development of inclusive technology by offering an affordable, adaptable, and practical alternative for assistive communication.