Abstract

The identification of the Indian Sign Language (ISL) is a challenge because it is quite complicated, with regard to its spatio-temporal, regional, and vulnerability of the video data involved. In this study, a Privacy-Saving Hybrid Deep and Machine Learning Framework is proposed that can be used in real-time to identify the ISL in edge devices. The research seeks to handle the dual challenge of high recognition accuracy and data security by offloading processing to the edge, hence securing the privacy of users by local processing. The proposed framework adopts a hybrid architecture, in which deep learning image-based feature extractors, namely MoViNet and Compact Inflated 3D ConvNet (I3D), serve as feature extractors to obtain high-resolution spatio-temporal gesture representations. Random Forest (RF) and Support Vector Classifier (SVC) are then used to classify these features. To enhance the privacy measure, the architecture will use differential privacy and local anonymity of data, where all raw data is not sent to third parties. The system was strictly tested with the help of the ISLVID25K and ISL-CSLTR datasets. The statistical findings indicate that the MoViNet+RF model produced the highest accuracy of 100.0% at the word-level recognition, and the I3D+RF model had a maximum accuracy of 99.2%. In a more complicated classification at the sentence level, the scheme had a high accuracy of 93.33%. The performance of the edge hardware performance benchmarks indicates an inference latency of less than 40 milliseconds and a small memory footprint of about 25 MB, which makes it very mobile-friendly. The results prove the hybrid method to be efficient in terms of balancing between computational efficiency and state-of-the-art accuracy. This framework will be a safe and accessible communication tool in privacy-restrictive settings that accommodates the Deaf and Hard-of-Hearing community by adding layers of privacy protection and real-time functionality to the system.