Classificação de sinais dinâmicos de Libras relacionados ao contexto hospitalar utilizando redes neurais recorrentes

Abstract

Limited communication between healthcare professionals and deaf patients still represents a significant obstacle to ensuring inclusive and equitable care, especially in clinical contexts in which the speed and accuracy of information are essential. In this scenario, this work proposes a method for the classification of ten dynamic signs of Brazilian Sign Language (Libras) related to the healthcare domain, encompassing the construction of a dataset as well as the development and evaluation of machine learning models. The constructed dataset involved the participation of professionals fluent in Libras, ensuring linguistic quality and diversity of execution. A total of 492 videos were recorded, corresponding to ten signs commonly used in hospital contexts, performed by nine individuals under controlled conditions, with variations in clothing, appearance, and gesture repetition in order to increase sample variability. The preprocessing stage included frame selection through probabilistic sampling and the extraction of three-dimensional hand and body pose landmarks, which correspond to anatomical reference points that describe the configuration and movement of the body over time. These landmarks were organized into temporal sequences, preserving the dynamics of the gestures. Based on this data, different classification approaches were investigated using neural network models with Long Short-Term Memory (LSTM) layers, capable of modeling the temporal dependencies present in the sequences of sign movements. Initially, the problem was formulated with ten classes. The analysis of confusion matrices revealed specific confusions between pairs of signs, which motivated the adoption of a hierarchical strategy based on specialist models, in which the problem was reorganized into seven classes, complemented by classifiers dedicated to the conflicting pairs. This strategy resulted in significant improvements in accuracy and result stability, achieving an average accuracy of 92.44% in the experiments. Evaluation on an additional test dataset confirmed the system's good generalization capability, with an average accuracy of 98.92% and a low standard deviation. The results indicate that the combination of a well-structured dataset, appropriate preprocessing techniques, and a hierarchical architecture of specialist models constitutes a promising solution for Libras sign recognition, contributing to the reduction of communication barriers and with potential application in healthcare communication support systems.