Abstract
Vision-based sign language recognition has the potential to bridge communication barriers within the deaf community. This paper explores a vision-based system designed for real-time sign language recognition, specifically focusing on South African Sign Language (SASL). We implement a streamlined pipeline that combines Google's MediaPipe Holistic framework for spatial feature extraction with a long short-term memory (LSTM) network for temporal modelling. Our experiments utilise a subset of the ASL Citizen dataset, featuring five word-level glosses, " SCHOOL, " " TIMEOUT, " " MORNING, " " THANK YOU, " and " I LOVE YOU " , as a proxy for SASL vocabulary. Key point sequences from both hands and body pose, extracted via MediaPipe, are input into a two-layer LSTM, followed by dense layers for classification. The system is trained using a TensorFlow TFRecord pipeline with sequences consisting of 30 frames per sign, employing the Adam optimiser and categorical cross-entropy loss. Empirical results reveal that the model achieves an accuracy of only 35.5% on held-out test clips, with approximately 32.5% peak training accuracy and 35.5% validation accuracy. These modest results underline the challenges posed by limited data and signer variation in SASL recognition. We analyse several factors contributing to the model's suboptimal performance, such as class confusion and motion ambiguity, and evaluate real-time webcam behaviour through an OpenCV demo. This work serves as a proof of concept that the combination of MediaPipe and LSTM can operate in real time for sign recognition, while emphasising the need for larger, language-specific datasets and more advanced models to enhance accuracy. 1