Gesture based video extension for deaf users

Gesture based video extension for deaf users

The Gesture based video extension for dumb and deaf users is a tool designed to capture subtitles from YouTube videos and display them in the extension popup.

Created on 27th July 2024

Gesture based video extension for deaf users

Gesture based video extension for deaf users

The Gesture based video extension for dumb and deaf users is a tool designed to capture subtitles from YouTube videos and display them in the extension popup.

The problem Gesture based video extension for deaf users solves

Access to digital content remains a significant challenge for the deaf and hard of hearing community. Many online videos, whether educational, entertainment, or informational, rely heavily on spoken language, which can be inaccessible to individuals who are deaf or hard of hearing. Traditional solutions such as subtitles and closed captions are often limited in their effectiveness. They may not always accurately capture the nuances of spoken dialogue, and not all videos include these features. Additionally, existing captioning solutions do not provide real-time translation into sign language, which is essential for effective communication for those who use American Sign Language (ASL) or Indian Sign Language (ISL).

Our project addresses this issue by developing a browser extension that extracts audio from online videos, converts it into text, and translates this text into corresponding ASL or ISL gestures. This real-time translation enables deaf and hard of hearing viewers to access video content more intuitively and naturally. The use of gesture images as an interim solution will provide immediate benefits, while future enhancements will integrate dynamic 3D models for a more immersive experience. By bridging the gap between spoken language and sign language, our solution ensures that digital content is more inclusive and accessible, fostering greater participation and engagement from the deaf community. This innovative approach not only addresses the limitations of traditional captioning but also sets a new standard for digital accessibility.

Challenges we ran into

We faced several challenges during development, including capturing clear audio amidst background noise and achieving accurate speech-to-text conversion. Real-time processing was critical for synchronizing gestures with video, requiring efficient code optimization. Mapping words to ASL/ISL gestures and designing a non-intrusive user interface posed difficulties, as did integrating dynamic iClone 8 models, which led to using static images temporarily. Ensuring cross-platform compatibility also required extensive testing and adaptation. These challenges have guided our improvements and enhanced the extension’s overall functionality.

Discussion

Builders also viewed

See more projects on Devfolio