The problem Hear Aid solves

In today’s digital world, communication is key, yet millions of deaf or hard-of-hearing individuals face barriers due to limited accessibility. While American Sign Language (ASL) bridges the gap, many platforms lack seamless ASL support, creating a divide between hearing and non-hearing individuals. This gap is especially evident in smaller companies and startups, where real-time ASL interpretation is often absent.

Our ASL project tackles this issue by developing a real-time ASL translation application using advanced NLP techniques and Unity-powered 3D sign language simulations. By translating spoken or written text into ASL, we aim to break down communication barriers, making interactions—whether in meetings, classrooms, or virtual chats—more inclusive. Our goal is simple: to create a world where everyone has equal access to effective communication, regardless of their hearing ability.

Challenges we ran into

The challenges we ran into included the time-consuming process of animating each keyframe in Blender, which required a lot of patience to position the animations correctly. Mapping animations to words posed difficulties, especially with homographs (words with the same spelling but different meanings), which we addressed using POS tagging in NLP. We also had to handle words with the same meaning but different forms, solved by using semantic search with sentence transformers. Additionally, rendering the application as WebGL for web integration added complexity, as did handling idiomatic signs to ensure accurate ASL translation for phrases.