VisionCORS - ASL Sign Interpreter
At VisionCORS, we aim to make deaf/mute people's life easier by bridging the communication gap and helping them to become more confident, successful and happy.
Created on 26th November 2022
•
VisionCORS - ASL Sign Interpreter
At VisionCORS, we aim to make deaf/mute people's life easier by bridging the communication gap and helping them to become more confident, successful and happy.
The problem VisionCORS - ASL Sign Interpreter solves
The deaf/mute community is not a very well-connected community. They have to depend on non-verbal or written communication which makes them highly dependent on others to relay messages and it's a not a very reliable and time efficient way. This leads to a large communication gap between the deaf and the rest of the world. We aim at bridging this gap by creating a digital platform that will help the deaf/mute people communicate with others.This submission is our attempt at building something positive towards this goal.
It can be used to convert sign language (ASL) hand gestures to text and speech.The model can be trained to convert text/speech to hand gestures in sign language.Once it is launched as a mobile app or API, it will become very convenient and accessible to a much wider consumer spectrum. It won't just be restricted to a particular section of society. Thus, we aim to normalize the use of this technology to such an extent that affected people dont feel marginalized.
Challenges we ran into
The biggest hurdle we faced during the ideation phase was absence of a standard Indian Sign Language(ISL). There are various dialects in the Indian Sign Langauge which vary greatly as we move across the country. Therefore to have a standardized approach we opted for the use of American Sign Language(ASL).
The first challenge we faced was finding a proper dataset which could fit for our model and approach. Therefore we decided to proceed and construct our own dataset which severly limited the extent of expansion of our project given the time constraint.So we decided to go with 14 alphabets (A->O).
The second challenge that we encountered was while training, we observed that the lighting conditions and the surrounding mattered a lot for proper interpretation and the solution we came up with was data augmentation using pillow and changing real time lighting conditions to further expand our dataset for proper training for the model. We also tried masking while detection but it had constraints of it's own as the gestures of a few alphabets were quite similar in their orientation.
The last challenge that we faced was tuning and improving the accuracy of the trained model in the given frame of time along with preparing front and back end to deploy the application.