The problem Gesto solves
Hand Gesture Recognition technology is of immense importance in the modern world, be it in sign language translation, automation, music generation, or AR/VR systems.
Current hand gesture recognition models come with repetitive pre-trained data and very less scope of customizability. They are useful for detecting gestures pertaining to very few sign languages (mostly ASL), they have no option to train custom signs, and are computationally slow.
Also, there are no web applications that allow users to train their gestures and make predictions in real-time.
We present a first-of-its-kind web app, Gesto, for custom gesture training and recognition.
Our web app interactively gathers data from the user and uses an ML algorithm to train on it. Rather than conventional images and CNNs, we use efficient data formats and lightweight neural networks that provide state-of-the-art accuracy. Once trained, the predictions that our model makes on new data are visible to the user in real-time with zero latency.
We are demonstrating the applications of this system in the fields of automation, music synthesis, and novel sign language generation in our project.
Challenges we ran into
- Integration of Mediapipe with ReactJs and NextJs
- Generating Dataset for custom gestures of the user from limited time video clip
- Designing Neural Network architecture to achieve very less training time with high accuracy for custom gestures of users
- Training Models on a limited amount of data with high accuracy
- Finding out way to serve trained model so as to predict gestures in real-time with zero latency
- Designing minimalistic UI with great UX
Technologies used