Describe your project

Our project focuses on real-time fraud detection using Twilio, Google API, and an ML model. When an unknown caller contacts a user, a Twilio agent joins the call, listening to the conversation. Using Twilio's API, we obtain real-time audio converted to text through Google's speech-to-text API. Our machine learning model then analyzes this transcription to determine if the caller is engaging in fraudulent or suspicious activities. If fraud is detected, the Twilio agent alerts the user during the call.

In scope of this solution are the seamless integration of Twilio for call handling, Google's API for real-time speech transcription, and the machine learning model for detecting fraudulent behaviour during the call. The solution focuses on detecting fraud in real-time through live transcription and analysis.

Out of scope includes any manual investigation of fraud after the call has ended, complex multi-lingual support beyond the languages handled by Google's API, and the creation of a detailed user interface beyond basic alerts. We also exclude handling fraud detection outside of phone conversations, such as in emails or messages.

Future opportunities for the solution include expanding to support more languages and dialects, integrating biometric or voice recognition to improve fraud detection accuracy, and developing advanced features like getting to know about the latest scams with the help of new channels api’s to refine the model’s capabilities further. Additionally, expanding the system to other communication channels such as SMS or email is another possibility for the future.

Challenges we ran into

1. Integrating the conference call feature with Twilio was one of the biggest issue as it was completely new for us as a team. Moreover, we had to perform live transcription on the Twilio conference call.
2. Defining the best user experience for this project was also quite challenging as the main focus of the idea is to intimate the user about the fraud calls without actually ruining the actual call experience.
3. Defining the architecture of the project was also a major challenge as it was very important to have the proper flow of data without much delay.