Intellicare is a multilingual, voice-based AI assistant engineered specifically for sensitive mental health-related conversations. Unlike text-based agents, Intellicare leverages real-time voice interaction and contextual memory to provide a more human-like, empathetic experience. Designed for high-stakes applications such as suicide hotlines and mental health support, Intellicare combines speech recognition, memory-augmented conversations, advanced safety guardrails, and real-time inference pipelines with sub-10 second latency. It can be accessed directly by simply calling a phone number.

The main challenge we ran into was using the twilio API to interface with our backend. We faced difficulties in getting the audio to stop recording only once the user is silent for a certain timeout. We overcome this issue by ensuring that our API doesnt try to access files its not supposed to, which is when the file is yet to be created. We tried out async and failed, and settled for a reasonable timeout.

An issue we faced with the ASR API is that it would sometime return texts in say Kannada, in Latin. This would further make it harder for us to use TTS since it requires a target language. We overcame this by using the languange detection endpoint, as a safety layer.

When we integrated the memory module through mem0, we found it not very effective since a lot of our text was in indic scripts, we overcame it by using an indic sentence transformer model for embeddings.

When we wanted to guard rail the models, it was tricky to set up the whole pipeline without increasing the latency by a lot. We utilized multi-threading to make sure the guardrail models running parallel and do not initerfere the main task.

while integrating self harm detection, we faced a lot of issues to analyze the audio directly since models aren't trained on indic languages, we used sarvam translate to translate the text to english and then analyzed those for self harm classification.

Another issue we faced is our LLM sometimes returning responses that could be seen as harmful. This is a very important part of a system like ours, since the topic is so sensitive. We fixed this by using guard rails.