The problem InsightPilot solves

In today’s digital world, customer feedback is pivotal in order to figure out what our users exactly need, their pain points and preferences. There is a plethora of data out there in the form of user reviews, tweets, comments etc that remains untapped to a major extent.

In this project, we build an end to end system, capable of ingesting data from multiple sources, denoising the data and clustering into relevant action items. We have built a dynamic hierarchical clustering algorithm in order to segment new custom reviews. In order to answer specific queries on the tickets, we have integrated a chatbot functionality within DevRev. We also produce an interactive dashboard from which end users can easily infer various insights of the customer requests such as type of issues, common customer pain points etc.

Our working model :

1)Our DevRev snapin is integrated with a timer automation which polls our data sources (Twitter and Playstore ) every 10 minutes and ingests unique customer reviews in real time.

2)The ingested data is sent to our analytics microservice written using FastAPI where we compute semantic embeddings for each review and store it in a Vector Database.

3)For each review received, we run a nearest neighbour search on the vector space using cosine similarity and compute the most probable cluster that the new review must belong to

4)These data points are returned to the snap-in, where we dynamically create either a feature or issue ticket, and assign it the relevant cluster tags.

5)In order to answer user queries on any tickets, we have a Retrieval Augmented Generation (RAG) pipeline which takes in the user query, fetches the closest reviews from vectordb, and passes the query along with the data subset to GPT 3.5 LLM Model in order to get summarised responses. This has been integrated directly with the DevRev UI as a bot.

6)UI dashboard to gain deep insights on existing issues and competitor analysis to improve performance

Challenges we ran into

Key Challenges

1)It took a lot of experimentation in order to figure out a clustering algorithm which allows us to dynamically compute and assign clusters to incoming tickets. For this we experiemented with different LLM Models, tried different distance measures and also tweaked our prompts multiple times get the best result.
2)It was also challenging to implement the chat mechanism and integrate it with DevRev. We needed to experiment with the model to ensure hallucination is minimal and most responses are returned from the existing tickets created.
3)Since we were completely new to devrev, it took some time to get comfortable with the new terminology, and methods to deploy a snap in.
We are proud to say that we have compared the existing "Smart Cluster" funcitonality provided in DevRev with our Dynamic Hierarchial Clustering algorithm and have seen much better results with our solution