The problem Nutshell solves

Nutshell allows students to quickly summarise their entire video lectures in just a few minutes. Students can make use of Nutshell whenever they need the key points from the class but they don't have enough time to go through it all by themselves. Whether it's for a quick revision, catching up on missed classes, or simply a reference for what happened, Nutshell helps students by providing them with the most important parts of the class without them having to watch a second of the recorded video.
And the best part is that it's not just limited to online lectures. Anyone in need of summarising a video-based session can make use of Nutshell and get all the meat. Be it online conferences, meetings, presentations, seminars, or workshops, you just have to upload the recorded video with clear audio, and Nutshell takes care of the rest.

Challenges we ran into

There were quite a few challenges that we faced throughout the completion of the project. Since we were trying to make use of different APIs and libraries to accomplish the task, it was challenging to make everything work with everything else.

Video to audio conversion

This was a particularly important step as we wanted to use speech-to-text technologies for transcribing the video and later summarising it. But we realised that in order to proceed, we needed to convert the video into audio format. We tested a few libraries until we settled on Moviepy to accomplish this task.

Speech-to-text

The first choice was to go with Google's speech recog library but there is a limit of 1min on that. Hence the fix there was to divide the file into chunks using PyDub's split-on-silence method. That took a long time to process, however. Hence we researched a bit more and decided to go with Deepgram STT API to handle the transcribing of audio.

Summarising

At first, we were using our own summariser that we wrote implementing the TextRank extractive summary algorithm in Python. That worked well but since it was all in Python, it was slow. We found that the NLP library Gensim also has a TextRank implementation and is more optimized for our use and hence we ended up using that.