One of the major challenges we faced was handling the generation of embeddings for very long videos. When attempting to generate embeddings for frequent timestamps across these videos, we encountered Google Gemini AI's rate limiting errors due to excessive API calls in a short span.
How We Overcame It:
Rate Limiting Implementation:
To resolve this, we implemented a custom rate limiter that restricts the number of embedding requests per certain number of timestamps. This ensures that our system stays within the API rate limits and prevents request failures, while still maintaining reasonable coverage across the video.
Reducing Redundancy with Cosine Similarity:
Another hurdle was that generating embeddings for every small clip resulted in many highly similar segments. Displaying these individually would overwhelm the user and reduce the usefulness of the navigation feature.
To solve this, we applied pairwise cosine similarity to compare embeddings of adjacent timestamps. If their similarity score exceeded a set threshold, these timestamps were merged into a single, meaningful segment, thus reducing clutter and making the navigation more user-friendly.
This not only prevented redundant API calls but also made sure that the user gets longer, meaningful clips instead of fragmented and overly short ones.