Challenges We Ran Into

1. Handling Large-Scale Data

• Hurdle: With over 180,000 projects on Devfolio, efficiently managing and comparing this extensive dataset posed a major challenge. Processing such a volume could easily lead to slow performance and high resource usage.
• Solution: We leveraged a combination of optimized data structures, efficient database indexing, and preprocessing techniques to handle and streamline the data, enabling faster similarity checks without compromising accuracy.

2. Dynamic Content Parsing and Scraping

• Hurdle: Many Devfolio project pages use dynamic content that required us to adapt our scraping techniques for consistency and accuracy. Changing class names and dynamic content required solutions that could adapt without breaking with each page load.
• Solution: To overcome this, we implemented a smart scraping approach using robust tools and selector fallback methods, allowing us to reliably access the required data regardless of page structure changes.

3. Textual Crux Extraction and Vectorization

• Hurdle: Summarizing each project’s description into a concise "crux" for efficient comparison was challenging, especially with the variation in text length and structure across projects.
• Solution: We incorporated an NLP model to automatically generate concise descriptions, capturing each project’s essence. These summaries were then vectorized to enable high-speed similarity calculations, achieving efficient and relevant comparisons.

4. Ensuring Accuracy in Similarity Detection

• Hurdle: Balancing accuracy and speed in detecting similarities was complex. Setting thresholds too high or low could lead to either missed detections or excessive false positives.
• Solution: We fine-tuned the similarity thresholds based on testing data to ensure accurate results, refining the model’s parameters to strike a balance between precision and recall.