Describe your project

In Scope:

• A unique app overlay feature allows users to interact with the app while shopping online. When users click the overlay, they instantly receive product health analysis without leaving the shopping app.
• Users can search products or scan the barcode of packaged products to get health analysis.
• The core functionality includes analyzing products using Gemini to provide their health rating, positives and negatives, and allergens present.
• Nutritional information is shown in an easily visualized portion, such as per packet or bowl.
• Diet compliance and healthier alternatives are also given.
• Gemini verifies product claims by analyzing the product ingredients.
• Uses the Open Food Facts API for fetching product information.
• Data Collection by prompting users to contribute by capturing images of the product, which are processed using OCR and sent to Gemini to extract the necessary information and stored in Firestore.
• Calorie Counter allows users to input meals and receive information on calorie content using Gemini
• The Gemini-powered chatbot helps users navigate the app, answering questions about how to use it, and providing assistance in real-time.

Out Of Scope:

• The app will not work in offline mode or provide cached results when the internet is unavailable.
• Cannot extend app support to wearable devices

Future Opportunities:

• Post Processing of Gemini response for enhanced credibility and accuracy.
• Predictive analytics to preemptively flag products that are not compliant with a user’s dietary needs based on shopping history or trends.
• Multilingual Support.
• The app provides better product recommendations but doesn’t yet offer real-time ingredient substitution.
• By adding a sustainability score, the app could inform users about the environmental impact of products, encouraging users to make eco-friendly food choices.
• Expanding the dataset to other product categories

Challenges we ran into

1. App Overlay Implementation

• Implementing the overlay required complex permissions, like "SYSTEM_ALERT_WINDOW", and had to work only on certain apps like Zepto and Blinkit, making Android's security restrictions a hurdle.
• We documented necessary permissions, provided a guided onboarding process to help users enable overlays, and utilized an activity recognition service to ensure the overlay was only activated on supported apps. Rigorous testing helped ensure stability across devices and versions.

2. Configuration Conflict Between Google ML Kit and Firestore

• A significant bug arose when integrating Google ML Kit for OCR and Firestore for data storage. Both libraries had conflicting dependencies that caused build failures and app crashes.
• We resolved this conflict by forcing the version that works for firestore:
  configurations.all {
  resolutionStrategy {
  force 'com.squareup.okio:okio:1.17.5'
  }
  }

3. Fetching Data from API for Health Analysis

• Retrieving and filtering data from the API for health analysis took longer than expected, leading to slower app performance and user frustration.
• To speed up API data fetching, we optimized our API calls by batching requests and pre-fetching essential data in the background. We also implemented a loading state UI to manage user expectations, while background data filtering ensured smoother performance once the data was loaded. Additionally, we cached previous results to reduce repeated API queries, significantly improving response times.

4. OCR from Overlay

• Capturing screen text via OCR from the overlay in real-time without performance delays was difficult.
• We used a Kotlin method channel for efficient screen capture, employed a lightweight OCR library, and optimized the image processing pipeline for better recognition accuracy. A feedback loop was built to allow users to correct OCR results, and caching reduced load times for frequently accessed data.