The problem VisualSense solves

Low vision impairment significantly impacts individuals' quality of life, limiting their ability to perform daily activities independently. It underscores the importance of accessible technology and assistive devices to enhance independence and inclusion for affected individuals, including AI-powered solutions like object detection models for image accessibility.
VisualSense provides specific UI catering to specific type of impairments. Visually impaired individuals can identify objects in their surroundings or aspects of images through audio prompts, whilst receiving audio outputs from the model.

• Functionality: The existing system relies on real-time visual assistance from volunteers, while VisualSense uses VQA (Visual Question Answering) based image analysis.
• Target Scenario: The existing system is designed for immediate understanding of surroundings, whereas VisualSense is better suited for detailed analysis of specific images.
• User Interaction: The existing system requires interaction with sighted volunteers, while VisualSense allows for independent questioning and analysis by the user.
• Privacy: The existing system might have lower privacy due to volunteer interaction, while VisualSense offers potentially higher privacy with its automated analysis.
• Scalability: The existing system's scalability is limited by the availability of volunteers, whereas VisualSense has the potential to be more scalable.

Challenges we ran into

• Identifying requirements for the types of low vision impairments due to less awareness and scarcity of existing documentation about the problem.
• Integrating machine learning with the backend.
• Generating full sentence responses with the transformer VQA model and existing datasets.
• Adding shortcut keys to provide a different UX tailored for visually impaired individuals, and enhancing it.