The problem Scooby solves

The need for using keyboard and mouse is eliminated in future. Hence there will be a commendable decrease in time and cost of manufacturing of computers and other electronic products making them cheaper by the day. This can be used in automated robots, cars or any other devices which require minimal human interaction to function. Remote access can be applied to this system to access it from anywhere in the world. Nowadays there are smart security systems installed in many homes. The root of these systems is generally controlled by the main computer of the house. If the user who is away form his house can remotely access and give commands wirelessly to his main computer, he can basically control his whole house sitting at any other place in the world. The CPUs that will be manufactured in coming future will be having much more capacity than the ones now. These CPUs combined with the state of the art Image Processing, Natural Language Processing and Speech-To-Text algorithms will surely prove to be revolutionary in the field of computer automation. Moreover this same program can be applied in linux too which requires just changing the automation libraries of windows and linux.

Challenges we ran into

Integration of different modules of python. Libraries like Spacy, Pytesseract, PILLOW, Pywinauto didnt support the same configurations we had on our system. Once we got rid of that our biggest problem was to manage clicking on screens with various strings matching the requires string. We Gave each match a number to be identified with and stores their coordinates in an array. Then we showed the user all the options and asked him to specify which one he wants to click depending on number. Hence the user would give us the number to click which would give us the coordinates of that element. One more slight problem was of inaccuracy in speech recognition but we tackled it eventually.