The problem Finance Tracker solves

Tracking our expenditures was a problem as it takes a lot of effort to maintain the handwritten ledger or even if you use an App you need to add the product, price, and other details manually, and later summing the items up is another headache. So, we started with an idea to make an App that helps any common man of various age groups to be able to keep an account of his expenses. The App we made in this Hackathon helps you with:

• Adding Item Details Manually
• Scanning your bills and adding all the Items without any need to type,
• Categorizing your item details into 6 categories we have defined currently,
• Present a Pie Chart denoting various fields where you have spent.
• Fetches details about Stock Market and Bitcoin prices.

We had few time constraints and there were many things we wanted to implement in order to make our App the ultimate solution for tracking your finance and investment. Few things we will love to change or add are:

• The Data Analytics need to be even more good, and we need to learn a bit about personal finance about the parameters on which we can recommend our users some investment options in various domains,
• The OCR on Bills and Receipts is a big problem yet because of the quality of Bills and huge variations in their layout so we can improve upon those points and build a model similar to Google Clouds Receipt Parser which is a quite Expensive service for any new App Service.
• We can use Account Aggregator in order to include those transaction details which are made online or details about EMI's and credit cards, hence we will be able to create a more suitable Analytic Tool.

We have currently deployed our Node.js Backend. We have not deployed OCR part and the Flask Backend which connects ML to our App, so please bear with us. You can see the UI and other parts of our App by installing Expo App from PlayStore and Scan the QR code here: https://exp.host/@neelbavarva/finance-tracker

Challenges we ran into

Certain packages use fs in order to create and read files on local storage, which was causing an error on react-native, cause we were running it on the expo. To overcome this we just edited and commented on those parts in node modules as we were uploading images to the cloud and we didn't need them.

The problems we faced in the initial stage was about deciding how to use an OCR and how to do these work. It was also a headache to obtain the correct output from the OCR, to reduce this problem we improved the contrast of the scanned bill and then uploaded it, we also shifted to GCP for better output. But even after doing this, we could not get favourable outcome and hence we tried Adaptive Gaussian Thresholding which gave us a much better result, but still, the output text was not distinguishing the difference between columns and hence instead of using Text obtained we used the Poly Bound from Google Cloud Visions response and the obtained it ourselves if the two-pieces of texts are in the same column, or same words part, in the same row, etc. It showed us good results with a regular bill layout.

We have written ML codes before but were not prepared to connect them with React Native Apps, so that was a problem we faced and a good amount of time went on researching. Then we tried writing the entire code in tf.js but that didn't work as model needed to be loaded using fs and it was not possible with the expo. So we changed the model to MultinomialNB and saved the trained model to pickle format and connected it using Flask Backend. To properly train the model, we have used scraped data as per categories from Indian Mart and it had 70K products from these 6 categories.