The problem setodata solves

424+ INGOs currently actively exist in Nepal. Additionally, 50k+ NGOs directly work together as a force to make a difference in various sectors; public health, rural development etc. However, due to lesser accessibility to data and higher costs in building in-house tech; most of research and societal understanding is heavily based on qualitative factors such as; participatory techniques, interviews, surveys or focus group.

Setodata as a research analysis tool aims to bring more publically available yet in-accessible ones to easy user experience, onboarding and direct decision making level data for better, quicker and more efficient market analysis and understanding.

Challenges we ran into

Firstly, since the data is heavily and majorly relied upon government's websites; portals and public databases it was difficult to cater to different params; data types and even file types for the same files spanning different months.

This made us work through multiple approaches over how we utilized our data pipeline to sometimes use hardcoded parsing method while other time using xlsx file system to take in the columns and rows.

Additionally; few government websites had ssl issues that made us forcefully brutforce into the website when making basic curl requests.

Moreover, we are doubtlful these websites will be up and running with the same data existent; hence we had to make database schema based on different data types and tables; to backup on our own system and not totally rely over government unstable websites.