The problem Sol Scraper solves

Features

• create APIs from on-chain data which is not limited to just the generics (hash, fees etc)
• add dApp specific information to the APIs
• filter data using custom queries

Design Details

• Scraper: create/save queries which will run as background jobs to fetch and parse on chain data
• Query: specify the contract, filters, and fields you would like to export
• Storage: Centralised storage for now in the form of MongoDB to store query metadata and the results from the queries

Challenges I ran into

• Understanding the working of existing dApps including their transactions, programs, accounts to form meaningful queries
• Parsing of queries : for extracting values from complex programs like Serum, there is a need to understand the structure of the data present in the event queues to find fill/bid trades