Waiting for your dashboard to queue and run costs you and your users valuable time. Further, long, repeated queries quickly damages both usability and maintainability.
We overcame a number of hurdles:
To begin with, though the Dune API is effective at fetching query results, it did not have dashboard nor SQL endpoints. Thus, we had to reverse engineer the frontend requests to extract the necessary data.
Many SQL parsers did not make the cut for our needs. We tested a number of parsers, including sqlparse and pglast, before settling on sqlglot.
Lastly, Dune does not yet allow the querying the results of other queries . Therefore, we extracted a sample dataset using the Dune API, and carried out the local validation using docker and postgresql.
Discussion