The problem Lens Substream solves

Decentralized social media generates a huge amount of on-chain data. This on-chain data needs a low-latency interface for it to remain accessible for all, along with adding the ability to perform real-time analysis and monitoring.

Substreams are a data solution developed for The Graph Network. They allow developers to write Rust modules composing data streams with low-cost caching and archiving of blockchain data, high throughput processing, and cursor-based reorgs handling.

This project builds a (basic) Substream in Rust to monitor on-chain Lens protocol data and provide a low-latency data streaming interface. Potential uses of such a tool could be for real-time analysis and monitoring of Lens data. Since the data is streamed, it can be directly injected into a variety of databases (BigQuery, Clickhouse, Kafka etc.), opening up a lot of distinct analytics use cases.

A few unrelated citations:

1. Gizmodo: Here Are 30 of the Hundreds of Subreddits Going Dark to Protest Reddit's API Changes
2. Verge: Twitter just closed the book on academic research
3. 9to5Google: Reddit’s API changes kill Android’s best third-party apps, including RIF and Sync

Challenges I ran into

• Substreams are an extremely new technology. The most actively developed implementation by the team (uniswap-v3) is still in beta. Therefore, the documentation is extremely limited. I worked around this by simple trial and error, playing around with things until they worked.
• Lens protocol is complex and involves multiple moving parts, this implementation only covers a small fraction of it. However, given the modular nature of the substream, this functionality can be added in the future.
• Given the novelty of working with Substreams, I would particularly like to thank Alex and Ørjan from The Graph for helpful discussions.