https://github.com/dineshpinto/lens-substream
Decentralized social-media data streaming with Rust (Best New Subgraph @ ETH Prague 2023)
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.6%) to scientific vocabulary
Keywords
Repository
Decentralized social-media data streaming with Rust (Best New Subgraph @ ETH Prague 2023)
Basic Info
Statistics
- Stars: 4
- Watchers: 2
- Forks: 0
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
Lens Substream
A Rusty way to stream Lens data using Substreams.
Decentralized social media generates a huge amount of on-chain data. This on-chain data needs a low-latency interface for it to remain accessible for all, along with adding the ability to perform real-time analysis and monitoring.
Substreams are a data solution developed for The Graph Network. They allow developers to write Rust modules composing data streams with low-cost caching and archiving of blockchain data, high throughput processing, and cursor-based reorgs handling.
This project builds a (basic) Substream in Rust to monitor on-chain Lens protocol data and provide a low-latency data streaming interface. Potential uses of such a tool could be for real-time analysis and monitoring of Lens data. Since Substreams are streamed, the data can be directly injected into a variety of databases (BigQuery, Clickhouse, Kafka etc.), opening up a lot of distinct analytics use cases.
A few unrelated citations:
- Gizmodo: Here Are 30 of the Hundreds of Subreddits Going Dark to Protest Reddit's API Changes
- Verge: Twitter just closed the book on academic research
- 9to5Google: Reddit’s API changes kill Android’s best third-party apps, including RIF and Sync
Installation
- Copy this repo
- Get an API key from StreamingFast
- Install the Substreams CLI
- Add the API key to your environment variables
bash
export STREAMINGFAST_KEY='xxx'
export SUBSTREAMS_API_TOKEN=$(curl https://auth.streamingfast.io/v1/auth/issue -s --data-binary '{"api_key":"'$STREAMINGFAST_KEY'"}' | jq -r .token)
Using Makefile defaults
bash
make codegen
make build
make stream
Manual compilation
Generate protobuf
bash
substreams protogen ./substreams.yaml --exclude-paths="sf/substreams,google"
Compile
bash
cargo build --target wasm32-unknown-unknown --release
Start Substream
bash
substreams run -e polygon.streamingfast.io:443 substreams.yaml map_transfers -s 43764445 -t 100
Limitations
Tuplesare not currently supported byrust-ethereum/ethabi( see open issue), so they have been removed from the Lens Protocol Profiles (LPP) ABI. Currently, only implements protobuf interfaces for Lens collects and mirrors.
Owner
- Name: Dinesh Pinto
- Login: dineshpinto
- Kind: user
- Location: Switzerland/Germany
- Website: dineshpinto.github.io
- Twitter: dineshkpinto
- Repositories: 6
- Profile: https://github.com/dineshpinto
quantum info PhD student @ EPFL, pythonista & rustacean
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: 10 months ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0