A not-so-technical introduction to building a blockchain indexer using Rust 🦀, simultaneously serving a GraphQL process and subscribing to block events.
The blockchain has been one of the most revolutionary technologies of the 21st century, destroying the boundaries I refer to as "economic borders" — allowing value (money) to flow without bounds and selling trust as a commodity, allowing parties that would never trust each other to do business without needing to engineer that trust. As beautiful as this technology may be, it has its downsides, one of which we will be addressing in this project.
Querying data directly from the blockchain is not scalable; it's like trying to drink from a firehose — messy, chaotic, and downright inefficient. Also, consider that after a while the node this data is being queried from is pruned and historic data can no longer be accessed. To access historical data, archive nodes have to be employed, and that can be very expensive. "Indexers" step in to solve this problem.
The core principle behind an indexer is querying or subscribing to events happening on a blockchain, storing these events and their metadata in a database, and serving that data to applications more efficiently. Some indexers go further, providing functionality to programmatically process this event data as the consuming application pleases. TheGraph and SubGraph are the most popular indexing service providers in the web3 ecosystem.
Building a Blockchain Indexer
A production-level indexing software involves the following components:
- A component for querying events from the blockchain using a filter that may include
from_block,event_signature, andaddress. - A component to subscribe to events happening on the blockchain, filtering them down to events concerning the consuming application (filters may include
from_block,event_signature, andaddress). - A component for serving this data to the consuming application — most indexing software uses GraphQL as it is very flexible and efficient.
These are the components that will be built in this rusty implementation.
Chronicle Architecture
In essence, Chronicle is a blockchain indexing tool, indexing transactions and modules' (smart contract, parachain, and so on) events. Chronicle stores this data in a Postgres DB and renders it using a GraphQL interface. It is designed to achieve high performance and high reliability by adapting Rust's concurrent paradigm and exploring a modular architecture — developers can hand-pick modules to build whatever they need, treating the components of Chronicle as pieces of Lego.
Repo: developeruche/chronicle
Chronicle needs at least 2 threads — one for indexing events (i.e. querying and subscribing to events) and another for serving the GraphQL server through which the consuming application grabs data. The async, multi-threading, and message-passing functionality is tightly coupled to the Rust tokio library.
Chronicle is, to an extent, built as a playground for intermediate Rust engineers to have fun with, as a good chunk of the development practices and Rust-centred design patterns are inspired by Substrate, Hyperbridge, and Rundler. Contributors will see a lot of not-so-entry-level code, and some shitty code too, written when laziness got to me. :)
Here are some highlights many people would look forward to:
- Async Rust
- A lot of heavy lifting using Traits and Generics
- Multi-threading with Rust
- Thread-channel message passing in Rust
Contributing to Chronicle
I somehow see Chronicle as a developer social experiment — to see how far an open-source project with no sort of internal maintenance can get to a production-ready stage for vast use cases. My recommended steps to contributing to the Chronicle codebase are: first and foremost read this article, visit the developer's documentation, read through the code comments, take down some already-existing issues, create and tackle more issues, and have rusty fun.
The crab soup is on me. :)
Built with 🦀 by Developer Uche, twitter.com/developeruche