-
Notifications
You must be signed in to change notification settings - Fork 155
WIP: Update Substreams documentation #760
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
@@ -4,43 +4,36 @@ title: Substreams | |||||||||||
|
||||||||||||
 | ||||||||||||
|
||||||||||||
Substreams is a powerful blockchain indexing technology developed for The Graph Network. It enables developers to write Rust modules, compose data streams alongside the community, and provide extremely high-performance indexing due to parallelization in a streaming-first approach. | ||||||||||||
Substreams is a powerful blockchain indexing technology developed to improve The Graph Network: | ||||||||||||
- Substreams allows you to speed up the indexing times of your subgraph. | ||||||||||||
- Substreams allows you to index non-EVM data (Solana, Injective, Starknet, Vara...) | ||||||||||||
|
||||||||||||
With Substreams, developers can quickly extract data from different blockchains (Ethereum, BNB, Solana, ect.) and send it to various locations of their choice, such as a Postgres database, a Mongo database, or a Subgraph. Additionally, Substreams packages enable developers to specify which data they want to extract from the blockchain. | ||||||||||||
This is possible because your subgraph can use a Substreams package as a data source, thus creating a Substreams-powered Subgraph (SpS). | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
According to "The Graph Style Guide", we shouldn't capitalize "subgraph", even in the phrase "Substreams-powered subgraphs". (@bermchain, should we make that document public, and maybe link to it from the docs' CONTRIBUTING guidelines?) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh and I propose removing the acronym because it's not used anywhere else on this page, but maybe it's used somewhere else and I'm not aware? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed @benface with lowercase subgraph and not using the acronym unless it's recurring. I like the idea of publicizing the style guide in the Docs! Let's do it -- I'll select sections that make sense to share. Also, re: "Substreams is" vs. "Substreams are" -- I believe it's the former because Substreams is a collective noun for the technology (i.e., powered by Substreams tech, not powered by a Substream). |
||||||||||||
A Substreams package a binary file that contains all the definitions of the data that you want to extract from the blockchain (similar to the `mapping.ts` file of your subgraph). | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
|
||||||||||||
## How Substreams Works in 4 Steps | ||||||||||||
The following example shows a subgraph manifest (`subgraph.yaml`) that uses a Substreams package as data source, thus receiving the data extracted by the Substreams. | ||||||||||||
Then, you can use this data in your `mapping.ts` and perform all the transformations needed as a usual subgraph. | ||||||||||||
Comment on lines
+14
to
+15
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
I'm thinking this is why the Prettier check is failing, but it could be more than one thing. If you're using VS Code, I recommend installing the Prettier extension with "auto-format on save" enabled. |
||||||||||||
|
||||||||||||
1. **You write a Rust program, which defines the transformations that you want to apply to the blockchain data.** For example, the following Rust function extracts relevant information from an Ethereum block (number, hash, and parent hash). | ||||||||||||
// example | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oops, I just realized this was a WIP; sorry I didn't pay attention to the title. 😅 I'll stop reviewing now unless you want me to continue. |
||||||||||||
|
||||||||||||
```rust | ||||||||||||
fn get_my_block(blk: Block) -> Result<MyBlock, substreams::errors::Error> { | ||||||||||||
let header = blk.header.as_ref().unwrap(); | ||||||||||||
## Find Substreams packages | ||||||||||||
|
||||||||||||
Ok(MyBlock { | ||||||||||||
number: blk.number, | ||||||||||||
hash: Hex::encode(&blk.hash), | ||||||||||||
parent_hash: Hex::encode(&header.parent_hash), | ||||||||||||
}) | ||||||||||||
} | ||||||||||||
``` | ||||||||||||
There are many Substreams packages already built that you can simply import into your subgraph. | ||||||||||||
[substreams.dev](https://substreams.dev) is a Substreams registry where you can explore different Substreams packages for several blockchain networks. | ||||||||||||
|
||||||||||||
2. **You wrap up your Rust program into a WASM module just by running a single CLI command.** | ||||||||||||
// image | ||||||||||||
|
||||||||||||
3. **The WASM container is sent to a Substreams endpoint for execution.** The Substreams provider feeds the WASM container with the blockchain data and the transformations are applied. | ||||||||||||
## Create Your Own Substreams Package | ||||||||||||
|
||||||||||||
4. **You select a [sink](https://substreams.streamingfast.io/documentation/consume/other-sinks), a place where you want to send the transformed data** (a Postgres database or a Subgraph, for example). | ||||||||||||
If you can't find a Substreams package that fits your needs, you can create your own package. | ||||||||||||
Creating a Substreams involves a Rust function that selects which specific data you want to extract from the network. | ||||||||||||
|
||||||||||||
## Substreams Documentation | ||||||||||||
However, the Substreams CLI allows you to auto-generate a Substreams project without writing any code. | ||||||||||||
You can import Solana transactions or Vara extrinsics in your subgraph by using the auto-generation tools provided by The Graph. | ||||||||||||
|
||||||||||||
The official Substreams documentation is currently maintained by the StreamingFast team [on the StreamingFast website](https://substreams.streamingfast.io/). | ||||||||||||
In the folllowing example, you run the `substreams init` command (similar to the `graph init` command), where you can select which kind of project you want to initialize. | ||||||||||||
You can choose among a variety of project generations (Solana transactions, EVM events and calls, Starknet transactions...). | ||||||||||||
|
||||||||||||
To learn about the latest version of Substreams CLI, which enables developers to bootstrap a Substreams project without any code, please check [Substreams Codegen](https://streamingfastio.medium.com/substreams-codegen-no-code-tool-to-bootstrap-your-project-a11efe0378c6). | ||||||||||||
// image | ||||||||||||
|
||||||||||||
### Getting Started | ||||||||||||
|
||||||||||||
- In order to develop and deploy a Substreams, [you must install the Substreams CLI](https://substreams.streamingfast.io/documentation/consume/installing-the-cli). | ||||||||||||
- Then, run your first Substreams by following the [Quickstart Tutorial](https://substreams.streamingfast.io/documentation/develop/init-project). | ||||||||||||
|
||||||||||||
### Expand Your Knowledge | ||||||||||||
|
||||||||||||
- Take a look at the [Ethereum Explorer Tutorial](https://substreams.streamingfast.io/tutorials/evm) to learn about the basic transformations you can create with Substreams. | ||||||||||||
From the generated project, you can run `substreams codegen subgraph` to convert the Substreams project into a ready-to-deploy subgraph project! |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
|
||
|
||
Getting started with a Solana Substreams-powered Subgraph is really easy by using the auto-generation tools contained in the `substreams init` command. Essentially it requires two steps: creating a Substreams project and creating a Subgraph project based on that Substreams project. | ||
|
||
> **Note**: It is recommended that you run everything inside the Substreams Development Environment. Take a look at the [Getting Started page](./getting-started.md) or directly clone the [substreams-starter](https://github.com/streamingfast/substreams-starter) repository. | ||
|
||
## Generate the Substreams Project | ||
|
||
The first step is creating a Substreams project and generating a Substreams package (`.spkg` file) that the subgraph can use as data source. | ||
|
||
1. Run `substreams init`, which will show the different options to intialize your project. | ||
|
||
// image | ||
|
||
2. Choose the preferred option to initialize your Solana project: | ||
|
||
- `sol-minimal`: will create a very simple Substreams, just extracting raw data from the block (it will generate Rust code). | ||
- `sol-transactions`: will create a Substreams that extracts Solana transactions filtered by one or several Program IDs. (it will NOT generate Rust code, as it relies on the Solana Foundational Modules). | ||
|
||
3. Complete the rest of information needed, such as the project name or the Program IDs that you want to use to filter the transactions. | ||
|
||
4. Once you are done answering all the questions, a Substreams project will be generated in the specified folder. | ||
|
||
// image | ||
|
||
5. Follow the instructions provided to authenticate, build and run the Substreams project. | ||
|
||
To build the project and **generate the Substreams package (`.spkg`)**: | ||
|
||
```bash | ||
substreams build | ||
``` | ||
|
||
To autenthicate with one of the Substreams providers: | ||
|
||
```bash | ||
substreams auth | ||
``` | ||
|
||
To run the Substreams in your command-line terminal: | ||
|
||
```bash | ||
substreams gui | ||
``` | ||
|
||
## Generate the Subgraph Project | ||
|
||
From the Substreams `.spkg` file, which contains all the data that you want to extract from the blockchain, you can generate a subgraph project. | ||
|
||
1. Run `substreams codegen subgraph`. | ||
A new folder, `subgraph`, will be created. | ||
|
||
2. The generated project is standard subgraph project, where the `subgraph.yaml` file uses the Substreams package as data source. By default, the code contained in the `mappings.ts` file is pretty simple. It is up to you to decide what entities you want to create with the data extracted from the Substreams. | ||
|
||
3. Follow the instructions provided to build your subgraph. | ||
|
||
To install the Node modules: | ||
|
||
```bash | ||
npm install | ||
``` | ||
|
||
To generate the GraphQL schema of the subgraph (i.e. the output of the subgraph): | ||
|
||
```bash | ||
npm run codegen | ||
``` | ||
|
||
To generate Protobuf schema of the Substreams (i.e. the output of the Substreams and the input of the subgraph): | ||
|
||
```bash | ||
npm run protogen | ||
``` | ||
|
||
To build your subgraph: | ||
|
||
```bash | ||
npm run build | ||
``` | ||
|
||
(All the previous command starting with `npm run` are wrappers defined in the `package.json` file). | ||
|
||
6. If you have a Graph Node instance running locally, you can deploy and test your subgraph. | ||
It is recommended that you run everything inside the [substreams-starter environment](https://github.com/streamingfast/substreams-starter). | ||
|
||
> **Note**: To run the following command, you need a working Substreams Developer Environment. If you are running Graph Node outside of the Developer Environment, take a look at the `package.json` file of the project to review the Graph CLI command executed. | ||
|
||
To create the subgraph: | ||
|
||
```bash | ||
npm run create-local | ||
``` | ||
|
||
To deploy the subgraph: | ||
|
||
```bash | ||
npm run deploy-local | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
--- | ||
title: Getting Started with Substreams | ||
--- | ||
|
||
## Getting Started | ||
|
||
The easiest way to get started with Substreams is by using one of the auto-generation tools. The Substreams CLI provides the `substreams init` command, which will allow you to initialize your Substreams project for several blockchains (EVM, Solana, Injective...). | ||
|
||
The Substreams CLI is pretty similar to The Graph CLI: you install it in your computer and execution the different commands available. | ||
|
||
### The Development Environment | ||
|
||
To facilitate the development of Substreams-powered Subgraphs, a local development environment for VSCode is available, the [substreams-starter](https://github.com/streamingfast/substreams-starter) project. | ||
|
||
The development environment is a VSCode extension that spins up all the necessary dependencies in Docker containers: | ||
- A terminal-based environment with the Substreams CLI pre-installed, so you don't have to install | ||
- A Graph Node instance, which you can uses to test your subgraph locally. | ||
- A PostgreSQL instance, which is used by the Graph Node to store all the necessary data for your subgraphs. | ||
- An IPFS instance, which is used by the Graph Node to store the _build_ files of the subgraph. | ||
|
||
You can run the development environment locally (you need both VSCode and Docker) or remotely through GitHub codespaces. | ||
|
||
#### Run the Development Environment Locally | ||
|
||
To run it locally, simply open the [substreams-starter](https://github.com/streamingfast/substreams-starter) in your local VSCode application. | ||
|
||
#### Run the Development Environment Remotely | ||
|
||
To run it in GitHub codespace, open [this link](https://github.com/codespaces/new/streamingfast/substreams-starter?machine=standardLinux32gb) (`https://github.com/codespaces/new/streamingfast/substreams-starter?machine=standardLinux32gb`) in your browser. Log in with GitHub and you will get access to a VSCode instance in your browser. | ||
|
||
### Initialize Your Substreams Project | ||
|
||
In the development environment, you will have access to the `substreams init` command. Depending on the type of project that you want to initialize, the flow will be different. Choose which blockchain you want to index: | ||
|
||
|
||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should it be "Substreams allow you"?