diff --git a/website/pages/en/substreams.mdx b/website/pages/en/substreams.mdx index 710e110012cc..10c2701d0e27 100644 --- a/website/pages/en/substreams.mdx +++ b/website/pages/en/substreams.mdx @@ -4,43 +4,36 @@ title: Substreams ![Substreams Logo](/img/substreams-logo.png) -Substreams is a powerful blockchain indexing technology developed for The Graph Network. It enables developers to write Rust modules, compose data streams alongside the community, and provide extremely high-performance indexing due to parallelization in a streaming-first approach. +Substreams is a powerful blockchain indexing technology developed to improve The Graph Network: +- Substreams allows you to speed up the indexing times of your subgraph. +- Substreams allows you to index non-EVM data (Solana, Injective, Starknet, Vara...) -With Substreams, developers can quickly extract data from different blockchains (Ethereum, BNB, Solana, ect.) and send it to various locations of their choice, such as a Postgres database, a Mongo database, or a Subgraph. Additionally, Substreams packages enable developers to specify which data they want to extract from the blockchain. +This is possible because your subgraph can use a Substreams package as a data source, thus creating a Substreams-powered Subgraph (SpS). +A Substreams package a binary file that contains all the definitions of the data that you want to extract from the blockchain (similar to the `mapping.ts` file of your subgraph). -## How Substreams Works in 4 Steps +The following example shows a subgraph manifest (`subgraph.yaml`) that uses a Substreams package as data source, thus receiving the data extracted by the Substreams. +Then, you can use this data in your `mapping.ts` and perform all the transformations needed as a usual subgraph. -1. **You write a Rust program, which defines the transformations that you want to apply to the blockchain data.** For example, the following Rust function extracts relevant information from an Ethereum block (number, hash, and parent hash). +// example -```rust -fn get_my_block(blk: Block) -> Result { - let header = blk.header.as_ref().unwrap(); +## Find Substreams packages - Ok(MyBlock { - number: blk.number, - hash: Hex::encode(&blk.hash), - parent_hash: Hex::encode(&header.parent_hash), - }) -} -``` +There are many Substreams packages already built that you can simply import into your subgraph. +[substreams.dev](https://substreams.dev) is a Substreams registry where you can explore different Substreams packages for several blockchain networks. -2. **You wrap up your Rust program into a WASM module just by running a single CLI command.** +// image -3. **The WASM container is sent to a Substreams endpoint for execution.** The Substreams provider feeds the WASM container with the blockchain data and the transformations are applied. +## Create Your Own Substreams Package -4. **You select a [sink](https://substreams.streamingfast.io/documentation/consume/other-sinks), a place where you want to send the transformed data** (a Postgres database or a Subgraph, for example). +If you can't find a Substreams package that fits your needs, you can create your own package. +Creating a Substreams involves a Rust function that selects which specific data you want to extract from the network. -## Substreams Documentation +However, the Substreams CLI allows you to auto-generate a Substreams project without writing any code. +You can import Solana transactions or Vara extrinsics in your subgraph by using the auto-generation tools provided by The Graph. -The official Substreams documentation is currently maintained by the StreamingFast team [on the StreamingFast website](https://substreams.streamingfast.io/). +In the folllowing example, you run the `substreams init` command (similar to the `graph init` command), where you can select which kind of project you want to initialize. +You can choose among a variety of project generations (Solana transactions, EVM events and calls, Starknet transactions...). -To learn about the latest version of Substreams CLI, which enables developers to bootstrap a Substreams project without any code, please check [Substreams Codegen](https://streamingfastio.medium.com/substreams-codegen-no-code-tool-to-bootstrap-your-project-a11efe0378c6). +// image -### Getting Started - -- In order to develop and deploy a Substreams, [you must install the Substreams CLI](https://substreams.streamingfast.io/documentation/consume/installing-the-cli). -- Then, run your first Substreams by following the [Quickstart Tutorial](https://substreams.streamingfast.io/documentation/develop/init-project). - -### Expand Your Knowledge - -- Take a look at the [Ethereum Explorer Tutorial](https://substreams.streamingfast.io/tutorials/evm) to learn about the basic transformations you can create with Substreams. +From the generated project, you can run `substreams codegen subgraph` to convert the Substreams project into a ready-to-deploy subgraph project! \ No newline at end of file diff --git a/website/pages/en/substreams/getting-started-solana.md b/website/pages/en/substreams/getting-started-solana.md new file mode 100644 index 000000000000..f3984faff1b3 --- /dev/null +++ b/website/pages/en/substreams/getting-started-solana.md @@ -0,0 +1,98 @@ + + +Getting started with a Solana Substreams-powered Subgraph is really easy by using the auto-generation tools contained in the `substreams init` command. Essentially it requires two steps: creating a Substreams project and creating a Subgraph project based on that Substreams project. + +> **Note**: It is recommended that you run everything inside the Substreams Development Environment. Take a look at the [Getting Started page](./getting-started.md) or directly clone the [substreams-starter](https://github.com/streamingfast/substreams-starter) repository. + +## Generate the Substreams Project + +The first step is creating a Substreams project and generating a Substreams package (`.spkg` file) that the subgraph can use as data source. + +1. Run `substreams init`, which will show the different options to intialize your project. + +// image + +2. Choose the preferred option to initialize your Solana project: + +- `sol-minimal`: will create a very simple Substreams, just extracting raw data from the block (it will generate Rust code). +- `sol-transactions`: will create a Substreams that extracts Solana transactions filtered by one or several Program IDs. (it will NOT generate Rust code, as it relies on the Solana Foundational Modules). + +3. Complete the rest of information needed, such as the project name or the Program IDs that you want to use to filter the transactions. + +4. Once you are done answering all the questions, a Substreams project will be generated in the specified folder. + +// image + +5. Follow the instructions provided to authenticate, build and run the Substreams project. + +To build the project and **generate the Substreams package (`.spkg`)**: + +```bash +substreams build +``` + +To autenthicate with one of the Substreams providers: + +```bash +substreams auth +``` + +To run the Substreams in your command-line terminal: + +```bash +substreams gui +``` + +## Generate the Subgraph Project + +From the Substreams `.spkg` file, which contains all the data that you want to extract from the blockchain, you can generate a subgraph project. + +1. Run `substreams codegen subgraph`. +A new folder, `subgraph`, will be created. + +2. The generated project is standard subgraph project, where the `subgraph.yaml` file uses the Substreams package as data source. By default, the code contained in the `mappings.ts` file is pretty simple. It is up to you to decide what entities you want to create with the data extracted from the Substreams. + +3. Follow the instructions provided to build your subgraph. + +To install the Node modules: + +```bash +npm install +``` + +To generate the GraphQL schema of the subgraph (i.e. the output of the subgraph): + +```bash +npm run codegen +``` + +To generate Protobuf schema of the Substreams (i.e. the output of the Substreams and the input of the subgraph): + +```bash +npm run protogen +``` + +To build your subgraph: + +```bash +npm run build +``` + +(All the previous command starting with `npm run` are wrappers defined in the `package.json` file). + +6. If you have a Graph Node instance running locally, you can deploy and test your subgraph. +It is recommended that you run everything inside the [substreams-starter environment](https://github.com/streamingfast/substreams-starter). + +> **Note**: To run the following command, you need a working Substreams Developer Environment. If you are running Graph Node outside of the Developer Environment, take a look at the `package.json` file of the project to review the Graph CLI command executed. + +To create the subgraph: + +```bash +npm run create-local +``` + +To deploy the subgraph: + +```bash +npm run deploy-local +``` \ No newline at end of file diff --git a/website/pages/en/substreams/getting-started.md b/website/pages/en/substreams/getting-started.md new file mode 100644 index 000000000000..5142b852fe4e --- /dev/null +++ b/website/pages/en/substreams/getting-started.md @@ -0,0 +1,37 @@ +--- +title: Getting Started with Substreams +--- + +## Getting Started + +The easiest way to get started with Substreams is by using one of the auto-generation tools. The Substreams CLI provides the `substreams init` command, which will allow you to initialize your Substreams project for several blockchains (EVM, Solana, Injective...). + +The Substreams CLI is pretty similar to The Graph CLI: you install it in your computer and execution the different commands available. + +### The Development Environment + +To facilitate the development of Substreams-powered Subgraphs, a local development environment for VSCode is available, the [substreams-starter](https://github.com/streamingfast/substreams-starter) project. + +The development environment is a VSCode extension that spins up all the necessary dependencies in Docker containers: +- A terminal-based environment with the Substreams CLI pre-installed, so you don't have to install +- A Graph Node instance, which you can uses to test your subgraph locally. +- A PostgreSQL instance, which is used by the Graph Node to store all the necessary data for your subgraphs. +- An IPFS instance, which is used by the Graph Node to store the _build_ files of the subgraph. + +You can run the development environment locally (you need both VSCode and Docker) or remotely through GitHub codespaces. + +#### Run the Development Environment Locally + +To run it locally, simply open the [substreams-starter](https://github.com/streamingfast/substreams-starter) in your local VSCode application. + +#### Run the Development Environment Remotely + +To run it in GitHub codespace, open [this link](https://github.com/codespaces/new/streamingfast/substreams-starter?machine=standardLinux32gb) (`https://github.com/codespaces/new/streamingfast/substreams-starter?machine=standardLinux32gb`) in your browser. Log in with GitHub and you will get access to a VSCode instance in your browser. + +### Initialize Your Substreams Project + +In the development environment, you will have access to the `substreams init` command. Depending on the type of project that you want to initialize, the flow will be different. Choose which blockchain you want to index: + + + +