Skip to content

Commit 9b60a63

Browse files
Update README.md template; proofread docs (#741)
1 parent 196aed4 commit 9b60a63

File tree

30 files changed

+3036
-2137
lines changed

30 files changed

+3036
-2137
lines changed

.github/workflows/installer.yml

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -45,16 +45,20 @@ jobs:
4545
- name: dipdup new
4646
run: dipdup new --quiet
4747

48-
- name: Replace DipDup in project with current HEAD
49-
run: cd dipdup_indexer; pdm add -d git+https://github.com/dipdup-io/dipdup.git@${{ github.sha }}
48+
- name: Replace DipDup in project with current HEAD (until 7.0)
49+
run: cd dipdup_indexer; pdm remove dipdup; pdm add ..
5050

5151
- name: dipdup init
5252
run: cd dipdup_indexer; dipdup init
5353

54-
- name: Run lint
55-
run: cd dipdup_indexer; pdm run lint
54+
- name: pdm all
55+
run: cd dipdup_indexer; pdm all
5656

57-
- name: Deploy scripts to GitHub Pages
57+
- name: Copy installer to scripts
58+
if: contains(github.ref, 'next')
59+
run: cp src/dipdup/install.py scripts/install.py
60+
61+
- name: Publish scripts to GitHub Pages
5862
if: contains(github.ref, 'next')
5963
uses: peaceiris/actions-gh-pages@v3
6064
with:

docs/1.getting-started/2.core-concepts.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,25 +13,25 @@ Before proceeding to development, let's take a look at basic DipDup concepts.
1313

1414
## Big picture
1515

16-
DipDup is an SDK for building custom backends for decentralized applications, or for short, indexers. DipDup indexers are off-chain services that aggregate blockchain data from various sources and store it in a database.
16+
DipDup is an SDK for building custom backends for decentralized applications, or, indexers. DipDup indexers are off-chain services that aggregate blockchain data from various sources and store it in a database.
1717

18-
Each indexer consists of a YAML configuration file and a Python package with models, handlers, and other code. The configuration file describes the inventory of the indexer, i.e., what contracts to index and what data to extract from them. It supports templates, environment variables, and use syntax similar to Subgraph manifest files. The Python package contains models, callbacks and queries. Models describe the domain-specific data structures you want to store in the database. Callbacks implement the business logic, i.e., how to convert blockchain data to your models. Other files in the package are optional and can be used to extend DipDup functionality.
18+
Each indexer consists of a YAML configuration file and a Python package with models, handlers, and other code. The configuration file describes the inventory of the indexer, i.e., what contracts to index and what data to extract from them. It supports templates, and environment variables, and uses a syntax similar to Subgraph manifest files. The Python package contains models, callbacks and queries. Models describe the domain-specific data structures you want to store in the database. Callbacks implement the business logic, i.e., how to convert blockchain data to your models. Other files in the package are optional and can be used to extend DipDup functionality.
1919

20-
As a result, you get a service responsible for filling the database with the indexed data. Then you can use it to build a custom API backend or integrate with existing ones. DipDup provides Hasura GraphQL Engine integration to expose indexed data via REST and GraphQL with little to no effort, but you can also use other API engines like PostgREST or develop one in-house.
20+
As a result, you get a service responsible for filling the database with indexed data. Then you can use it to build a custom API backend or integrate with existing ones. DipDup provides Hasura GraphQL Engine integration to expose indexed data via REST and GraphQL with little to no effort, but you can also use other API engines like PostgREST or develop one in-house.
2121

2222
<!-- TODO: SVG include doesn't work -->
2323

2424
![Generic DipDup setup and data flow](../assets/dipdup.svg)
2525

2626
## Storage layer
2727

28-
DipDup uses PostgreSQL or SQLite as a database backend. All the data is stored in a single database schema created on the first run. Make sure it's used by DipDup exclusively, since changes in index configuration or models require DipDup to drop the whole database schema and start indexing from scratch. You can, however, mark specific tables as immune to preserve them from being dropped.
28+
DipDup uses PostgreSQL or SQLite as a database backend. All the data is stored in a single database schema created on the first run. Make sure it's used by DipDup exclusively since changes in index configuration or models require DipDup to drop the whole database schema and start indexing from scratch. You can, however, mark specific tables as immune to preserve them from being dropped.
2929

3030
DipDup does not support database schema migration. Any change in models or index definitions will trigger reindexing. Migrations introduce complexity and do not guarantee data consistency. DipDup stores a hash of the SQL version of the DB schema and checks for changes each time you run indexing.
3131

32-
DipDup applies all updates atomically block by block, ensuring data integrity. If indexing is interrupted, next time DipDup starts, it will check the database state and continue from the last block processed. DipDup state is stored in the database per-index and can be used by API consumers to determine the current indexer head.
32+
DipDup applies all updates atomically block by block, ensuring data integrity. If indexing is interrupted, the next time DipDup starts, it will check the database state and continue from the last block processed. DipDup state is stored in the database per index and can be used by API consumers to determine the current indexer head.
3333

34-
Index is a set of contracts and rules for processing them as a single entity. Your config can contain more than one index, but they are processed in parallel and cannot share data as execution order is not guaranteed.
34+
An index is a set of contracts and rules for processing them as a single entity. Your config can contain more than one index, but they are processed in parallel and cannot share data as execution order is not guaranteed.
3535

3636
## Handling chain reorgs
3737

docs/1.getting-started/3.config.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ Use `config export`{lang="sh"} and `config env`{lang="sh"} commands to check the
4343

4444
## Environment variables
4545

46-
DipDup supports compose-style variable expansion with optional default value. Use this feature to store sensitive data outside of the configuration file and make your app fully declarative. If required variable is not set, DipDup will fail with an error. You can use these placeholders anywhere throughout the configuration file.
46+
DipDup supports compose-style variable expansion with an optional default value. Use this feature to store sensitive data outside of the configuration file and make your app fully declarative. If a required variable is not set, DipDup will fail with an error. You can use these placeholders anywhere throughout the configuration file.
4747

4848
```yaml [dipdup.yaml]
4949
database:
@@ -57,7 +57,7 @@ There are multiple ways to pass environment variables to DipDup:
5757
- Export them in the shell before running DipDup
5858
- Create the env file and pass it to DipDup with the `-e` CLI option
5959

60-
For every config file in `deploy` project directory, DipDup will create a corresponding `.env.default` file with all the variables used in the config. Copy it removing `.default` suffix and fill in the values.
60+
For every config file in the `deploy` project directory, DipDup will create a corresponding `.env.default` file with all the variables used in the config. Copy it, remove the `.default` suffix and fill in the values.
6161

6262
```sh [deploy/.env.default]
6363
POSTGRES_HOST=localhost

docs/1.getting-started/4.package.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,11 @@ description: "DipDup is a Python framework for building smart contract indexers.
66

77
# Package structure
88

9-
Each DipDup project consists of a YAML config and a Python package of specific structure. It could be placed anywhere, but needs to be importable. The package name is defined in the config file.
9+
Each DipDup project consists of a YAML config and a Python package of a specific structure. It could be placed anywhere, but needs to be importable. The package name is defined in the config file.
1010

1111
To generate all necessary directories and files according to config run the `init` command. You should run it every time you significantly change the config file.
1212

13-
The structure of resulting package is the following:
13+
The structure of the resulting package is the following:
1414

1515
| Path | Description |
1616
| ------------------------ | --------------------------------------------------------------- |
@@ -29,7 +29,7 @@ There's also a bunch on files in the root directory: .ignore files, pyproject.to
2929

3030
## ABIs and typeclasses
3131

32-
DipDup uses contract type information to generate dataclasses for developers to work with strictly typed data. Theses dataclasses are generated automatically from contract ABIs. In most cases, you don't need to modify them manually. The process is roughly the following:
32+
DipDup uses contract type information to generate dataclasses for developers to work with strictly typed data. These dataclasses are generated automatically from contract ABIs. In most cases, you don't need to modify them manually. The process is roughly the following:
3333

3434
1. Contract ABIs are placed in the `abi` directory; either manually or during init.
3535
2. DipDup converts these ABIs to intermediate JSONSchemas.
@@ -76,4 +76,4 @@ indexer
7676
└── bar
7777
```
7878

79-
The same applies to handler callbacks. Callback alias still needs to be a valid Python module path: lowercase letters, underscores, and dots.
79+
The same applies to handler callbacks. The callback alias still needs to be a valid Python module path: lowercase letters, underscores, and dots.

docs/1.getting-started/5.models.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,21 +6,21 @@ description: "DipDup is a Python framework for building smart contract indexers.
66

77
# Models
88

9-
To store indexed data in the database, you need to define models. Our storage layer is based on [Tortoise ORM](https://tortoise.github.io/index.html). It's fast, flexible, and have a syntax similar to Django ORM. We have extended it with some useful features like copy-on-write rollback mechanism, caching, and more.
9+
To store indexed data in the database, you need to define models. Our storage layer is based on [Tortoise ORM](https://tortoise.github.io/index.html). It's fast, flexible, and has a syntax similar to Django ORM. We have extended it with some useful features like a copy-on-write rollback mechanism, caching, and more.
1010

11-
We plan to make things official and fork this library under a new name, but it's not ready yet. For now let's call our implemetation **DipDup ORM**.
11+
We plan to make things official and fork this library under a new name, but it's not ready yet. For now, let's call our implementation **DipDup ORM****.
1212

1313
Before we begin to dive into the details, here's an important note:
1414

1515
::banner{type="warning"}
16-
Please, don't report DipDup issues to the Tortoise ORM bugtracker! We patch it heavily to better suit our needs, so it's not the same library anymore.
16+
Please, don't report DipDup issues to the Tortoise ORM bug tracker! We patch it heavily to better suit our needs, so it's not the same library anymore.
1717
::
1818

1919
You can use [Tortoise ORM docs](https://tortoise.github.io/examples.html) as a reference. We will describe only DipDup-specific features here.
2020

2121
## Defining models
2222

23-
Project models should be placed in `models` directory in the project root. By default, `__init__.py` module created on project initialization, but you can use any structure you want. Models from nested packages will be discovered automatically.
23+
Project models should be placed in the `models` directory in the project root. By default, the `__init__.py` module is created on project initialization, but you can use any structure you want. Models from nested packages will be discovered automatically.
2424

2525
Here's an example containing all available fields:
2626

@@ -34,7 +34,7 @@ Some limitations are applied to model names and fields to avoid ambiguity in Gra
3434

3535
- Table names must be in snake_case
3636
- Model fields must be in snake_case
37-
- Model fields must differ from table name
37+
- Model fields must differ from the table name
3838

3939
## Basic usage
4040

@@ -62,7 +62,7 @@ See `demo_uniswap` project for real-life examples.
6262

6363
## Differences from Tortoise ORM
6464

65-
This section describes differences between DipDup and Tortoise ORM. Most likely won't notice them, but it's better to be aware of them.
65+
This section describes the differences between DipDup and Tortoise ORM. Most likely won't notice them, but it's better to be aware of them.
6666

6767
### Fields
6868

@@ -86,7 +86,7 @@ In Tortoise ORM each subsequent call creates a new queryset using expensive `cop
8686

8787
### Transactions
8888

89-
DipDup manage transactions automatically for indexes opening one for each level. You can't open another one. Entering a transaction context manually with `in_transaction()` will return the same active transaction. For hooks, there's `atomic` flag in configuration.
89+
DipDup manages transactions automatically for indexes opening one for each level. You can't open another one. Entering a transaction context manually with `in_transaction()` will return the same active transaction. For hooks, there's `atomic` flag in the configuration.
9090

9191
<!-- ::banner{type="note"}
9292

docs/1.getting-started/6.datasources.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ Index datasources, ones that can be attached to a specific index, are prefixed w
2525

2626
All datasources now share the same code under the hood to communicate with underlying APIs via HTTP. Their configs have an optional section `http` to configure connection settings. You can use it to set timeouts, retry policies, and other parameters.
2727

28-
Each datasource kind has its defaults. Usually, there's no reason to alter these settings unless you use self-hosted instances. In example below, default values are shown:
28+
Each datasource kind has its defaults. Usually, there's no reason to alter these settings unless you use self-hosted instances. In the example below, default values are shown:
2929

3030
```yaml [dipdup.yaml]
3131
datasources:
@@ -48,7 +48,7 @@ datasources:
4848

4949
## Ratelimiting
5050

51-
Ratelimiting is implenented using "leaky bucket" algorithm. The number of consumed "drops" can be set with each request (defaults to 1), and the bucket is refilled with a constant rate. If the bucket is empty, the request is delayed until it's refilled.
51+
Ratelimiting is implemented using the "leaky bucket" algorithm. The number of consumed "drops" can be set with each request (defaults to 1), and the bucket is refilled with a constant rate. If the bucket is empty, the request is delayed until it's refilled.
5252

5353
```python
5454
response = await datasource.request(

docs/1.getting-started/7.indexes.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,14 +20,15 @@ Multiple indexes are available for different workloads. Every index is linked to
2020
| [tezos.tzkt.operations_unfiltered](6.tezos_tzkt_operations_unfiltered.md) | Tezos | TzKT | untyped operations |
2121
| [tezos.tzkt.token_transfers](7.tezos_tzkt_token_transfers.md) | Tezos | TzKT | TZIP-12/16 token transfers |
2222

23-
Indexes can join multiple contracts considered as a single application. Also, contracts can be used by multiple indexes of any kind, but make sure that they are independent of each other and indexed data don't overlap.
23+
Indexes can join multiple contracts considered as a single application. Also, contracts can be used by multiple indexes of any kind, but make sure that they are independent of each other and that indexed data don't overlap.
24+
2425
<!-- TODO: here was a link to a place that doesnt exist now -- Make sure to visit {{ #summary getting-started/core-concepts.md#atomicity-and-persistency }}. -->
2526

2627
Handler is a callback function, called when new data has arrived from the datasource. The handler receives the data item as an argument and can perform any actions, e.g., store data in the database, send a notification, or call an external API.
2728

2829
## Using templates
2930

30-
Index definitions can be templated to reduce the amount of boilerplate code. To create an index from template use the following syntax:
31+
Index definitions can be templated to reduce the amount of boilerplate code. To create an index from the template use the following syntax:
3132

3233
```yaml [dipdup.yaml]
3334
templates:

docs/3.datasources/2.abi_etherscan.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ description: "DipDup is a Python framework for building smart contract indexers.
55
network: "ethereum"
66
---
77

8-
[Etherscan](https://etherscan.io/) is a popular Ethereum blockchain explorer. It provides a public API to fetch ABIs of verified contracts. DipDup can use it API to fetch ABIs for contracts being indexed.
8+
[Etherscan](https://etherscan.io/) is a popular Ethereum blockchain explorer. It provides a public API to fetch ABIs of verified contracts. DipDup can use its API to fetch ABIs for contracts being indexed.
99

1010
To use this datasource, add the following section in config:
1111

docs/3.datasources/3.coinbase.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ datasources:
1414
1515
## Authorization
1616
17-
If you have a Coinbase API key, you can set it in config and, optionally, increase the ratelimit according to your subscription plan. Otherwise, you will be limited to 10 requests per second.
17+
If you have a Coinbase API key, you can set it in the config and, optionally, increase the ratelimit according to your subscription plan. Otherwise, you will be limited to 10 requests per second.
1818
1919
```yaml [dipdup.yaml]
2020
datasources:

docs/3.datasources/4.evm_node.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ description: "DipDup is a Python framework for building smart contract indexers.
55
network: "ethereum"
66
---
77

8-
DipDup can connect to any EVM-compatible node via JSON-RPC. It can be used as a "last mile" datasource for EVM indexes (data that not in Subsquid Archives yet) or as a standalone datasource for handlers and hooks.
8+
DipDup can connect to any EVM-compatible node via JSON-RPC. It can be used as a "last mile" datasource for EVM indexes (data that is not in Subsquid Archives yet) or as a standalone datasource for handlers and hooks.
99

1010
Examples below show how to connect to Infura and Alchemy nodes for Ethereum mainnet indexes. You can also use your own node, but make sure it has all the necessary data (e.g. archive node).
1111

@@ -39,7 +39,7 @@ datasources:
3939
Don't initialize web3 clients manually! It will break the connection pooling, lock the event loop, and kill your dog.
4040
::
4141

42-
To access the client, use `web3` property of the datasource. Underlying web3 client is asynchronous, so you should use `await` keyword to call its methods.
42+
To access the client, use `web3` property of the datasource. The underlying web3 client is asynchronous, so you should use `await` keyword to call its methods.
4343

4444
```python
4545
web3: AsyncWeb3 = ctx.get_evm_node_datasource('mainnet_node').web3

0 commit comments

Comments
 (0)