Update metadata, fix package discovery (#758)

Author: droserasprout

.github/workflows/docs.yml

@@ -57,12 +57,13 @@ jobs:
       - name: Build docs
         run: pdm run docs_build
 
-      - name: Commit and push frontend
-        if: github.ref == 'refs/heads/next'
-        run: |
-          cd ../interface
-          git config --local user.email "[email protected]"
-          git config --local user.name "GitHub Action"
-          git add .
-          git diff-index --quiet HEAD || git commit -m "${{ github.event.head_commit.message }}"
-          git push origin $FRONTEND_BRANCH
+      # FIXME: Reenable before 7.0.0rc3 release
+      # - name: Commit and push frontend
+      #   if: github.ref == 'refs/heads/next'
+      #   run: |
+      #     cd ../interface
+      #     git config --local user.email "[email protected]"
+      #     git config --local user.name "GitHub Action"
+      #     git add .
+      #     git diff-index --quiet HEAD || git commit -m "${{ github.event.head_commit.message }}"
+      #     git push origin $FRONTEND_BRANCH

CHANGELOG.md

@@ -8,8 +8,12 @@ The format is based on [Keep a Changelog], and this project adheres to [Semantic
 
 ### Fixed
 
+- ci: Fixed dipdup package metadata.
+- cli: Generate base template from replay only when --base flag is set.
 - index: Remove Python limitation on large int<->str conversions.
 - index: Fixed crash when parsing typed transactions with empty parameter.
+- package: Don't create empty pyproject.toml during init.
+- package: Fixed discovery of the package when workdir is project root.
 
 ## [6.5.10] - 2023-08-02
 

docs/0.quickstart.md

@@ -10,72 +10,79 @@ This page will guide you through the steps to get your first selective indexer u
 
 Let's create an indexer for the [tzBTC FA1.2 token contract](https://tzkt.io/KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn/operations/). Our goal is to save all token transfers to the database and then calculate some statistics of its holders' activity.
 
-`demo_token` demo is discussed in this guide; you can find it in the list of available templates when running `dipdup new` command.
+## Install DipDup
 
 A modern Linux/macOS distribution with Python 3.11 installed is required to run DipDup.
 
-## Create a new project
-
-### Interactively (recommended)
-
-You can initialize a hello-world project interactively by choosing configuration options in the terminal. The following command will install DipDup for the current user:
+We have a [convenient installer](https://dipdup.io/install.py) that will install DipDup and all the required dependencies for you. It uses pipx to install DipDup as a CLI application and make it available everywhere for the current user. Run the following command in your terminal:
 
 ```shell [Terminal]
 curl -Lsf https://dipdup.io/install.py | python3
 ```
 
-Now, let's create a new project:
+If you don't want to use our installer, create a new project directory you can install DipDup manually. You can use any Python package manager you like, but we recommend [PDM](https://pdm.fming.dev/latest/). Here are some spells to get you started:
 
 ```shell [Terminal]
-dipdup new
-```
+# pipx
+pipx install dipdup datamodel-code-generator pdm
 
-Follow the instructions; the project will be created in the current directory. You can skip reading the rest of this page and slap `dipdup run` instead.
+# PDM
+pdm init --python 3.11
+pdm use .venv
+pdm add "dipdup>=7.0.0rc2,<8"
+$(pdm venv activate)
 
-### From scratch
+# Poetry
+poetry init --python ">=3.11,<3.12"
+poetry add "dipdup>=7.0.0rc2,<8"
+poetry shell
 
-If you don't want to use our installer, you can install DipDup manually. You can use any Python package manager you like, but we recommend [PDM](https://pdm.fming.dev/latest/).
+# pip + venv
+python -m venv .venv
+. .venv/bin/activate
+echo "dipdup>=7.0.0rc2,<8" > requirements.txt
+pip install -r requirements.txt -e .
+```
+
+## Create a project
+
+DipDup CLI has a built-in project generator. Run the following command in your terminal:
 
 ```shell [Terminal]
-# Create a new project directory
-mkdir {{ project.package }}; cd {{ project.package }}
+dipdup new
+```
 
-# PDM
-pdm init --python ">=3.11,<3.12"
-pdm add dipdup
-pdm venv activate
+For educational purposes, we'll create a project from scratch, so choose `[none]` network and `demo_blank` template.
 
-# Plain pip
-python -m venv .venv
-. .venv/bin/activate
-pip install dipdup
+::banner{type="note"}
+Want to skip a tutorial? Choose `Tezos` and `demo_token` instead!
+::
 
-# or Poetry
-poetry init --python ">=3.11,<3.12"
-poetry add dipdup
-poetry shell
-```
+Follow the instructions; the project will be created in the new directory.
 
 ## Write a configuration file
 
-DipDup configuration is stored in YAML files of a specific format. Create a new file named `dipdup.yaml` in the project root with the following content:
+In the project root, you'll find a file named `dipdup.yaml`. It's the main configuration file of your indexer. We will discuss it in detail in the [Config](1.getting_started/3.config.md) section; for now just replace its content with the following:
 
 ```yaml [dipdup.yaml]
 {{ #include ../src/demo_token/dipdup.yaml }}
 ```
 
 ## Generate types and stubs
 
-Now it's time to generate typeclasses and callback stubs. Run the following command:
+Now it's time to generate typeclasses and callback stubs based on definitions from config. Examples below use `demo_token` as a package name; yours may differ.
+
+Run the following command:
 
 ```shell [Terminal]
 dipdup init
 ```
 
-DipDup will create a Python package `demo_token` having the following structure:
+DipDup will create a Python package `demo_token` with everything you need to start writing your indexer. Use `package tree` command to see the generated structure:
 
-```shell
-demo_token [src/demo_token]
+```shell [Terminal]
+$ dipdup package tree
+demo_token [.]
 ├── abi
 ├── configs
 │   ├── dipdup.compose.yaml
@@ -108,24 +115,25 @@ demo_token [src/demo_token]
 │   ├── tzbtc/tezos_parameters/mint.py
 │   ├── tzbtc/tezos_parameters/transfer.py
 │   └── tzbtc/tezos_storage.py
-├── py.typed
-├── __init__.py
-└── pyproject.toml
+├── demo_token.py
+└── py.typed
 ```
 
 That's a lot of files and directories! But don't worry, we will need only `models` and `handlers` sections in this guide.
 
 ## Define data models
 
-Our schema will consist of a single model `Holder` having several fields:
+DipDup supports storing data in SQLite, PostgreSQL and TimescaleDB databases. We use custom ORM based on Tortoise ORM as an abstraction layer.
+
+First, you need to define a model class. Our schema will consist of a single model `Holder` with the following fields:
 
 - `address` — account address
-- `balance` — in tzBTC
-- `volume` — total transfer/mint amount bypassed
+- `balance` — token amount held by the account
+- `volume` — total amount of transfer/mint calls
 - `tx_count` — number of transfers/mints
 - `last_seen` — time of the last transfer/mint
 
-Put the following content in the `models/__init__.py` file:
+Here's how to implement this model in DipDup:
 
 ```python [models/__init__.py]
 {{ #include ../src/demo_token/models/__init__.py }}
@@ -153,7 +161,7 @@ Three methods of tzBTC contract can alter token balances — `transfer`, `mint`,
 
 And that's all! We can run the indexer now.
 
-## Next steps 
+## Next steps
 
 Run the indexer in-memory:
 
@@ -176,4 +184,14 @@ cp .env.default .env
 docker-compose up
 ```
 
-DipDup will fetch all the historical data and then switch to realtime updates. Your application data has been successfully indexed!
+DipDup will fetch all the historical data and then switch to realtime updates. You can check the progress in the logs.
+
+If you use SQLite, run a query to check the data:
+
+```bash
+sqlite3 demo_token.sqlite 'SELECT * FROM holder LIMIT 10'
+```
+
+If you run a Compose stack, check open `http://127.0.0.1:8080` in your browser to see the Hasura console (exposed port may differ). You can use it to explore the database and build GraphQL queries.
+
+Congratulations! You've just created your first DipDup indexer. Proceed to the Getting Started section to learn in-depth about DipDup configuration and features.

docs/1.getting-started/1.installation.md

@@ -46,18 +46,19 @@ Then use the snippets below to create a new Python project and add DipDup as a d
 PDM is a very powerful package manager with a lot of features. Also, it can run scripts from pyproject.toml as npm does. It's a good choice for both beginners and advanced users.
 
 ```shell [Terminal]
-pdm init --python ">=3.10,<3.11"
-pdm add dipdup
-pdm venv activate
+pdm init --python 3.11
+pdm use .venv
+pdm add "dipdup>=7.0.0rc2,<8"
+$(pdm venv activate)
 ```
 
 #### Poetry
 
 Poetry is another popular tool to manage Python projects. It's less stable and often slower than PDM, but it's still a good choice.
 
 ```shell [Terminal]
-poetry init --python ">=3.10,<3.11"
-poetry add dipdup
+poetry init --python ">=3.11,<3.12"
+poetry add "dipdup>=7.0.0rc2,<8"
 poetry shell
 ```
 
@@ -68,7 +69,7 @@ Finally, if you prefer to do everything manually, you can use pip. It's the most
 ```shell [Terminal]
 python -m venv .venv
 . .venv/bin/activate
-echo "dipdup~={{ project.dipdup_version }}" >> requirements.txt
+echo "dipdup>=7.0.0rc2,<8" >> requirements.txt
 pip install -r requirements.txt -e .
 ```
 

docs/1.getting-started/4.package.md

@@ -11,21 +11,22 @@ To generate all necessary directories and files according to config run the `ini
 
 The structure of the resulting package is the following:
 
-| Path                     | Description                                                                                                                          |
-| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ |
-| :file_folder: `abi`      | Contract ABIs used to generate typeclasses                                                                                           |
-| :file_folder: `configs`  | Environment-specific configs to merge with the root one                                                                              |
-| :file_folder: `deploy`   | Dockerfiles, compose files, and default env variables for each environment                                                                                    |
-| :file_folder: `graphql`  | Custom GraphQL queries to expose with Hasura engine                                                                                                    |
-| :file_folder: `handlers` | User-defined callbacks to process contract data                                                                                      |
-| :file_folder: `hasura`   | Arbitrary Hasura metadata to apply during configuration                                                                              |
-| :file_folder: `hooks`    | User-defined callbacks to run manually or by schedule                                                                                |
-| :file_folder: `models`   | DipDup ORM models to store data in the database                                                                                      |
-| :file_folder: `sql`      | SQL scripts and queries to run manually or on specific events                                                                        |
-| :file_folder: `types`    | Automatically generated Pydantic dataclasses for contract types                                                                      |
-| `dipdup.yaml`            | Root DipDup config; can be expanded with env-specific files                                                                          |
-| `pyproject.toml`         | Python package metadata (introduced in PEP 518; see [details](https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/)) |
-|                          |                                                                                                                                      |
+| Path                       | Description                                                                                                                          |
+| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
+| :file_folder: `abi`        | Contract ABIs used to generate typeclasses                                                                                           |
+| :file_folder: `configs`    | Environment-specific configs to merge with the root one                                                                              |
+| :file_folder: `deploy`     | Dockerfiles, compose files, and default env variables for each environment                                                           |
+| :file_folder: `graphql`    | Custom GraphQL queries to expose with Hasura engine                                                                                  |
+| :file_folder: `handlers`   | User-defined callbacks to process contract data                                                                                      |
+| :file_folder: `hasura`     | Arbitrary Hasura metadata to apply during configuration                                                                              |
+| :file_folder: `hooks`      | User-defined callbacks to run manually or by schedule                                                                                |
+| :file_folder: `models`     | DipDup ORM models to store data in the database                                                                                      |
+| :file_folder: `sql`        | SQL scripts and queries to run manually or on specific events                                                                        |
+| :file_folder: `types`      | Automatically generated Pydantic dataclasses for contract types                                                                      |
+| `dipdup.yaml`              | Root DipDup config; can be expanded with env-specific files                                                                          |
+| `{{ project.package }}.py` | A little helper to let you import the package from the root directory.                                                               |
+| `pyproject.toml`           | Python package metadata (introduced in PEP 518; see [details](https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/)) |
+|                            |                                                                                                                                      |
 
 There are also a bunch files in the root directory: .ignore files, PEP 561 marker, etc. Usually, you won't need to modify them.