Minor documentation and CI improvements (#747)

Author: droserasprout

.github/workflows/build.yml

@@ -12,7 +12,8 @@ on:
       - 'src/**'
       - 'Dockerfile'
       - 'pyproject.toml'
-      - 'pdm.lock'
+      # NOTE: Dumped with `pdm run update`
+      - 'requirements.txt'
       - '.github/workflows/build.yml'
 
 jobs:

.github/workflows/docs.yml

@@ -1,6 +1,4 @@
-# NOTE: to run pipeline in act
-# change ${{ secrets.GITHUB_TOKEN }} to $GITHUB_TOKEN
-# and uncommit Install GH
+# NOTE: To run this pipeline in act replace secrets.GITHUB_TOKEN with $GITHUB_TOKEN and uncomment the first step
 
 name: Docs
 concurrency: 
@@ -26,6 +24,16 @@ jobs:
     name: Docs
     runs-on: ubuntu-22.04
     steps:
+      # NOTE: Uncomment for local testing with act
+      # - name: Install GitHub CLI (act only)
+      #   run: |
+      #     type -p curl >/dev/null || (sudo apt update && sudo apt install curl -y)
+      #     curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \
+      #     && sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \
+      #     && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
+      #     && sudo apt update \
+      #     && sudo apt install gh -y
+
       - name: Check out the repo
         uses: actions/checkout@v3
 
@@ -41,15 +49,6 @@ jobs:
       - name: Install project
         run: pdm install
 
-      # - name: Install GH
-      #   run: |
-      #     type -p curl >/dev/null || (sudo apt update && sudo apt install curl -y)
-      #     curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg \
-      #     && sudo chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg \
-      #     && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
-      #     && sudo apt update \
-      #     && sudo apt install gh -y
-
       - name: Clone frontend
         run: |
           gh auth setup-git

.github/workflows/installer.yml

@@ -6,7 +6,6 @@ on:
   push:
     paths:
       - 'src/**'
-      - 'scripts/install.py'
       - 'pyproject.toml'
       - 'pdm.lock'
       - '.github/workflows/installer.yml'

.github/workflows/release.yml

@@ -27,29 +27,20 @@ jobs:
         with:
           install: true
 
-      - name: Log in to the registry
+      - name: Log in to Docker Hub
         uses: docker/login-action@v1
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
 
-      - name: Log in to the registry
+      - name: Log in to GHCR
         uses: docker/login-action@v1
         with:
           registry: ${{ env.DOCKER_REGISTRY }}
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
-      - name: Tag image for ghcr registry
-        uses: docker/metadata-action@v3
-        with:
-          images: dipdup/dipdup
-          flavor: |
-            latest=false
-          tags: |
-            ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}/dipdup
-
-      - name: Set up metadata
+      - name: Set up Docker metadata
         id: meta
         uses: docker/metadata-action@v3
         with:

CONTRIBUTING.md

@@ -2,21 +2,23 @@
 
 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).
 
-## General
+## For contributors
+
+### General
 
 - All code in this repository MUST be licensed under the [MIT License](./LICENSE.md).
 - Python code in this repository MUST run on Python 3.11. Using modern language features is encouraged.
 - Python code in this repository MUST run in Linux, macOS, Docker, and `amd64`/`arm64` environments. Windows SHOULD be supported as well.
-- We use the [Poetry](https://python-poetry.org/docs/#installation) package manager and GNU Make to set up the development environment. You SHOULD install both tools and run `make help` to see available shortcuts.
-- Developers SHOULD have fun while contributing to the project.
+- We use the PDM package manager to set up the development environment. You SHOULD install it and run `pdm run -l` to see available shortcuts.
+- Have fun!
 
-## GitHub
+### Git workflow
 
-- Branch names MUST follow `prefix/short-description` format. Prefixes currently in use: `feat` for features, `fix` for bugfixes, `docs` for documentation, `exp` for experiments, `aux` for everything else.
+- Branch names MUST follow `prefix/short-description` format. Prefixes currently in use: `feat` for features, `fix` for bugfixes, `docs` for documentation, `exp` for experiments, `ci` for GHA and Docker stuff, `aux` for everything else.
 - Commits in pull requests MUST be squashed when merging to `next`.
 - Issues and pull requests MUST have a descriptive title; they SHOULD be linked to each other, appropriately labeled, and assigned to maintainers while in progress.
 
-## Codestyle
+### Codestyle
 
 We use `isort` and `black` for autoformatting, `ruff` for linting, and `mypy` for typechecking. All checks MUST pass before merging the code to default branch. Everything not enforced by these tools is up to the developer. But here are some recommendations:
 
@@ -26,13 +28,13 @@ We use `isort` and `black` for autoformatting, `ruff` for linting, and `mypy` fo
 - Some methods and attributes made private to avoid polluting the public API. Feel free to access them from the outside if you know what you are doing.
 - Finally, about exact language features. F-string formatting is preferred over other syntax. Be careful with walrus operator. Don't forget else in conditional expressions. Listen to you mom. We have no consensus about the match-case yet.
 
-## Packaging
+### Packaging
 
 - All dependencies MUST be declared in `pyproject.toml` file.
 - Non-development dependencies MUST be pinned to non-breaking versions (e.g. `^1.2.3`).
 - Core dependencies that we patch MUST be pinned to specific versions (e.g. `1.2.3`).
 
-## Changelog
+### Changelog
 
 - All changes that affect user (developer) experience MUST be documented in the CHANGELOG.md file.
 - Changes that significantly affect DipDup maintainers' experience MAY be documented in the CHANGELOG.md file.
@@ -42,47 +44,41 @@ We use `isort` and `black` for autoformatting, `ruff` for linting, and `mypy` fo
 ## Documentation
 
 - A page in Release Notes SHOULD accompany all major releases.
-- All internal links MUST be created with `{{ #summary ...` shortcodes.
-- All values used in project templates MUST be replaced with `{{ #cookiecutter ...` shortcodes.
 
-# DipDup maintainer guide
+## For maintainers
 
-## Security
+### Security
 
 - GitHub alerts about dependencies that contain vulnerabilities MUST be investigated and resolved as soon as possible.
 - Security-related bugfixes MUST be mentioned in the changelog under the "Security" section.
 
-## Privacy
+### Privacy
 
 - Crash reporting MUST be opt-in (disabled by default) both in config and project templates.
 - Sentry events and crash reports MUST NOT contain any sensitive information (IP addresses, hostnames, etc.)
 - DipDup SHOULD NOT perform network requests to APIs not defined in config as datasources. Current exceptions: GitHub.
 
-## Docker images
+### Docker images
 
-- DipDup dockerfiles use autogenerated `requirements.txt` files. Maintainers MUST run `make update` script on every change in dependencies.
+- DipDup dockerfiles use autogenerated `requirements.txt` files. Maintainers MUST run `pdm run update` script on every change in dependencies.
 - Docker images for stable releases MUST be published on Docker Hub. They MAY also be published on GHCR.
 - Maintainers MAY publish arbitrary images on GHCR and remove them when not needed.
 
-## Installer
+### Installer
 
 - Installer module MUST depend on Python stdlib only.
 
-## Scaffolding
+### Scaffolding
 
 - Project templates SHOULD cover all index types available in DipDup.
 - They also MAY contain additional features and integrations.
 
-## Demo projects
+### Demo projects
 
-- Demos are stored in `demos` root directory. They MUST be generated automatically from project templates using replay files.
-- Maintainers SHOULD run `make demos replays` command regularly to ensure that demo projects are up to date.
+- Demos are stored in `src` directory. They MUST be generated automatically from project templates using replay files.
+- Maintainers SHOULD run `pdm demos` command regularly to ensure that demo projects are up to date.
 
-## Releases
+### Releases
 
 - Release versions MUST conform to [Semantic Versioning](https://semver.org/). Releases that introduce breaking changes MUST be major ones.
 - Only the latest major version is supported in general. Critical fixes MAY be backported to the previous major release. To do so, create an `aux/X.Y.Z` branch from the latest stable tag, bump the DipDup version manually, and add a new tag.
-
-## Tests
-
-## Code Review

README.md

@@ -23,7 +23,7 @@ DipDup is a Python framework for building smart contract indexers. It helps deve
 
 This project is maintained by the [Baking Bad](https://bakingbad.dev/) team.
 <br>
-Development is supported by [Tezos Foundation](https://tezos.foundation/) and [OnlyDust](https://www.onlydust.xyz)
+Development is supported by [Tezos Foundation](https://tezos.foundation/) and [OnlyDust](https://www.onlydust.xyz).
 
 ## Thanks
 

docs/0.quickstart.md

@@ -21,7 +21,7 @@ A modern Linux/macOS distribution with Python 3.11 installed is required to run
 You can initialize a hello-world project interactively by choosing configuration options in the terminal. The following command will install DipDup for the current user:
 
 ```shell [Terminal]
-curl -Lsf https://dipdup.io/install.py | python3.11
+curl -Lsf https://dipdup.io/install.py | python3
 ```
 
 Now, let's create a new project:

docs/1.getting-started/4.package.md

@@ -3,7 +3,7 @@ title: "Package"
 description: "Each DipDup project consists of a YAML config and a Python package of a specific structure. It could be placed anywhere, but needs to be importable. The package name is defined in the config file."
 ---
 
-# Package structure
+# Project package
 
 Each DipDup project consists of a YAML config and a Python package of a specific structure. It could be placed anywhere, but needs to be importable. The package name is defined in the config file.
 

docs/1.getting-started/8.handlers.md

@@ -3,7 +3,7 @@ title: "Handlers"
 description: "After defining the index in the configuration file and initializing the project, you can start implementing handlers. DipDup generates callback stubs for all handlers specified in the configuration file. You can find them in the handlers directory."
 ---
 
-# Implementing handlers
+# Handlers
 
 After defining the index in the configuration file and initializing the project, you can start implementing handlers. DipDup generates callback stubs for all handlers specified in the configuration file. You can find them in the `handlers`{lang="python"} directory.
 

docs/13.faq.md renamed to docs/12.faq.md

@@ -8,7 +8,9 @@ nested: Resources
 
 This page contains answers to the most popular questions about DipDup guts. If you landed here - congrats, you're writing a pretty advanced indexer!
 
-## How to index similar but not identical contracts as a single entity?
+## Indexing
+
+### How to index similar but not identical contracts as a single entity?
 
 Multiple contracts can provide the same interface (like FA1.2 and FA2 standard tokens) but have a different storage structure. If you try to use the same typename for them, indexing will fail. However, you can modify typeclasses manually. Modify `types/<typename>/storage.py` file and comment out unique fields that are not important for your index:
 
@@ -24,23 +26,25 @@ class ContractStorage(BaseModel):
 
 Don't forget `Extra.ignore` Pydantic hint, otherwise storage deserialization will fail. To restore the original typeclass, remove modified file and run `dipdup init` again. You can also add `--force` flag to overwrite all ABIs and typeclasses.
 
-## How to use off-chain datasources?
+### How to use off-chain datasources?
 
 DipDup provides convenient helpers to process off-chain data like market quotes or IPFS metadata. Follow the tips below to use them most efficiently.
 
 - Do not perform off-chain requests in handers until necessary. Handlers need to be as fast as possible not to block the database transaction. Use hooks instead, enriching indexed data on-demand.
 - Use generic `http` datasource for external APIs instead of plain `aiohttp` requests. It makes available the same features DipDup uses for internal requests: retry with backoff, rate limiting, Prometheus integration etc.
 - Database tables that store off-chain data can be marked as immune, to speed up reindexing.
 
-## How to process indexes in a specific order?
+### How to process indexes in a specific order?
 
 Indexes of all kinds are fully independent. They are processed in parallel, have their own message queues, and don't share any state. It is one of the essential DipDup concepts, so there's no "official" way to manage the order of indexing.
 
 Avoid using sync primitives like `asyncio.Event` or `asyncio.Lock` in handlers. Indexing will be stuck forever, waiting for the database transaction to complete.
 
 Instead, save raw data in handlers and process it later with hooks when all conditions are met. For example, process data batch only when all indexes in the `dipdup_index` table have reached a specific level.
 
-## How to perform database migrations?
+## Database
+
+### How to perform database migrations?
 
 DipDup does not provide any tooling for database migrations. The reason is that schema changes almost always imply reindexing when speaking about indexers. However, you can perform migrations yourself using any tool you like. First, disable schema hash check in config:
 
@@ -75,7 +79,7 @@ diff old-schema.sql new-schema.sql
 
 Now you can prepare and execute an `ALTER TABLE` query manually or using SQL hooks.
 
-## I get `schema_modified` error, but I didn't change anything
+### I get `schema_modified` error, but I didn't change anything
 
 DipDup compares the current schema hash with the one stored in the database. If they don't match, it throws an error. If models were not modified, most likely the reason is incorrect model definitions. e.g. if you define a timestamp field like this…
 
@@ -104,7 +108,7 @@ You need to make the following change in models.py:
 
 We plan to improve field classes in future releases to accept callables as default values.
 
-## Why am I getting `decimal.InvalidOperation` error?
+### Why am I getting `decimal.InvalidOperation` error?
 
 If your models contain `DecimalField`s, you may encounter this error when performing arithmetic operations. It's because the value is too big to fit into the current decimal context.